#integration-testing #json-file #json-configuration #integration-tests #command-arguments #test-files #cargo-command

app tiny-integration-tester

This Rust program is used to perform simple integration tests on programs through a RUST dyn lib or a JSON configuration file

18 unstable releases (3 breaking)

0.4.6 Jun 13, 2023
0.4.5-draft Jun 13, 2023
0.3.2 Jun 3, 2023
0.2.3 Jun 1, 2023
0.1.4 Jun 1, 2023

#248 in Development tools

MIT/Apache

58KB
946 lines

This Rust program is used to perform simple integration tests on programs through a RUST dyn lib or a JSON configuration file.

Usage: tiny-integration-tester [options]

Arg Long name Description Platforms
-h --help print this help menu ALL
-v --version print the program version ALL
-f --file set lib dir or json tests file name ALL
-j --json use json file instead of rust lib ALL
-a --args send arguments to the test dyn lib ALL
-c --toolchain set compilation toolchain ALL
-t --timeout set default timeout(ms) for all tests *NIX ONLY
-o, --output display output of commands by default *NIX ONLY

This program has been tested on Linux. Although it compiles for Windows, features such as timeout and the ability to display outputs are not supported

USING RUST DYN LIB :


When you use the tiny_integration_tester command on a crate, whether it's an executable or a library, a integration_tester_{package_name} folder is generated and two basic tests, cargo test and cargo build, are run.

The contents of the new lib.rs file should look like this :

//! This crate contains integration tests for {package_name}.

mod boilerplate_structure;
mod testable;

pub use boilerplate_structure::{Attribute, IntTest};
pub use testable::Testable;

use std::ffi::OsStr;

/// Test for `cargo build` command.
#[derive(Debug)]
pub struct CargoBuilder;

// YOU CAN IMPLEMENT THE `Testable` TRAIT TO CREATE A NEW TEST ...
#[cfg_attr(rustfmt, rustfmt_skip)]
impl Testable for CargoBuilder {
    fn command(&self) -> &OsStr { "cargo".as_ref() }
    fn name(&self) -> &str { "Build program" }
    fn args(&self) -> Vec<&OsStr> { vec!["build".as_ref()] }
    fn terminate(&self) -> bool { true }
    fn show_output(&self) -> Option<bool> { Some(true) }
    fn output(&mut self, status: i32, _out: &[u8], _err: &[u8]) -> bool { status == 0 }
}

/// This main function will be exported.
#[no_mangle]
#[allow(non_snake_case)]
pub fn TESTS(_args: &[&str]) -> Result<Vec<Box<dyn Testable>>, Box<dyn std::error::Error>> {
    // ... OR YOU CAN USE THE `IntTest` STRUCTURE, WHICH IS A VERY HANDY BOILERPLATE.
    let cargo_test = Box::new(
        IntTest::builder("cargo")
            .args(["test"])
            .output(|status: i32, _out: &[u8], _err: &[u8]| status == 0)
            .set_attribute(Attribute {
                name: "Basic cargo test".to_string(),
                terminate: true,
                show_output: Some(true),
                ..Default::default()
            }),
    );

    Ok(vec![Box::new(CargoBuilder {}), cargo_test])
}

Each of the tests must implement the Testable trait and they must be returned as a Box<dyn Testable> through a collection of type Vec<Box<dyn Testable>>.

In order to achieve this, you have the choice between creating a new structure and implementing the Testable trait for it, or you can make use of the boilerplate structure IntTest for which the Testable trait implementations are already defined in the boilerplate_structure.rs file.

You also have the possibility to pass arguments from the tiny_integration_tester command line with the -a argument. You will find the list of these arguments as a parameter of the main test function in the form &[&str]. You can of course use getopt on it.

The best thing to do is to generate the documentation for the crate with cargo doc and look at all the existing implementations, which already do just about everything!

Do not change either the name or the prototype of the function : pub fn TESTS(_args: &[&str]) -> Result<Vec<Box<dyn Testable>>, Box<dyn std::error::Error>>

However, there is one last restriction, it is imperative to compile the dynamic library with the same version of rustc that was used to compile tiny_integration_tester, use the -c option if necessary. I hope to find a satisfactory solution in the future.

FOR JSON USE :


After creating a new project with the command cargo new hello, create a file named test.json like this :

[
    {
        "command" : "./target/debug/hello",
        "stdout" : "Hello, world!\n"
    },
    {
        "command" : "./target/debug/hello",
        "stdout" : "Hello, dummy!\n"
    }
]

test.json is the default json filename.

Then simply run cargo build then tiny-integration-tester -j (with -j argument) :

Executing command: "./target/debug/hello" with args: []
OK!
Executing command: "./target/debug/hello" with args: []
FAIL!
result: 1 / 2

You can set these values into the json test file :

Field Definition Type Comment
command command to run String Mandatory
stdout FD stdout String
stderr FD stderr String
stdin FD stdin String
args command arguments Array of String
envs command envs args Array of (String, String) json tuple is [1, 2]
env_clear clear environment Boolean clear all env if true
status process exit status Hexadecimal String 0xHHLL depends of OS
test false if it is no a test Boolean no timeout effect
terminate true if exit in failure Boolean exit tester on failure
timeout kill process after X ms Unsigned Integer UNIX ONLY. 0 = infinity
show_output true for showing output Boolean UNIX ONLY.

Here, for instance, we run the command cargo build --release, setting arguments to - "args" : ["build", "--release"] ,waiting for the command to succeed - "status" : "0x0". There is no time limit - "timeout" : 0, and the output will be displayed on the screen - "show_output" : true. Finally, if an error occurs, we stop the tests - "terminate" : true :

[
    {
        "command" : "cargo",
        "args" : ["build", "--release"],
        "status" : "0x0",
        "timeout" : 0,
        "terminate" : true,
        "show_output" : true
    }
]

Same example as below :

[
    {
        "command" : "echo",
        "args" : ["hello", "world"],
        "status" : "0x0",
        "show_output" : true
    },
    {
        "command" : "cargo",
        "args" : ["test", "--release"],
        "status" : "0x0",
        "timeout" : 0,
        "terminate" : true,
        "show_output" : true
    }
]

Also, we check if stdout or stderr match a given pattern :

[
    {
        "command" : "./target/release/fibonacci",
        "args" : ["3"],
        "stdout" : "fibo(3) = 2\n",
        "timeout" : 1000
    },
    {
        "command" : "./target/release/fibonacci",
        "args" : ["1", ""],
        "status" : "0xff00",
        "stderr" : "Usage: target/debug/fibonacci POSITIF_NUMBER\n"
    },
    {
        "command" : "./target/release/fibonacci",
        "args" : ["7"],
        "stdout" : "fibo(7) = 13\n"
    },
    {
        "timeout" : 500,
        "command" : "./target/release/fibonacci",
        "args" : ["15"],
        "status" : "0x0",
        "stdout" : "fibo(15) = 610\n"
    },
    {
        "command" : "./target/release/fibonacci",
        "args" : ["164neuf"],
        "status" : "0xff00",
        "stderr" : "Bad number format\n"
    },
    {
        "command" : "./target/release/fibonacci",
        "status" : "0xff00",
        "stderr" : "Bad number format\n"
    }
]

We can set test to false so that it does not count as a test :

[
    {
        "command" : "rustc",
        "args" : ["readline-tester.rs"],
        "test" : false
    },
    {
        "command" : "rustc",
        "args" : ["env-tester.rs"],
        "test" : false
    }
]

We can also record data for stdin or manage environment variables with env_clear and envs :

[
    {
        "command" : "./readline-tester",
        "status" : "0x0",
        "stdin" : "carrots are cooked\nbananas\n",
        "stdout" : "1 carrots are cooked\n2 bananas\n"
    },
    {
        "command" : "./env-tester",
        "stdout" : "1 = foo\n2 = bar\n",
        "env_clear" : true,
        "envs" : [["1", "foo"], ["2", "bar"], ["CLUTTER_IM_MODULE", "dummy"]],
        "show_output" : true
    }
]

Be careful with trailing commas at the end in a JSON file.

Dependencies

~5–16MB
~205K SLoC