Test harness for Kailua.

Most integration tests in Kailua are composed of the input, the expected output and expected reports to kailua_diag::Report. This cannot be easily done with unit tests, so this library simplifies the creation of such test harness for individual applications.

A crate can add the following section to its Cargo.toml...

[[test]]
name = "integration-test"
harness = false

...and put the following to the corresponding source file.

extern crate kailua_env;
extern crate kailua_diag;
extern crate kailua_test;

use std::cell::RefCell;
use std::rc::Rc;
use std::collections::HashMap;
use kailua_env::{Source, Span};
use kailua_diag::Report;

struct Testing;
impl kailua_test::Testing for Testing {
    fn run(&self, source: Rc<RefCell<Source>>, span: Span,
           filespans: &HashMap<String, Span>, report: Rc<Report>) -> String {
        format!("Expected test output")
    }
}

fn main() {
    kailua_test::Tester::new("integration-test", Testing).scan("src/tests").done();
}

This will scan all *.lua files in src/tests and test all them according to the command-line options. It is possible to add more methods to Testing to further customize the test application.

Test Format

this part is ignored, put some descriptions here.


--8<-- test-name-1 -- options options ...
--@v Error: this error should occur in the following line
local x = 'some testing code' --@< Note: this note should occur in this exact line
--@^ Warning: this warning should occur in the preceding line

--@v-vvv Fatal: this fatal error should span following three lines
do
    return
end

--! expected output here; long line can be concatenated \
--!     like this, where preceding whitespaces are ignored in this line.


-->8-- test-name-2
this test is ignored unless `-f` is given.


--8<-- test-name-3
local a = require 'x'

--& x
-- it is possible to use multiple input files.
return 42

--&
-- duplicate input file names are invalid, except for the empty file name.
-- this means that the preceding file (named or not) should be trimmed following whitespaces,
-- required by parser tests which tend to be sensitive to the final whitespaces.
-- the code block itself gets ignored.

More on report formats:

• The explicit line can be also given (--@3-8 ...); the line number is 1-based and renumbered for each code block.

• The line number can be also missing, in which case the report should not have any associated span.

• All report Kinds except for Kind::Info can be used. The info report is normally used for debugging informations, so it cannot be suppressed nor checked against.

Supported test options:

• exact means that the test should not tolerate additional reports (normally only enabled when -e is given).

• feature:FEATURE enables the test only when a particular feature is enabled. There can be multiple feature options, all of them are required to enable the test. Note that this is different from Cargo features; you should manually set the flag with Tester::feature in the main function.

• feature:!FEATURE enables the test only when a particular feature is disabled.

Test invocation

cargo test (or if you have multiple test binaries, cargo test --test NAME) will do all the jobs. The process will exit on any error.

The test binary itself can be given options. Note that this will conflict with Cargo's own options, so they should be in principle given after --: cargo test -- --option-for-tester. Since this is annoying you can also replace preceding hyphens with pluses: cargo test ++option-for-tester will be same and easier to run.

Supported options:

• -v, --verbose: Displays all test outputs even when the test passes.

• -e, --exact-diags: Fails the test when additional reports exist. This is same to put the exact option to all tests.

• -h, --highlight-mismatch: Highlights any mismatch in the reports or output.

• -l LOCALE, --message-locale LOCALE: Let reports use given locale. Likely to fail most tests as the reports are compared by their localized texts.

• -p, --stop-on-panic: Stops on the first panic. Also enables RUST_BACKTRACE=1.

• -pp: Same to -p but will enable RUST_BACKTRACE=full instead.

• -f, --force: Run the explicitly ignored (-->8--) tests.

• -ff: Same to -f but will also run tests which are ignored by feature options.

• The implementation can supply its own options by adding Testing::augment_args method.

The remaining argument, if any, is a regular expression for filtering test names.