60 releases (26 breaking)
0.27.1 | Oct 7, 2024 |
---|---|
0.26.5 | Sep 11, 2024 |
0.25.0 | Jul 24, 2024 |
0.22.2 | Feb 27, 2024 |
0.1.0 | Jul 28, 2022 |
#15 in Programming languages
24,152 downloads per month
Used in 4 crates
260KB
6.5K
SLoC
A test runner that builds tests with rustc or cargo (or any other compiler with some configuration effort) and compares the output of the compiler with a file that you check into git. This allows you to test how your libraries show up to your users when the library is used wrongly and emits errors.
Usage
See examples directory for how to use this in your own crate.
To be able to use it with cargo test
, you need to put
[[test]]
name = "your_test_file"
harness = false
into your Cargo.toml
, otherwise cargo test
will only look for #[test]
s and
not run your fn main()
that actually executes ui_test
Implicit (and possibly surprising) behavior
- Tests are run in order of their filenames (files first, then recursing into folders). So if you have any slow tests, prepend them with a small integral number to make them get run first, taking advantage of parallelism as much as possible (instead of waiting for the slow tests at the end).
cargo test --test your_test_name -- --help
lists the commands you can specify for filtering, blessing and making your tests less verbose.- Since
cargo test
on its own runs all tests, usingcargo test -- --check
will not work on its own, butcargo test -- --quiet
andcargo test -- some_test_name
will work just fine, as the CLI matches.
- Since
- if there is a
.stdin
file with the same filename as your test, it will be piped as standard input to your program.
Supported comment annotations
If your test tests for failure, you need to add a //~
annotation where the error is happening
to ensure that the test will always keep failing at the annotated line. These comments can take two forms:
//~ LEVEL: XXX
matches by error level and message textLEVEL
can be one of the following (descending order):ERROR
,HELP
,WARN
orNOTE
- If a level is specified explicitly, all diagnostics of that level or higher need an annotation. To avoid this see
//@require-annotations-for-level
- This checks the output before normalization, so you can check things that get normalized away, but need to be careful not to accidentally have a pattern that differs between platforms.
- if
XXX
is of the form/XXX/
it is treated as a regex instead of a substring and will succeed if the regex matches.
//~ CODE
matches by diagnostic code.CODE
can take multiple forms such as:E####
,lint_name
,tool::lint_name
.- This will only match a diagnostic at the
ERROR
level.
In order to change how a single test is tested, you can add various //@
comments to the test.
Any other comments will be ignored, and all //@
comments must be formatted precisely as
their command specifies, or the test will fail without even being run.
//@ignore-C
avoids running the test when conditionC
is met.C
can betarget: XXX YYY
, which checks whether the target triple containsXXX
orYYY
.C
can behost: XXX YYY
, which checks whether the host triple containsXXX
orYYY
.C
can also bebitwidth:
followed by one or more space separated integer size like64
,32
or16
.C
can also beon-host
, which will only run the test during cross compilation testing.
//@only-C
only runs the test when conditionC
is met. The conditions are the same as withignore
.//@needs-asm-support
only runs the test when the target supportsasm!
.//@stderr-per-bitwidth
produces one stderr file per bitwidth, as they may differ significantly sometimes//@error-in-other-file: XXX
can be used to check for errors that can't have//~
patterns due to being reported in other files.//@revisions: XXX YYY
runs the test once for each space separated name in the list- emits one stderr file per revision
//~
comments can be restricted to specific revisions by adding the revision name after the~
in square brackets://~[XXX]
//@
comments can be restricted to specific revisions by adding the revision name after the@
in square brackets://@[XXX]
- Note that you cannot add revisions to the
revisions
command.
- Note that you cannot add revisions to the
//@compile-flags: XXX
appendsXXX
to the command line arguments passed to the rustc driver- you can specify this multiple times, and all the flags will accumulate
//@rustc-env: XXX=YYY
sets the env varXXX
toYYY
for the rustc driver execution.- for Miri these env vars are used during compilation via rustc and during the emulation of the program
- you can specify this multiple times, accumulating all the env vars
//@normalize-stderr-test: "REGEX" -> "REPLACEMENT"
replaces all matches ofREGEX
in the stderr withREPLACEMENT
. The replacement may specify$1
and similar backreferences to paste captures.- you can specify multiple such commands, there is no need to create a single regex that handles multiple replacements that you want to perform.
//@require-annotations-for-level: LEVEL
can be used to change the level of diagnostics that require a corresponding annotation.- this is only useful if there are any annotations like
HELP
,WARN
orNOTE
, as these would automatically require annotations for all other diagnostics of the same or higher level.
- this is only useful if there are any annotations like
//@check-pass
requires that a test has no error annotations, emits no errors, and exits successfully with exit/status code 0.//@edition: EDITION
overwrites the default edition (2021) to the given edition.//@no-rustfix
do not run rustfix on tests that have machine applicable suggestions.//@aux-build: filename
looks for a file in theauxiliary
directory (within the directory of the test), compiles it as a library and links the current crate against it. This allows you import the crate withextern crate
or just viause
statements. This will automatically detect aux files that are proc macros and build them as proc macros.//@run
compiles the test and runs the resulting binary. The resulting binary must exit successfully. Stdout and stderr are taken from the resulting binary. Any warnings during compilation are ignored.- You can also specify a different exit code/status that is expected via e.g.
//@run: 1
or//@run: 101
(the latter is the standard Rust exit code for panics). - run tests collect the run output into
.run.stderr
and.run.stdout
respectively. - if a
.run.stdin
file exists, it will be piped as standard input to your test's execution.
- You can also specify a different exit code/status that is expected via e.g.
Significant differences to compiletest-rs
ignore-target-xxx
andonly-target-xxx
requires thetarget-
prefix before thexxx
substring to be matched against target triples, whereas compiletest allows plainignore-xxx
without thetarget-
prefix. The substringxxx
must also be a substring of target triples, and special collections such asmacos
/unix
in compiletest is not supported.- only supports
ui
tests - tests are run in named order, so you can prefix slow tests with
0
in order to make them get run first aux-build
s require specifying nested aux builds explicitly and will not allow you to reference siblingaux-build
s' artifacts.
Dependencies
~11–22MB
~287K SLoC