#testing #tests #golden

goldentests

A golden file testing library where tests can be configured within the same test file

11 releases

0.3.7 Sep 21, 2020
0.3.6 Aug 3, 2020
0.3.4 Jul 11, 2020
0.2.2 Jul 4, 2020

#65 in Testing

Download history 14/week @ 2020-11-09 21/week @ 2020-11-16 20/week @ 2020-11-23 6/week @ 2020-11-30 11/week @ 2020-12-07 15/week @ 2020-12-14 2/week @ 2020-12-21 40/week @ 2020-12-28 22/week @ 2021-01-04 20/week @ 2021-01-11 21/week @ 2021-01-18 13/week @ 2021-01-25 6/week @ 2021-02-01 16/week @ 2021-02-08 122/week @ 2021-02-15 13/week @ 2021-02-22

202 downloads per month

Custom license

4.5MB
322 lines

Golden Tests

Travis (.org) crates.io docs.rs

Golden tests is a golden file testing library configured so that tests can be created and edited from the test files alone without ever touching the rust source code of the test.

Why golden tests?

Golden tests allow you to specify the output of some command within a file and automatically ensure that that output doesn't change. If it does, goldentests will show an error-diff showing the expected and actual output. This way, whenever the output of something changes a human can see the change and decide if it should be kept or is a bug and should be reverted.

What are golden tests useful for?

Golden tests are especially useful for applications that take a file as input and produce output of some kind. For example: compilers and config-parsers (well, parsers in general) are two such applications that can benefit from automated golden tests. In the case of a config parser, you would be able to provide many config examples as tests and ensure that your parser was able to read the files with the expected stdout/stderr output and exit code.

Example cargo test Output

example image

Getting Started

To get started plop this into your Cargo.toml:

goldentests = "0.3.0"

And create an integration test in tests/goldentests.rs. The specific name doesn't matter as long as the test can be picked up by cargo. A typical usage looks like this:

use goldentests::{ TestConfig, TestResult };

#[test]
fn run_golden_tests() -> TestResult<()> {
    let config = TestConfig::new("target/debug/my-binary", "my-test-path", "// ")?;
    config.run_tests()
}

This will tell goldentests to find all files recursively in my-test-path and run target/debug/my-binary to use the files in some way to produce the expected output. For example, if we're testing a compiler for a C-like language a test file for us may look like this:

puts("Hello, World!");

// args: --run
// expected stdout:
// Hello, World!

This will run the command target/debug/my-binary --run my-test-path/example.c and will issue an error if the output of the command is not "Hello, World!".

Note that there are test keywords args: and expected stdout: embedded in the comments. This is what the "// " parameter was in the rust example. You can change this parameter to change the prefix that goldentests looks for when parsing a file. For most languages, this should be a comment of some kind. E.g. if we we're testing haskell, we would use -- as the test-line prefix.

Advanced Usage

Here is the full set of keywords goldentests looks for in the file:

  • args: <single-line-string>: Anything after this keyword will be used as the command-line arguments for the program that was specified when creating the TestConfig.
  • expected stdout: <multi-line-string>: This keyword will continue reading characters, appending them to the expected stdout output until it reaches a line that does not start with the test prefix ("// " in the example above). If the stdout when running the program differs from the string given here, an appropriate error will be issued with a given diff. Defaults to "".
  • expected stderr: <multi-line-string>: The same as expected stdout: but for the stderr stream. Also defaults to "".
  • expected exit status: [i32]: If specified, goldentests will issue an error if the exit status differs to what is expected. Defaults to None (exit status is ignored by default).

You can even configure the specific keywords used if you want. For any further information, check out goldentest's documentation here.

Dependencies