Goldenscript

A Rust testing framework loosely based on Cockroach Labs' datadriven framework for Go. It combines several testing techniques that make it easy and efficient to write and update test cases:

• Golden master testing (aka characterization testing or historical oracle)
• Data-driven testing (aka table-driven testing or parameterized testing)
• Keyword-driven testing

A goldenscript is a plain text file that contains a set of arbitrary input commands and their expected text output, separated by ---:

command
---
output

command argument key=value
---
output

The commands are executed by a provided Runner. The expected output is usually not written by hand, but instead generated by running tests with the environment variable UPDATE_GOLDENFILES=1 and then verified by inspection before it is checked in to version control. Tests will fail with a diff if they don't match the expected output.

This approach is particularly useful when testing complex stateful systems, such as computer language parsing, operations on a key/value store, concurrent transactions in a SQL database, or communication between a cluster of Raft nodes. It can be very tedious and labor-intensive to write and assert such cases by hand, so scripting and recording these interactions often yields much better test coverage at a fraction of the cost.

Internally, the goldenfile crate is used to manage golden files.

Documentation

See the crate documentation for more information.

Example

Below is an example goldenscript for the dateparser timestamp parsing crate. It supports a single parse command taking a timestamp argument, and outputs the parsed timestamp in RFC 3339 format.

Running cargo test asserts that the runner's output matches the file, failing with a diff otherwise. Running UPDATE_GOLDENFILES=1 cargo test (re)populates the file with the runner’s output, verified by inspection.

parse 2024-04-30
---
2024-04-30T00:00:00+00:00

# Test various other date formats.
parse 2024-Apr-30
parse 2024.04.30
parse 04/30/2024
---
2024-04-30T00:00:00+00:00
2024-04-30T00:00:00+00:00
2024-04-30T00:00:00+00:00

# Test some error cases.
parse 30.04.2024
parse 30/04/2024
parse 30/04/24
---
Error: 30.04.2024 did not match any formats.
Error: 30/04/2024 did not match any formats.
Error: 30/04/24 did not match any formats.

# Strings containing special characters must be quoted using " or '.
parse "2024-04-30 11:55:32"
parse '2024年04月30日11时55分32秒'
---
2024-04-30T11:55:32+00:00
2024-04-30T11:55:32+00:00

The corresponding runner for this script:

struct DateParserRunner;

impl goldenscript::Runner for DateParserRunner {
    fn run(&mut self, command: &goldenscript::Command) -> Result<String, String> {
        // Only accept a parse command with a single argument.
        if command.name != "parse" {
            return Err(format!("invalid command {}", command.name))
        }
        if command.args.len() != 1 {
            return Err("parse takes 1 argument".to_string())
        }

        // Parse the timestamp, and output the RFC 3339 timestamp or error string.
        let input = &command.args[0].value;
        match dateparser::parse_with(input, &chrono::offset::Utc, chrono::NaiveTime::MIN) {
            Ok(datetime) => Ok(datetime.to_rfc3339()),
            Err(error) => Ok(format!("Error: {error}")),
        }
    }
}

#[test]
fn dateparser() -> std::io::Result<()> {
    goldenscript::run(&mut DateParserRunner, "tests/scripts/dateparser")
}

For more information on goldenscript syntax and features, see the crate documentation.