arbtest

A powerful property-based testing library with a tiny API and a small implementation.

use arbtest::arbtest;

#[test]
fn all_numbers_are_even() {
    arbtest(|u| {
        let number: u32 = u.arbitrary()?;
        assert!(number % 2 == 0);
        Ok(())
    });
}

lib.rs:

A powerful property-based testing library with a tiny API and a small implementation.

use arbtest::arbtest;

#[test]
fn all_numbers_are_even() {
    arbtest(|u| {
        let number: u32 = u.arbitrary()?;
        assert!(number % 2 == 0);
        Ok(())
    });
}

Features:

• single-function public API,
• no macros,
• automatic minimization,
• time budgeting,
• fuzzer-compatible tests.

The entry point is the arbtest function. It accepts a single argument --- a property to test. A property is a function with the following signature:

/// Panics if the property does not hold.
fn property(u: &mut arbitrary::Unstructured) -> arbitrary::Result<()>

The u argument is a finite random number generator from the arbitrary crate. You can use u to generate pseudo-random structured data:

let ints: Vec<u32> = u.arbitrary()?;
let fruit: &str = u.choose(&["apple", "banana", "cherimoya"])?;

Or use the derive feature of the arbitrary crate to automatically generate arbitrary types:

#[derive(arbitrary::Arbitrary)]
struct Color { r: u8, g: u8, b: u8 }

let random_color = u.arbitrary::<Color>()?;

Property function should use randomly generated data to assert some interesting behavior of the implementation, which should hold for any values. For example, converting a color to string and then parsing it back should result in the same color:

#[test]
fn parse_is_display_inverted() {
    arbtest(|u| {
        let c1: Color = u.arbitrary();
        let c2: Color = c1.to_string().parse().unwrap();
        assert_eq!(c1, c2);
        Ok(())
    })
}

After you have supplied the property function, arbtest repeatedly runs it in a loop, passing more and more arbitrary::Unstructured bytes until the property panics. Upon a failure, a seed is printed. The seed can be used to deterministically replay the failure.

thread 'all_numbers_are_even' panicked at src/lib.rs:116:9:
assertion failed: number % 2 == 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

arbtest failed!
    Seed: 0xa88e234400000020

More features are available with builder-style API on the returned ArbTest object.

Time Budgeting

arbtest(property).budget_ms(1_000);
arbtest(property).budget(Duration::from_secs(1));

The budget function controls how long the search loop runs, default is one hundred milliseconds. This default can be overridden with ARBTEST_BUDGET_MS environmental variable.

Size Constraint

arbtest(property)
    .size_min(1 << 4)
    .size_max(1 << 16);

Internally, arbitrary::Unstructured is just an &[u8] --- a slice of random bytes. The length of this slice determines how much randomness your tests gets to use. A shorter slice contains less entropy and leads to a simpler test case.

The size_min and size_max parameters control the length of this slice: when looking for a failure, arbtest progressively increases the size from size_min to size_max.

Note when trying to minimize a known failure, arbtest will try to go even smaller than size_min.

Replay and Minimization

arbtest(property).seed(0x92);
arbtest(property).seed(0x92).minimize();

When a seed is specified, arbtest uses the seed to generate a fixed Unstructured and runs the property function once. This is useful to debug a test failure after a failing seed is found through search.

If in addition to seed minimize is set, then arbtest will try to find a smaller seed which still triggers a failure. You could use budget to control how long the minimization runs.

When the Code Gets Run

The arbtest function doesn't immediately run the code. Instead, it returns an ArbTest builder object that can be used to further tweak the behavior. The actual execution is triggered from the ArbTest::drop. If panicking in drop is not your thing, you can trigger the execution explicitly using ArbTest::run method:

let builder = arbtest(property);
drop(builder); // This line actually runs the tests.

arbtest(property).run(); // Request the run explicitly.

Errors

Property failures should be reported via a panic, for example, using assert_eq! macros. Returning an Err(arbitrary::Error) doesn't signal a test failure, it just means that there isn't enough entropy left to complete the test. Instead of returning an arbitrary::Error, a test might choose to continue in a non-random way. For example, when testing a distributed system you might use the following template:

while !u.is_empty() && network.has_messages_in_flight() {
    network.drop_and_permute_messages(u);
    network.deliver_next_message();
}
while network.has_messages_in_flight() {
    network.deliver_next_message();
}

Imports

Recommended way to import:

[dev-dependencies]
arbtest = "0.3"

#[cfg(test)]
mod tests {
    use arbtest::{arbtest, arbitrary};

    fn my_property(u: &mut arbitrary::Unstructured) -> arbitrary::Result<()> { Ok(()) }
}

If you want to #[derive(Arbitrary)], you need to explicitly add Cargo.toml dependency for the arbitrary crate:

[dependencies]
arbitrary = { version = "1", features = ["derive"] }

[dev-dependencies]
arbtest = "0.3"

#[derive(arbitrary::Arbitrary)]
struct Color { r: u8, g: u8, b: u8 }

#[cfg(test)]
mod tests {
    use arbtest::arbtest;

    #[test]
    fn display_parse_identity() {
        arbtest(|u| {
            let c1: Color = u.arbitrary()?;
            let c2: Color = c1.to_string().parse();
            assert_eq!(c1, c2);
            Ok(())
        });
    }
}

Note that arbitrary is a non-dev dependency. This is not strictly required, but is helpful to allow downstream crates to run their tests with arbitrary values of Color.

Design

Most of the heavy lifting is done by the arbitrary crate. Its arbitrary::Unstructured is a brilliant abstraction which works both for coverage-guided fuzzing as well as for automated minimization. That is, you can plug arbtest properties directly into cargo fuzz, API is fully compatible.

Property function uses &mut Unstructured as an argument instead of T: Arbitrary, allowing the user to generate any T they want imperatively. The smaller benefit here is implementation simplicity --- the property type is not generic. The bigger benefit is that this API is more expressive, as it allows for interactive properties. For example, a network simulation for a distributed system doesn't have to generate "failure plan" upfront, it can use u during the test run to make dynamic decisions about which existing network packets to drop!

A "seed" is an u64, by convention specified in hexadecimal. The low 32 bits of the seed specify the length of the underlying Unstructured. The high 32 bits are the random seed proper, which is feed into a simple xor-shift to generate Unstructured of the specified length.

If you like this crate, you might enjoy https://github.com/graydon/exhaustigen-rs as well.