5 unstable releases
0.3.1 | Feb 27, 2022 |
---|---|
0.3.0 | Sep 10, 2021 |
0.2.1 | Dec 5, 2020 |
0.2.0 | Sep 27, 2020 |
0.1.0 | Jan 12, 2019 |
#424 in Testing
706 downloads per month
Used in diceprop
305KB
6.5K
SLoC
dicetest
Framework for writing tests with randomly generated test data.
Status of this crate
The author does not consider this crate as stable yet. Changes will be documented in the changelog.
Example
Here's an example of an incorrect sort function tested with dicetest:
fn bubble_sort<T: Ord>(slice: &mut [T]) {
let len = slice.len();
for _ in 0..len {
for j in 1..len - 1 {
let jpp = j + 1;
if slice[j] > slice[jpp] {
slice.swap(j, jpp);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use dicetest::prelude::*;
#[test]
fn result_of_bubble_sort_is_sorted() {
Dicetest::repeatedly().run(|mut fate| {
let mut v = fate.roll(dice::vec(dice::u8(..), ..));
hint!("unsorted: {:?}", v);
bubble_sort(&mut v);
hint!(" sorted: {:?}", v);
let is_sorted = v.windows(2).all(|w| w[0] <= w[1]);
assert!(is_sorted);
})
}
}
Running cargo test
produces the following output:
The test failed after 31 passes.
# Config
- seed: 3713861809241954222
- start limit: 0
- end limit: 100
- passes: 200
# Counterexample
- run code: "/yiA1sab3S4UnCf4ozyMpxMxzg1NtFybCuYLHy0/oscDAAAAAAAAAA=="
- limit: 3
- hints:
- unsorted: [201, 209, 2]
- sorted: [201, 2, 209]
- error: assertion failed: is_sorted
You can rerun the counterexample by setting an environment variable:
DICETEST_DEBUG=/yiA1sab3S4UnCf4ozyMpxMxzg1NtFybCuYLHy0/oscDAAAAAAAAAA== cargo test
Or you can modify the test:
Dicetest::debug("/yiA1sab3S4UnCf4ozyMpxMxzg1NtFybCuYLHy0/oscDAAAAAAAAAA==").run(|mut fate| {
// ...
})
Features
These features are available:
- Generators for many libstd types (
u8
,String
,Vec
, etc.). - Generators for functions (
FnMut
,FnOnce
,Fn
). - Generator combinators (
map
,flat_map
,zip
, etc.). - Integration of
rand::distributions::Distribution
. - Integration of
quickcheck::Arbitrary
(without shrinking). - Configurable test runner.
- Utilities for debugging tests (
hints
andstats
).
These features are missing:
- Shrinking of counterexamples.
- Custom pseudorandom number generators.
- Own type class for arbitrary types.
Alternatives
- Write down your test data and use a loop.
- Use the crate quickcheck.
- Use the crate proptest.
Guide
This section will guide you through the most important concepts and features of dicetest.
Pseudorandomness
The type Seed
allows to determine the pseudorandomness. You can either use a fixed
Seed
or a random Seed
:
use dicetest::Seed;
println!("{:?}", Seed(42));
// Output: Seed(42)
println!("{:?}", Seed::random());
// Output: Seed(8019292413750407764)
The Seed
can be used to initialize the pseudorandom number generator Prng
. For each
Seed
the Prng
provides a different infinite pseudorandom sequence of u64
s
use dicetest::{Prng, Seed};
fn print_random_values(mut prng: Prng) {
for _ in 0..3 {
print!("{:?}, ", prng.next_number());
}
println!("...");
}
print_random_values(Prng::from_seed(Seed(42)));
// Output: 16628028624323922065, 3476588890713931039, 59688652182557721, ...
print_random_values(Prng::from_seed(Seed(42)));
// Output: 16628028624323922065, 3476588890713931039, 59688652182557721, ...
print_random_values(Prng::from_seed(Seed::random()));
// Output: 4221507577048064061, 15374206214556255352, 4977687432463843847, ...
print_random_values(Prng::from_seed(Seed::random()));
// Output: 11086225885938422405, 9312304973013875005, 1036200222843160301, ...
Dice
Although Prng
can only generate pseudorandom u64
s, the u64
s can be used for constructing
more complex values. The traits DieOnce
and Die
represents Prng
-based generators for
values of any type.
An implementor of DieOnce
is a generator that can be used a single time
(similar to FnOnce
).
use dicetest::prelude::*;
let xx = "xx".to_string();
let yy = "yy".to_string();
// This generator implements `DieOnce`.
// It chooses one of the `String`s without cloning them.
let xx_or_yy_die = dice::one_of_once().two(xx, yy);
An implementor of Die
is a generator that can be used infinite times (similar to Fn
).
use dicetest::prelude::*;
let xx = "xx".to_string();
let yy = "yy".to_string();
// This generator implements `Die`.
// It chooses one of the `String`s by cloning them.
let xx_or_yy_die = dice::one_of().two(xx, yy);
// This generator uses `xx_or_yy_die` to generate three `String`s at once.
let three_xx_or_yy_die = dice::array::<_, _, 3>(xx_or_yy_die);
Generators can be easily implemented and composed:
use dicetest::prelude::*;
// A classic die that generates a number between 1 and 6 with uniform distribution.
let classic_die = dice::one_of().six::<u8>(1, 2, 3, 4, 5, 6);
// A loaded die that generates the number 6 more frequently.
let loaded_die =
dice::weighted_one_of().six::<u8>((1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (2, 6));
// This die generates the result of the function.
let die_from_fn = dice::from_fn(|_| 42);
// This die generates always the same `String` by cloning the original one.
let foo_die = dice::just("foo".to_string());
// This die generates an arbitrary byte.
let byte_die = dice::u8(..);
// This die generates a non-zero byte.
let non_zero_byte_die = dice::u8(1..);
// This die generates a `Vec` that contains an arbitrary number of arbitrary bytes.
let bytes_die = dice::vec(dice::u8(..), ..);
// This die generates a `Vec` that contains up to 10 arbitrary bytes.
let up_to_ten_bytes_die = dice::vec(dice::u8(..), ..=10);
// This die generates an arbitrary wrapped byte.
struct WrappedByte(u8);
let wrapped_byte_die = dice::u8(..).map(WrappedByte);
// This die generates a permutation of `(0..=n)` for an arbitrary `n`.
let permutation_die = dice::length(0..).flat_map(|n| {
let vec = (0..=n).collect::<Vec<_>>();
dice::shuffled_vec(vec)
});
The struct Fate
is necessary for using DieOnce
or Die
. It contains two parameters:
Prng
: Provides the pseudorandomu64
s that the implementor ofDieOnce
orDie
can use for constructing more complex values. The implementor should only use this as its source of randomness.Limit
: The upper limit for the length of dynamic data structures generated by the implementor ofDieOnce
orDie
. The implementor is allowed to freely interpret or even ignore this value.
use dicetest::prelude::*;
use dicetest::{Limit, Prng};
// Provides the randomness for the generator and will be mutated when used.
let mut prng = Prng::from_seed(0x5EED.into());
// Limits the length of dynamic data structures. The generator has only read access.
let limit = Limit(5);
// Contains all parameters necessary for using `DieOnce` or `Die`.
let mut fate = Fate::new(&mut prng, limit);
// Generator for a `Vec` with an arbitrary length.
let vec_die = dice::vec(dice::u8(..), ..);
// Generates a `Vec`. Although `vec_die` can generate a `Vec` with an arbitrary length,
// the length of the actual `Vec` is limited by `limit`.
let vec = fate.roll(vec_die);
assert!(vec.len() <= 5);
println!("{:?}", vec);
// Output: [252, 231, 153, 0]
Tests
If you want to write a test with randomly generated test data you can use the test
builderDicetest
:
- It can be configured via source code or environment variables.
- It runs your test repeatedly with different seeds.
- It logs useful information that helps you to debug your test.
use dicetest::prelude::*;
#[test]
fn test_foo() {
// Runs your test with default configuration.
Dicetest::repeatedly().run(|fate| {
// Write your test here.
});
}
#[test]
fn test_bar() {
// Runs your test with custom configuration.
Dicetest::repeatedly().passes(10000).run(|fate| {
// Write your test here.
});
}
The closure contains your test. With the passed fate
you can generate test data and make
assertions. If the closure panics, Dicetest
catches the panic, logs the test result to
stdout and resumes the panic.
Hints
Hints can be used to analyze a single test run. In most cases you want to analyze the counterexample. Use it to reveal what test data were generated or which branches were taken:
use dicetest::prelude::*;
#[test]
fn test_foo() {
Dicetest::repeatedly().run(|mut fate| {
let x = fate.roll(dice::u8(1..=5));
hint_debug!(x);
let y = fate.roll(dice::u8(1..=3));
if y != x {
hint!("took branch if with y = {}", y);
assert_eq!(3, y);
} else {
hint!("took branch else");
}
})
}
Running the test produces the following output:
The test failed after 0 passes.
# Config
- seed: 10929669535587280453
- start limit: 0
- end limit: 100
- passes: 200
# Counterexample
- run code: "JfXG0LRXjKUMu+YmdrF38/GstRdeLAeMRTKskCQcgNoAAAAAAAAAAA=="
- limit: 0
- hints:
- x = 5
- took branch if with y = 1
- error: assertion failed: `(left == right)`
left: `3`,
right: `1`
Stats
Stats can be used to analyze multiple test runs. Use it to reveal the distribution of generated test data or the probability of branches:
use dicetest::prelude::*;
#[test]
fn test_foo() {
Dicetest::repeatedly().run(|mut fate| {
let x = fate.roll(dice::u8(1..=5));
stat_debug!(x);
let y = fate.roll(dice::u8(1..=3));
if y != x {
stat!("branch", "if with y = {}", y)
} else {
stat!("branch", "else");
}
})
}
Running the test with the environment variable DICETEST_STATS_ENABLED=true
produces
the following output:
The test withstood 200 passes.
# Config
- seed: 5043079553183914912
- start limit: 0
- end limit: 100
- passes: 200
# Stats
- branch:
- 29.50% (59): if with y = 1
- 27.50% (55): if with y = 3
- 22.50% (45): if with y = 2
- 20.50% (41): else
- x:
- 31.50% (63): 1
- 22.00% (44): 5
- 17.00% (34): 2
- 15.50% (31): 4
- 14.00% (28): 3
Environment variables
You can use environment variables to configure your tests without changing the source code.
See the documentation of Dicetest
for a full list of supported environment variables.
Here are some examples:
- You want to debug the counterexample of
mytest
with its run code (copied from the test result):
DICETEST_DEBUG=ABIDje/+CYVkmmCVTwKJ2go6VrzZWMjO2Bqc9m3b3h0DAAAAAAAAAA== cargo test mytest
- You want to reproduce the result of
mytest
with its seed (copied from the test result):
DICETEST_SEED=795359663177100823 cargo test mytest
- You want to see the stats of
mytest
:
DICETEST_STATS_ENABLED=true cargo test -- --show-output mytest
- You want to run
mytest
with more passes and bigger test data:
DICETEST_PASSES_MULTIPLIER=10 DICETEST_LIMIT_MULTIPLIER=2 cargo test mytest
- You want to run
mytest
with a single test run and see the test result:
DICETEST_MODE=once cargo test -- --show-output mytest
Feature flags
There are several feature flags for disabling runtime overhead or enabling additional features at compile time.
hints
(enabled by default)
Enables or disables the hints feature at compile time. If disabled, all hints operations are no-ops.
stats
(enabled by default)
Enables or disables the stats feature at compile time. If disabled, all stats operations are no-ops.
rand_core
(disabled by default)
If enabled, dicetest::Prng
and dicetest::Fate
implements the rand_core::RngCore
trait.
rand_full
(disabled by default, alias for rand_core,rand
)
If enabled, Fate::roll_distribution
and dice::from_distribution
are available.
This allows to generate values and create Die
s from implementations
of rand::distributions::Distribution
.
use dicetest::prelude::*;
use dicetest::{Limit, Prng};
let mut prng = Prng::from_seed(0x5EED.into());
let limit = Limit(5);
let mut fate = Fate::new(&mut prng, limit);
// Generate a value from a `rand::distributions::Distribution`
let byte: u8 = fate.roll_distribution(rand::distributions::Standard);
println!("{:?}", byte);
// Output: 28
// Create a `Die` from a `rand::distributions::Distribution`
let byte_die = dice::from_distribution(rand::distributions::Standard);
let bytes_die = dice::vec(byte_die, 1..);
let bytes: Vec<u8> = fate.roll(bytes_die);
println!("{:?}", bytes);
// Output: [236, 205, 151, 229]
quickcheck_full
(disabled by default, alias for rand_core,quickcheck
)
If enabled, Fate
implements the quickcheck::Gen
trait and Fate::roll_arbitrary
and
dice::arbitrary
are available. This allows to generate values and create Die
s for types
that implements quickcheck::Arbitrary
.
use dicetest::prelude::*;
use dicetest::{Limit, Prng};
let mut prng = Prng::from_seed(0x5EED.into());
let limit = Limit(5);
let mut fate = Fate::new(&mut prng, limit);
// Generate a value of a type that implements `quickcheck::Arbitrary`
let byte: u8 = fate.roll_arbitrary();
println!("{:?}", byte);
// Output: 0
// Create a `Die` for a type that implements `quickcheck::Arbitrary`
let byte_die = dice::arbitrary();
let bytes_die = dice::vec(byte_die, 1..);
let bytes: Vec<u8> = fate.roll(bytes_die);
println!("{:?}", bytes);
// Output: [1, 4, 4, 2]
License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Dependencies
~74–610KB
~10K SLoC