skipif

skipif provides an attribute macro called skip_if. This macro allows the user to specify that a rust test case should be skipped if a specified condition is met. skip_if currently supports the following conditions:

• missing_env(VAR1, VAR2, ...). If the specified environment variable(s) is not set on the cargo test process then the test case should be skipped.

This is useful for tests that integrate with external systems such as Kubernetes, databases, cloud APIs, etc.

When one or more specified conditions are met then the macro rewrites the test case name to append _SKIPPED to avoid the misleading appearance of a test case passing as might normally be the case for a test requiring an environment variable to specify an external integration config.

Motivation

Programming language test runners often offer some way to programmatically mark a test case as skipped at runtime. A good example of this is Golang, where test cases can call the testing.T.SkipNow method to specify that the current running test case should be marked as skipped. In the Golang case, it's up to the programmer to determine whether a test case should be skipped for any reason and this is how they do it.

In Rust, we have the builtin #[ignore] attribute macro, but it lacks any mechanism for programmatically determining at runtime whether or not to skip a given test. Instead, the best recourse we have is either a macro like skip_if OR returning early indicating success or failure. For example:

#[test]
fn my_supercool_test_succeeds() {
  if std::env::var("DATABASE_URL").is_err() {
    // variable is unset. oh well, i guess this test passes
    return;
  }
}

#[test]
fn my_supercool_test_fails() {
  // variable is unset. oh well, i guess this test fails
  assert!(std::env::var("DATABASE_URL").is_ok());
}

How skip_if Works

This is what a test case utilizing skip_if looks like:

#[skipif::skip_if(missing_env(DATABASE_URL))]
#[test]
fn my_supercool_test() -> std::result::Result<(), ()> {
  assert!(false);
}

During the macro expansion phase of the cargo test compilation, the macro checks for the presence of an environment variable named DATABASE_URL. If found, it expands to essentially the same test fn as shown. If the variable isn't found, the following happens:

• the test name gets _SKIPPED appended
• the test body is removed such that the test case trivially passes
• the test fn output signature is set to -> ()

So we end up with a test case that looks more like:

#[test]
fn my_supercool_test_SKIPPED() {}

The advantage, compared to the my_supercool_test_succeeds example shown above is that instead of ending up with the following in our test suite output:

test my_supercool_test ... ok

we get:

test my_supercool_test_SKIPPED ... ok

The _SKIPPED at the end gives developers runnign the test case the information they need to know that the test case didn't merely pass but was SKIPPED.

Many folks might prefer the behavior in the my_supercool_test_fails example above and that's fine. skip_if is intended for people who don't want that behavior.

Gotchas

Attribute macro order

Attribute macro order matters here. The [skipif::skip_if(...)] macro must precede any [test]-like macro (eg [tokio::test], [sqlx::test]). If the skip_if macro doesn't get chance to rename the test function then a [#test]-like macro will capture the wrong function name to be executed in test main.

Recompilation

This macro leads to different behaviors at test run time based on the conditions present during test compile time. But the cargo compilation step doesn't automatically recognize, for example with the missing_env contidion) that an environment variable relevant to tests has changed.

Taking the my_supercool_test example above as an example, imagine the following sequence of events:

1. DATABASE_URL is unset and you run cargo test.
2. The cargo test process compiles the test binary.
   • During compilation, the macro is expanded and we end up with my_supercool_test_SKIPPED.
3. The cargo test process executes the compiled binary.
4. The test case my_supercool_test_SKIPPED trivially passes.
5. You realize you need to set DATABASE_URL so you do so without changing any code. You run cargo test again.
6. The cargo test process executes the compiled binary.
7. The test case my_supercool_test_SKIPPED trivially passes.

But wait, why didn't the second invocation of cargo test re-compile the test binary? Because cargo (or rustc, I don't know) caches build output and only re-compiles if one of the compilation inputs changes. In the case of my_supercool_test and the missing_env condition, arbitrary environment variables are not considered to be compilation inputs.

However, not all is lost! There is a workaround to fix this in the form of [cargo build scripts][builtscript]. Specifically, you can use the cargo:rerun-if-env-changed=<SUPERCOOL_ENVVAR> instruction from a build script to tell cargo to consider <SUPERCOOL_ENVVAR> as a compilation input. Here is an example build script that would do the trick for my_supercool_test:

fn main() {
  println!("cargo:rerun-if-env-changed=DATABASE_URL")
}

Just drop that into build.rs right next to Cargo.toml and you would get test cases that recompile whenever the DATABASE_URL value changes. The downside of this approach is that everything else in your crate would also get rebuilt ¯_(ツ)_/¯ whenever the variable changes.

Similar projects

• needs_env_var