#failure #panic #chaos #fault-injection

emergency_brake

emergency_brake is a simple and easy to use process or service monitor that will immediately terminate the execution of a process or service on a critical dependency failure

1 unstable release

0.1.0 Sep 25, 2023

#606 in Testing

Apache-2.0

12KB
147 lines

Emergency Brake

emergency_brake is a simple and easy to use process or service monitor that will immediately terminate the execution of a process or service on a critical dependency failure.

Usage

eBrake creates a moving sample window of the last N samples. If the number of failures in the sample window exceeds the threshold, the process or service will be terminated. The sample window is a circular buffer, so the oldest sample will be replaced by the newest sample.

use emergency_brake::*;

fn main() {
    let sample_window_size = 25;
    let threshold = 3;
    let mut ebrake = EBrake::new(sample_window_size, threshold);
    loop:
        // Check service status
        let service_status: bool = check_service_status('service.foo.com');
        // Add the sample to the sample window and trigger if necessary
        ebrake.add_sample(service_status);
        ebrake.trigger();
        // Do something critical
        ...
}

lib.rs:

eBrake creates a moving sample window of the last N samples. If the number of failures in the sample window exceeds the threshold, the process or service will be terminated. The sample window is a circular buffer, so the oldest sample will be replaced by the newest sample.

Examples

This will use the sample and trigger functions separately.

use emergency_brake::*;
let sample_window_size = 25;
let failure_threshold = 3;
let mut ebrake = EBrake::new(sample_window_size, failure_threshold);
for _ in 0..sample_window_size {
   ebrake.add_sample(true);
}
assert_eq!(ebrake.trigger(&Trigger::Panic), false);

This will use the trigger_on_sample function.

use emergency_brake::*;
let sample_window_size = 25;
let failure_threshold = 3;
let mut ebrake = EBrake::new(sample_window_size, failure_threshold);
for _ in 0..sample_window_size {
  ebrake.trigger_on_sample(true, &Trigger::Panic);
}
assert_eq!(ebrake.trigger(&Trigger::Panic), false);

Kelsea Blackwell (c) 2023 See LICENSE for licensing information.

Dependencies

~0.3–11MB
~131K SLoC