emit_file

Current docs

Emit diagnostic events to rolling files.

lib.rs:

Emit diagnostic events to rolling files.

All file IO is performed on batches in a dedicated background thread.

This library writes newline delimited JSON by default, like:

{"ts_start":"2024-05-29T03:35:13.922768000Z","ts":"2024-05-29T03:35:13.943506000Z","module":"my_app","msg":"in_ctxt failed with `a` is odd","tpl":"in_ctxt failed with `err`","a":1,"err":"`a` is odd","lvl":"warn","span_id":"0a3686d1b788b277","span_parent":"1a50b58f2ef93f3b","trace_id":"8dd5d1f11af6ba1db4124072024933cb"}

Getting started

Add emit and emit_file to your Cargo.toml:

[dependencies.emit]
version = "0.11.0-alpha.21"

[dependencies.emit_file]
version = "0.11.0-alpha.21"

Initialize emit using a rolling file set:

fn main() {
let rt = emit::setup()
.emit_to(emit_file::set("./target/logs/my_app.txt").spawn())
.init();

// Your app code goes here

rt.blocking_flush(std::time::Duration::from_secs(30));
}

The input to [set] is a template for log file naming. The example earlier used ./target/logs/my_app.txt. From this template, log files will be written to ./target/logs, each log file name will start with my_app, and use .txt as its extension.

File naming

Log files are created using the following naming scheme:

{prefix}.{date}.{counter}.{id}.{ext}

where:

• prefix: A user-defined name that groups all log files related to the same application together.
• date: The rollover interval the file was created in. This isn't necessarily related to the timestamps of events within the file.
• counter: The number of milliseconds since the start of the current rollover interval when the file was created.
• id: A unique identifier for the file in the interval.
• ext: A user-defined file extension.

In the following log file:

log.2024-05-27-03-00.00012557.37c57fa1.txt

the parts are:

• prefix: log.
• date: 2024-05-27-03-00.
• counter: 00012557.
• id: 37c57fa1.
• ext: txt.

When files roll

Diagnostic events are only ever written to a single file at a time. That file changes when:

1. The application restarts and FileSetBuilder::reuse_files is false.
2. The rollover period changes. This is set by FileSetBuilder::roll_by_day, FileSetBuilder::roll_by_hour, and FileSetBuilder::roll_by_minute.
3. The size of the file exceeds FileSetBuilder::max_file_size_bytes.
4. Writing to the file fails.

Durability

Diagnostic events are written to files in asynchronous batches. Under normal operation, after a call to emit::Emitter::blocking_flush, all events emitted before the call are guaranteed to be written and synced via Rust's std::fs::File::sync_all method. This is usually enough to guarantee durability.

Handling IO failures

If writing a batch fails while attempting to write to a file then the file being written to is considered poisoned and no future attempts will be made to write to it. The batch will instead be retried on a new file. Batches that fail attempting to sync are not retried. Since batches don't have explicit transactions, it's possible on failure for part or all of the failed batch to actually be present in the original file. That means diagnostic events may be duplicated in the case of an IO error while writing them.

Troubleshooting

If you're not seeing diagnostics appear in files as expected, you can rule out configuration issues in emit_file by configuring emit's internal logger, and collect metrics from it:

use emit::metric::Source;

fn main() {
// 1. Initialize the internal logger
//    Diagnostics produced by `emit_file` itself will go here
let internal = emit::setup()
.emit_to(emit_term::stdout())
.init_internal();

let mut reporter = emit::metric::Reporter::new();

let rt = emit::setup()
.emit_to({
let files = emit_file::set("./target/logs/my_app.txt").spawn();

// 2. Add `emit_file`'s metrics to a reporter so we can see what it's up to
//    You can do this independently of the internal emitter
reporter.add_source(files.metric_source());

files
})
.init();

// Your app code goes here

rt.blocking_flush(std::time::Duration::from_secs(30));

// 3. Report metrics after attempting to flush
//    You could also do this periodically as your application runs
reporter.emit_metrics(&internal.emitter());
}

Diagnostics include when batches are written, and any failures observed along the way.