metrics-tracing-context

A crate to use tracing context as metrics labels.

lib.rs:

Use tracing::span! fields as metrics labels.

The metrics-tracing-context crate provides tools to enable injecting the contextual data maintained via span! macro from the tracing crate into the metrics.

Usage

First, set up tracing and metrics crates:

use metrics_tracing_context::{MetricsLayer, TracingContextLayer};
use metrics_util::layers::Layer;
use tracing_subscriber::layer::SubscriberExt;

// Prepare tracing.
let subscriber = my_subscriber.with(MetricsLayer::new());
tracing::subscriber::set_global_default(subscriber).unwrap();

// Prepare metrics.
let recorder = TracingContextLayer::all().layer(my_recorder);
metrics::set_global_recorder(recorder).unwrap();

Then emit some metrics within spans and see the labels being injected!

use tracing::{span, Level};
use metrics::counter;

let user = "ferris";
let span = span!(Level::TRACE, "login", user);
let _guard = span.enter();

counter!("login_attempts", "service" => "login_service").increment(1);

The code above will emit a increment for a login_attempts counter with the following labels:

• service=login_service
• user=ferris

Implementation

The integration layer works by capturing all fields that are present when a span is created, as well as fields recorded after the fact, and storing them as an extension to the span. If a metric is emitted while a span is entered, any fields captured for that span will be added to the metric as additional labels.

Be aware that we recursively capture the fields of a span, including fields from parent spans, and use them when generating metric labels. This means that if a metric is being emitted in span B, which is a child of span A, and span A has field X, and span B has field Y, then the metric labels will include both field X and Y. This applies regardless of how many nested spans are currently entered.

Duplicate span fields

When span fields are captured, they are deduplicated such that only the most recent value is kept. For merging parent span fields into the current span fields, the fields from the current span have the highest priority. Additionally, when using Span::record to add fields to a span after it has been created, the same behavior applies. This means that recording a field multiple times only keeps the most recently recorded value, including if a field was already present from a parent span and is then recorded dynamically in the current span.

Span fields and ancestry

Likewise, we capture the sum of all fields for a span and its parent span(s), meaning that if you have the following span stack:

root span        (fieldA => valueA)
 ⤷ mid-tier span (fieldB => valueB)
    ⤷ leaf span  (fieldC => valueC)

Then a metric emitted while within the leaf span would get, as labels, all three fields: A, B, and C. As well, this layer does not deduplicate the fields. If you have two instance of the same field name, both versions will be included in your metric labels. Whether or not those are deduplicated, and how they're deduplicated, is an exporter-specific implementation detail.

In addition, for performance purposes, span fields are held in pooled storage, and additionally will copy the fields of parent spans. Following the example span stack from above, the mid-tier span would hold both field A and B, while the leaf span would hold fields A, B, and C.

In practice, these extra memory consumption used by these techniques should not matter for modern systems, but may represent an unacceptable amount of memory usage on constrained systems such as embedded platforms, etc.