6 releases

0.3.0	Apr 15, 2024
0.2.2	Nov 19, 2023
0.2.1	Oct 4, 2023
0.2.0	Sep 24, 2023
0.1.1	Aug 22, 2023

#16 in Visualization

190 downloads per month

MIT/Apache

155KB
2.5K SLoC

Kommitted - Measure Kafka consumers lag

Measure Kafka Consumer Offset Lag and Time Lag

Kommitted is a service to measure the Lag (i.e. Latency) of Kafka consumers. It works with all consumers that commit their offsets into Kafka (i.e. the standard way), as it consumes the internal __consumer_offsets topic.

Metrics are exported following the Prometheus Exposition formats.

Please see DESIGN.md for details about the overall architecture, dependencies and other details.

Features

Track Offset for all consumers
Track Offset Lag for all consumers
Track Time Lag for all consumers
Offset and Lag metrics are tracked with all contextual information to identify exact topic partition assignments
Exposes additional metrics to track status of Kafka cluster (topics, members, brokers, partitions)
Exposes Kafka-polling metrics, to assess its own performance
Metrics exposed in Prometheus format, at /metrics endpoint
REST API to build further automation on top of it (e.g. auto-scaling logics that depend on Consumer Group lag)

All of this comes based on:

A fast and efficient Rust implementation, built on Tokio
The widely used librdkafka, the de-facto standard for Kafka Clients (outside of Java)

Please see the complete list of exposed Metrics, for further details.

Getting started

To install kommitted, you need to compile it yourself, or use the Docker image. If you have the Rust Toolchain already setup, then just run:

$ cargo install kommitted

In Docker

Kommitted is now available as a Docker Image: kafkesc/kommitted on the Docker Hub registry. Both linux/amd64 and linux/arm64 images are available, based on Debian slim images.

The ENTRYPOINT is the kommitted binary itself, so you can just pass arguments to the container execution.

Usage

Kommitted supports compact (-h) and extended (--help) usage instructions. Use the former for a quick look up; use the latter to better understand what each argument can do.

Compact: `kommitted -h`

Usage: kommitted [OPTIONS] --brokers <BOOTSTRAP_BROKERS>

Options:
  -b, --brokers <BOOTSTRAP_BROKERS>
          Initial Kafka Brokers to connect to (format: 'HOST:PORT,...')
      --client-id <CLIENT_ID>
          Client identifier used by the internal Kafka (Admin) Client [default: kommitted]
      --kafka-conf <CONF_KEY:CONF_VAL>
          Additional configuration used by the internal Kafka (Admin) Client (format: 'CONF_KEY:CONF_VAL').
      --cluster-id <CLUSTER_ID>
          Override identifier of the monitored Kafka Cluster
      --history <SIZE_PER_PARTITION>
          For each Topic Partition, how much history of offsets to track in memory. [default: 3600]
      --history-ready-at <FULLNESS_PERCENT_PER_PARTITION>
          How full `--history` of Topic Partition offsets has to be (on average) for service to be ready. [default: 0.3]
      --host <HOST>
          Host address to listen on for HTTP requests. [default: 127.0.0.1]
      --port <PORT>
          Port to listen on for HTTP requests. [default: 6564]
  -v, --verbose...
          Verbose logging.
  -q, --quiet...
          Quiet logging.
  -h, --help
          Print help (see more with '--help')
  -V, --version
          Print version

Extended: `kommitted --help`

Usage: kommitted [OPTIONS] --brokers <BOOTSTRAP_BROKERS>

Options:
  -b, --brokers <BOOTSTRAP_BROKERS>
          Initial Kafka Brokers to connect to (format: 'HOST:PORT,...').

          Equivalent to '--config=bootstrap.servers:host:port,...'.

      --client-id <CLIENT_ID>
          Client identifier used by the internal Kafka (Admin) Client.

          Equivalent to '--config=client.id:my-client-id'.

          [default: kommitted]

      --kafka-conf <CONF_KEY:CONF_VAL>
          Additional configuration used by the internal Kafka (Admin) Client (format: 'CONF_KEY:CONF_VAL').

          To set multiple configurations keys, use this argument multiple times.
          See: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md.

      --cluster-id <CLUSTER_ID>
          Override identifier of the monitored Kafka Cluster.

          If set, it replaces the value `cluster.id` from the Brokers' configuration. This can be useful when `cluster.id` is not actually
          set.

      --history <SIZE_PER_PARTITION>
          For each Topic Partition, how much history of offsets to track in memory.

          Offsets data points are collected every 500ms, on average: so, on average,
          30 minutes of data points is 3600 offsets, assuming partition offsets are
          regularly produced to.

          Once this limit is reached, the oldest data points are discarded, realising
          a "moving window" of offsets history.

          [default: 3600]

      --history-ready-at <FULLNESS_PERCENT_PER_PARTITION>
          How full `--history` of Topic Partition offsets has to be (on average) for service to be ready.

          This value will be compared with the average "fullness" of each data structure containing
          the offsets of Topic Partitions. Once passed, the service can start serving metrics.

          The value must be a percentage in the range `[0.0%, 100.0%]`.

          [default: 0.3]

      --host <HOST>
          Host address to listen on for HTTP requests.

          Supports both IPv4 and IPv6 addresses.

          [default: 127.0.0.1]

      --port <PORT>
          Port to listen on for HTTP requests.

          [default: 6564]

  -v, --verbose...
          Verbose logging.

          * none    = 'WARN'
          * '-v'    = 'INFO'
          * '-vv'   = 'DEBUG'
          * '-vvv'  = 'TRACE'

          Alternatively, set environment variable 'KOMMITTED_LOG=(ERROR|WARN|INFO|DEBUG|TRACE|OFF)'.

  -q, --quiet...
          Quiet logging.

          * none    = 'WARN'
          * '-q'    = 'ERROR'
          * '-qq'   = 'OFF'

          Alternatively, set environment variable 'KOMMITTED_LOG=(ERROR|WARN|INFO|DEBUG|TRACE|OFF)'.

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Connect to Kafka cluster requiring `SASL_SSL`

$ kommitted \
    --brokers {{ BOOTSTRAP_BROKERS or BROKER_ENDPOINT }} \
    --config security.protocol:SASL_SSL \
    --config sasl.mechanisms=PLAIN \
    --config sasl.username:{{ USERNAME or API_KEY }} \
    --config sasl.password:{{ PASSWORD or API_SECRET }} \  
    ...

Log verbosity

Kommitted follows the long tradition of -v/-q to control the verbosity of it's logging:

Arguments	Log verbosity level	Default
`-qq...`	`OFF`
`-q`	`ERROR`
none	`WARN`	x
`-v`	`INFO`
`-vv`	`DEBUG`
`-vvv...`	`TRACE`

It uses log and env_logger, and so logging can be configured and fine-tuned using the Environment Variable KOMMITTED_LOG. Please take a look at env_logger doc for more details.

License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Dependencies

~49–67MB
~1M SLoC