#proxy #http #kafka #split

archived app kafka-proxy

Takes in HTTP Posts, and sends them to a Kafka Server

11 releases (1 stable)

Uses old Rust 2015

1.0.0 Jan 19, 2017
0.10.1 Dec 22, 2016
0.9.0 Nov 1, 2016

#131 in #kafka

38 downloads per month

MIT license

597 lines


Kafka-Proxy is a small webserver that takes in an HTTP Post, and then throws that post off to a Kafka Topic. The way it's currently setup it requires an SSL Certificate/Key to talk to the Kafka Instances, because setting up SSL Certs isn't that hard, and you should do it even if you think you don't need too.

Right now the Kafka Client, and http webserver run on different threads. Although rust is very fast you shouldn't take the "200 OK" as meaning it has been posted in the kafka topic. This is so we can process http requests at the rate of thousands per second, and still guarantee that they'll be posted to kafka.

If a message fails to send in kafka it will create a unique file inside of a folder called "kafka_rust". Kafka Rust will attempt to send messages from this folderthat have failed on reboot. This will hopefully increase the need for human checking. Even more so for payloads without timestamps in the message.


  1. Make sure you have rust installed, and ready to build. Go Here, or use this one-line bash statement: curl -sSf https://static.rust-lang.org/rustup.sh | sh.

  2. To build the dev version simply run: cargo build, to build a release run: cargo build --release. (Note: Kafka-Rust requires you to have LibSnappy in your path: To build kafka-rust you'll need libsnappy-dev on your local machine. If that library is not installed in the usual path, you can export the LD_LIBRARY_PATH and LD_RUN_PATH environment variables before issueing cargo build.).

  3. Run the binary located in target/<debug|release>/kafka-proxy passing in the necessary env vars.

Setting up Stats features

Kafka-Proxy allows reporting of HTTP/Kafka stats to some of the most popular solutions out there. The two of these are prometheus, and statsd (called "Graphite" in some places in the source). In order to use one of these simply enable: stats-prometheus, or stats-statsd features at build time, and setup the env vars.

Setting up Error Notifying

Kafka-Proxy allows alerting when we fail to send to kafka so you can fix the problem manually. Right now the only supported reporter is over slack, but if you want it to report somewhere else feel free to open an issue/PR implementing it.

In order to use slack simply enable the feature: reporter-slack at build time, and setup the env vars.

Env Vars

It should be noted env vars can be passed through CLI Opts in v0.7.0

Name Optional Function
GRAPHITE_HOST Sometimes The IPv4 Address of the Graphite Host to POST results to for reporting with statsd.
KAFKA_BROKERS No A comma seperated list of brokers for kafka. Right now this has to be in the form: ip:port. Hostname resolution is coming soon.
KAFKA_PROXY_CERT_PATH No The path to the certificate file to connect to kafka with.
KAFKA_PROXY_KEY_PATH No The path to the key file to connect to kafka with.
PANIC_ON_BACKUP Yes Whether the program should crash if we fail to backup a message that failed to send to kafka.
PROXY_PORT No The port for the HTTP Webserver to listen on.
SLACK_WEBHOOK Sometimes The Slack Webhook URL to connect to slack.
SLACK_CHANNEL Yes The slack channel to post to. Defaults to "#general".
NO_SSL Yes Whether to blacklist ssl.

Finally logging is setup through the rust crate log, and env_logger. As such the logging level printed to STDOUT/STDERR is determined by the env var: RUST_LOG. Which I recommend setting at INFO, unless you need DEBUG for the advanced logs.

Supported Kafka Versions

We support the same versions of kafka as kafka-rust. Which happens to be:

kafka-rust is tested against Kafka 0.8.2.x and regularly used
against Kafka 0.9 servers. However, efforts to implement support
for new features from the Kafka 0.9 release are just in their


~397K SLoC