Kafka-Proxy

Kafka-Proxy is a small webserver that takes in an HTTP Post, and then throws that post off to a Kafka Topic. The way it's currently setup it requires an SSL Certificate/Key to talk to the Kafka Instances, because setting up SSL Certs isn't that hard, and you should do it even if you think you don't need too.

Right now the Kafka Client, and http webserver run on different threads. Although rust is very fast you shouldn't take the "200 OK" as meaning it has been posted in the kafka topic. This is so we can process http requests at the rate of thousands per second, and still guarantee that they'll be posted to kafka.

If a message fails to send in kafka it will create a unique file inside of a folder called "kafka_rust". Kafka Rust will attempt to send messages from this folderthat have failed on reboot. This will hopefully increase the need for human checking. Even more so for payloads without timestamps in the message.

Installation

1. Make sure you have rust installed, and ready to build. Go Here, or use this one-line bash statement: curl -sSf https://static.rust-lang.org/rustup.sh | sh.

2. To build the dev version simply run: cargo build, to build a release run: cargo build --release. (Note: Kafka-Rust requires you to have LibSnappy in your path: To build kafka-rust you'll need libsnappy-dev on your local machine. If that library is not installed in the usual path, you can export the LD_LIBRARY_PATH and LD_RUN_PATH environment variables before issueing cargo build.).

3. Run the binary located in target/<debug|release>/kafka-proxy passing in the necessary env vars.

Setting up Stats features

Kafka-Proxy allows reporting of HTTP/Kafka stats to some of the most popular solutions out there. The two of these are prometheus, and statsd (called "Graphite" in some places in the source). In order to use one of these simply enable: stats-prometheus, or stats-statsd features at build time, and setup the env vars.

Setting up Error Notifying

Kafka-Proxy allows alerting when we fail to send to kafka so you can fix the problem manually. Right now the only supported reporter is over slack, but if you want it to report somewhere else feel free to open an issue/PR implementing it.

In order to use slack simply enable the feature: reporter-slack at build time, and setup the env vars.

Env Vars

It should be noted env vars can be passed through CLI Opts in v0.7.0

Finally logging is setup through the rust crate log, and env_logger. As such the logging level printed to STDOUT/STDERR is determined by the env var: RUST_LOG. Which I recommend setting at INFO, unless you need DEBUG for the advanced logs.

Supported Kafka Versions

We support the same versions of kafka as kafka-rust. Which happens to be:

kafka-rust is tested against Kafka 0.8.2.x and regularly used
against Kafka 0.9 servers. However, efforts to implement support
for new features from the Kafka 0.9 release are just in their
beginnings.