#serde #kafka

kafka-serde

serialization and deserialization for the Kafka protocol

1 unstable release

0.1.0 Feb 12, 2021

#328 in Encoding

Download history 256/week @ 2021-02-25 166/week @ 2021-03-04 107/week @ 2021-03-11 120/week @ 2021-03-18 124/week @ 2021-03-25 56/week @ 2021-04-01 89/week @ 2021-04-08 148/week @ 2021-04-15 129/week @ 2021-04-22 13/week @ 2021-04-29 169/week @ 2021-05-06 160/week @ 2021-05-13 72/week @ 2021-05-20 112/week @ 2021-05-27 80/week @ 2021-06-03 94/week @ 2021-06-10

508 downloads per month

Apache-2.0 OR MIT

35KB
973 lines

kafka-serde

Rust's serde implementation for the Kafka protocol.

This allows you to serialize and deserialize kafka payloads. It can be used as a building block for a native-rust kafka client.

Usage

Serializing the kafka request header:

use serde::Serialize;
use std::io::{Write, Cursor};

#[derive(Serialize, Debug)]
struct RequestHeader {
    api_key: i16,
    api_version: i16,
    correlation_id: i32,
    client_id: &'static str,
}

let req = RequestHeader {
    api_key: 0,
    api_version: 0,
    correlation_id: 1,
    client_id: ""
};

let mut x = Cursor::new(Vec::<u8>::new());
kafka_serde::to_writer(x, &req).unwrap();

Deserializing the kafka response header:

use serde::Serialize;

#[derive(Deserialize, Default, Debug, Clone)]
struct ResponseHeader {
    pub correlation: i32,
}

let data : Vec<u8> = [0x0, 0x0, 0x0, 0x1];
let resp: ResponseHeader = kafka_serde::from_bytes(&data).unwrap();

Support

All Kafka protocol types are listed here

  • The fixed-size types are supported and map to their native rust types (int8 -> i8, etc).
  • Strings can be deserialized to both String and &str, and similarly for bytes
  • UUID is not supported (yet)
  • variable-size types like VARLONG and COMPACT_STRING are not supported (yet)

Dependencies

~0.6–1.2MB
~29K SLoC