Amazon Kinesis Client Library for Rust

This package provides a Rust interface to the Amazon Kinesis Client Library (KCL) MultiLangDaemon, which is part of the Amazon KCL for Java.

This interface manages the interaction with the MultiLangDaemon so that developers can focus on implementing their record processor executable.

There is a provided Docker image that sets up the correct JARs using the Amazon KCL for Python.

A settings file is also required for the MultiLangDaemon to correctly set up your processor. A sample of this can be found in the examples.

Basic Usage

A more complete example can be found in the example

use kcl::checkpointer::Checkpointer;
use kcl::reader::StdinReader;
use kcl::writer::StdoutWriter;
use kcl::{run, Processor, Record};
use serde::Deserialize;

#[derive(Deserialize)]
struct DummyPayload;
struct BaseApp;

impl Processor<StdoutWriter, StdinReader> for BaseApp {
    fn initialize(&mut self, _shard_id: &str) {}

    fn process_records(
        &mut self,
        data: &[Record],
        _checkpointer: &mut Checkpointer<StdoutWriter, StdinReader>,
    ) {
        for record in data {
            match record.json::<DummyPayload>() {
                Ok(data) => {}
                Err(e) => {}
            }
        }
    }
    fn lease_lost(&mut self) {}
    fn shard_ended(&mut self, _checkpointer: &mut Checkpointer<StdoutWriter, StdinReader>) {}
    fn shutdown_requested(&mut self, _checkpointer: &mut Checkpointer<StdoutWriter, StdinReader>) {}
}

fn main() {
    run(&mut BaseApp {});
}

Docker

An example consumer of this Docker Image would be:

Compile with al2 because amazoncoretto uses al2

FROM amazonlinux:2 as builder
RUN yum update -y && yum install -y gcc
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
COPY . .
RUN cargo build --release

FROM ghcr.io/validus-risk-management/amazon-kinesis-client-rust:latest as runner
COPY my-configs/app.properties app.properties
COPY --from=builder target/release/my-app target/release/my-app

The default entrypoint should meet most requirements:

CMD ["java", "-cp", "/usr/local/lib/jars/*", "software.amazon.kinesis.multilang.MultiLangDaemon", "--properties-file", "app.properties"]

Additional configuration can be found here.