payload-offloading-for-aws

Large Payload offloading to S3, compatible with existing AWS Java libraries.

When sending or receiving via SNS/SQS, large message payloads will be offloaded to or download from S3 in this format:

[
    "software.amazon.payloadoffloading.PayloadS3Pointer",
    {
        "s3BucketName": "offloading-bucket",
        "s3Key": "9cc30888-0e2e-40da-b594-c9ca2cd58722"
    }
]

It's compatible with the common offloading Java library, which is used by the following AWS Java libraries:

• amazon-sqs-java-extended-client-lib
• amazon-sns-java-extended-client-lib

Usage

Add dependency to Cargo.toml:

rust-payload-offloading-for-aws = { version = "0.1", features = ["sns", "sqs"] }

Setup AWS SDK interceptors and use the SDK as usual:

use payload_offloading_for_aws::offload;
use aws_sdk_sns::{config::{Config as SnsConfig}};

async fn send_sns_and_receive_sqs() {
    let aws_config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .load()
        .await;

    let sns_s3_offload_interceptor = offload::sns::S3OffloadInterceptor::new(
        &aws_config,
        offload::id_provider::RandomUuidProvider::default(), // bucket key generator
        "my-bucket".to_owned(), // offloading bucket
        255000, // max non-offload body size
    );

    // or use a helper function `offloading_client(&aws_config, "my-bucket".to_owned(), 25000)`
    let sns_client = aws_sdk_sns::Client::from_conf(
        SnsConfig::new(&aws_config)
            .to_builder()
            // Set offloading interceptor
            .interceptor(sns_s3_offload_interceptor)
            .build(),
    );

    // Send SNS using AWS SDK as usual
    let _ = sns_client
        .publish()
        .topic_arn("TEST-TOPIC".to_owned())
        .set_message(Some("TEST BODY".to_owned()))
        .send()
        .await
        .unwrap();

    // Another option to setup AWS client
    let sqs_client = offload::sqs::offloading_client(&aws_config, "my-bucket", 25000);

    // Receive message from SQS using the AWS SDK as usual.
    // If the message body exceeds the offloading size limit,
    // it will be automatically downloaded from S3 behind the scenes.
    let _ = sqs_client
        .receive_message()
        .queue_url("TEST-QUEUE")
        .send()
        .await
        .unwrap();
}

Alternatively you can download offloaded bodies manually like this:

use payload_offloading_for_aws::offload;

async fn download_offloaded_payload_explicitly() -> Result<(), String> {
    let aws_config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .load()
        .await;

    let s3_client = aws_sdk_s3::Client::from_conf(
        aws_sdk_s3::Config::from(&aws_config)
            .to_builder()
            .force_path_style(true)
            .build(),
    );

    let raw_payload = r#"[
        "software.amazon.payloadoffloading.PayloadS3Pointer",
        {
            "s3BucketName": "offloading-bucket",
            "s3Key": "9cc30888-0e2e-40da-b594-c9ca2cd58722"
        }
    ]"#;

    // It's only `Some` when download was needed, failover to original payload
    let res = offload::sqs::try_downloading_body(&s3_client, raw_payload)
        .map_err(|e| e.to_string())?
        .unwrap_or(raw_payload.to_owned());

    Ok(())
}

Additional References

AWS SDK Interceptors discussion

AWS SDK interceptor examples:

• Replaying body
• Compressing body

License

Licensed under either of

• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in time by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.