sup3

A standalone S3 tool

Warning

This is alpha

Anything and everything could go wrong. The Rust AWS SDK used is also in developer preview.

Usage goal

• CI and other minimal environments where pulling in python or building dependencies might be undesirable
• Eventually, as a consistent, reliable, fast and fun S3 tool for interactive use

Design

• Act as much as possible like existing unix commands (e.g. cp, ls, cat)
• Progress reporting by default where it makes sense (e.g. cp, down, up)

Features

• Cross-platform
• Uses authorisation support from the Rust AWS SDK
• Streaming async transfers (files not read into memory)
• Upload resume on remote errors (provided by the SDK)
• No startup delay
• No runtime non-platform dependencies (e.g. libc, libm, libgcc_s)
• Concurrent transfers
• Optional progress reporting
• Recursive upload support
• Recursive download support
• Remote globbing (e.g. sup3 ls s3://bucket/media/**/highres*.png .)
• Binary size reduction
• Upload arguments, e.g. ACLs
• Config file support
• List only files or only directories
• List paging
• Server to server copy
• Custom endpoints for other S3-compatible hosts (--endpoint)
• A sync operation (as part of copy/up/down or separate)

Commands

• Upload (upload|down) (local files to S3 remote)
• Download (download|down) (S3 URIs to local file/directory)
• Remove (rm) (S3 URIs)
• List (ls) (1..N S3 URIs)
• List Buckets (list-buckets|lb)
• Copy (cp)
• Cat (S3 URIs)
• Make Bucket (mb) (S3 URIs)

Speed

Meaurements performed on an i7-7700HQ, with an approximately 50ms RTT to the S3 server.

Download - 20MiB file

Download - 224byte file

ls - 63,186 files from bucket by substring