#s3 #amazon-s3 #aws #cli

app sup3

A standalone S3 uploader

1 unstable release

0.8.12 Nov 28, 2022

#1315 in Filesystem

MIT license

255KB
1.5K SLoC

sup3

Standalone Binaries

A standalone S3 tool

Recording of download with progress enabled

Warning

This is alpha

Anything and everything could go wrong. The Rust AWS SDK used is also in developer preview.

Usage goal

  • CI and other minimal environments where pulling in python or building dependencies might be undesirable
  • Eventually, as a consistent, reliable, fast and fun S3 tool for interactive use

Design

  • Act as much as possible like existing unix commands (e.g. cp, ls, cat)
  • Progress reporting by default where it makes sense (e.g. cp, down, up)

Features

  • Cross-platform
  • Uses authorisation support from the Rust AWS SDK
  • Streaming async transfers (files not read into memory)
  • Upload resume on remote errors (provided by the SDK)
  • No startup delay
  • No runtime non-platform dependencies (e.g. libc, libm, libgcc_s)
  • Concurrent transfers
  • Optional progress reporting
  • Recursive upload support
  • Recursive download support
  • Remote globbing (e.g. sup3 ls s3://bucket/media/**/highres*.png .)
  • Binary size reduction
  • Upload arguments, e.g. ACLs
  • Config file support
  • List only files or only directories
  • List paging
  • Server to server copy
  • Custom endpoints for other S3-compatible hosts (--endpoint)
  • A sync operation (as part of copy/up/down or separate)

Commands

  • Upload (upload|down) (local files to S3 remote)
  • Download (download|down) (S3 URIs to local file/directory)
  • Remove (rm) (S3 URIs)
  • List (ls) (1..N S3 URIs)
  • List Buckets (list-buckets|lb)
  • Copy (cp)
  • Cat (S3 URIs)
  • Make Bucket (mb) (S3 URIs)

Speed

Meaurements performed on an i7-7700HQ, with an approximately 50ms RTT to the S3 server.

Download - 20MiB file

Command Mean [s] Min [s] Max [s] Relative CPU User [s] CPU System [s] CPU Relative
sup3 cp s3://bucket/file dir/ 1.306 ± 0.047 1.249 1.416 1.00 0.095 0.153 100%
aws s3 cp s3://bucket/file dir/ 1.855 ± 0.106 1.724 2.044 142% ± 10% 0.565 0.122 277%

Download - 224byte file

Command Mean [s] Min [s] Max [s] Relative CPU User [s] CPU System [s] CPU Relative
sup3 cp s3://bucket/file dir/ 0.246 ± 0.006 0.234 0.255 100% 0.004 0.007 100%
aws s3 cp s3://bucket/file dir/ 0.910 ± 0.013 0.892 0.937 370% ± 11% 0.394 0.050 4036%

ls - 63,186 files from bucket by substring

Command Mean [s] Min [s] Max [s] Relative CPU User [s] CPU System [s] CPU Relative
sup3 ls --substring s3://bucket/substring >/dev/null 10.053 ± 0.694 9.253 10.487 100% 0.485 0.141 100%
aws s3 ls s3://bucket/substring >/dev/null 22.050 ± 3.809 19.643 26.441 219% ± 41% 11.772 0.143 1900%

Dependencies

~33–46MB
~758K SLoC