6 releases (breaking)

0.4.0 Oct 11, 2020
0.3.1 Sep 1, 2020
0.3.0 Aug 28, 2020
0.2.0 Aug 17, 2020
0.0.1 Sep 5, 2019

#1200 in Command line utilities

23 downloads per month

Custom license



s3m a "fault-tolerant" command-line tool for storing streams of data in s3 buckets.

crates.io Build Status Build Status - develop

Problem trying to solve

There are streams of data that can not be lost besides that when created, they degrade the performance of running systems, for example, if the stream is a backup of a database, every time the stream is produced it may lock the entire cluster (depends on the options and tools used mysqldump/xtrabackup for example), however, the effort to create the stream is proportional to the size of the database, in most of the cases the bigger the database is the more time and CPU and Memory is required.

In the case of backups, normally the streams are piped to a compression tool and later put in the disk, in some cases writing to the existing disk where the database is or to a remote mount endpoint, is not possible due to size constraints and the compressed backup should be streamed to am s3 bucket (X provider), therefore if for some reason the connection gets lost while streaming almost before finishing, the whole backup procedure could be corrupted and in worst scenario everything should start all over again.

The aim of s3m is to make as much as possible “fault-tolerant” the storage procedure of the incoming stream so that even if the server lost network connectivity, the stream could still be received and queued so that when the network re-establishes it can continue where it was left and resume the upload without the need to start all over again.



~593K SLoC