#provider #remote #hash #tor #channel #storage #byte

unmaintained bin+lib maimo

Just Another Storage

3 releases

0.0.3 Feb 11, 2022
0.0.2 Feb 11, 2022
0.0.1 Feb 11, 2022

#99 in #tor

47 downloads per month

Unlicense

325KB
964 lines

Maimo

Just Another Storage

Crates.io License: Unlicense Lines of code

Tree being struck by lightning (royalty-free stock photo from pixabay)

  • Remote storage: Open storage channels to a remote storage provider over Tor
  • Lightweight: Platform-optimized using Blake3-based hash, verified streaming, and encryption algorithms
  • Local files can be deleted: Periodic verification of remotely stored data
  • Remote files can be retrieved: Files can be proven to exist remotely and sent back
  • Low write-amplification: Bytes stored are only slightly larger than bytes read

Caution! This is experimental, potentially dangerous technology that hasn't yet been audited!

Notable Dependencies

Roadmap

0.0.1 - Experiment

  • Bao encoding
  • Bao verification
  • Bao extraction

0.0.2 - File storage

  • Sled for path lookup
  • file SQL
    • Schema
    • Insert
    • Query
  • Encode files in the Maimo Data folder, and store them in a configured storage volume
  • Decode files stored in a configured storage volume, and restore them to the Maimo Data folder
  • Verify a random slice of a file (accounting for files of varying sizes)
  • Display encoded file list
  • Paths are indexed in sqlite (path, file name, file size, creation & modification dates, file hash)
  • Multiple files can be stored

0.0.3 - Crate & Testing

  • Exports all CLI commands as functions
  • Serial integration tests added for crate
  • Needs more documentation and tests

0.0.4 - Tor networking

  • Generate Onion v3 address
  • peer SQL
    • Schema
    • Insert
    • Query
  • Open & Receive TCP socket over Tor hidden service

0.0.5 - Authenticated encryption

  • Authentication between storage client and storage provider using Onion v3 addresses
  • Blake3 keyed hashes as a MAC
  • Files are encrypted using XChaCha8Blake3Siv authenticated encryption
    • Caution! Experimental encryption!
  • CSPRNGs where RNGs are used
  • Use randomized padding instead of zeroed padding
  • Zeroization of private keys after dropped from memory

0.1.0 - Proof of Concept

Goal: A storage client that can compress, encrypt, and store data on a remote storage provider using Tor. The storage client can check periodically that the data is still present and consistent on the remote storage provider against only a local 32-byte Blake3 hash without a full local reference copy, allowing the client to delete its local data, trusting that it can retrieve it later in-full. The storage client can then retrieve the data from the storage provider and decode it on-disk.

  • Storage client can open a storage channel to storage provider over Tor
    • Storage provider generates Onion v3 address to provide to storage client out-of-band
    • Storage client generates Onion v3 address of their own
    • TCP socket is established from storage client to storage provider over Tor hidden service
  • Storage client can store data on storage provider
    • Storage client can supply their node with specified path to data to store remotely
    • Data is encoded using Bao, hashed with Blake3, and transmitted over TCP socket over Tor circuit
    • Blake3 hash is persisted locally
    • Optional: Delete the local data
  • Storage client can periodically verify the data they sent is still present and consistent over time
    • Storage client asks for a 4KB slice of data at a random offset of their choosing from the storage storage provider
    • Storage client checks 4KB slice against the same offset against local Bao Blake3 hash
  • Storage client can retrieve data from storage provider over storage channel
    • Data is written to disk at specified path
  • Files are compressed using zstd dictionary compression
  • Individual files can be retrieved from storage provider
  • Files can be removed
  • Files can be overwritten, with old revisions still retrievable
  • The number of older revisions can be configured
  • Embeddable library available, with documentation
  • Parallel processing for lots of files

0.1.1

  • Complex volume and storage layouts

Future

After this basic functionality exists, more exciting features are planned, including apps on the Start9 Embassy and Umbrel!

Dependencies

~46MB
~747K SLoC