remozipsy - Remote Zip Sync

Overview

remozipsy is a crate enabling incremental, differential synchronization of zip files from a remote location. It avoids downloading the entire archive and instead fetches only updated parts (by using CRC matching). This optimizes download times over slow connections.

Use Cases:

• Efficiently updating large datasets with periodic releases
• Quick Game assets updates on Airshipper to increase player experiences

Features

• 🚀 Parallel file downloads from remote server to reduce latency
• 🗜️ Local CRC comparison to avoid redundant data transfers
• 🧰 Abstractable interfaces (RemoteZip, FileSystem) enabling custom storage backends
• 🛠️ Prebuild implementations for reqwest and tokio::fs
• 📡 Designed for performance in low-bandwidth networks

Installation

Add the following dependency to your Cargo.toml:

[dependencies]
remozipsy = "0.0.3"

Usage

Example: Sync remote zip with local filesystem

See examples/sync_remote_zip.rs for a detailed example.

use remozipsy::{Config, Statemachine, reqwest::ReqwestRemoteZip, tokio::TokioLocalStorage};

#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let remote = ReqwestRemoteZip::with_url("https://getsamplefiles.com/download/zip/sample-1.zip")?;
    let local = TokioLocalStorage::new("./extract", Vec::new());
    let state = Statemachine::new(remote, local, Config::default());

    while let Some((progress, next_state)) = state.progress().await {
        state = next_state;
        println!("Progress: {progress:?}");
        tokio::task::yield_now().await;
    }
    Ok(())
}

• The main idea is to progress the internal statemachine with statemachine.progress().

Algorithm Overview

1. Fetch zip’s central directory (zip metadata)
2. Traverse filesystem, compare existing files by CRC
3. Determine which files to download/delete
4. Download updated/new files in parallel (using tokio tasks)
5. Unzip downloaded files locally
6. Cleanup deleted outdated files

Why remozipsy works

1. Zip structure: Central directory contains all metadata at the end of the file, seperated from the actuall stored information
2. HTTP Range requests: HTTP allows the RANGE header to only fetch part of a ressource, luckily many webservers (and github releases) support this feature
3. Zip Individuall compression: all files with in an archive are compressed individually.

For more details, see the documentation and explore examples.

Contributing

Pull Requests welcome! Start by checking open issues or feature requests in our GitLab repo.

License

MIT License or APACHE-2.0 License