#update #synchronization #swupdate #delta-update

bin+lib bita

Tool for fast file synchronization over http

4 releases

✓ Uses Rust 2018 edition

0.2.0 Apr 11, 2019
0.1.2 Feb 22, 2019
0.1.1 Feb 12, 2019
0.1.0 Jan 28, 2019

#86 in Compression

31 downloads per month

MIT license

148KB
3.5K SLoC

bita

Build Status

bita is a tool aiming for fast and low bandwidth file synchronization over http.

The file to synchronize could be any file where data is expected to change partially or completely between updates. Any local file that might contain data of the source file may be used as seed while cloning.

In a software update system (IoT device/embedded system) with an A/B partition setup bita can be used to update the B partition while using the A partition as seed. The result written to the B parition will be an exact clone of the remote file but the only data fetched from remote is the data which actually differ between the A partition and the remote file.


On compression the input is scanned for chunk boundaries using a rolling hash. With the default setting a suitable boundary should be found every ~64 KiB. Each found chunk is assigned a strong hash using blake2. If the chunk is unique it will be compressed and inserted into the final archive. A dictionary which describes all chunks by their strong hash and the order for how to rebuild the input file is also attached to the archive.

On clone the dictionary is first fetched from the remote archive. Then bita scans all the given seed files for chunks which are present in the dictionary. Any matching chunk found in a seed will be inserted into the output file. When all seeds has been consumed the chunks still missing is fetched from the remote archive, then unpacked and inserted into the output file.

Each chunk of data is verified by the strong hash before written to the output.


Installing from crates.io

olle@home:~$ cargo install bita

Building from source

olle@home:~$ cargo build` or `$ cargo build --release

Example usage

Compress

Create a compressed archive file.cba from stdin stream:

olle@host:~$ gunzip -c file.gz | bita compress --compression ZSTD --compression-level 9 file.cba

Create an compressed archive file.ext4.cba from file file.ext4:

olle@host:~$ bita compress file.ext4 file.ext4.cba

Clone

Clone file at http://host/new.tar.cba using seed another_old.tar and stdin (-):

olle@device:~$ gunzip -c old.tar.gz | bita clone --seed another_old.tar --seed - http://host/new.tar.cba new.tar

Clone using block device /dev/mmcblk0p1 as seed and /dev/mmcblk0p2 as target:

olle@device:~$ bita clone --seed /dev/mmcblk0p1 http://host/file.ext4.cba /dev/mmcblk0p2

Similar Tools

Dependencies

~18MB
~352K SLoC