zarrs_icechunk

icechunk store support for the zarrs Rust crate.

Icechunk is a transactional store that enables git-like version control of Zarr hierarchies.

zarrs_icechunk can read data in a range of archival formats (e.g., netCDF4, HDF5, etc.) that are converted to icechunk-backed "virtual Zarr datacubes" via VirtualiZarr (example below).

Version Compatibility Matrix

See doc/version_compatibility_matrix.md.

Examples

Basic Usage and Version Control

use icechunk::{Repository, RepositoryConfig, repository::VersionInfo};
use zarrs_icechunk::AsyncIcechunkStore;

// Create an icechunk repository
let storage = icechunk::new_in_memory_storage().await?;
let config = RepositoryConfig::default();
let repo = Repository::create(Some(config), storage, HashMap::new()).await?;

// Do some array/metadata manipulation with zarrs, then commit a snapshot
let session = repo.writable_session("main").await?;
let store = Arc::new(AsyncIcechunkStore::new(session));
let array: Array<AsyncIcechunkStore> = ...;
let snapshot0 = store.session().write().await.commit("Initial commit", None).await?;

// Do some more array/metadata manipulation, then commit another snapshot
let session = repo.writable_session("main").await?;
let store = Arc::new(AsyncIcechunkStore::new(session));
let array: Array<AsyncIcechunkStore> = ...;
let snapshot1 = store.session().write().await.commit("Update data", None).await?;

// Checkout the first snapshot
let session = repo.readonly_session(&VersionInfo::SnapshotId(snapshot0)).await?;
let store = Arc::new(AsyncIcechunkStore::new(session));
let array: Array<AsyncIcechunkStore> = ...;

Virtualise NetCDF as Zarr (via VirtualiZarr)

Decode a virtual Zarr array /examples/data/test.icechunk.zarr:

cargo run --example virtualizarr_netcdf

This references /examples/data/test[0,1].nc hosted in this repository over HTTP. /examples/data/test.icechunk.zarr was created with /examples/virtualizarr_netcdf.py.

Licence

zarrs_icechunk is licensed under either of

• the Apache License, Version 2.0 LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0 or
• the MIT license LICENSE-MIT or http://opensource.org/licenses/MIT, at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.