#array #zarr #codec #multi-dimensional #metadata #format #subset

zarrs

A library for the Zarr storage format for multidimensional arrays and metadata

46 releases (15 breaking)

0.17.1 Oct 17, 2024
0.17.0-beta.3 Sep 25, 2024
0.16.1 Jul 30, 2024
0.12.5 Mar 17, 2024
0.6.0 Nov 16, 2023

#142 in Encoding

Download history 269/week @ 2024-07-07 53/week @ 2024-07-14 10/week @ 2024-07-21 667/week @ 2024-07-28 235/week @ 2024-08-04 168/week @ 2024-08-11 227/week @ 2024-08-18 54/week @ 2024-08-25 123/week @ 2024-09-01 42/week @ 2024-09-08 284/week @ 2024-09-15 566/week @ 2024-09-22 405/week @ 2024-09-29 57/week @ 2024-10-06 217/week @ 2024-10-13 82/week @ 2024-10-20

788 downloads per month
Used in 3 crates

MIT/Apache

1.5MB
29K SLoC

zarrs

Latest Version zarrs documentation msrv downloads build codecov

zarrs is a Rust library for the Zarr storage format for multidimensional arrays and metadata. It supports:

A changelog can be found here. Correctness issues with past versions are detailed here.

Developed at the Department of Materials Physics, Australian National University, Canberra, Australia.

Getting Started

Example

use zarrs::group::GroupBuilder;
use zarrs::array::{ArrayBuilder, DataType, FillValue, ZARR_NAN_F32};
use zarrs::array::codec::GzipCodec; // requires gzip feature
use zarrs::array_subset::ArraySubset;
use zarrs::storage::ReadableWritableListableStorage;
use zarrs::filesystem::FilesystemStore; // requires filesystem feature

// Create a filesystem store
let store_path: PathBuf = "/path/to/hierarchy.zarr".into();
let store: ReadableWritableListableStorage =
    Arc::new(FilesystemStore::new(&store_path)?);

// Write the root group metadata
GroupBuilder::new()
    .build(store.clone(), "/")?
    // .attributes(...)
    .store_metadata()?;

// Create a new V3 array using the array builder
let array = ArrayBuilder::new(
    vec![3, 4], // array shape
    DataType::Float32,
    vec![2, 2].try_into()?, // regular chunk shape (non-zero elements)
    FillValue::from(ZARR_NAN_F32),
)
.bytes_to_bytes_codecs(vec![
    Arc::new(GzipCodec::new(5)?),
])
.dimension_names(["y", "x"].into())
.attributes(serde_json::json!({"Zarr V3": "is great"}).as_object().unwrap().clone())
.build(store.clone(), "/array")?; // /path/to/hierarchy.zarr/array

// Store the array metadata
array.store_metadata()?;
println!("{}", serde_json::to_string_pretty(array.metadata())?);
// {
//     "zarr_format": 3,
//     "node_type": "array",
//     ...
// }

// Perform some operations on the chunks
array.store_chunk_elements::<f32>(
    &[0, 1], // chunk index
    &[0.2, 0.3, 1.2, 1.3]
)?;
array.store_array_subset_ndarray::<f32, _>(
    &[1, 1], // array index (start of subset)
    ndarray::array![[-1.1, -1.2], [-2.1, -2.2]]
)?;
array.erase_chunk(&[1, 1])?;

// Retrieve all array elements as an ndarray
let array_ndarray = array.retrieve_array_subset_ndarray::<f32>(&array.subset_all())?;
println!("{array_ndarray:4}");
// [[ NaN,  NaN,  0.2,  0.3],
//  [ NaN, -1.1, -1.2,  1.3],
//  [ NaN, -2.1,  NaN,  NaN]]

zarrs Ecosystem

Crate Description Version Docs
Core
zarrs The core library for manipulating Zarr hierarchies zarrs_ver zarrs_doc
zarrs_metadata Zarr metadata support zarrs_metadata_ver zarrs_metadata_doc
zarrs_storage The storage API for zarrs zarrs_storage_ver zarrs_storage_doc
Stores
zarrs_filesystem A filesystem store zarrs_filesystem_ver zarrs_filesystem_doc
zarrs_object_store object_store store support zarrs_object_store_ver zarrs_object_store_doc
zarrs_opendal opendal store support zarrs_opendal_ver zarrs_opendal_doc
zarrs_http A synchronous http store zarrs_http_ver zarrs_http_doc
zarrs_zip A storage adapter for zip files zarrs_zip_ver zarrs_zip_doc
zarrs_icechunk icechunk store support zarrs_icechunk_ver zarrs_icechunk_doc
Bindings
zarrs_ffi A subset of zarrs exposed as a C/C++ API zarrs_ffi_ver zarrs_ffi_doc

zarrs_tools

zarrs_tools_ver zarrs_tools_doc

  • A reencoder that can change codecs, chunk shape, convert Zarr V2 to V3, etc.
  • Create an OME-Zarr hierarchy from a Zarr array.
  • Transform arrays: crop, rescale, downsample, gradient magnitude, gaussian, noise filtering, etc.
  • Benchmarking tools and performance benchmarks of zarrs.

Licence

zarrs is licensed under either of

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Dependencies

~11–23MB
~334K SLoC