8 unstable releases (3 breaking)
new 0.4.0 | Apr 20, 2024 |
---|---|
0.3.0 | Feb 22, 2024 |
0.2.3 | Feb 6, 2024 |
0.2.1 | Jan 26, 2024 |
0.1.1 | Dec 11, 2023 |
#418 in Encoding
202 downloads per month
260KB
6K
SLoC
zarrs_tools
Various tools for creating and manipulating Zarr v3 data with the zarrs rust crate.
zarrs_tools is experimental and in limited production use. Correctness issues with zarrs affecting past versions of zarrs_tools are detailed here.
Tools
- zarrs_reencode: reencode an array.
- Can change the chunk size, shard size, codecs, fill value, chunk key encoding separator, and attributes.
- zarrs_binary2zarr (feature
binary2zarr
): create an array from piped binary data. - zarrs_ncvar2zarr (feature
ncvar2zarr
): convert a netCDF variable to an array.- Supports multi-file datasets where a variable has been split along a single dimension.
The following tools are highly experimental and have had limited production testing:
- zarrs_filter (feature
filter
): apply simple image filters (transformations) to an array. - zarrs_ome (feature
ome
): convert an array to OME-Zarr.- Outputs OME-Zarr
0.5-dev
. This revision is currently recognised by Neuroglancer for Zarr V3.
- Outputs OME-Zarr
See docs/ for tool documentation.
zarrs
Benchmarking
zarrs_reencode
: suitable for round trip benchmarking.zarrs_benchmark_read_sync
(featurebenchmark
): benchmark the zarrs sync API.zarrs_benchmark_read_async
(featurebenchmark
): benchmark the zarrs async API.
See docs/benchmarks.md for some benchmark measurements.
Install
From crates.io
cargo install --all-features zarrs_tools
From source
cargo install --all-features --path .
# cargo install --all-features --git https://github.com/LDeakin/zarrs_tools
Enabling SIMD intrinsics
Encoding and decoding performance may be improved with avx2
/sse2
enabled (if supported).
This can be enabled by compiling with either of:
RUSTFLAGS="-C target-cpu=native"
RUSTFLAGS="-C target-feature=+avx2,+sse2"
Enabling non-default zarrs
codecs
Non-default zarrs
codecs (see zarrs
crate features) can be enabled by passing them as feature flags.
For example:
cargo install zarrs_tools --all-features --features zarrs/bitround,zarrs/zfp,zarrs/bz2,zarrs/pcodec
Licence
zarrs_tools
is licensed under either of
- the Apache License, Version 2.0 LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0 or
- the MIT license LICENSE-MIT or http://opensource.org/licenses/MIT, at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Dependencies
~20–59MB
~1M SLoC