8 releases
0.3.3 | Nov 10, 2024 |
---|---|
0.3.2 | Nov 10, 2024 |
0.3.1 | Oct 26, 2024 |
0.2.1 | Oct 17, 2024 |
0.1.1 | Oct 2, 2024 |
#217 in Filesystem
493 downloads per month
Used in 5 crates
170KB
4K
SLoC
Fusio
fusio
provides Read and Write traits to operate on multiple storage backends (e.g., local disk, Amazon S3) across various asynchronous runtimes—both poll-based (tokio) and completion-based (tokio-uring, monoio)—with:
- lean: binary size is at least 14× smaller than others.
- minimal-cost abstraction: compared to bare storage backends, trait definitions allow dispatching file operations without extra overhead.
- extensible: exposes traits to support implementing storage backends as third-party crates.
fusio
is now at preview version, please join our community to attend its development and semantics / behaviors discussion.
Why do we need fusio
?
In developing Tonbo, we needed a flexible and efficient way to handle file and file system operations across multiple storage backends—such as memory, local disk, and remote object storage. We also required compatibility with various asynchronous runtimes, including both completion-based runtimes and event loops in languages like Python and JavaScript.
fusio
addresses these needs by providing:
- offers traits that allow dispatch of file and file system operations to multiple storage backends.
- usable in diverse async runtimes, not only disk but also network I/O.
- ideal for embedded libs like Tonbo.
- can be extended via third-party crates, enabling custom asynchronous file and file system implementations.
For more context, please check apache/arrow-rs#6051.
How to use it?
Installation
fusio = { version = "*", features = ["tokio"] }
Examples
Runtime agnostic
fusio
supports switching the async runtime at compile time. Middleware libraries can build runtime-agnostic implementations, allowing the top-level application to choose the runtime.
Object safety
fusio
provides two sets of traits:
Read
/Write
/Seek
/Fs
are not object-safe.DynRead
/DynWrite
/DynSeek
/DynFs
are object-safe.
You can freely transmute between them.
File system traits
fusio
has an optional Fs trait (use default-features = false
to disable it). It dispatches common file system operations (open, remove, list, etc.) to specific storage backends (local disk, Amazon S3).
S3 support
fusio
has optional Amazon S3 support (enable it with features = ["tokio-http", "aws"]
); the behavior of S3 operations and credentials does not depend on tokio
.
When to choose fusio
?
Overall, fusio
carefully selects a subset of semantics and behaviors from multiple storage backends and async runtimes to ensure native performance in most scenarios. For example, fusio
adopts a completion-based API (inspired by monoio) so that file operations on tokio
and tokio-uring
have the same performance as they would without fusio
.
compare with object_store
object_store
is locked to tokio and also depends on bytes
. fusio
uses IoBuf
/ IoBufMut
to allow &[u8]
and Vec<u8>
to avoid potential runtime costs. If you do not need to consider other async runtimes, try object_store
; as the official implementation, it integrates well with Apache Arrow and Parquet.
compare with opendal
fusio
does not aim to be a full data access layer like opendal
. fusio
keeps features lean, and you are able to enable features and their dependencies one by one. The default binary size of fusio
is 245KB, which is smaller than opendal
(439KB). If you need a full ecosystem of DAL (tracing, logging, metrics, retry, etc.), try opendal.
Also, compared with opendal::Operator
, fusio exposes core traits and allows them to be implemented in third-party crates.
Roadmap
- abstractions
- file operations
- (partial) file system operations
- storage backend implementations
- disk
- tokio
- tokio-uring
- monoio
- network
- HTTP client trait wi
- network storage runtime support
- tokio (over reqwest)
- monoio (over hyper-tls)
- tokio-uring (over hyper-tls)
- Amazon S3
- Azure Blob Storage
- Cloudflare R2
- in-memory
- disk
- conditional operations
- extensions
- parquet support
- object_store support
Credits
monoio
: all core traits—buffer, read, and write—are highly inspired by it.futures
: its design of abstractions and organization of several crates (core, util, etc.) to avoid coupling have influencedfusio
's design.opendal
: Compile-time poll-based/completion-based runtime switching inspiresfusio
.object_store
:fusio
adopts S3 credential and path behaviors from it.
Dependencies
~3–18MB
~259K SLoC