130 releases (47 breaking)

0.47.3 Jul 7, 2024
0.47.1 Jun 21, 2024
0.45.1 Mar 2, 2024
0.44.0 Dec 31, 2023
0.3.0 Mar 25, 2022

#23 in Filesystem

Download history 13558/week @ 2024-04-05 13058/week @ 2024-04-12 15452/week @ 2024-04-19 11911/week @ 2024-04-26 13738/week @ 2024-05-03 14830/week @ 2024-05-10 13844/week @ 2024-05-17 12600/week @ 2024-05-24 11456/week @ 2024-05-31 11777/week @ 2024-06-07 13104/week @ 2024-06-14 12832/week @ 2024-06-21 13100/week @ 2024-06-28 12216/week @ 2024-07-05 12768/week @ 2024-07-12 10666/week @ 2024-07-19

50,637 downloads per month
Used in 55 crates (36 directly)

Apache-2.0

3MB
59K SLoC

Apache OpenDAL™: Access Data Freely

Build Status Latest Version Crate Downloads chat

OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services.

OpenDAL Architectural

Services

OpenDAL supports the following storage services:

Type Services
Standard Storage Protocols ftp http sftp webdav
Object Storage Services azblob cos gcs obs oss s3
b2 openstack_swift upyun vercel_blob
File Storage Services fs alluxio azdls azfile chainsafe compfs
dbfs gridfs hdfs hdfs_native ipfs webhdfs
Consumer Cloud Storage Service aliyun_drive gdrive onedrive dropbox icloud koofr
pcloud seafile yandex_disk
Key-Value Storage Services cacache cloudflare_kv dashmap memory etcd
foundationdb persy redis rocksdb sled
redb tikv atomicserver
Database Storage Services d1 mongodb mysql postgresql sqlite surrealdb
Cache Storage Services ghac memcached mini_moka moka vercel_artifacts
Git Based Storage Services huggingface

Layers

OpenDAL supports the following storage layers to extend the behavior:

Name Depends Description
AsyncBacktraceLayer async-backtrace Add Efficient, logical 'stack' traces of async functions for the underlying services.
AwaitTreeLayer await-tree Add a Instrument await-tree for actor-based applications to the underlying services.
BlockingLayer tokio Add blocking API support for non-blocking services.
ChaosLayer rand Inject chaos into underlying services for robustness test.
ConcurrentLimitLayer tokio Add concurrent request limit.
DtraceLayer probe Support User Statically-Defined Tracing(aka USDT) on Linux
LoggingLayer log Add log for every operations.
MetricsLayer metrics Add metrics for every operations.
MinitraceLayer minitrace Add minitrace for every operations.
OtelTraceLayer opentelemetry::trace Add opentelemetry::trace for every operations.
PrometheusClientLayer prometheus_client Add prometheus metrics for every operations.
PrometheusLayer prometheus Add prometheus metrics for every operations.
RetryLayer backon Add retry for temporary failed operations.
ThrottleLayer governor Add a bandwidth rate limiter to the underlying services.
TimeoutLayer tokio Add timeout for every operations to avoid slow or unexpected hang operations.
TracingLayer tracing Add tracing for every operations.

Quickstart

use opendal::Result;
use opendal::layers::LoggingLayer;
use opendal::services;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // Pick a builder and configure it.
    let mut builder = services::S3::default();
    builder.bucket("test");

    // Init an operator
    let op = Operator::new(builder)?
        // Init with logging layer enabled.
        .layer(LoggingLayer::default())
        .finish();

    // Write data
    op.write("hello.txt", "Hello, World!").await?;

    // Read data
    let bs = op.read("hello.txt").await?;

    // Fetch metadata
    let meta = op.stat("hello.txt").await?;
    let mode = meta.mode();
    let length = meta.content_length();

    // Delete
    op.delete("hello.txt").await?;

    Ok(())
}

Examples

Name Description
Basic Show how to use opendal to operate storage service.
Concurrent Upload Show how to perform upload concurrently to a storage service.
Multipart Upload Show how to perform a multipart upload to a storage service.

Contributing

Check out the CONTRIBUTING guide for more details on getting started with contributing to this project.

Branding

The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.

For more details, see the Apache Product Name Usage Guide.

License and Trademarks

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation.

Dependencies

~9–60MB
~1M SLoC