nydus-storage

The core storage subsystem for Nydus Image Service to:

• Fetch blob objects from storage backend such as Registry, OSS, S3, local disk and file systems etc.
• Load data from storage backend on demand.
• Cache blob objects on local storage.

Support

Platforms:

• x86_64
• aarch64

Operating Systems:

• Linux
• MacOS

License

This code is licensed under Apache-2.0 or BSD-3-Clause.

lib.rs:

Chunked blob storage service to support Rafs filesystem.

The Rafs filesystem is blob based filesystem with chunk deduplication. A Rafs filesystem is composed up of a metadata blob and zero or more data blobs. A blob is just a plain object storage containing data chunks. Data chunks may be compressed, encrypted and deduplicated by content digest value. When Rafs file is used for container images, Rafs metadata blob contains all filesystem metadatas, such as directory, file name, permission etc. Actually file contents are split into chunks and stored into data blobs. Rafs may built one data blob for each container image layer or build a single data blob for the whole image, according to building options.

The nydus-storage crate is used to manage and access chunked blobs for Rafs filesystem, which contains three layers:

• Backend: access raw blob objects on remote storage backends.
• Cache: cache remote blob contents onto local storage in forms optimized for performance.
• Device: public APIs for chunked blobs

There are several core abstractions provided by the public APIs:

• BlobInfo: provides information about blobs, which is typically constructed from the blob array in Rafs filesystem metadata.
• BlobDevice: provides access to all blobs of a Rafs filesystem, which is constructed from an array of BlobInfo objects.
• BlobChunkInfo: provides information about a data chunk, which is loaded from Rafs metadata.
• BlobIoDesc: a blob IO descriptor, containing information for a continuous IO range within a chunk.
• BlobIoVec: a scatter/gather list for blob IO operation, containing one or more blob IO descriptors

To read data from the Rafs filesystem, the Rafs filesystem driver will prepare a BlobIoVec object and submit it to the corresponding BlobDevice object to actually execute the IO operations.