2 releases
new 0.1.3 | Sep 1, 2024 |
---|---|
0.1.2 | Aug 31, 2024 |
0.1.1 |
|
0.1.0 |
|
0.0.0 |
|
#54 in Database implementations
428 downloads per month
92KB
2K
SLoC
Append Only Log
Introducation
When developing infrastructure softwares, write-ahead log or append-only log plays an important role, and people re-implement same
funcationalities multiple times, but actually, the core for append-only log is just atomic append
, append_batch
, replay
, and rewrite
.
This crate provides generic purpose append-only log implementation, there are two kinds of implementations based on std::fs::File
and memory map.
-
aol::fs::AppendLog
:Generic append-only log implementation based on
std::fs::File
.-
It is good for:
- The encoded entry size is smaller than
64
bytes. - Manifest file.
- Write is not too frequently.
- The encoded entry size is smaller than
-
Pros:
- It is growable, do not require pre-allocated.
- Support automatically rewrite.
- No holes in the file.
-
-
aol::memmap::AppendLog
:Generic append-only log implementation based on
memmap
.-
It is good for:
- Any append-only log, if you do not care about pre-allocating the file and you know you data will never larger than the pre-allocating size.
-
Pros:
- Support automatic rewrite.
- No holes in the file.
- As this implementation is backed by an ARENA, no allocation required for both read and write.
- Fast read and write performance, backed by memory map, no extra I/O required.
-
File Structure
+----------------------+--------------------------+-----------------------+
| magic text (4 bytes) | external magic (2 bytes) | magic (2 bytes) |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+
| op (1 bit) | custom flag (7 bits) | len (4 bytes) | data (N bytes) | checksum (8 bytes) |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+
| op (1 bit) | custom flag (7 bits) | len (4 bytes) | data (N bytes) | checksum (8 bytes) |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+
| ... | ... | ... | ... | ... |
+----------------------+--------------------------+-----------------------+-----------------------+-----------------------+
Installation
[dependencies]
aol = "0.1"
License
aol
is under the terms of both the MIT license and the
Apache License (Version 2.0).
See LICENSE-APACHE, LICENSE-MIT for details.
Copyright (c) 2024 Al Liu.
Dependencies
~0.2–9MB
~91K SLoC