4 releases
Uses new Rust 2024
0.1.0 |
|
---|---|
0.1.0-m2 | May 18, 2025 |
0.1.0-m1 | May 8, 2025 |
0.1.0-alpha.2 | Apr 29, 2025 |
#262 in Database interfaces
380 downloads per month
110KB
2.5K
SLoC
Xaeroflux
⚠️ Work in progress – NOT READY FOR PRODUCTION USE ⚠️
Introduction
Xaeroflux is a mobile-friendly, decentralized append-only storage and indexing engine powering truly cloud-less, peer-to-peer applications (e.g. social networks, marketplaces, identity systems). It combines:
- Event-first design: every change is an append-only Event
- Two-tier Merkle-indexing:
- Per-page Merkle trees for efficient bulk page proofs
- Merkle Mountain Range (MMR) for instant per-event proofs and low-latency diffs
- Plug-and-play networking: transport-agnostic sync protocol over Iroh (see: https://github.com/n0-computer/iroh for more details) Libp2p/QUIC/WebRTC/Tor
- Mobile-optimized I/O: zero-copy serialization, mmap-backed pages, minimal threads
- Privacy & resilience: end-to-end encryption, fallback to Tor, never-say-die mesh sync
Quickstart
Open and use Xaeroflux just like a local embedded database—background sync happens automatically via iroh, iroh-blob, and iroh-gossip:
use xaeroflux::Core;
// NOTE THIS CODE IS A POSSIBLE EXAMPLE OF HOW IT MAY FUNCTION
// THIS IS STILL EVOLVING!
fn main() -> anyhow::Result<()> {
// initialize and open database (background sync starts automatically)
let mut core = Core::builder()
.with_page_size(16 * 1024)
.with_segment_pages(1024)
.with_sync_backends(&["iroh", "iroh-blob", "iroh-gossip"] )
.open("/path/to/xaeroflux.db")?;
// append a new Event payload
let ev = core.new_event(b"Hello, Xaeroflux!");
core.append(ev);
// read back all events like a local DB
for event in core.iter_events() {
println!("Event: {:?}", event.data);
}
// get current MMR root for diffing or audit
let root = core.mmr_root();
println!("MMR root: {}", hex::encode(root));
// Core runs sync in background; just exit when done
Ok(())
}
Architecture Overview
1. Serialization
- Zero-copy reads via [rkyv]
- Zero-copy writes via [bytemuck] where possible
- Deserialize only on demand
2. Storage Layer
-
Append-Only Event Log (AOF)
- Backed by LMDB: fast, B+-tree mmap pages for raw Event bytes
- Each
Event
is serialized with a small header and archived bytes
-
Segment Writer Actor
- Buffers Events in a thread-safe channel (crossbeam)
- Packs into a fixed-size page buffer (e.g. 16 KiB)
mmap.flush_range
per-page, rolls over after N pages into new.seg
file
-
Per-Page Merkle Trees
- On each full page: build a Merkle tree over its event-hashes
- Store page’s nodes in a
XaeroOnDiskMerklePage
blob alongside raw bytes - Enables O(log P) proofs within any page
-
Merkle Mountain Range (MMR)
- Immediate per-event root updates
- Append: push leaf, carry-merge equal-height peaks → update in-memory peaks +
root
- Persist peaks/root metadata so you can resume on restart
- Enables instant O(log E) inclusion proofs and minimal diffing
3. Networking Layer
-
Xaeroflux Sync Protocol (XSP) over any bidirectional byte stream
- HELLO handshake (capabilities: SYNC, CRDT, MCP…)
- PEAKS ↔ DIFF: peer exchanges MMR peak-roots
- PAGE_REQUEST / PAGE_RESPONSE for missing pages
- SYNC_DONE
-
Built on Iroh which is based on Libp2p (QUIC, TCP, WebRTC)
-
Uses Arti Tor for completely anonymous mode!
-
Optional CRDT and MCP (Model Context Protocol) integrations
4. Plumbing & Runtime
- Intra-process comms: crossbeam channels, thread-pool handlers
- I/O: MIO +
mmap
for non-blocking page watches and efficient flushes - Threads: native OS threads (no async/await baggage)
Testing & Quality
# Run all tests
cargo test --all
# Enforce code style
cargo fmt --all
cargo clippy --all -- -D warnings
Contributing
- Fork the repo
- Create a feature branch
- Add tests & documentation
- Open a PR and iterate
See CONTRIBUTING.md for details.
License
Mozilla Public License Version 2.0 See LICENSE for full text.
Dependencies
~56–92MB
~1.5M SLoC