4 releases

Uses new Rust 2024

0.1.0 Apr 11, 2025
0.1.0-m2 May 18, 2025
0.1.0-m1 May 8, 2025
0.1.0-alpha.2 Apr 29, 2025

#262 in Database interfaces

Download history 109/week @ 2025-04-09 13/week @ 2025-04-16 121/week @ 2025-04-23 95/week @ 2025-04-30 147/week @ 2025-05-07

380 downloads per month

MPL-2.0 license

110KB
2.5K SLoC

Xaeroflux

⚠️ Work in progress – NOT READY FOR PRODUCTION USE ⚠️

Introduction

Xaeroflux is a mobile-friendly, decentralized append-only storage and indexing engine powering truly cloud-less, peer-to-peer applications (e.g. social networks, marketplaces, identity systems). It combines:

  1. Event-first design: every change is an append-only Event
  2. Two-tier Merkle-indexing:
    • Per-page Merkle trees for efficient bulk page proofs
    • Merkle Mountain Range (MMR) for instant per-event proofs and low-latency diffs
  3. Plug-and-play networking: transport-agnostic sync protocol over Iroh (see: https://github.com/n0-computer/iroh for more details) Libp2p/QUIC/WebRTC/Tor
  4. Mobile-optimized I/O: zero-copy serialization, mmap-backed pages, minimal threads
  5. Privacy & resilience: end-to-end encryption, fallback to Tor, never-say-die mesh sync

Quickstart

Open and use Xaeroflux just like a local embedded database—background sync happens automatically via iroh, iroh-blob, and iroh-gossip:

use xaeroflux::Core;

// NOTE THIS CODE IS A POSSIBLE EXAMPLE OF HOW IT MAY FUNCTION
// THIS IS STILL EVOLVING!
fn main() -> anyhow::Result<()> {
    // initialize and open database (background sync starts automatically)
    let mut core = Core::builder()
        .with_page_size(16 * 1024)
        .with_segment_pages(1024)
        .with_sync_backends(&["iroh", "iroh-blob", "iroh-gossip"] )
        .open("/path/to/xaeroflux.db")?;

    // append a new Event payload
    let ev = core.new_event(b"Hello, Xaeroflux!");
    core.append(ev);

    // read back all events like a local DB
    for event in core.iter_events() {
        println!("Event: {:?}", event.data);
    }

    // get current MMR root for diffing or audit
    let root = core.mmr_root();
    println!("MMR root: {}", hex::encode(root));

    // Core runs sync in background; just exit when done
    Ok(())
}

Architecture Overview

1. Serialization

  • Zero-copy reads via [rkyv]
  • Zero-copy writes via [bytemuck] where possible
  • Deserialize only on demand

2. Storage Layer

  • Append-Only Event Log (AOF)

    • Backed by LMDB: fast, B+-tree mmap pages for raw Event bytes
    • Each Event is serialized with a small header and archived bytes
  • Segment Writer Actor

    • Buffers Events in a thread-safe channel (crossbeam)
    • Packs into a fixed-size page buffer (e.g. 16 KiB)
    • mmap.flush_range per-page, rolls over after N pages into new .seg file
  • Per-Page Merkle Trees

    • On each full page: build a Merkle tree over its event-hashes
    • Store page’s nodes in a XaeroOnDiskMerklePage blob alongside raw bytes
    • Enables O(log P) proofs within any page
  • Merkle Mountain Range (MMR)

    • Immediate per-event root updates
    • Append: push leaf, carry-merge equal-height peaks → update in-memory peaks + root
    • Persist peaks/root metadata so you can resume on restart
    • Enables instant O(log E) inclusion proofs and minimal diffing

3. Networking Layer

  • Xaeroflux Sync Protocol (XSP) over any bidirectional byte stream

    1. HELLO handshake (capabilities: SYNC, CRDT, MCP…)
    2. PEAKS ↔ DIFF: peer exchanges MMR peak-roots
    3. PAGE_REQUEST / PAGE_RESPONSE for missing pages
    4. SYNC_DONE
  • Built on Iroh which is based on Libp2p (QUIC, TCP, WebRTC)

  • Uses Arti Tor for completely anonymous mode!

  • Optional CRDT and MCP (Model Context Protocol) integrations

4. Plumbing & Runtime

  • Intra-process comms: crossbeam channels, thread-pool handlers
  • I/O: MIO + mmap for non-blocking page watches and efficient flushes
  • Threads: native OS threads (no async/await baggage)

Testing & Quality

# Run all tests
cargo test --all

# Enforce code style
cargo fmt --all
cargo clippy --all -- -D warnings

Contributing

  1. Fork the repo
  2. Create a feature branch
  3. Add tests & documentation
  4. Open a PR and iterate

See CONTRIBUTING.md for details.


License

Mozilla Public License Version 2.0 See LICENSE for full text.

Dependencies

~56–92MB
~1.5M SLoC