memmap3

Safe memory-mapped I/O with zero-copy persistent data structures.

memmap3 is a complete drop-in replacement for memmap2 that adds a huge additional list of 'safe' features. All memmap2 functionality is available, plus powerful new APIs that eliminate unsafe code while providing automatic persistence.

🎯 Core Features Summary

• 🔄 Auto-Persistence: Changes persist automatically via memory mapping
• 🔒 Zero Unsafe: Safe Rust APIs for all operations
• ⚡ Thread-Safe: Atomic fields work across processes
• 🎭 Convenience Operators: << for intuitive data manipulation
• 📏 Fixed-Size + Unlimited: Fixed layouts + segmented growth when needed
• 🧩 Rich Type System: Primitives, atomics, strings, collections, nested structs
• 📊 Multidimensional: N-dimensional arrays for join-like operations
• 🔌 Drop-in Compatible: All memmap2 code works unchanged

Quick Start

Add to your Cargo.toml:

[dependencies]
memmap3 = "0.1"

Create persistent data structures:

use memmap3::prelude::*;

#[mmap_struct]
struct Counter {
    #[mmap(atomic)]
    value: u64,
    name: [u8; 32],
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create persistent data
    let mut counter = MmapStruct::<Counter>::create("/tmp/counter.mmap")?;
    &mut counter.name << "my_counter";
    
    // Atomic operations work across processes
    let old_value = counter.value.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
    println!("Counter: {} -> {}", old_value, old_value + 1);
    
    Ok(())
}

Key Features

🔒 Safe API

• Zero unsafe code required in user applications
• Compile-time validation through #[mmap_struct] macro
• Runtime bounds checking for all operations
• UTF-8 validation for string operations

⚡ High Performance

• Zero-copy access to persistent data
• OS-level caching through memory mapping
• Atomic operations for thread-safe access
• Minimal overhead with direct memory access

🧩 Drop-in Compatibility

// Just change your import:
use memmap2::{Mmap, MmapOptions};  // From this
use memmap3::{Mmap, MmapOptions};  // To this
// All existing memmap2 code works unchanged!

🔧 The #[mmap_struct] Macro

Transform regular Rust structs into persistent, memory-mapped types:

use memmap3::prelude::*;

#[mmap_struct]
struct MyData {
    // Regular fields (primitives, enums)
    id: u64,
    active: bool,
    
    // Atomic fields (thread-safe across processes)
    #[mmap(atomic)]
    counter: u64,
    
    // Auto-detected strings (default for [u8; N])
    name: [u8; 32],
    
    // Explicit raw binary data
    #[mmap(raw)]
    key: [u8; 32],
    
    // Fixed-capacity vectors  
    #[mmap(vec)]
    scores: [u32; 10],
    
    // String arrays
    #[mmap(string_array)]
    tags: [[u8; 16]; 5],
    
    // Unlimited growth storage
    #[mmap(segmented)]
    events: [u64; 0],
}

🎭 Convenience Operators

Smooth, intuitive syntax for common operations:

// String assignment
&mut my_string << "Hello";

// Vector append  
&mut my_vec << item;

// HashMap insertion
&mut my_map << ("key", "value");

// HashMap key assignment using macro
hashmap_index!(my_map["key"] << "value");

// Segmented append
&mut my_segmented << item;

📊 Complete Type System

Primitive Types (Native Support)

• Integers: u8, u16, u32, u64, u128, i8, i16, i32, i64, i128, usize, isize
• Floats: f32, f64
• Bool: bool
• Char: char (stored as u32)
• C-style Enums: Any enum with #[repr(C)]

Atomic Types (#[mmap(atomic)])

Transform any primitive to thread-safe atomic:

• u64 → MmapAtomicU64 (fetch_add, compare_exchange, etc.)
• bool → MmapAtomicBool (load, store, etc.)
• Plus all integer types + usize/isize

String Types

• Auto-detected: [u8; N] → MmapString<N> (default behavior)
• Explicit: #[mmap(string)] to force string behavior
• Raw binary: #[mmap(raw)] to keep as byte array

Collection Types

• Vectors: #[mmap(vec)] transforms [T; N] → MmapVec<T, N>
• String Arrays: #[mmap(string_array)] transforms [[u8; LEN]; N] → MmapStringArray<N, LEN>
• Hash Maps: MmapHashMap<K, V> for key-value storage

Advanced Types

• Segmented: #[mmap(segmented)] for unlimited growth (breaks fixed-size constraint)
• Nested Structs: Full support for complex nested structures
• Multidimensional Arrays: [[[u8; X]; Y]; Z] for N-dimensional data
• Serde Integration: Store any Serialize/Deserialize type

🔗 Multidimensional & Join-like Operations

Create sophisticated data structures for complex queries:

use memmap3::prelude::*;

#[mmap_struct]
struct BitIndex {
    // 3D lookup table for fast joins
    index: [[[u8; 256]; 256]; 256],
    
    // User data indexed by multiple dimensions  
    users_by_age_location: [[u64; 100]; 50], // [age][location] → user_id
    
    // Unlimited growth using segmented
    #[mmap(segmented)]
    overflow_data: [u64; 0],
}

🏗️ Fixed-Size Constraint + Segmented Growth

• Most types: Fixed size at compile time for predictable layout
• Exception: #[mmap(segmented)] allows unlimited growth
• Benefit: Known memory layout enables direct pointer arithmetic

⚙️ How Auto-Persistence Works

1. Memory Mapping: File mapped into process memory
2. Direct Access: Struct fields map directly to file bytes
3. OS Sync: Operating system handles sync to disk
4. Cross-Process: Multiple processes can share same file safely

Core Types

MmapStruct

Safe wrapper for memory-mapped structs with automatic file management:

use memmap3::prelude::*;

#[mmap_struct]
struct GameState {
    level: u32,
    score: u64,
    player_name: [u8; 32],
}

let mut game = MmapStruct::<GameState>::create("game.mmap")?;
game.level = 5;
game.score = 12345;
&mut game.player_name << "Player1";

Atomic Fields

Thread-safe operations across processes:

use memmap3::prelude::*;

#[mmap_struct]
struct Metrics {
    #[mmap(atomic)]
    requests: u64,
    #[mmap(atomic)]
    errors: u64,
}

let metrics = MmapStruct::<Metrics>::create("metrics.mmap")?;
metrics.requests.fetch_add(1, Ordering::SeqCst);

Collections

Persistent data structures with dynamic sizing:

use memmap3::prelude::*;

// Persistent hash map
let mut cache = MmapHashMap::<&str, &str>::create("cache.mmap")?;
cache.insert("key1", "value1")?;

// Multiple assignment syntaxes available:
&mut cache << ("key2", "value2");           // Tuple insertion
hashmap_index!(cache["key3"] << "value3");  // Index-style assignment
cache.set("key4", "value4")?;               // Direct set method

// Persistent vector
let mut log = MmapVec::<u64, 1000>::new();
log.push(1234567890); // timestamp

// Persistent string
let mut name = MmapString::<64>::new();
&mut name << "Alice";

Supported Types

Primitive Types (no attribute needed)

• Integers: u8, u16, u32, u64, u128, i8, i16, i32, i64, i128, usize, isize
• Floats: f32, f64
• Bool: bool
• Char: char (stored as u32)
• C-style Enums: Any enum with #[repr(C)]

Transformed Types (via attributes)

Atomic Types (#[mmap(atomic)])

Transforms regular types to atomic equivalents for thread-safe access:

• u8 → MmapAtomicU8, u16 → MmapAtomicU16, etc.
• bool → MmapAtomicBool

Strings (#[mmap(string)])

Transforms byte arrays to UTF-8 strings with helper methods:

• [u8; N] → MmapString<N>
• Provides .set(), << operator, .as_str(), .clear()

Vectors (#[mmap(vec)])

Transforms arrays to fixed-capacity vectors:

• [T; N] → MmapVec<T, N>
• Provides .push(), .pop(), .len(), .iter(), indexing

String Arrays (#[mmap(string_array)])

Transforms 2D byte arrays to arrays of strings:

• [[u8; LEN]; N] → MmapStringArray<N, LEN>
• Provides indexed access to multiple strings

Running Examples

See the examples/ directory for complete working examples:

cargo run --example 01-002_macro_usage
cargo run --example 03-001_atomic_operations  
cargo run --example 02-002_string_operations
cargo run --example 04-003_event_log
cargo run --example 14_hashmap

Examples

Simple Configuration Storage

use memmap3::prelude::*;

#[mmap_struct]
struct Config {
    max_connections: u32,
    timeout_ms: u32,
    debug_mode: bool,
    server_name: [u8; 64],
}

let mut config = MmapStruct::<Config>::create("app.config")?;
config.max_connections = 1000;
config.timeout_ms = 5000;
&mut config.server_name << "production-server";
// Configuration persists across application restarts

Inter-Process Communication

use memmap3::prelude::*;
use std::sync::atomic::Ordering;

#[mmap_struct]
struct SharedData {
    #[mmap(atomic)]
    message_count: u64,
    #[mmap(atomic)]
    worker_status: u32,
    shared_buffer: [u8; 1024],
}

// Process A
let shared = MmapStruct::<SharedData>::create("/tmp/ipc.mmap")?;
shared.message_count.store(42, Ordering::SeqCst);

// Process B can read the same data
let shared = MmapStruct::<SharedData>::open("/tmp/ipc.mmap")?;
let count = shared.message_count.load(Ordering::SeqCst);
println!("Messages: {}", count); // Prints: Messages: 42

Persistent Analytics

use memmap3::prelude::*;
use std::sync::atomic::Ordering;

#[mmap_struct]
struct Analytics {
    #[mmap(atomic)]
    page_views: u64,
    #[mmap(atomic)]
    unique_visitors: u64,
    #[mmap(atomic)]
    conversion_rate: u64, // Fixed-point arithmetic
}

let analytics = MmapStruct::<Analytics>::create("analytics.mmap")?;

// Record events
analytics.page_views.fetch_add(1, Ordering::SeqCst);
analytics.unique_visitors.fetch_add(1, Ordering::SeqCst);

// Calculate conversion rate (example: 2.5% as 250 basis points)
let rate = (conversions * 10000) / page_views;
analytics.conversion_rate.store(rate, Ordering::SeqCst);

How It Works

The #[mmap_struct] attribute macro:

1. Automatically adds #[repr(C)] for predictable memory layout
2. Transforms annotated fields to thread-safe types
3. Generates trait implementations for memory mapping
4. Handles file creation, opening, and validation

Thread Safety

• Regular fields: Single writer, multiple readers (protected by OS page cache)
• Atomic fields: Multiple concurrent writers and readers
• String fields: Single writer with UTF-8 validation

Using with Arc for Shared Access

You can wrap MmapStruct in Arc for safe sharing across threads:

use memmap3::prelude::*;
use std::sync::Arc;
use std::sync::atomic::Ordering;

#[mmap_struct]
struct Config {
    port: u16,
    #[mmap(atomic)]
    counter: u64,
}

// Single-threaded use
let mut data = MmapStruct::<Config>::create("config.mmap")?;
data.port = 8080;  // Can modify all fields

// Multi-threaded use with Arc
let shared = Arc::new(MmapStruct::<Config>::create("config.mmap")?);

// Clone for each thread
let thread_data = shared.clone();
std::thread::spawn(move || {
    // Atomic fields can be modified safely
    thread_data.counter.fetch_add(1, Ordering::Relaxed);
    
    // Regular fields are read-only when shared via Arc
    println!("Port: {}", thread_data.port);
});

Access Patterns:

• No Arc: Full read/write access to all fields (single-threaded)
• With Arc: Atomic fields writable, regular fields read-only (multi-threaded)
• With Arc<Mutex<...>>: If you need to modify regular fields from multiple threads

String Handling

Automatic string operations for byte arrays:

use memmap3::prelude::*;

#[mmap_struct]
struct User {
    id: u64,
    name: [u8; 32],    // Automatically treated as string
    email: [u8; 128],  // Automatically treated as string
}

let mut user = MmapStruct::<User>::create("user.mmap")?;
user.id = 12345;

// Write strings using << operator
&mut user.name << "Alice Smith";
&mut user.email << "alice@example.com";

// Read strings back
println!("User: {} ({})", user.name.as_str(), user.email.as_str());

Feature Flags

Enable additional functionality:

[dependencies]
memmap3 = { version = "0.1", features = ["serde"] }

• serde - Serialize/deserialize support for complex types

Documentation

• API Reference - Complete API documentation
• Cookbook Examples - Real-world usage patterns
• Migration Guide - Upgrading from memmap2

Use Cases

• Inter-process communication - Shared configuration and state
• Event logging - High-performance ring buffers
• ML model weights - Instant loading of large models
• Game save states - Fast persistence and loading
• Database indexes - Memory-mapped B-trees

Performance

memmap3 provides zero-copy access with minimal overhead:

• Direct memory access - No serialization/deserialization
• OS page cache integration - Automatic memory management
• Lazy loading - Only accessed pages are loaded
• Cross-process sharing - Efficient memory usage

Safety Guarantees

Multiple layers of safety validation:

1. Compile-time validation - #[mmap_struct] macro checks
2. Runtime bounds checking - All array/collection access
3. Memory layout verification - Struct compatibility validation
4. UTF-8 validation - String operation safety
5. Atomic operation safety - Cross-process synchronization

License

This project is licensed under the MIT OR Apache-2.0 license.

Contributing

Contributions are welcome! Please see our GitHub repository for guidelines.