English | 中文

jdb_proto : System Core Protocol Layer

High-performance protocol definitions for distributed database communication with zero-copy memory optimization.

Table of Contents

• Project Overview
• Features
• Quick Start
• API Reference
• Design Architecture
• Technical Stack
• Project Structure
• Historical Context

Project Overview

jdb_proto provides the core protocol layer for the jdb distributed database system. It defines essential data structures, communication protocols, and unified error handling mechanisms for high-performance database operations.

The library implements a WiscKey-style storage architecture with zero-copy memory optimization, enabling efficient data access patterns in distributed environments.

Features

• Zero-copy Memory Optimization: Leverages Bytes for efficient memory management
• WiscKey Storage Pattern: Separates keys from values for optimal disk usage
• Async Communication: Built on tokio with high-performance async channels
• Type Safety: Compile-time guarantees for memory layout and data integrity
• Unified Error Handling: Comprehensive error propagation across system layers

Quick Start

Add to your Cargo.toml:

[dependencies]
jdb_proto = "0.1.1"
tokio = { version = "1.48", features = ["full"] }

Basic usage example:

use jdb_proto::*;
use bytes::Bytes;

#[tokio::main]
async fn main() -> Result<()> {
  // Create communication channels
  let (tx, rx) = new_shard_channel(1000);
  
  // Create a request
  let (resp_tx, resp_rx) = new_resp_channel();
  let req = Req::Set {
    key: Bytes::from("hello"),
    val: Bytes::from("world"),
    resp: resp_tx,
  };
  
  // Send request (in real implementation)
  tx.send(req).await?;
  
  Ok(())
}

API Reference

Core Data Types

Key and Val

pub type Key = Bytes;
pub type Val = Bytes;

Immutable byte sequences with zero-copy reference counting.

ValPtr

pub struct ValPtr {
  pub offset: u64,    // File offset (8 bytes)
  pub file_id: u32,   // Log file ID (4 bytes)
  pub len: u32,       // Data length (4 bytes)
}

Logical pointer to disk data with 16-byte fixed memory layout.

Communication Protocol

Request Types

pub enum Req {
  Set { key: Key, val: Val, resp: RespTx<()> },
  Get { key: Key, resp: RespTx<Option<Val>> },
  Del { key: Key, resp: RespTx<bool> },
}

Channel Types

• ShardTx/ShardRx: Multi-producer, single-consumer channels for shard communication
• RespTx/RespRx: One-shot channels for request-response patterns

Error Handling

pub enum Error {
  IO(std::io::Error),
  Internal(String),
  Busy,
  Shutdown,
  NotFound,
}

Design Architecture

graph TD
    Client[Client] -->|MPSC| Shard[Shard]
    Shard -->|Oneshot| Client
    
    Shard -->|File I/O| Storage[Storage Layer]
    
    subgraph "Protocol Layer"
        Req[Request Enum]
        ValPtr[Value Pointer]
        Channels[Async Channels]
    end
    
    Client --> Req
    Shard --> Req
    Req --> ValPtr
    Req --> Channels

The architecture follows a layered approach:

1. Protocol Layer: Defines communication primitives and data structures
2. Channel Layer: Handles async message passing between components
3. Storage Interface: Provides abstractions for disk operations

Module Interaction Flow

1. Client creates request using Req enum
2. Request sent through ShardTx to appropriate shard
3. Shard processes request and sends response via RespTx
4. Client receives response through RespRx

Technical Stack

• Core Language: Rust (Edition 2024)
• Async Runtime: Tokio
• Memory Management: bytes crate for zero-copy operations
• Channel Communication: crossfire for high-performance MPSC
• One-shot Communication: async-oneshot for request-response

Project Structure

src/
├── lib.rs      # Main protocol definitions
└── error.rs    # Unified error handling

tests/
└── main.rs     # Comprehensive test suite

Key Components

• Data Types: Zero-copy byte sequences and value pointers
• Communication: Async channels with configurable backpressure
• Error Handling: Unified error type with automatic conversion

Historical Context

The WiscKey architecture, which inspired ValPtr design, emerged from the University of Wisconsin-Madison in 2016. It challenged traditional LSM-tree designs by separating keys from values, reducing write amplification and improving SSD performance.

This protocol layer builds upon those principles while adding modern async communication patterns inspired by systems like TiKV and CockroachDB. The zero-copy design philosophy traces back to Linux's sendfile() system call (2003) and has evolved through modern frameworks like Cap'n Proto and FlatBuffers.

The MPSC channel design draws from Erlang's actor model (1986) and Go's channels (2009), adapted for Rust's ownership system while maintaining the safety guarantees that make Rust unique in systems programming.

About

This project is an open-source component of js0.site ⋅ Refactoring the Internet Plan.

We are redefining the development paradigm of the Internet in a componentized way. Welcome to follow us:

• Google Group
• js0site.bsky.social

jdb_proto : 系统核心协议层

面向分布式数据库通信的高性能协议定义，具备零拷贝内存优化特性。

目录

• 项目概述
• 核心特性
• 快速开始
• API 参考
• 设计架构
• 技术堆栈
• 目录结构
• 历史背景

项目概述

jdb_proto 为 jdb 分布式数据库系统提供核心协议层。它定义了基本数据结构、通信协议和统一错误处理机制，支持高性能数据库操作。

该库采用 WiscKey 风格的存储架构，具备零拷贝内存优化，在分布式环境中实现高效的数据访问模式。

核心特性

• 零拷贝内存优化：利用 Bytes 实现高效内存管理
• WiscKey 存储模式：分离键值以优化磁盘使用
• 异步通信：基于 tokio 的高性能异步通道
• 类型安全：编译时保证内存布局和数据完整性
• 统一错误处理：跨系统层的综合错误传播

快速开始

在 Cargo.toml 中添加：

[dependencies]
jdb_proto = "0.1.1"
tokio = { version = "1.48", features = ["full"] }

基础使用示例：

use jdb_proto::*;
use bytes::Bytes;

#[tokio::main]
async fn main() -> Result<()> {
  // 创建通信通道
  let (tx, rx) = new_shard_channel(1000);
  
  // 创建请求
  let (resp_tx, resp_rx) = new_resp_channel();
  let req = Req::Set {
    key: Bytes::from("hello"),
    val: Bytes::from("world"),
    resp: resp_tx,
  };
  
  // 发送请求（实际实现中）
  tx.send(req).await?;
  
  Ok(())
}

API 参考

核心数据类型

Key 和 Val

pub type Key = Bytes;
pub type Val = Bytes;

具备零拷贝引用计数的不可变字节序列。

ValPtr

pub struct ValPtr {
  pub offset: u64,    // 文件偏移量 (8 字节)
  pub file_id: u32,   // 日志文件 ID (4 字节)
  pub len: u32,       // 数据长度 (4 字节)
}

指向磁盘数据的逻辑指针，固定 16 字节内存布局。

通信协议

请求类型

pub enum Req {
  Set { key: Key, val: Val, resp: RespTx<()> },
  Get { key: Key, resp: RespTx<Option<Val>> },
  Del { key: Key, resp: RespTx<bool> },
}

通道类型

• ShardTx/ShardRx：用于分片通信的多生产者单消费者通道
• RespTx/RespRx：用于请求-响应模式的单次通道

错误处理

pub enum Error {
  IO(std::io::Error),
  Internal(String),
  Busy,
  Shutdown,
  NotFound,
}

设计架构

graph TD
    Client[客户端] -->|MPSC| Shard[分片]
    Shard -->|Oneshot| Client
    
    Shard -->|文件 I/O| Storage[存储层]
    
    subgraph "协议层"
        Req[请求枚举]
        ValPtr[值指针]
        Channels[异步通道]
    end
    
    Client --> Req
    Shard --> Req
    Req --> ValPtr
    Req --> Channels

架构采用分层设计：

1. 协议层：定义通信原语和数据结构
2. 通道层：处理组件间异步消息传递
3. 存储接口：提供磁盘操作抽象

模块交互流程

1. 客户端使用 Req 枚举创建请求
2. 请求通过 ShardTx 发送到相应分片
3. 分片处理请求并通过 RespTx 发送响应
4. 客户端通过 RespRx 接收响应

技术堆栈

• 核心语言：Rust (Edition 2024)
• 异步运行时：Tokio
• 内存管理：bytes crate 实现零拷贝操作
• 通道通信：crossfire 实现高性能 MPSC
• 单次通信：async-oneshot 实现请求-响应

目录结构

src/
├── lib.rs      # 主要协议定义
└── error.rs    # 统一错误处理

tests/
└── main.rs     # 综合测试套件

关键组件

• 数据类型：零拷贝字节序列和值指针
• 通信：可配置背压的异步通道
• 错误处理：具备自动转换的统一错误类型

历史背景

启发 ValPtr 设计的 WiscKey 架构诞生于 2016 年的威斯康星大学麦迪逊分校。它挑战了传统的 LSM-tree 设计，通过分离键值来减少写放大并提升 SSD 性能。

此协议层基于这些原则构建，同时融入了受 TiKV 和 CockroachDB 等系统启发的现代异步通信模式。零拷贝设计理念可追溯至 Linux 的 sendfile() 系统调用（2003 年），并通过 Cap'n Proto 和 FlatBuffers 等现代框架不断发展。

MPSC 通道设计借鉴自 Erlang 的 Actor 模型（1986 年）和 Go 的通道（2009 年），适配 Rust 的所有权系统，同时保持 Rust 在系统编程中独特的安全性保证。

关于

本项目为 js0.site ⋅ 重构互联网计划 的开源组件。

我们正在以组件化的方式重新定义互联网的开发范式，欢迎关注：

• 谷歌邮件列表
• js0site.bsky.social