35 releases (21 breaking)

new 0.22.1 Jan 21, 2025
0.21.1 Dec 16, 2024
0.21.0 Nov 27, 2024
0.1.0 May 20, 2024

#200 in Database implementations

Download history 126/week @ 2024-10-04 67/week @ 2024-10-11 66/week @ 2024-10-18 160/week @ 2024-10-25 416/week @ 2024-11-01 431/week @ 2024-11-08 754/week @ 2024-11-15 406/week @ 2024-11-22 234/week @ 2024-11-29 150/week @ 2024-12-06 222/week @ 2024-12-13 49/week @ 2024-12-20 59/week @ 2024-12-27 140/week @ 2025-01-03 149/week @ 2025-01-10 336/week @ 2025-01-17

684 downloads per month
Used in 26 crates (24 directly)

Apache-2.0

87KB
1.5K SLoC

Vortex Buffer

For now, a Vortex buffer is implemented as a very thin wrapper around the Tokio bytes crate. In the future, we may re-implement this ourselves to have more control over alignment (see https://github.com/tokio-rs/bytes/issues/437)


lib.rs:

A library for working with custom aligned buffers of sized values.

The vortex-buffer crate is built around bytes::Bytes and therefore supports zero-copy cloning and slicing, but differs in that it can define and maintain a custom alignment.

  • Buffer<T> and BufferMut<T> provide immutable and mutable wrappers around bytes::Bytes and bytes::BytesMut respectively.
  • ByteBuffer and ByteBufferMut are type aliases for u8 buffers.
  • BufferString is a wrapper around a ByteBuffer that enforces utf-8 encoding.
  • ConstBuffer<T, const A: usize> provides similar functionality to Buffer<T> except with a compile-time alignment of A.
  • buffer! and buffer_mut! macros with the same syntax as the builtin vec! macro for inline construction of buffers.

You can think of BufferMut<T> as similar to a Vec<T>, except that any operation that may cause a re-allocation, e.g. extend, will ensure the new allocation maintains the buffer's defined alignment.

For example, it's possible to incrementally build a Buffer<T> with a 4KB alignment.

use vortex_buffer::{Alignment, BufferMut};

let mut buf = BufferMut::<i32>::empty_aligned(Alignment::new(4096));
buf.extend(0i32..1_000);
assert_eq!(buf.as_ptr().align_offset(4096), 0)

Comparison

Implementation Zero-copy Custom Alignment Typed
vortex_buffer::Buffer<T> ✔️ ✔️ ✔️
arrow_buffer::ScalarBuffer<T> ✔️ ❌️️️ ✔️
bytes::Bytes ✔️ ❌️️️ ❌️️️
Vec<T> ❌️ ❌️️ ✔️

Features

The arrow feature can be enabled to provide conversion functions to/from Arrow Rust buffers, including arrow_buffer::Buffer, arrow_buffer::ScalarBuffer<T>, and arrow_buffer::OffsetBuffer.

Dependencies

~7MB
~111K SLoC