#byte-stream #byte #byte-slice #serialization #derive #serde

byteserde

A generic serialization/deserialization framework for mapping byte streams to/from arbitrary struct types with focus on performance

8 releases (5 breaking)

0.6.2 Jan 14, 2024
0.6.1 Jan 11, 2024
0.6.0 Dec 24, 2023
0.5.0 Nov 13, 2023
0.1.1 May 19, 2023

#351 in Encoding

Download history 234/week @ 2023-12-23 444/week @ 2023-12-30 736/week @ 2024-01-06 627/week @ 2024-01-13 251/week @ 2024-01-27 61/week @ 2024-02-03 168/week @ 2024-02-10 89/week @ 2024-02-17 314/week @ 2024-02-24 8/week @ 2024-03-02 8/week @ 2024-03-09 11/week @ 2024-03-16 18/week @ 2024-03-23 23/week @ 2024-03-30 3/week @ 2024-04-06

59 downloads per month
Used in 13 crates (9 directly)

Custom license

66KB
1K SLoC

master

Motivation

  • The motivation for this product is two fold:

    • To be able to map a byte slice &[u8] , typically acquired from a network, into a rust struct datamodel by simply using derive macro annotations and attributes auto-generate necessary code.

    • This comes very handy when the byte slice is not serialized using one of the existing and widely available protocols. Ex: An application specific C-Struct.

    • To be the fastest byte stream serializer/deserializer on the market for latency sensitive use cases. Benchmark results below show a performance summary of serializing & deserializing a Reference Struct using different frameworks available:

      • byteserde - ~15ns read/write
      • bincode - ~15ns read / ~100ns write
      • rmp-serde - ~215ns read/write
      • serde_json - ~600ns read/write - understandably slow due to strings usage

When to use and when to avoid this framework

Use

  • If you work with network protocols that deliver data in a byte stream format that is not matching one of the widely available standards, ex: bincode, protobuf. Use this product to efficiently map your byte stream into a rust struct.

  • You have a latency sensitive usecase. Note that this protocol does not add any schema information during serialization process and hence is equivalent of dumping the memory layout of the struct without padding

  • Example of protocols that are a perfect fit for this framework.

Avoid

  • If the byte stream is serialized or deserialized using a wideley available standard avoid this framework and instead the that respective standard to work with the byte stream

The project contains three craits

byteserde_derive@crates.io - byteserde_derive/Cargo.toml

  • contains derive macros that generates byteserde@crates.io traits
    • #[derive(ByteSerializeStack)] - generates ByteSerializeStack trait

    • #[derive(ByteSerializeHeap)] - generates ByteSerializeHeap trait

    • #[derive(ByteDeserializeSlice)] - generates ByteDeserializeSlice<T> trait

    • #[derive(ByteSerializedSizeOf)] - generates ByteSerializedSizeOf trait - this trait provides an associated method byte_size() which gives you a struct memory size in bytes without alignment. However it does not support types which heap allocate, ex: Vectors, Strings, or their derivations.

    • #[derive(ByteSerializedLenOf)] - generates ByteSerializedLenOf trait - this trait provides an instance method byte_len(&self) which gives you memory size in bytes without alignment of specific instance. It exists specifically to deal with types that ByteSerializedSizeOf trait does not support

  • For more examples follow here
  • NOTE: that Union and Unit structure are not supported, but it might change in the future.

byteserde@crates.io - byteserde/Cargo.toml

  • Highlights
    • ByteSerializerStack<CAP> - provides ultra fast serializer into a pre allocated byte array [u8; CAP] on stack, hence the name, it is very fast but at the cost of you needing to specify the size of the LARGEST struct you will attempt to serialize. If you reach the boundary of this preallocated byte array, your serialization will fail. This utility provides a reset features, which moves the internal counter to the begining, and allows you to recycle the buffer multiple times.

    • ByteSerializerHeap - provides a fast enough for most speed by serializing into a byte vector Vec<u8>, hence the name. This utility trades some performance in return for not having to worry about knowing the LARGEST struct size in advance.

    • ByteDeserializerSlice - takes a byte stream &[u8] irrespctive of heap vs stack allocation and turns it into a struct

byteserde_types@crates.io - byteserde_types/Cargo.toml

  • contains optional ascii string related types and macros, which are typically usefull when dealing with fixed length strings while parsing a byte stream, follow example section for more details.

Examples & Overview

  • Please refer to this document for a number of comprehensive examples and features overview.

Dependencies

~170KB