#io #network #tokio #protocol #async

fixed-buffer

Fixed-size buffers for network protocol parsers

15 unstable releases (4 breaking)

Uses new Rust 2021

0.5.0 Mar 21, 2022
0.4.0 Mar 21, 2022
0.3.1 Dec 30, 2021
0.3.0 Oct 9, 2021
0.1.3 Oct 30, 2020

#905 in Network programming

Download history 78/week @ 2022-10-07 22/week @ 2022-10-14 26/week @ 2022-10-21 79/week @ 2022-10-28 188/week @ 2022-11-04 167/week @ 2022-11-11 201/week @ 2022-11-18 145/week @ 2022-11-25 157/week @ 2022-12-02 200/week @ 2022-12-09 154/week @ 2022-12-16 106/week @ 2022-12-23 140/week @ 2022-12-30 244/week @ 2023-01-06 175/week @ 2023-01-13 222/week @ 2023-01-20

810 downloads per month
Used in fewer than 6 crates

Apache-2.0

47KB
394 lines

fixed-buffer

crates.io version license: Apache 2.0 unsafe forbidden pipeline status

This is a Rust library with fixed-size buffers, useful for network protocol parsers and file parsers.

Features

  • forbid(unsafe_code)
  • Depends only on std
  • Write bytes to the buffer and read them back
  • Lives on the stack
  • Does not allocate
  • Use it to read a stream, search for a delimiter, and save leftover bytes for the next read.
  • No macros
  • Good test coverage (99%)
  • Async support by enabling cargo features async-std-feature, futures-io, smol-feature, or tokio.

Limitations

  • Not a circular buffer. You can call shift() periodically to move unread bytes to the front of the buffer.

Examples

Read and handle requests from a remote client:

use fixed_buffer::{deframe_line, FixedBuf};
use std::io::Error;
use std::net::TcpStream;

fn handle_conn(mut tcp_stream: TcpStream) -> Result<(), Error> {
    let mut buf: FixedBuf<4096> = FixedBuf::new();
    loop {
        // Read a line
        // and leave leftover bytes in `buf`.
        let line_bytes = match buf.read_frame(
            &mut tcp_stream, deframe_line)? {
                Some(line_bytes) => line_bytes,
                None => return Ok(()),
            };
        let request = Request::parse(line_bytes)?;
        handle_request(request)?;
    }
}

For a complete example, see tests/server.rs.

Read and process records:

use fixed_buffer::FixedBuf;
use std::io::{Error, ErrorKind, Read};
use std::net::TcpStream;

fn try_process_record(b: &[u8]) -> Result<usize, Error> {
    if b.len() < 2 {
        return Ok(0);
    }
    if b.starts_with("ab".as_bytes()) {
        println!("found record");
        Ok(2)
    } else {
        Err(Error::new(ErrorKind::InvalidData, "bad record"))
    }
}

fn read_and_process<R: Read>(mut input: R) -> Result<(), Error> {
    let mut buf: FixedBuf<1024> = FixedBuf::new();
    loop {
        // Read a chunk into the buffer.
        if buf.copy_once_from(&mut input)? == 0 {
            return if buf.len() == 0 {
                // EOF at record boundary
                Ok(())
            } else {
                // EOF in the middle of a record
                Err(Error::from(
                    ErrorKind::UnexpectedEof))
            };
        }
        // Process records in the buffer.
        loop {
            let num_to_consume =
                try_process_record(buf.readable())?;
            if num_to_consume == 0 {
                break;
            }
            buf.try_read_exact(num_to_consume).unwrap();
        }
        // Shift data in the buffer to free up
        // space at the end for writing.
        buf.shift();
    }
}
#

The From<[u8; SIZE]> implementation is useful in tests. Example:

use core::convert::From;
assert_eq!(3, FixedBuf::from(*b"abc").len());

Alternatives

Changelog

  • v0.5.0 - Move ReadWriteChain and ReadWriteTake to new read-write-ext crate.
  • v0.4.0
    • From<&[u8]>
    • write_bytes to take AsRef<[u8]>
    • Rename try_read_exact to read_and_copy_exact.
    • Rename try_read_bytes to try_read_exact.
    • Remove empty, filled, read_byte, read_bytes, try_parse, and write_str.
    • deframe to allow consuming bytes without returning a frame
    • write_bytes to write as many bytes as it can, and return new NoWritableSpace error only when it cannot write any bytes. Remove NotEnoughSpaceError. The method now behaves like std::io::Write::write.
  • v0.3.1 - Implement From<NotEnoughSpaceError> and From<MalformedInputError> for String.
Older changelog entries

TO DO

Release Process

  1. Edit Cargo.toml and bump version number.
  2. Run ../release.sh

License: Apache-2.0

Dependencies

~0–4MB
~60K SLoC