3 releases (stable)

1.0.1 Feb 17, 2024
1.0.0 Feb 8, 2024
0.1.3 Oct 28, 2023

#152 in Caching

39 downloads per month

MIT/Apache

68KB
1.5K SLoC

TMFAlloc: Transactional Mapped File Allocator for Rust

Transactional memory-mapped file allocator inspired by POST++. Allows to merge data representation and storage tiers into one tier. May be used as fixed-schema client cache, embedded application data storage, etc.

Features

  • File-backed memory-mapped storage.
  • Implements std::alloc::Allocator trait, so usual std::collections::* (except Hash*), std::boxed::Box, etc. containers could be stored in and retreived from the file.
  • Single writer/multiple readers in multi-threaded code.
  • Write transactions exploit memory page protection and copy-on-write log file.
  • Storage has user-defined Root type generic parameter to store all application-specific parameters, collections, etc.
  • Storage file is flock-protected, so simultaneous processes access is possible.
  • Allocates the least but fittable free block with the lowest address among all equally-sized blocks.
  • Average allocation and deallocation cost is O(log(number of free blocks)) (plus possible file operations costs).
  • Allows allocation arena expansion.
  • Runs on Linux and Windows.
  • Runs on 32-bit and 64-bit CPUs.

Caveats on current limitations

  • Do not use the same storage Holder in both parent and fork-ed child processes.
  • It's not guaranteed if dangling pointers to unmapped storage memory could be avoided in case of some non-standard use.
  • Memory mapping address cannot be changed after storage initialization. Hence, explicit memory mapping address specification on storage initialization is recommended.
  • Memory allocation quantum is 32 or 64 bytes on, respectively, 32 or 64-bit architectures, that may be percieved as wasteful.

What's new in 1.0.1

  • tmfalloc::Allocator::{allocate, deallocate, grow, shrink} now panic if the current thread has not opened a write transation for the appropriate storage address space. This is to hopefully prevent possible misuse of leaked allocators.
  • tests::allocator_leak_should_panic to test for this panic.

To do list

  • Concurrent threads access tests to detect race conditions.
  • 64 bytes allocation quantum may be too much. Two RBTrees holding free blocks may be too slow. Any suggestions on how to improve this?
  • Main file page mlock-ing instead of log file immediate fsync in signal handler to increase write throughput and decrease latency.
  • 100% code lines test coverage. How to collect coverage of docs tests?
  • Do less RBTrees traversal on (de/re)allocations by (re)using already available pointers.

License

Apache License v2.0 or MIT License

Author and feedback

Vladimir Voznesenskiy <vvoznesensky@yandex.ru>. Looking for a Rust job.

Feedback is welcome. Please, send me an email, if you need more tests, etc.

Dependencies

~2–41MB
~631K SLoC