#memory #tracker #allocation #header

near-rust-allocator-proxy

Rust allocator proxy with added header

4 releases

0.3.0 Oct 11, 2021
0.2.9 Feb 23, 2021
0.2.8 Feb 19, 2021
0.2.7 Feb 10, 2021

#34 in Memory management

Download history 1830/week @ 2021-07-07 1353/week @ 2021-07-14 1536/week @ 2021-07-21 1702/week @ 2021-07-28 1698/week @ 2021-08-04 2311/week @ 2021-08-11 1813/week @ 2021-08-18 1281/week @ 2021-08-25 1341/week @ 2021-09-01 1288/week @ 2021-09-08 2226/week @ 2021-09-15 2596/week @ 2021-09-22 2221/week @ 2021-09-29 2594/week @ 2021-10-06 2896/week @ 2021-10-13 1919/week @ 2021-10-20

7,745 downloads per month
Used in 3 crates (2 directly)

Apache-2.0

17KB
333 lines

Track Rust memory usage by adding a 32 bytes header to all allocations. See https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html

Usage

You can use code below to enable usage of this library:

use near_rust_allocator_proxy::allocator::MyAllocator;
use jemallocator::Jemalloc

#[global_allocator]
static ALLOC: MyAllocator<Jemalloc> = MyAllocator::new(Jemalloc);

Design

  • header - For each memory allocation we add a 32 bytes header. This allows figuring out how memory was allocated by looking at memory dump of the process.
  • per thread memory usage stats - thread_memory_usage(tid) method can be used to get amount of memory allocated by thread
  • PRINT_STACK_TRACE_ON_MEMORY_SPIKE - if set to true a stack trace will be used on memory spike

Constants

  • ENABLE_STACK_TRACE - if enabled backtrace will get executed on each allocation and stack pointer will be added to the header
  • MIN_BLOCK_SIZE - if allocation size of below MIN_BLOCK_SIZE, we will only run backtrace SMALL_BLOCK_TRACE_PROBABILITY percentage of time
  • SMALL_BLOCK_TRACE_PROBABILITY - probability of running a stack trace for small allocations
  • REPORT_USAGE_INTERVAL - if printing memory spikes is enabled print if memory usage exceeded this value in bytes
  • PRINT_STACK_TRACE_ON_MEMORY_SPIKE - if true print stack trace when memory usage exceeds REPORT_USAGE_INTERVAL on given Rust thread

Header representation

Allocation structure:

  • magic - unique 8 bytes identifier, which is used to mark memory allocations
  • size - size in bytes
  • tid - thread id
  • stack - stack trace during time of allocation
#[repr(C)]
struct AllocHeader {
    magic: u64,
    size: u64,
    tid: u64,
    stack: [*mut c_void; STACK_SIZE],
}

TODO

  • Add methods to set the configuration instead of having to change the constants.
  • Add tests

Dependencies

~5MB
~99K SLoC