License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you shall be dual licensed as above, without any additional terms or conditions.

lib.rs:

This crate provides a cross-platform API for dealing with buffers backed by raw virtual memory.

Apart from providing protection and isolation between processes, paging, and memory mapped hardware, virtual memory serves to solve another critical issue: the issue of the virtual buffer. It allows us to reserve a range of memory only in the process's virtual address space, without actually committing any of the memory. This can be used to create a buffer that's infinitely growable and shrinkable in-place, without wasting any physical memory, nor even overcommitting any memory. It can also be used to create sparse data structures that don't overcommit memory.

The property of growing in-place is very valuable when reallocation is impossible, for example because the data structure needs to be concurrent or otherwise pinned. It may also be of use for single-threaded use cases if reallocation is too expensive (say, tens to hundreds of MB). However, it's probably easier to use something like Vec::with_capacity in that case.

See also the vec module for an implementation of a concurrent vector.

Reserving

Reserving memory involves allocating a range of virtual address space, such that other allocations within the same process can't reserve any of the same virtual address space for anything else. Memory that has been reserved has zero memory cost, however, it can't be accessed. In order to access any of the pages, you will have to commit them first.

Committing

A range of reserved memory can be committed to make it accessible. Memory that has been freshly committed doesn't use up any physical memory. It merely counts towards overcommitment, which may increase the likelihood of being OOM-killed, and may take up space for page tables and may use some space in the page file. A committed page is only ever backed by a physical page after being written to for the first time (being "faulted"), or when it was prefaulted.

Committed memory can be committed again without issue, so there is no need to keep track of which pages have been committed in order to safely commit some of them.

Decommitting

A range of committed memory can be decommitted, making it inaccessible again, and releasing any physical memory that may have been used for them back to the operating system. Decommitted memory is still reserved.

Reserved but uncommitted memory can be decommitted without issue, so there is no need to keep track of which pages have been committed in order to safely decommit some of them.

Unreserving

Memory that is unreserved is available for new allocations to reserve again.

Committed memory can be unreserved without needing to be decommitted first. However, it's not possible to unreserve a range of reserved memory, only the entire allocation.

Prefaulting

By default, each committed page is only ever backed by physical memory after it was first written to. Since this happens for every page, and can be slightly costly due to the overhead of a context switch, operating systems provide a way to prefault multiple pages at once.

Pages

A page refers to the granularity at which the processor's Memory Management Unit operates and varies between processor architectures. As such, virtual memory operations can only affect ranges that are aligned to the page size.

Cargo features