Uses new Rust 2021
|0.1.1||Oct 15, 2021|
|0.1.0||Oct 12, 2021|
#235 in Concurrency
Used in linux-rtic
An RTIC implementation for real-time Linux.
This implementation of RTIC is based on
std::thread by spawning a thread for each task priority group. Threads are initialized with
SCHED_FIFO real-time policy. Task priorities correspond 1:1 to Linux priorities and usually have a range of 1-99.
Scheduling of tasks is done by futex-queue, which cleverly utilizes futex syscall to wait on both immediate and scheduled (timed) tasks on a single syscall. No timer thread (and additional context switching) is required.
Original cortex-m-rtic uses Stack Resource Policy (SRP), but it is difficult to emulate in userspace Linux. Firstly, setting thread priority for each lock/unlock involves an expensive syscall (~10us on Raspberry Pi 4). Secondly, setting thread priority does not guarantee that lower priority thread will not run. Lower priority thread can be executed on a different core, or when higher priority thread is suspended (i.e. I/O syscall). While it is possible to fix memory safety issues by a backup synchronisation mechanism (mutex), the syscall overhead is too high for real-time applications.
To solve the issue, a pcp-mutex library was written, which implements Original Priority Ceiling Protocol (OPCP). This allows preserving two important properties of SRP: bounding priority inversion and statically preventing deadlocks. This mutex is lock-free in the fast path. Technical details are in the pcp-mutex README.
Scheduling tasks in userspace threads is slow due to context switching overhead (~10us on Raspberry Pi 4) and other approaches were explored:
- POSIX signals are used in the older linux-rtfm implementation. They are faster than thread context switching, however, tasks are limited to reentrant (signal safe) functions only, which forces to use
no_std. Also, resource locking slower, because of signal masking syscalls.
- Kernel threads are only marginally faster as most of the overhead seems to be in the scheduler itself. So losing userspace safety and std library didn't seem worth it.
- Hard interrupt context would be closest to what cortex-m-rtic does, but Linux does not support interrupt prioritization (only IRQ threads have priorities) and would require major kernel modifications.
Running examples requires Linux with PREEMPT-RT patched kernel for
SCHED_FIFO and root privileges. This requirement can be lifted by compiling with
--no-default-features, but then all tasks will share the same priority.
cargo build --release --example priority_inversion
Run (requires sudo for
sudo taskset -c 1 target/release/examples/priority_inversion
No real-time priorities:
cargo run --release --example priority_inversion --no-default-features
PREEMPT-RTkernel patch and compile kernel with
CONFIG_PREEMPT_RT_FULLto reduce non-preemptable sections in the kernel.
- Disable dynamic CPU frequency scaling. Either in kernel config or with
cpufreq-set -g performance.
isolcpuskernel parameter to run RTIC on an isolated core.
- Ensure that peripheral process (i.e.
spi0) is scheduled with real-time priority:
sudo chrt -f -p 50 $(pidof spi0).
- Try to limit scheduling between different priority tasks to reduce context switching overhead.
- Watch A Checklist for Writing Linux Real-Time Applications - John Ogness, Linutronix GmbH
This work was done as a part of my Thesis at University of Twente.