2 releases
0.1.1 | Oct 15, 2021 |
---|---|
0.1.0 | Oct 12, 2021 |
#1016 in Concurrency
23KB
242 lines
linux-rtic
An RTIC implementation for real-time Linux.
How it Works
This implementation of RTIC is based on std::thread
by spawning a thread for each task priority group. Threads are initialized with SCHED_FIFO
real-time policy. Task priorities correspond 1:1 to Linux priorities and usually have a range of 1-99.
Scheduling
Scheduling of tasks is done by futex-queue, which cleverly utilizes futex syscall to wait on both immediate and scheduled (timed) tasks on a single syscall. No timer thread (and additional context switching) is required.
Resource Locking
Original cortex-m-rtic uses Stack Resource Policy (SRP), but it is difficult to emulate in userspace Linux. Firstly, setting thread priority for each lock/unlock involves an expensive syscall (~10us on Raspberry Pi 4). Secondly, setting thread priority does not guarantee that lower priority thread will not run. Lower priority thread can be executed on a different core, or when higher priority thread is suspended (i.e. I/O syscall). While it is possible to fix memory safety issues by a backup synchronisation mechanism (mutex), the syscall overhead is too high for real-time applications.
To solve the issue, a pcp-mutex library was written, which implements Original Priority Ceiling Protocol (OPCP). This allows preserving two important properties of SRP: bounding priority inversion and statically preventing deadlocks. This mutex is lock-free in the fast path. Technical details are in the pcp-mutex README.
Other Notes
Scheduling tasks in userspace threads is slow due to context switching overhead (~10us on Raspberry Pi 4) and other approaches were explored:
- POSIX signals are used in the older linux-rtfm implementation. They are faster than thread context switching, however, tasks are limited to reentrant (signal safe) functions only, which forces to use
no_std
. Also, resource locking slower, because of signal masking syscalls. - Kernel threads are only marginally faster as most of the overhead seems to be in the scheduler itself. So losing userspace safety and std library didn't seem worth it.
- Hard interrupt context would be closest to what cortex-m-rtic does, but Linux does not support interrupt prioritization (only IRQ threads have priorities) and would require major kernel modifications.
Examples
Running examples requires Linux with PREEMPT-RT patched kernel for SCHED_FIFO
and root privileges. This requirement can be lifted by compiling with --no-default-features
, but then all tasks will share the same priority.
Build:
cargo build --release --example priority_inversion
Run (requires sudo for sched_setscheduler
syscall):
sudo target/release/examples/priority_inversion
Single core:
sudo taskset -c 1 target/release/examples/priority_inversion
No real-time priorities:
cargo run --release --example priority_inversion --no-default-features
Tips to Make Real-Time More Real
- Apply
PREEMPT-RT
kernel patch and compile kernel withCONFIG_PREEMPT_RT_FULL
to reduce non-preemptable sections in the kernel. - Disable dynamic CPU frequency scaling. Either in kernel config or with
cpufreq-set -g performance
. - Use
isolcpus
kernel parameter to run RTIC on an isolated core. - Ensure that peripheral process (i.e.
spi0
) is scheduled with real-time priority:sudo chrt -f -p 50 $(pidof spi0)
. - Try to limit scheduling between different priority tasks to reduce context switching overhead.
- Watch A Checklist for Writing Linux Real-Time Applications - John Ogness, Linutronix GmbH
Credits
This work was done as a part of my Thesis at University of Twente.
Dependencies
~5–13MB
~158K SLoC