13 releases
Uses old Rust 2015
0.2.9 | Feb 18, 2021 |
---|---|
0.2.8 | Feb 18, 2021 |
0.2.7 | Sep 16, 2020 |
0.2.4 | Aug 6, 2020 |
0.1.6 | Jul 28, 2020 |
#2 in #queueing
42 downloads per month
77KB
2K
SLoC
Round queue
The rq
program implements a very simple queuing system. This system
favors convention over configuration, and hence requires zero
configuration, but does impose a constraint on how you set up your
system. In particular, all users who use rq
must have home
directories within a single parent directory (this is typically
/home
, but could be some other directory). Also, rq
requires that
home directories be shared across any computers involved in the
cluster, and mounted at the same point on each computer.
This program is designed to be very easy to configure and use, and should be reliable. However, it is not designed to be efficient when there are large numbers of jobs or users. It is intended for a small computing cluster, where $O(N^2)$ operation is acceptable.
Installing rq
To install rq, you must have rust installed (see https://rustup.rs). Then just type
cargo install roundqueue
Using rq
Each user running rq
should run rq daemon
on each compute node on
which they wish jobs to be able to run. On my cluster, I do this on
users' behalf using systemd
to start them up. This means that with
$N$ users on $M$ nodes, you will have $NM$ daemons running.
To submit a job, you can run something like
rq run --job-name my-compute-job ./compute --flag-for-compute --other-flag
in the directory where you want this command to run. By default, jobs are assumed to run on a single cpu. You can see the running (and waiting to run) jobs by simply executing
rq
You can cancel a job by executing
rq cancel my-compute-job
Finally, you can see how busy the nodes are with
rq nodes
How does it work
The security design of rq
involves never communicating directly over
the network. Instead, communication is entirely done by creating and
reading files in the home directory (which is presumably shared over
the network). This simplifies the design of rq
, and presumably
increases its security, provided your file system is secure.
It is not necessary for all users to have the rq daemon
running on
all nodes, however, any nodes that a given user does not have the
daemon running on will not run that user's jobs.
Each users job information is stored in the $HOME/.roundqueue/
directory. Each job is a separate file in a subdirectory there.
rq run
Submitting a job consists of creating a single JSON file in
$HOME/.roundqueue/waiting/
. That's all.
Dependencies
~3–12MB
~123K SLoC