Turnstile

Turnstile is a Bash wrapper in the BootLoops repository that decides when a long, memory-hungry job may start on a Linux machine that other jobs share. You put it in front of the command you want to run and state roughly how much memory and how many CPU threads the job will use. Very large jobs then run one at a time in priority order, and smaller jobs run alongside only while memory and cores allow.

What it does

Turnstile is a repository utility for operating a shared machine. It is not one of the toolkit packages and computes no scientific result itself.

The usual way to make big jobs take turns on one machine is flock LOCKFILE command: whoever holds the lock file runs, everyone else waits. That has three problems:

turnstile_run.sh keeps the same kernel lock and adds a queue directory beside it, <lock>.q/. A job starts in token mode or in exempt mode. The four points below give the two modes, how the lock is held in token mode, and how threads are counted.

An operator steers the queue through three plain files in <lock>.q/ (listed under Routines), re-read at every poll, without touching running jobs.

Every run ends with an event=done line in <lock>.q/LEDGER.log. The line gives the job name, mode, est_rss_g, measured peak_rss_g, declared est_threads, measured peak threads peak_r, wall_s and rc. Those measured peaks are where the estimates for the next similar job should come from.

Exit code 75 means the start was refused (--no-wait) or timed out, 69 means the platform is not Linux, and 2 is a usage error. Anything else is the wrapped command's own code.

Turnstile has these limits:

Examples

Run the self-test. On a machine that meets the Requirements, run it from the root of a clone of the repository; the package directory is ops/turnstile/:

bash ops/turnstile/selftest.sh

The self-test uses a scratch lock in a temporary directory and never touches a real lock or a running job. Each check prints a PASS or FAIL line, a summary line counts passes and failures, and the exit status is 0 when nothing failed. The checks include priority order among three waiters, a PRIORITIES.conf override, and memory and HOLD refusals (exit code 75).

Start a large solver alone, ahead of lower-priority work. The job is expected to peak near 400 GiB with 16 worker threads, and everything after -- is the command (the example from the package's README):

ops/turnstile/turnstile_run.sh --lane solveA_big --class critical --est-rss 400 \
    --est-threads 16 --est-wall 12000 -- \
    nice -n 5 python3 -u solve_big.py ...

The wrapper registers solveA_big at priority 10 (the critical class), waits its turn, takes the lock, and passes the memory and thread checks. It then starts the command under taskset, the Linux utility that pins a process to a set of cores. While the command runs, the wrapper samples the command's memory every 30 seconds. The wrapper's own messages go to standard error. When the solver exits, the lock is released, the event=done line is written, and the wrapper exits with the solver's code.

Run two smaller jobs alongside the lock holder, then look at the queue. The scripts are called by bare name here; from the repository root, put ops/turnstile/ in front of each:

turnstile_run.sh --exempt --lane jobB_ramp --est-rss 60 --est-threads 8 -- python3 grow_table.py ...
turnstile_run.sh --exempt --lane jobB_watcher --est-rss 1 --est-threads 0 -- ./watch.sh
turnstile_status.sh

The first job starts as soon as 60 GiB fits under the memory rule and 8 more threads fit under the core cap. The second, declared with --est-threads 0, is treated as a negligible helper and skips the thread accounting. turnstile_status.sh prints available memory and HOLD state, the lock holder with its current and peak memory, the waiting jobs in service order, the running exempt jobs, the thread reservations, and the memory and core headroom.

Routines

The three scripts are in ops/turnstile/.

CommandWhat it does
turnstile_run.sh [options] -- <command ...>The launcher: token mode by default, --exempt to run alongside the lock holder
turnstile_status.sh [--lock PATH]Read-only report of holder, waiters, running exempt jobs, thread reservations and headroom
selftest.shThe self-test, run on a scratch lock

turnstile_run.sh takes these options.

OptionWhat it does
--lane NAMEJob name shown in the queue, the status report and the log, and matched by PRIORITIES.conf patterns
--class NAME, --prio NPriority by class (critical 10, high 20, normal 30, low 40, misc 50, the default) or as a number; lower runs first
--est-rss GExpected peak memory in GiB; required
--est-threads NExpected busy threads; omitted means the whole pinned CPU range, 0 means a helper with no thread accounting
--est-wall SExpected run time in seconds, stored in the queue entry
--exemptRun under the memory rule without the lock; refused when --est-rss is at or above TURNSTILE_EXEMPT_MAX_G
--no-wait, --timeout SExit with code 75 instead of polling, immediately or after S seconds
--lock PATH, --floor G, --poll SOverride the lock file, the memory floor and the poll interval for this call
--no-stagger, --no-gate, --no-width-gateOperator escapes: ignore the growth window, skip the memory check, or skip the wait on the core cap (threads still recorded)

The settings are environment variables. Every launcher on a machine must share TURNSTILE_LOCK.

VariableWhat it sets
TURNSTILE_LOCKThe lock file, by default TURNSTILE.lock under the system temporary directory
TURNSTILE_POLLSeconds between polls of the queue; an integer
TURNSTILE_MONSeconds between memory samples of a running job
TURNSTILE_FLOOR_GThe memory floor in GiB
TURNSTILE_RAMP_SECSThe growth window in seconds
TURNSTILE_EXEMPT_MAX_GThe estimate in GiB at or above which --exempt is refused; TURNSTILE_FORCE_EXEMPT=1 overrides
TURNSTILE_PINThe CPU list for taskset -c; all cores when unset
TURNSTILE_WIDTH_CAPThe core cap; the cores in TURNSTILE_PIN when unset
TURNSTILE_DRIFT_LOG_SECSMinimum seconds between width_drift log lines, written when a job's running threads exceed twice its declared count

The queue directory is the path of the lock file with .q in place of .lock. The wrapped command receives TURNSTILE_HELD=1 or TURNSTILE_EXEMPT=1, plus TURNSTILE_LANE and TURNSTILE_EST_THREADS. The default of each variable is given under "Env knobs" in the package README.md.

The operator files are in <lock>.q/.

FileWhat it does
PRIORITIES.confRe-orders waiting jobs: one <name-glob> <priority> per line, first match wins, default N allowed
HOLDWhile present, no new job starts in either mode; its text is shown as the reason in waiting messages and in turnstile_status.sh
COTENANT_RESERVE_GOne integer: the GiB kept free for another process, counted in every memory check

Requirements

Source and references

The code is ops/turnstile/ in the BootLoops repository, beside the tools/ tree rather than inside it. It is released under the MIT license.

← back to the tools index