Turnstile
Turnstile is a Bash wrapper in the BootLoops repository that decides when a long, memory-hungry job may start on a Linux machine that other jobs share. You put it in front of the command you want to run and state roughly how much memory and how many CPU threads the job will use. Very large jobs then run one at a time in priority order, and smaller jobs run alongside only while memory and cores allow.
What it does
Turnstile is a repository utility for operating a shared machine. It is not one of the toolkit packages and computes no scientific result itself.
The usual way to make big jobs take turns on one machine is flock LOCKFILE command: whoever holds the lock file runs, everyone else waits. That has three problems:
- Order. The kernel hands the lock over roughly in arrival order, so an urgent job cannot move ahead.
- Small jobs. Jobs too small to need the lock are not counted, so several starting together can exhaust memory that each alone would have fit in.
- Test, then start. A script that tests the lock with
flock -nand then starts its job as a separate step holds nothing while the job runs.
turnstile_run.sh keeps the same kernel lock and adds a queue directory beside it, <lock>.q/. A job starts in token mode or in exempt mode. The four points below give the two modes, how the lock is held in token mode, and how threads are counted.
- Token mode. By default a job registers in the queue directory with a name (
--lane), a priority (--classor--prio, lower number first) and its expected peak memory in GiB (--est-rss, required). Waiting wrappers poll every 15 seconds. When the lock is free, only the highest-priority live waiter takes it, and ties go to the longest wait. - Held lock. The lock's file descriptor (the open handle on the lock file) is handed to the job itself, so the lock stays held for the whole run even if the wrapper is killed. The kernel releases it when the job exits. Entries left by killed wrappers are removed at the next poll.
- Exempt mode. With
--exempt, for jobs under 300 GiB, the job does not take the lock. It starts only when available memory covers its own estimate, the not-yet-used part of every running job's estimate, any operator reserve, and a fixed memory floor (150 GiB by default). This is the memory rule. A new exempt job also waits while another exempt job is under five minutes old (the growth window) and still below 90% of its estimate, so that only one grows at a time. - Threads. Both modes declare a thread count with
--est-threads. A new job waits while the declared threads of running jobs plus its own would exceed the core cap. The core cap is the number of cores in the CPU range all jobs are pinned to (the whole machine unlessTURNSTILE_PINnarrows it). A job that declares no thread count is assumed to need every core and runs alone.
An operator steers the queue through three plain files in <lock>.q/ (listed under Routines), re-read at every poll, without touching running jobs.
Every run ends with an event=done line in <lock>.q/LEDGER.log. The line gives the job name, mode, est_rss_g, measured peak_rss_g, declared est_threads, measured peak threads peak_r, wall_s and rc. Those measured peaks are where the estimates for the next similar job should come from.
Exit code 75 means the start was refused (--no-wait) or timed out, 69 means the platform is not Linux, and 2 is a usage error. Anything else is the wrapped command's own code.
Turnstile has these limits:
- Everything is local to one machine, so this is not a cluster scheduler.
- Priorities order the waiting queue and never interrupt a running job.
- Nothing caps the memory of a job once it runs.
- Jobs started without the wrapper count only through the memory they already occupy. Their future growth and their threads are not counted, although a plain
flockon the same file still excludes correctly. - Linux only. On macOS
turnstile_run.shandturnstile_status.shexit with code 69 and a message, andselftest.shprintsSKIPlines and exits 0.
Examples
Run the self-test. On a machine that meets the Requirements, run it from the root of a clone of the repository; the package directory is ops/turnstile/:
bash ops/turnstile/selftest.sh
The self-test uses a scratch lock in a temporary directory and never touches a real lock or a running job. Each check prints a PASS or FAIL line, a summary line counts passes and failures, and the exit status is 0 when nothing failed. The checks include priority order among three waiters, a PRIORITIES.conf override, and memory and HOLD refusals (exit code 75).
Start a large solver alone, ahead of lower-priority work. The job is expected to peak near 400 GiB with 16 worker threads, and everything after -- is the command (the example from the package's README):
ops/turnstile/turnstile_run.sh --lane solveA_big --class critical --est-rss 400 \
--est-threads 16 --est-wall 12000 -- \
nice -n 5 python3 -u solve_big.py ...
The wrapper registers solveA_big at priority 10 (the critical class), waits its turn, takes the lock, and passes the memory and thread checks. It then starts the command under taskset, the Linux utility that pins a process to a set of cores. While the command runs, the wrapper samples the command's memory every 30 seconds. The wrapper's own messages go to standard error. When the solver exits, the lock is released, the event=done line is written, and the wrapper exits with the solver's code.
Run two smaller jobs alongside the lock holder, then look at the queue. The scripts are called by bare name here; from the repository root, put ops/turnstile/ in front of each:
turnstile_run.sh --exempt --lane jobB_ramp --est-rss 60 --est-threads 8 -- python3 grow_table.py ... turnstile_run.sh --exempt --lane jobB_watcher --est-rss 1 --est-threads 0 -- ./watch.sh turnstile_status.sh
The first job starts as soon as 60 GiB fits under the memory rule and 8 more threads fit under the core cap. The second, declared with --est-threads 0, is treated as a negligible helper and skips the thread accounting. turnstile_status.sh prints available memory and HOLD state, the lock holder with its current and peak memory, the waiting jobs in service order, the running exempt jobs, the thread reservations, and the memory and core headroom.
Routines
The three scripts are in ops/turnstile/.
| Command | What it does |
|---|---|
turnstile_run.sh [options] -- <command ...> | The launcher: token mode by default, --exempt to run alongside the lock holder |
turnstile_status.sh [--lock PATH] | Read-only report of holder, waiters, running exempt jobs, thread reservations and headroom |
selftest.sh | The self-test, run on a scratch lock |
turnstile_run.sh takes these options.
| Option | What it does |
|---|---|
--lane NAME | Job name shown in the queue, the status report and the log, and matched by PRIORITIES.conf patterns |
--class NAME, --prio N | Priority by class (critical 10, high 20, normal 30, low 40, misc 50, the default) or as a number; lower runs first |
--est-rss G | Expected peak memory in GiB; required |
--est-threads N | Expected busy threads; omitted means the whole pinned CPU range, 0 means a helper with no thread accounting |
--est-wall S | Expected run time in seconds, stored in the queue entry |
--exempt | Run under the memory rule without the lock; refused when --est-rss is at or above TURNSTILE_EXEMPT_MAX_G |
--no-wait, --timeout S | Exit with code 75 instead of polling, immediately or after S seconds |
--lock PATH, --floor G, --poll S | Override the lock file, the memory floor and the poll interval for this call |
--no-stagger, --no-gate, --no-width-gate | Operator escapes: ignore the growth window, skip the memory check, or skip the wait on the core cap (threads still recorded) |
The settings are environment variables. Every launcher on a machine must share TURNSTILE_LOCK.
| Variable | What it sets |
|---|---|
TURNSTILE_LOCK | The lock file, by default TURNSTILE.lock under the system temporary directory |
TURNSTILE_POLL | Seconds between polls of the queue; an integer |
TURNSTILE_MON | Seconds between memory samples of a running job |
TURNSTILE_FLOOR_G | The memory floor in GiB |
TURNSTILE_RAMP_SECS | The growth window in seconds |
TURNSTILE_EXEMPT_MAX_G | The estimate in GiB at or above which --exempt is refused; TURNSTILE_FORCE_EXEMPT=1 overrides |
TURNSTILE_PIN | The CPU list for taskset -c; all cores when unset |
TURNSTILE_WIDTH_CAP | The core cap; the cores in TURNSTILE_PIN when unset |
TURNSTILE_DRIFT_LOG_SECS | Minimum seconds between width_drift log lines, written when a job's running threads exceed twice its declared count |
The queue directory is the path of the lock file with .q in place of .lock. The wrapped command receives TURNSTILE_HELD=1 or TURNSTILE_EXEMPT=1, plus TURNSTILE_LANE and TURNSTILE_EST_THREADS. The default of each variable is given under "Env knobs" in the package README.md.
The operator files are in <lock>.q/.
| File | What it does |
|---|---|
PRIORITIES.conf | Re-orders waiting jobs: one <name-glob> <priority> per line, first match wins, default N allowed |
HOLD | While present, no new job starts in either mode; its text is shown as the reason in waiting messages and in turnstile_status.sh |
COTENANT_RESERVE_G | One integer: the GiB kept free for another process, counted in every memory check |
Requirements
- Runs on: Bash on Linux with
flock,taskset,setsid,nproc, GNUps,awkand/proc/meminfo. Turnstile has no macOS port. - Install: nothing to install.
- Self-test:
bash ops/turnstile/selftest.shfrom the repository root ends with a line of the form== selftest: N pass, 0 fail ==and exit status 0. The repository'srun_selftests.pycoverstools/only, so run this self-test by hand. - Optional:
TURNSTILE_TEST_DIR=<dir>chooses the scratch directory of the self-test.
Source and references
The code is ops/turnstile/ in the BootLoops repository, beside the tools/ tree rather than inside it. It is released under the MIT license.