Gate Throughput
One duty, shared by three seats: doctor owns the box, test-engineer owns
what a change must prove, release-manager owns the mint. All three are
spending the same scarce thing — a gate slot — and the sentence that keeps them
from contradicting each other is this:
The queue is what slows work, not the suite.
A single tsc --noEmit on one.ie/web is the 2.5GB item in the whole harness.
The stall measured on 2026-09-13 was a lock on the typecheck path, and not
one second of it was vitest. Anyone who reads "the tests are slow" and reaches
for the test files is optimising a thing that was never the constraint.
The number that settles it, measured 2026-09-13 from the land door's own
records (one.ie/web/src/data/deploy-runs.json): ten land runs, 9,304s of
wall clock, of which 8,413s (90.4%, 8 of the 10 runs) produced ZERO phases —
they never ran a gate at all, and two printed no slot after 1800s. Actual gate
compute across the same ten: 891s. That is 9.4× more time spent waiting
than testing, and it is the whole answer to "are the tests slowing us down."
Verdict: the tests are not what slows work. Admission to the queue is.
Do not quote "the suite is 87.3s" — that figure is stale. It was measured
2026-09-01 on ~1135 files. Same suite by vitest's own Duration, 1295–1317
files (+1.7%), measured 2026-09-11/12/13: 113.55s uncontended floor →
421.43s contended peak, a 3.71× spread on a suite that barely changed.
The internal shape proves it is contention and not work: import 158.46s / environment 71.86s at the floor against import 644.96s / environment 250.56s
at the peak — the same forks doing the same work 4× slower because the box is
paging. Quote the floor (113.55s) as the cost and the spread as the problem.
This file is the one copy. Three agent bodies point at it; none of them restates
it, because a fact copied three times drifts three ways.
What a gate costs, and what actually runs out
Memory, never cores. gate_price_mb (.claude/scripts/lib/govern.sh:199 →
_gov price-mb) prices one slot at ~2GB by default; GOVERN_GB_PER_CYCLE is
the operator override and it wins. Every gate is charged that price whatever it
actually weighs, which is the design working and also the thing to know:
| Thing |
Measured 2026-09-13 |
Charged |
single-file vitest |
152M across 4 procs |
~2GB |
| full vitest pool (8 forks) |
1.6G |
~2GB |
tsc --noEmit on one.ie/web |
up to 2.5G |
~2GB |
So a one-file test queued 26 minutes behind five verify-fast runs — the
cheapest possible question paying the most expensive possible price for a seat.
That asymmetry is the throughput problem in one line, and it is why "run the
cheapest lane that can still go red" is a throughput rule and not a style
preference.
Worktrees cost DISK; sessions, browsers and dev servers cost RAM. Measured
the same day: one astro dev + workerd pair 2.4G, six Claude sessions
~2.6G with two MCP servers each. Sweeping landed worktrees is hygiene and
reclaims gigabytes of disk — it is not the throughput fix, and pricing it as one
is how the box stays at gates funded 1.
bash .claude/scripts/health.sh is the reading: load, swap, gates funded,
slots held, orphans, plus the four estate surfaces. gates funded 1 names the
binding constraint out loud.
Admission control — never queue a gate whose answer is already memoised
The rule: probe the memo BEFORE you take a slot. A cache hit needs no memory,
no forks and no slot, so it must never wait behind a real gate.
This is not a proposal — the mechanism exists and is proven, and
test-cached.sh states the reason in its own docblock: verify-fast.sh once
"took a GOVERNOR SLOT and then asked whether there was anything to run",
measured 2026-09-09 at 855s and 1324s queued for work that was a memo hit
costing ~0.
| Affordance |
Where |
Answers |
TEST_CACHE_PROBE=1 |
test-cached.sh |
would this HIT? exit 0 hit / 1 miss, runs nothing |
TEST_CACHE_KEY_ONLY=1 |
test-cached.sh:184 |
prints the key, runs nothing |
tsc-cached.sh --probe <folder> |
branch feat/govern-order |
green (0) / unknown (1), takes neither slot nor lock |
Where it belongs: at the CALLER, not in gate-run.sh. gate-run.sh receives
an opaque argv (bash test-full.sh) and cannot know the memo key without running
the thing. Teaching it to guess from argv is the same defect as inferring gate
weight from argv — see the next section. The cached scripts already answer
correctly; the caller must ask before it wraps.
The one unconverted caller, measured 2026-09-13. verify-fast.sh:338
(_memo_hit), release.sh, do-close.sh and close-metrics.sh all probe
first. deploy.sh does not — grep -c TEST_CACHE_PROBE .claude/scripts/deploy.sh
is 0. Its two heavy gates go through gate_start_heavy → gate_start_governed,
which wraps the command in gate-run.sh and buys the slot up front
(deploy.sh:620-634). Measured that night at 22:45: the deploy-vitest gate
queued 300s and then reported both lanes as cache HITs — 300 seconds of a
1-slot box spent delivering an answer that needed 0 seconds of compute.
What must NOT be skippable, or the memo becomes a lie:
- Only a PASS is ever memoised.
test-cached.sh writes its stamp only after
a green run and its --self-test asserts "a red run recorded no stamp";
tsc-cached.sh records only a 0 count. So a miss is never "it failed" — it
is unknown, and unknown must compute.
- A probe that ERRORS is
unknown, never green. Three-valued and only one
value skips: green → skip; unknown → run; any other exit → run. An
erroring probe that skipped would be an unrun check reading as a pass, which
is the single failure this harness exists to prevent.
- A probe must never compute and never stamp. A probe that computed would
mint the green half and hide the very cost it exists to avoid.
- The fast lane's own two invariants still hold underneath: an empty diff
falls back to the FULL suite, never to a pass; and the pinned suites
always run.
How it goes red (the cheap proof — no suite, a stub probe and four cases):
drive the caller with a probe that returns green / unknown / exit 2 /
garbage and assert skip / run / run / run. Then the live half: with a warm
memo assert no lock-slot-* was ever created; with TEST_CACHE_DISABLE=1
assert the gate does take a slot and does run.
Class pricing — light and heavy are different pools
Today every gate is charged the same flat price, and gate-run.sh printed it
verbatim on 2026-09-13: memory funds 1 at 2300MB each (config 2). A
single-file vitest measured at 152M is charged 2300M — overpriced 15× —
while tsc at 2.5G is UNDER-priced. The governor is simultaneously starving
the cheapest questions and over-admitting the most expensive one. That inversion
is why a one-file test waited 26 minutes.
Two rules, both non-negotiable, and the evidence for each:
- The caller DECLARES light. Never infer it from argv.
verify-fast is the
counterexample that settles it: its argv is a tiny bash verify-fast.sh and
it shells out to a 2.5GB tsc. Measured 2026-09-13, the same lane cost
54s and 837s. Any argv-based classifier calls that light and is
wrong by 15×.
- A light gate's child must NOT inherit the light lane.
gate-run.sh:31
execs straight through on GOVERN_IN_GATE=1, so today a child inherits the
parent's slot whatever it weighs — a light parent would launder an unbounded
heavy child.
How to honour rule 2 without re-creating the deadlock. The obvious fix —
"a heavy child inside a light parent takes its own heavy slot" — is hold-and-wait,
exactly the shape that deadlocked this box three times on 2026-09-13 (see below).
So: light is a PROMISE about a leaf, enforced by a checker, with a loud refusal
as the runtime backstop. Declare light only for a gate that shells out to
nothing heavy; enforce it the way governor-doors-check.sh already enumerates
package scripts and follows one hop through bash <x>.sh; and if a heavy gate
is requested while the ambient class is light, refuse loudly rather than exec
through. A refusal costs one run; a silent laundering costs the box.
Price it, never free. A light slot is ~0.4G (152M measured, ~2.6× headroom)
with its own cap, and gate_headroom must still be able to lower both pools —
a probe that cannot read memory returns 99 and must never silently serialise
the fleet.
The deadlock — LOCK→SLOT against SLOT→LOCK
Two orderings exist in the same harness, and they hold each other's resource:
- LOCK→SLOT.
tsc-cached.sh:127 takes the per-folder lock
(gate_lock "$_key"), and then, still holding it, routes the compute through
gate-run.sh (:155) to queue for a slot.
- SLOT→LOCK.
gate-run.sh verify-fast -- verify-fast.sh holds a slot, and its
child tsc-cached queues for the folder lock that the standalone above is
sitting on.
Neither can finish. Three occurrences in 35 minutes on 2026-09-13, 8–28 minutes
each at 0% CPU — one of them a production release mint.
Measured live while this file was being written, same box, same hour:
lock-slot-1 pid 10866 bash gate-run.sh verify-fast -- verify-fast.sh 0.0% CPU 08:23
lock-tsc-one.ie_web pid 13011 bash tsc-cached.sh one.ie/web 0.0% CPU 08:06
pid 10866's ancestry contains gate-run.sh; pid 13011's does not — its parent is
a session shell. A fourth process (13546) was queueing behind 13011 for the same
lock. That is the pair, exactly.
gate-reaper.sh cannot see this and never will. It kills only a provably
dead owner — ppid == 1, a gate-shaped command naming this repo, and real CPU
burned (gate-reaper.sh:17-22). A deadlock is two live processes at 0% CPU
with living parents. Every condition the reaper checks says both are fine.
A fix is on branch feat/govern-order — 5ba2d9297 fix(governor): take the SLOT before the LOCK, with a red-proof checker at
.claude/scripts/govern-order-check.sh. It was not on dev when this was
written (dev head d66c63429), and a worktree runs the governor code IT has, not
the one on trunk — so a tree picks the fix up only when it rebases. Check with
git log dev --oneline | grep -i govern before assuming a tree is safe.
The cheap cut
Kill the ONE process holding a folder lock with NO slot — that is, no
gate-run.sh anywhere in its ancestry. It is the standalone that took a lock it
could not pay for; dropping it releases the lock and the slot owner finishes the
work.
D="${TMPDIR}one-govern"
for d in "$D"/lock-*; do printf '%-26s %s\n' "$(basename "$d")" "$(cat "$d/pid" 2>/dev/null)"; done
# then walk each pid up and look for gate-run.sh:
q=<pid>; while [ -n "$q" ] && [ "$q" != 1 ]; do ps -o pid=,ppid=,args= -p "$q"; q=$(ps -o ppid= -p "$q" | tr -d ' '); done
Two ways to get this wrong, both measured:
- Killing the slot owner instead throws away a whole verify run — minutes of
work already done, and the lock is still held afterwards.
- Killing lock holders repeatedly just promotes the next standalone. Three in
a row went the same way before the lock finally landed on an in-gate owner and
tsc ran at 121% CPU. If the second cut does not free the queue, stop
cutting and read the pairing again.
A pile of 10 standalone tsc-cached on one lock, oldest 50 minutes, was
measured on 2026-09-13. Ten processes, one answer, none of them computing it.
The memo key, and why worktrees cannot share a pass
Measured 2026-09-13, and this is the duplication to name. The same selection
(signals-parity) asked in five checkouts through TEST_CACHE_KEY_ONLY=1 — a
lookup that runs nothing — produced five different keys:
. HEAD=8b9876162 key=6a3eb01ee06743b6
worktrees/dev HEAD=d66c63429 key=e390855f4920b7a4
worktrees/inbox-find HEAD=ef2c286ac key=d5882811f1934e70
worktrees/ehc-polish HEAD=8323fcb70 key=b604bfdc9e44109e
worktrees/tasks-board HEAD=db3e3a649 key=e1ba495fa02275eb
The key includes git rev-parse HEAD and git diff HEAD across the whole
repo, so two trees that differ only in an unrelated markdown file can never
share a pass. Same day: 18 worktrees at 18 distinct HEADs, and 69 distinct
tsc-one.ie_web fingerprints (≥69 real computes, one every ~19 minutes for 22
hours). Zero of the 69 were avoidable by cross-tree sharing — the memo is
working exactly as designed; there is simply no duplicate tree to share with.
Do not "fix" the tsc memo: it is not broken, and the fingerprints prove it.
The reaching set is SMALL and enumerable — measured 2026-09-13
CLAUDE.md § Tests forbids narrowing the key by extension and says why, then
names the real fix: "fix the reaching, not the key." Here is the count that
tells you which way to go, taken over one.ie/web:
|
|
| test files total |
1316 |
| files that touch the filesystem at all |
202 (15.3%) |
files that reach outside one.ie/web |
29 (2.2%) |
Those 29 reach six roots, and nothing else: channels/src (9) ·
pay/backend (8) · packages/{sdk,mcp,cli} (6) · schema/{one.tql,codegen} (3)
· .claude/scripts (3) · text/ (2 files, reading 2 named documents —
signals-catalog.md and blocks-agents-docs.md).
Doctrine: the reaching set is small and enumerable, so the key MAY hash that
set instead of the whole-repo diff — but only behind a checker that fails when
a test reaches somewhere undeclared. Two named files in text/ are why an
edit to any of the hundreds of other documents there currently invalidates a
1317-file suite memo. That is the duplication worth removing.
The guard is the whole design, not a detail. A wrong key that replays a
stale pass is the worst outcome in this repo — worse than every queueing loss
on this page, because it is a green gate over a red tree. So the enumeration is
derived mechanically (grep the suite for out-of-package reaches) and a checker
refuses to run when a target is not in the declared set — the shape
factory-repo.sh --check-portability already uses for unclassified scripts.
Narrowing without that checker is forbidden.
Several features in flight at once
Measured 2026-09-13: four land.sh runs and a release mint queued together.
What is shareable is already shared, and the rule is worth knowing before you
add anything:
- The receipt is content-addressed, so a neighbour's suite run counts as
yours. test-cached.sh keys on the tree, not the runner — the 2026-09-05
production ship cost 77s and read full suite REUSED because another session
had run that exact tree minutes earlier. On the promote path concurrent
sessions subsidise each other.
- The tsc memo is shared by design —
tsc-cached.sh's key is byte-identical
to do-reconcile.sh's so a reconcile and N cycle gates on one tree cost one
tsc between them. It shares nothing today only because no two trees are equal.
- What must stay serial: anything mutating one shared resource. The
real-TypeDB lane runs against one production cluster shared by every
session, which is why its fixtures are epoch-named and swept by age rather
than wholesale (tests/helpers/probe-sweep.ts). And a release mint is never
shared and never cut — see below.
Batch the promote rather than minting per merge: one gate pays for N merges,
and the receipt binds to a sha, so promoting a slightly-older green sha is the
design working, not a compromise.
The false RED — an unrun check is not a red
This is the reading rule every seat here must know, because the output looks
exactly like a failure:
[gate-run] verify-fast: no slot after 1800s — machine saturated. (gate-run.sh:60)
✗ typecheck FAILED
That check never ran. gate_slot timed out at GOVERN_QUEUE_WAIT (default
1800s, gate-run.sh:59), the wrapper exited 1, and the caller printed its
failure line for a command that was never executed. A starved gate and a broken
gate print the same word.
On a contended box, raise the wait rather than believe the word:
GOVERN_QUEUE_WAIT=5400 (and TSC_CACHE_WAIT for the folder lock, default 900s,
tsc-cached.sh:127). Measured 2026-09-13: the first --gates-only mint printed
✗ typecheck FAILED for checks that never ran; the re-run with
GOVERN_QUEUE_WAIT=5400 went green in 580s with 13276 tests passing. Nothing
about the tree had changed.
An unrun gate is never a red, and never a pass. Report it as unrun and say
which wait bounded it.
Do not cut a gate that belongs to a release mint
A mint is the most expensive gate on the box and the one whose loss costs most —
and it looks identical to a stuck verify from the outside. The tell is the
tree it runs from. release.sh runs its gates as
( cd "$WT" && DEPLOY_YES=1 bash .claude/scripts/deploy.sh … ) where $WT is
.release/ (release.sh:228,244), so every process in that mint's tree has
.release in its cwd and in its script path:
lsof -a -d cwd -p <pid> | tail -1 # prints the process's cwd — verified on a live pid
ps -o args= -p <pid> # a mint names .../.release/.claude/scripts/deploy.sh
Anything under .release/ is a release mint: leave it, and say in your report
that you left it. If a mint is what is starving, the remedy is to stop feeding
the queue — not to cut the mint.
What you may do alone, and what needs Tony
| Safe alone |
Needs Tony |
reaping a provably dead owner (gate-reaper.sh --once — ~60ms, silent when clean) |
closing his Claude sessions |
| breaking a proven deadlock by the cheap cut above |
quitting Chrome or an editor |
sweeping worktrees that are fully landed (git rev-list --count origin/main..<branch> = 0) |
killing another session's LIVE gate — one burning CPU, or one you have not paired |
raising GOVERN_QUEUE_WAIT / TSC_CACHE_WAIT for your own call |
anything in .release/ |
The rule underneath the table: something that looks idle is often someone's
live work. A 0% CPU process in a deadlock pair is proven stuck by its pairing,
not by its CPU — prove the pairing before you cut, or propose and let a human
decide.
Cannot run
Say cannot-run and stop when: the lock directory is empty or unreadable
($TMPDIR/one-govern — a different GOVERN_DIR means you are reading someone
else's map); a pid you are about to judge has already exited so its ancestry
cannot be walked; you can see a lock holder but cannot establish whether a
gate-run.sh is in its ancestry; or the number you would report came off a
gate-run.sh wall clock rather than the tool's own — an 8-fork run once showed a
20-minute wall clock that was governor queueing, not runtime.
Return { ok: false, reason: "<which, and what would make it decidable>" } and
close with warn. Never name a process as safe to kill from a reading you could
not complete — the cost of a wrong cut is someone's hour, and it has been paid.
Out of scope
- Taking the measurement — clocks, baselines, swap direction:
perf-audit.
- Deciding what a change must prove —
test-strategy.
- Deciding whether a release may ship —
release-gate.
- The governor's mechanics — locks, slots, claims,
run_bounded, the memory
arithmetic: .claude/scripts/CLAUDE.md § Machine governor, which a subagent
can Read by path even though it inherits no CLAUDE.md.
- Changing the governor. Editing
govern.sh, gate-run.sh or tsc-cached.sh
is a build task with a red-proof checker attached, not a remedy.