Our build engine ranks its own backlog. Every finished change gets scored on an eight-question rubric — security, simplicity, structure, speed, integration, reuse, test coverage, UX — and the score feeds a priority queue that decides what the engine builds next. It’s a real feedback loop: closed work moves a weight, the weight moves the ranking, the ranking moves the next pick.
It’s also a closed loop in the bad sense. The rubric is generated by the same system whose work it’s grading. We checked the trend this week: every one of the eight dimensions sitting flat between 0.85 and 1.00, for weeks. That’s not health. That’s a grader that ran out of ways to surprise itself.
The fix sounds obvious once you say it: let production grade the homework too. The harder part, and the part worth writing up, is that the obvious fix has a failure mode that’s nearly undetectable, and we came within one skipped step of shipping it broken.
What “production graded” actually means here
We already had a local JSON mirror (task-paths.json) that our ranking script (do-rank.py) reads: closed cycles from learnings.md, plus anything an operator marks by hand with --mark. What we didn’t have was a path from production into that file. Three real signals were sitting in our D1 database, unread:
- A settled promise. Every feature we ship carries a machine-checkable proof; when it’s re-run and passes, we
mark()apromise:<slug>→proofedge with a real strength. Direct, clean, already slug-keyed. - A
/dobuild cycle that actually closed in a live workspace, not just in our own commit log, recorded by the production event handler as a tagged row (do:learn+slug:<x>) the moment it happened. - One actor grading another actor’s delivered work: a
world:outcomesignal that, if anyone called it with a task-slug tag, would land as a scoreable edge. Nobody does yet. We shipped the mapping anyway and said so in the docs, because a real, unused mechanism beats a fabricated one.
The promise’s own draft language lumped in a fourth source — the router’s shape→model learning edges — as if they were slug-scoped too. They aren’t; we checked the actual write path (shape:<x> → model:<y>, no slug in either id) before wiring anything, and dropped that source rather than fake a join that doesn’t exist in the data.
The design call that mattered: no network
The pulling script (outcome-pull.ts) reads the local D1 mirror — the one our existing session-start sync already refreshes from production — never the network directly. That was deliberate: our ranking script has been zero-network since day one, and a puller that reaches out over HTTP on every session start is one flaky connection away from adding latency (or a hang) to something that used to be instant. Reading a database that was already being synced for other reasons cost us nothing new.
The merge rule is the whole safety property of the feature: production data lands in a prod sub-object, full overwrite on every pull, and the local fields (strength, resistance, closed, everything --mark and cycle-closes already own) are never touched. Two pulls of the same snapshot are byte-identical. A local mark can never be clobbered by a production sync, and vice versa.
Where it almost went wrong
The puller is best-effort by design: if the local database isn’t reachable, it logs a line and exits 0 rather than blocking a session start over a missing cache. Sensible. Also, if you’re not careful, a booby trap.
A genuinely broken query — wrong working directory, a malformed result shape, a CLI flag that changed — hits the exact same catch block as “offline.” Both print a warning, both exit 0, both write nothing. Every offline fixture test we wrote — and we wrote nineteen of them, all passing — never once touched the real database call. A reviewer walking through the diff caught it before I did: we had proven the feature’s logic was correct and never once proven the feature actually ran.
So we ran it for real. Spun up local D1, applied migrations, inserted a synthetic settled-promise row and a synthetic cycle-close row by hand, executed the puller with no flags, and read the output file back:
"task:smoke-test": {
"prod": {
"strength": 4.5,
"sources": ["do:learn", "promise:proof"],
"traversals": 2
}
}
4.5 is 4 (the promise mark) plus 0.5 (the cycle-close signal), summed correctly, provenance-tagged correctly, written to disk correctly. Then we copied a real snapshot of our production ranking data into the test, injected a prod block on one entry, and confirmed the ranked-priority view actually re-sorted around it — not just that the math was right in isolation, but that a real row with real siblings rendered without breaking anything nearby.
That live pass also caught something the fixtures physically could not: our TypeScript writer sorted only the top-level JSON keys before saving, while the Python reader sorts every nested level. Same data, two different byte orders, a spurious diff on every single sync from here to eternity if we’d shipped it. One-line fix, invisible without a real round trip through both writers.
The number that matters more than 19/19
Nineteen passing fixture assertions felt like proof. It was proof of the mapping logic (real proof, worth having) and zero evidence the wiring between “read the database” and “the mapping logic” actually held. For a feature whose entire job is to notice when something silently fails, shipping it on the strength of tests that could never observe its own silent failure would have been a little on the nose.
The ranking script’s self-check is at 45 out of 45 assertions now, up from 41, and the two new ones are load-bearing rather than decorative: a synthetic slug with a production mark ranks strictly above an identical slug with a production warning, checked by actual comparison, not a string match on the word “pass.” If you’re building anything that grades itself and then acts on the grade, that’s the bar — prove the number moves the decision, not just that the number exists.