← Skills

test-strategy

test-strategy

When an agent or user needs to decide what a change must prove, which test lane to run, or whether an existing check is real. Also use when the user says "what should I test," "which tests do I run," "is this covered," "why is the suite slow," "did the tests actually run," "add a test for this," or "this passed but I don't trust it." Use this whenever the question is about proof rather than code. For measuring runtime, see perf-audit. For reading a diff, see diff-review. For whether the proof is enough to ship, see release-gate.

Test Strategy

You decide what a change must prove and whether the proof can fail. A green
suite that cannot go red is theatre.

The loop, in order

  1. Write the red-first test. For a fix, name the test that goes RED on the
    old code and GREEN on the new. State that order explicitly. A test added
    after a fix that passes on both is documentation, not a gate.

  2. Pick the lane and say which. The suite is ~1135 files and ~87s.

    Lane Runs When
    verify:fast sdk build + tsc --noEmit + vitest related <changed> + pinned suites Every edit
    verify (FULL) The whole suite + ratchets + honesty checks Review gate, /close, ./deploy

    Take FULL, never fast, when: it is the last cycle of a plan · the change
    touched schema/, packages/sdk/, or auth/authority code · a file was
    renamed or deleted · a fast pass just went red and you are confirming the
    fix. A fast pass is never reported as a full pass.

  3. Keep the fast lane's two invariants. Any change to it must preserve
    both, or the lane becomes a liar:

    • An empty diff falls back to the FULL suite — never to a pass. A lane
      that goes green by selecting zero tests is worse than the slow one.
    • The pinned suites always run. vitest related walks the import graph;
      config, parity, and boundary gates import nothing from what they guard —
      one reads a .toml.

  4. Apply the docblock rule. A test touching the DOM carries
    // @vitest-environment jsdom on line 1. environment defaults to node
    because only ~230 of ~1135 files need a DOM. Forget it and you get
    ReferenceError: document is not defined immediately — loud and
    self-correcting, which is why there is no central list.

  5. Prove the checker can fail. Break the thing the gate guards and confirm
    RED before trusting it. A checker that stays green against gutted code has
    proven nothing about the code.

Hard rules

  • Never reach for environmentMatchGlobs — vitest 4 removed it and ignores
    it silently, so the config looks right and does nothing.
  • Never mock TypeDB in integration tests — use real TypeDB or skip.
  • An exit code is not the gate result. Read the last line. (0 test) is a
    load failure. 144 is unrun. 141 is SIGPIPE from producer | grep -q under
    pipefail — which returns 141 when it matches, on a large producer.
  • An unrun gate is not a pass. "Not measured, box saturated" demotes the
    outcome; it never counts as green.

Cannot run

You apply that last rule to everyone else's gates. It applies to you.

Say cannot-run — never "the lane is fine" — when the diff is empty or was
never handed over, when the suite could not load ((0 test), 144, 141), or when
the box was too saturated to trust a timing. Return
{ ok: false, reason: "<which of those, and the line that says so>" } and close
with warn. A lane you could not observe is not a lane you approved.

Out of scope

  • Tuning worker counts or measuring wall clock — see perf-audit.
  • Writing the implementation the test covers.
  • Declaring a release shippable — see release-gate.