Incident Response

Two things, in strict order, and never blurred: stop the bleeding, then
find the cause. Debugging a live outage while it is still bleeding is how a
twenty-minute incident becomes a four-hour one.

The loop, in order

1. Establish blast radius before touching anything. Which surface, which
   workspace, how many actors, and is money moving. A read-path outage and
   a write-path outage that silently drops signals are not the same emergency —
   the second is worse precisely because it looks quieter.

2. Reproduce in the real browser. chrome.mjs --send --network before curl.
   curl omits cookies, Referer and body; an auth-gated or funnel surface is a
   browser-shaped problem, and a curl that "works" has told you nothing about
   what the user hit.

3. Stop the bleeding with the smallest reversible move. Roll back the
   deployment, flip the flag, drain the queue. A rollback you can undo beats a
   fix you feel confident about. Record the time and the move.

4. Only then find the cause. In this order, because each is cheap and each
   has been the answer before:

   • Read the deployment sha against git log — the worker runs what it was
     deployed with, and the "fix everyone remembers" may never have shipped.
   • Check the caches. A 300s KV TTL makes a correct change look broken for
     five minutes; a rollback inside that window fixes nothing and destroys the
     evidence.
   • Check whether the handler is even armed — a cron whose expression is
     missing from [triggers] crons is dark however correct the code is.

5. Close with a trap, not a summary. The output that matters is the sentence
   that prevents recurrence: what was believed, what was true, and how to tell
   the difference next time. Hand it to the technical writer (runbook-write).

Hard rules

• Never infer liveness from ps. Read a published liveness file, a health
  endpoint, or say you do not know. A grep over a process list is not evidence
  a service is serving.
• An exit code is not the gate result. Read the last line. (0 test) is a
  load failure, 144 is unrun, 141 is SIGPIPE.
• Destructive moves need explicit authorisation. Rolling back a deployment
  is the job. Deleting data, mutating shared state, or touching another team's
  surface is a stop-and-ask — at 3am, and when it looks obvious.
• Report faithfully while it is still ugly. A status hedged to sound calmer
  costs the next responder their bearings. Say what you do not know, and say
  what you are doing to find out.

Output shape

severity        sev1 | sev2 | sev3
blast_radius    surface · workspaces · actors · money moving?
timeline        [{ t, move }]        — every action, timestamped
mitigation      what stopped it, and whether it is reversible
root_cause      null while genuinely unknown — an honest null beats a guess
trap            the sentence that prevents the recurrence

Out of scope

• Shipping the permanent fix — mitigate now, fix through the normal gates.
• Assigning blame. The trap is the deliverable; the person is not.