incident-response

When production is broken or degraded and someone needs the bleeding stopped before the cause is found. Also use when the user says "the site is down," "production is broken," "users are reporting," "something's wrong in prod," "did we break it," "roll it back," or "why is this failing live." Use this whenever a live surface is failing right now. For a suspected-but-not-live weakness, see security-audit. For a slow-but-working system, see perf-audit. For writing the trap up durably afterwards, see runbook-write.

Incident Response

Two things, in strict order, and never blurred: stop the bleeding, then
find the cause. Debugging a live outage while it is still bleeding is how a
twenty-minute incident becomes a four-hour one.

The loop, in order

Establish blast radius before touching anything. Which surface, which
workspace, how many actors, and is money moving. A read-path outage and
a write-path outage that silently drops signals are not the same emergency —
the second is worse precisely because it looks quieter.
Reproduce in the real browser. chrome.mjs --send --network before curl.
curl omits cookies, Referer and body; an auth-gated or funnel surface is a
browser-shaped problem, and a curl that "works" has told you nothing about
what the user hit.
Stop the bleeding with the smallest reversible move. Roll back the
deployment, flip the flag, drain the queue. A rollback you can undo beats a
fix you feel confident about. Record the time and the move.
Only then find the cause. In this order, because each is cheap and each
has been the answer before:
- Read the deployment sha against git log — the worker runs what it was
  deployed with, and the "fix everyone remembers" may never have shipped.
- Check the caches. A 300s KV TTL makes a correct change look broken for
  five minutes; a rollback inside that window fixes nothing and destroys the
  evidence.
- Check whether the handler is even armed — a cron whose expression is
  missing from [triggers] crons is dark however correct the code is.
Close with a trap, not a summary. The output that matters is the sentence
that prevents recurrence: what was believed, what was true, and how to tell
the difference next time. Hand it to the technical writer (runbook-write).

Hard rules

Never infer liveness from ps. Read a published liveness file, a health
endpoint, or say you do not know. A grep over a process list is not evidence
a service is serving.
An exit code is not the gate result. Read the last line. (0 test) is a
load failure, 144 is unrun, 141 is SIGPIPE.
Destructive moves need explicit authorisation. Rolling back a deployment
is the job. Deleting data, mutating shared state, or touching another team's
surface is a stop-and-ask — at 3am, and when it looks obvious.
Report faithfully while it is still ugly. A status hedged to sound calmer
costs the next responder their bearings. Say what you do not know, and say
what you are doing to find out.

Output shape

severity        sev1 | sev2 | sev3
blast_radius    surface · workspaces · actors · money moving?
timeline        [{ t, move }]        — every action, timestamped
mitigation      what stopped it, and whether it is reversible
root_cause      null while genuinely unknown — an honest null beats a guess
trap            the sentence that prevents the recurrence

Out of scope

Shipping the permanent fix — mitigate now, fix through the normal gates.
Assigning blame. The trap is the deliverable; the person is not.