- 01Runbooks fail at the moment they are needed most because they are written for the author, not the 3 a.m. on-call. The structure must be optimized for the reader under stress.
- 02Seven blocks, in order: Trigger, Decide-this-is-it, Contain, Investigate, Decide-escalate, Recover, Close. Anything else is supplementary.
- 03Cut all background, theory, and 'why this exists' content from the runbook itself. Link to it in a sidebar; it does not belong in the execution path.
- 04Untested runbooks are documentation, not runbooks. Annual tabletop test minimum; quarterly is better; chaos-day exercises best.
- 05The single best diagnostic for a runbook's quality: time how long it takes a new on-call to complete the runbook against a synthetic incident. If > 25% over expert time, the runbook needs work.
A runbook that requires the reader to context-switch through three different documentation systems at 3 a.m. is not a runbook. The structure has to survive sleep deprivation, partial information, and the specific anxiety of not knowing whether the current situation matches the playbook.
This piece is the runbook-writing standard we hand to clients during SOC reorganizations. It is short on theory and long on structure, because the only thing that matters is whether the on-call can execute it cold.
Why most runbooks fail.
Most runbooks are written by an expert who lived through one incident, intending to capture what they did. The result reads beautifully to the author and unhelpfully to anyone else. The fix is to write the runbook for a competent on-call who has never seen this incident type before — a definition that includes the author six months from now.
The seven-block structure
Every block has one job and ends before it strays. The on-call should be able to scroll-stop on any block and know what they are doing.
/RUNBOOK · BLOCK_STRUCTURE
| # | Block | One-sentence purpose |
|---|---|---|
| 1 | Trigger | What specific signal invokes this runbook |
| 2 | Decide (is this it?) | Quick yes/no checks: do I have the right runbook? |
| 3 | Contain | First actions to limit damage, in strict order |
| 4 | Investigate | Queries with expected output; build the picture |
| 5 | Decide (escalate?) | Named criteria + named contacts; not 'page someone' |
| 6 | Recover | Restoration steps with verification |
| 7 | Close | Postmortem trigger + required artifacts |
A worked example — phishing-derived credential compromise
Below is a complete runbook in the seven-block format. Notice how each block is short, executable, and assumes the reader is competent but tired.
/RUNBOOK · phishing-derived credential compromise
# 1 · TRIGGER
# ITDR alert: "Stolen-session token replay (geo-mismatch)"
# OR
# IdP alert: "Successful sign-in from unknown ASN within 1h of MFA fatigue"
# OR
# User self-report through helpdesk → tagged 'phish-creds'
# 2 · DECIDE — is this the right runbook?
# YES if all of:
# ✓ Authenticated session in IdP / SaaS within last 24h
# ✓ Source IP / fingerprint mismatch with prior 30d
# ✓ NO laptop loss or theft reported (different runbook)
# 3 · CONTAIN (do these in order, do NOT skip)
# a. IdP: revoke all sessions for user_id (UI / 'kill sessions')
# b. IdP: require step-up + factor re-enroll on next sign-in
# c. PAM: invalidate any cached creds for that user
# d. SaaS: invoke 'force re-auth' on Salesforce, M365, GitHub
# e. Email: pull-back any sent mail in last 4h via journaling rule
# 4 · INVESTIGATE
# Q1: SIEM > sourcetype=okta event=user.session.start user=$USER
# earliest=-72h | stats count by ip,asn,fingerprint
# expected: 1 ASN historically; 1 NEW ASN in last 24h
# Q2: SIEM > sourcetype=ms365 user=$USER OperationName=*
# earliest=-24h | stats count by Operation
# expected: read-heavy normal, watch for *Forwarding*, *Send*
# Q3: GitHub audit log: user=$USER action=oauth_authorization*
# expected: 0 new oauth grants in window
# 5 · DECIDE — escalate?
# ESCALATE TO: ir-on-call (page) + legal-on-call (page) IF any of:
# - Any 'Forwarding' rule created in mailbox in window
# - Any new OAuth app authorized in window
# - Any GitHub repo deleted, secret accessed, or release published
# - Any payment / wire-related M365 access in window
# OTHERWISE: continue.
# 6 · RECOVER
# a. Confirm user has re-enrolled phishing-resistant MFA
# b. Issue temporary device-bound credentials only
# c. Re-grant session ONLY after Q1 / Q2 / Q3 are clean
# d. 7-day enhanced monitoring tag on user_id
# 7 · CLOSE
# - Postmortem REQUIRED if escalation triggered
# - Postmortem OPTIONAL if contained pre-escalation
# - Required artifacts: SIEM exports for Q1-Q3, IdP audit, screenshots
# - File ticket: phish-IOC update for security-awareness team
What to ruthlessly cut
Most runbooks bloat for the same reasons. Below is the cut list. If you find any of these in a runbook, remove them; link to a separate background doc if needed.
- 01Background ('In 2023 we observed an increase in...')
- 02Theory ('Phishing-derived credential compromise typically follows the kill chain...')
- 03Justification ('We respond this way because the EDPB recommendation...')
- 04Author voice ('When I worked through a similar incident...')
- 05Anything in passive voice — execution steps must be imperative
- 06Anything that says 'consider' instead of 'do' — decide once, write it once
The 'consider' trap.
When a runbook says 'consider notifying legal,' it is asking the on-call to make a decision under stress. Either the runbook should say 'page legal-on-call IF X' (with X concretely defined) or it should not mention legal at all. Decisions that the runbook author can make once, in calm, must not be deferred to an on-call at 3 a.m.
Test cadence — the only honest measure of runbook quality
An untested runbook is documentation. The test cadence below is what we recommend by runbook severity. Test failures are not embarrassing; they are the entire point of testing.
/RUNBOOK · TEST_CADENCE
| Runbook severity | Minimum cadence | Test type |
|---|---|---|
| Critical (data exfil, ransomware) | Quarterly | Tabletop + chaos-day execution |
| High (creds, internal lateral) | Semi-annual | Tabletop |
| Medium (malware containment) | Annual | Tabletop |
| Low (lost device, policy violation) | Annual | Walk-through review |
Three quality KPIs for a runbook program
/RUNBOOK_PROGRAM_KPIS
| KPI | Target | What it proves |
|---|---|---|
| New-on-call-vs-expert time ratio | ≤ 1.25 | Runbook is reader-optimized |
| Test pass rate (no edits required) | ≥ 80% per cycle | Runbooks are kept current |
| Median age since last test | ≤ 12 months | Cadence discipline is real |
A 30-day program
- 01Days 0–7 — Inventory all current runbooks. Tag by severity. Identify the top 10 most-invoked. Schedule their tests.
- 02Days 7–21 — Re-write the top 10 in the seven-block format. Cut bloat. Add expected outputs to every query.
- 03Days 21–30 — Run tabletops with on-calls who did not write the runbooks. Capture every place they paused, asked a question, or guessed. Fix those places before publishing.
Monday morning — the test that proves your runbook program is real.
Pick any runbook in your inventory at random. Hand it to an on-call who has not used it before. Time them through a synthetic invocation. If they finish within 25% of expert time without asking for help, the runbook is good. If not, you have your week's work — and a representative sample of how the rest of the inventory probably looks.
- ▸Random-sample one runbook from your inventory
- ▸Hand it cold to a new on-call against a synthetic incident
- ▸Time them; tally pauses and questions
- ▸Each pause = a future fix; each question = a future cut
Closing — runbooks are written for the worst night of the year
The on-call who picks up your runbook at 3 a.m. is competent, exhausted, and uncertain. The runbook's only job is to make their next 30 minutes safer than improvisation would have been. Seven blocks, no theory, no passive voice, queries with expected outputs, escalation criteria with named names, tested at a known cadence. That is the standard. Hold the line, and the worst night of the year becomes the night the runbook quietly did its job.