Incident simulator

Generic scenarios only — ack the pager, pick runbook-style steps, see your score.

INCOMING PAGE

Dashboard

SEV2 · public RPC p95 latency 4.2s (threshold 800ms) · 12 min

Probe OK rate 94% · max_lag 4200ms · burn 8× (6h window)

First triage move? Check probe logs and upstream status Restart the worker blindly Disable all alerts

Upstream looks degraded. Next? Verify with manual JSON-RPC off-path Page the whole company Wait until morning

Confirmed provider-side slowness. Note incident and watch error budget Delete historical metrics

SEV2 · probe host /var 96% used · writes failing

Last successful probe 4m ago · cron queue backing up

Immediate check? Find largest log or temp directories Format the volume

Old artifacts eating space. Prune safely and document retention Turn off probes permanently

Probes recovering. Confirm cron caught up and close incident Skip post-incident review

SEV3 · edge cert expires in 36h

Health checks green · cert probe yellow

Best next step? Confirm auto-renew path and expiry clock Disable HTTPS

Auto-renew stuck. Manual renew and fix ACME challenge Wait for users to report

New cert live. Verify chain externally and resolve Close without checking

SEV1 · 5xx rate 40% after release v2.8.4

Error budget burn 20× · rollback available

You just acked. First action? Compare deploy time with error spike Scale to zero

Deploy correlation confirmed. Rollback to last known good Forward-fix without plan

Errors dropping after rollback. Hold window and schedule postmortem Redeploy same build

Customer comms? Brief status note if user-facing SLO hit Stay silent

SEV2 · status CNAME points to retired host

External synthetics failing · edge 530 errors

Validate impact? dig from outside your network Flush laptop DNS only

Wrong CNAME in zone. Fix record and wait for TTL Rename the product

Propagation clearing. Re-run synthetics and resolve Resolve after one local ping

SEV2 · API DB pool wait 30s · timeouts climbing

Connections at max · CPU normal

Initial hypothesis? Look for leaks or slow queries Reboot database immediately

Stuck transactions found. Kill long runners and patch leak Raise pool max without limit

Pool healthy again. Alert on saturation and resolve Disable DB metrics