Runbook
How this site is operated. Detailed steps with account access live in the private repo docs.
SLO policy
- 99.5% rolling 30-day availability per probed RPC (216-minute error budget).
- Pipeline: 99% of expected probe runs in 24 hours (every 2 minutes).
- Burn alerts: 1h rate ≥ 14.4× (SEV2); 6h rate ≥ 6× (SEV3).
- SEV1 pages: major board outage, consecutive hard downs, or health check stale.
- SEV2 pages: open degraded/burn incidents affecting the board narrative.
- SEV3 tickets: slow burn and handoff items unless escalated.
Board stale
SEV1 When /api/health returns 503, probe or daily rollup metadata is too old. Static pages still load; live JSON may be stale.
- Check status and
/api/healthJSON. - Look for cron or Worker errors in Cloudflare observability.
- Wait one probe interval (2 min) after upstream recovery.
Endpoint permanently dead
SEV2 One public RPC stays down after provider outage.
- Verify the URL off-site (manual JSON-RPC).
- Disable or replace the endpoint in upstream config, re-verify, redeploy.
- Board updates on the next successful probe.
D1 or request quota pressure
SEV3 Free-tier D1 and Worker request limits are modeled in code and tested in CI.
- Daily rollup prunes old hourly and incident rows.
- If quota email arrives: reduce probe fan-out (fewer endpoints), never add extra crons.
- Request exhaustion: static site may work while API returns errors — see banner on home.
Cron never runs
Two schedules only: every 2 minutes and 00:07 UTC daily. If metadata never updates, cron triggers or deployment is wrong — redeploy after fixing config.
Budget at a glance
8 keyless RPC probes · 720 probe runs/day · 2 cron slots · $0 Cloudflare free tier target.