An on-call runbook is a single page that tells whoever answers the page what to check first, who to wake next, and how to roll back before customers pile into support. For a two-to-five person team running Docker Compose on an OVH VPS or a small k3s cluster, it replaces tribal knowledge in one engineer's head with links to Grafana dashboards, kubectl commands, and escalation phone numbers. You do not need a 40-page SRE handbook—a structured template updated after every incident is enough to cut mean time to recovery from hours to minutes.
DarDev maintains runbooks for dardev-vps and client stacks in Tunis. This template aligns with alert-fatigue-routing severities, logs-metrics-traces dashboards, and runbook-rollback-bad-deploy commands—store it in git beside your manifests.
What belongs in a one-page runbook
Every runbook answers five questions at 2 a.m.: Is this real? How bad? What do I click? Who else wakes up? When do I roll back? Use bullets and deep links—not essays. One page per user-visible service (api, worker, edge, Postgres) plus a platform overview for SSH and backup restore.
- Service owner and backup contact with Africa/Tunis timezone and phone or Telegram handle
- Severity table: critical pages immediately, warning next business day, info dashboard only
- Symptom-based triage: if 5xx rate high → open this Grafana panel; if disk full → this Loki query
- Rollback procedure with copy-paste commands and expected recovery time
- Escalation ladder: primary ack in 15 min, secondary at 30 min, founder or vendor at 60 min
- Post-incident: link to incident-postmortem-culture one-pager and where to file action items
Runbook template sections (copy this outline)
- Header block
Service name, production URL, on-call rotation calendar link, last reviewed date, and runbook maintainer. Stale dates signal neglect—review monthly.
- Architecture sketch
One diagram or bullet list: ingress → app containers → Postgres → Redis. Name the VPS hostname or cluster context. New hires should orient in sixty seconds.
- Health checks
Public /health URL, internal metrics endpoint, and smoke-test script path. Note expected response codes and latency baseline from grafana-dashboards-small-teams.
- Alert playbook
For each critical Alertmanager rule: symptom, dashboard deep link, first command to run, rollback trigger, and comms template for status channel.
- Dependencies and vendors
DNS provider, TLS cert renewal, object storage, payment gateway status page. External outages look like internal bugs without this list.
- Access and secrets
Where SSH keys live, which vault or 1Password folder holds production creds, and break-glass procedure if primary engineer is unreachable.
Link every critical alert to a runbook anchor
Add a runbook URL footer in Alertmanager templates—one anchor per critical rule. Empty runbooks cause silences, not fixes. Backfill your three noisiest alerts first.
Paste the same PromQL from promql-deploy-health into both the alert rule and runbook so on-call does not hunt repos mid-outage.
Triage flow for self-hosted stacks
On one or two nodes, check host health first—node_exporter disk and memory—then the failing container. Compose: docker compose ps and logs --tail=100. Kubernetes: kubectl get pods -A and describe the crash loop. Error rate spiked after deploy? Roll back before log diving.
- Ack the page in Telegram or Slack within fifteen minutes—silence without ack hides team burnout
- Open the service overview dashboard; confirm user-visible symptoms match the alert
- Check recent deploys, cert expiry, and upstream vendor status in parallel
- Execute the runbook remediation or rollback section
- Post a one-line status update and schedule post-mortem if users were impacted
Maintenance cadence that keeps runbooks honest
Runbooks rot faster than code. Monthly: primary on-call clicks every dashboard link in the runbook and fixes broken URLs. After each incident, update the runbook before post-mortem ends—novel fixes become bullets, not Slack threads.
Quarterly game day: time how long a backup engineer finds the rollback section. Three repos opened first means navigation is wrong.
Common mistakes
- Runbook says check logs with no Loki URL or grep example
- One giant wiki page for twelve services—nobody scrolls at 3 a.m.
- Rollback commands for staging image tags, not production SHAs
- Escalation list with ex-employees and dead phone numbers
- Severity definitions that disagree with Alertmanager labels
- No timezone on on-call calendar—Africa/Tunis vs UTC confusion pages the wrong person
How DarDev helps
DarDev Services observability engagements deliver Prometheus rules, Alertmanager routes, Grafana dashboards, and runbook templates wired together—not dashboards nobody reads during incidents. We scope to your VPS or cluster size and hand off a git repo your team can maintain. Request an assessment at dardev.net/products.
How long should an on-call runbook be?
One to two printed pages per service. If it exceeds that, split into a platform overview plus service-specific children. On-call needs answers in under sixty seconds, not a manual.
Where should we store runbooks?
Version-controlled markdown in your infra or GitOps repo is best. Notion or Confluence work if you review links monthly. Never only in one engineer's local notes.
Do two-person teams need formal on-call rotation?
Yes—a simple primary and secondary weekly swap with a shared calendar prevents both engineers assuming the other saw the Telegram message. Formal does not mean PagerDuty on day one.
What is the first runbook section to write?
Rollback for your highest-traffic service. Most SME incidents are bad deploys; runbook-rollback-bad-deploy covers the commands—paste them into your template before writing triage prose.
How often should runbooks be updated?
After every critical incident, monthly link review, and quarterly game day. If alert-fatigue-routing changes severity labels, update the runbook severity table the same day.
