Alert fatigue happens when every threshold breach pages the same two engineers at 3 a.m.—until they silence the channel and miss the real outage. Fix it with three severity tiers, Alertmanager routes that match business hours, and chat delivery (Telegram or Slack) that separates wake-up pages from next-day tickets. A two-person platform team can run production safely with roughly eight to twelve alert rules, not eighty.
DarDev runs dardev-vps and client clusters with the same Prometheus → Alertmanager → Telegram pattern we recommend to Tunisian SaaS teams. Routing decides who learns about a failure and how loudly—not how many graphs you built.
Start with symptoms, not infrastructure
The fastest path to fatigue is alerting on CPU percentage while users still get 200 responses. Page on user-visible failure: HTTP 5xx rate above baseline, checkout latency p95 beyond SLO, or queue depth growing for more than ten minutes. Warn on capacity trends: disk free below twenty percent, memory pressure on the node running your only Postgres replica. Info belongs in Grafana and a daily digest—not a push notification.
- Critical — revenue or data loss risk; page immediately via Telegram or SMS
- Warning — degrading but served; Slack channel during business hours, ticket overnight
- Info — capacity planning; weekly email or dashboard annotation only
If you have not defined golden signals yet, read logs-metrics-traces first. Alert rules without latency, traffic, errors, and saturation context become arbitrary numbers. promql-deploy-health shows PromQL we attach to GitLab deploy gates—pair those queries with routes so a post-release spike pages the person who merged, not everyone.
Alertmanager routing for small teams
Prometheus evaluates rules; Alertmanager deduplicates, groups, inhibits, and delivers. For two engineers, keep the route tree shallow: one root receiver, child routes by severity label, optional routes by service label for the mailer stack versus Hesabi API. Use group_wait (thirty seconds) and group_interval (five minutes) so ten pods restarting do not produce ten messages—they produce one grouped alert with a count.
Inhibition rules stop noise at the source: if NodeNotReady fires, suppress disk alerts on that node. Mute deploy-window noise unless error rate stays elevated fifteen minutes after the job finishes.
Telegram and Slack delivery that actually gets read
Telegram fits Tunisia teams well: mobile push works on spotty mobile data, bots are free, and a dedicated ops group keeps work alerts out of family WhatsApp. Create @DarDevOpsBot (or equivalent), add Alertmanager webhook via alertmanager-bot or prometheus-bot, and restrict the group to on-call engineers plus one founder escalation contact.
Slack suits teams already in Slack for product chat. Route critical alerts to #incidents with an @oncall group—not @channel. Warnings go to #platform-feed with runbook links. Pick one primary wake-up channel; duplicating to Telegram and Slack creates ack confusion.
- Message title: severity, service, short symptom (not raw PromQL)
- Body: current value, threshold, Grafana dashboard deep link
- Footer: runbook URL—oncall-runbook-template is our starting skeleton
- Action buttons where supported: silence 2h, open dashboard
On-call rotation when you are only two people
Formal rotation beats informal whoever sees the phone. Two-person teams often run primary/secondary weekly: primary acks within fifteen minutes, secondary covers if primary is on a flight or client site. Document handoff in a shared calendar block—Friday 17:00 Africa/Tunis swap with five-minute verbal sync: open incidents, recent deploys, cert expiries.
PagerDuty can wait. A shared calendar, Alertmanager receiver labels (oncall=primary vs oncall=secondary), and Telegram mention mapping suffice until you exceed one critical alert per week. Escalate unacked critical after thirty minutes—never for disk-at-seventy-percent warnings.
Tactics that reduce fatigue without hiding outages
- Review fired alerts monthly—delete rules that never caused action
- Require for: duration on flaky checks (disk full for 10m, not 1m)
- Use runbook-rollback-bad-deploy links in deploy-related alerts
- Cap total critical rules at twelve until team grows past five engineers
- Track mean time to ack and mean time to silence—rising silence rate means fatigue
Rising silence rate means fatigue—fix thresholds or delete the rule. Burn-in new rules in warning-only mode for one week before promoting to critical. Trim kube-prometheus defaults; single-node k3s does not need etcd pages.
Common mistakes
- Paging on container restart count during normal rolling deploys
- Identical routes for production and staging
- No grouping—thundering herd when one upstream fails
- Runbooks that say check logs with no Loki or dashboard link
- Founder on every route because nobody trusted severity labels
How DarDev helps
DarDev Services observability engagements include Alertmanager review: severity labels, inhibition, chat receivers, and a rule budget matched to headcount. Request a scoped assessment at dardev.net/products.
How many alert rules should a two-person team run?
Aim for eight to twelve critical and warning rules total across services. More than twenty usually means unfired noise or missing inhibition—not better coverage.
Telegram or Slack for on-call in Tunisia?
Telegram for critical mobile pages when email reliability is uncertain. Slack if the team already lives there for product work. Use one primary wake-up channel, not both.
Do we need PagerDuty?
Not until weekly critical volume and escalation policy outgrow a spreadsheet plus Alertmanager receivers. Many SME teams run months on Telegram and a shared calendar.
How do we stop deploy-related alert storms?
Group alerts, add short group_wait, use deploy annotations in Grafana, and mute non-error alerts during the deploy window unless elevated error rate persists fifteen minutes after.
DarDev setup help?
Observability solution via DarDev Services—Prometheus rules, Alertmanager routes, chat delivery, and runbook templates sized to your VPS or cluster.
