deveshThe problem
On-call was burning people out. Engineers were paged dozens of times a week, mostly for alerts that were stale, duplicated, or purely informational. Real incidents got the same response as noise, postmortems assigned blame instead of producing fixes, and the rota was becoming a reason people quit.
The approach
Alert audit. Every alert was inventoried and forced to justify itself against one rule: if this fires at 3 a.m., is there an action a human must take right now? Alerts that failed became tickets, dashboard panels, or were deleted. The pager was reserved for user-impacting, actionable events tied to SLO burn rates.
Runbooks attached to every page. No alert ships without a runbook link: what this means, how to confirm, first three remediation steps, when to escalate. The 3 a.m. engineer starts from a checklist, not a blank terminal.
PagerDuty structured properly. Clear escalation policies, service ownership mapped to teams, severity levels with defined response expectations, and a schedule with humane handoffs.
Blameless postmortems that produce change. A lightweight template focused on contributing factors and concrete follow-ups — each with an owner and a deadline tracked to completion. Postmortems stopped being documents nobody reads and became the backlog of reliability work.
The outcome
- Pages down 70%; the remaining pages are real and actionable.
- Alert fatigue eliminated — measured not just in numbers but in engineers
volunteering back onto the rota.
- Median acknowledgement-to-action time dropped because every page arrives
with its runbook.
- The postmortem follow-up completion rate became a tracked metric, turning
incidents into a flywheel of fixes instead of a cycle of repeats.