DevOps Incident Response Runbook: From Alert to Resolution in 15 Minutes

Most teams don't have an incident response problem — they have a preparation problem. This runbook documents the exact framework we use to achieve p95 MTTR < 15 minutes for Sev-1/Sev-2 incidents across Kubernetes, AWS, and multi-cloud environments.

Philosophy: Principles Over Procedures

"We don't rise to the level of our expectations; we fall to the level of our training." — Archilochus, adapted for SRE

Every procedure in this runbook is built on four non-negotiable principles:

1. SLO-First: Alert on user-impacting symptoms (error rate, latency, availability), not internal metrics (CPU, memory).
2. Automation-First: If a human runs it twice, it becomes a script. If a script runs it twice, it becomes a controller.
3. Blameless by Default: Postmortems analyze systems, not people. "Who" is never in the template; "what" and "why" are.
4. Documentation as Code: Runbooks live in Git, versioned with infrastructure, tested in CI.

Phase 0: Pre-Incident Preparation (Do This Now)

0.1 Define Your SLOs & Error Budgets

Before writing runbooks, define what "healthy" means per service:

Error budget burn rate triggers alerting tiers:

# PrometheusRule example
groups:
- name: slo-burn-rate
  rules:
  - alert: HighErrorBudgetBurn
    expr: |
      (job:slo_errors_per_request:ratio_rate5m / job:slo_errors_per_request:ratio_rate1h) > 14.4
    for: 2m
    labels:
      severity: critical
      team: platform
    annotations:
      summary: "Error budget burning at 14.4x normal rate"
      runbook_url: "https://runbooks.company.com/slo-burn"

0.2 Service Catalog with Runbook Links

Every service in Backstage/GitOps repo has:

• Owner team + on-call rotation (PagerDuty/Opsgenie schedule)
• Dependency graph (upstream/downstream services)
• Runbook URL: https://runbooks.company.com/{service-name}
• Dashboard URL: https://grafana.company.com/d/{service-name}
• Recent deployments (ArgoCD/Flux sync status)

0.3 War Room Infrastructure (Pre-Provisioned)

• Slack: #incident-{incident-id} channel auto-created via bot
• Zoom: Permanent meeting link in runbook header
• Notion/Confluence: Incident template pre-populated
• StatusPage: One-click public/private status update

Phase 1: Detection & Triage (0-2 Minutes)

1.1 Alert Routing Rules

1.2 First Responder Checklist (First 2 Minutes)

☐ Acknowledge alert in PagerDuty/Opsgenie
☐ Open runbook URL from alert annotation
☐ Join war room Zoom (link in runbook)
☐ Post in #incident-{id}: "🔴 IC: {name} acknowledged. Investigating {service}."
☐ Check dashboard: Is SLO actually breached? (False positive rate target: < 5%)
☐ Check recent deployments: ArgoCD/Flux diff, GitHub releases
☐ Check dependency health: Upstream/downstream dashboards

1.3 Triage Decision Tree

                    ALERT FIRED
                         │
                         ▼
              ┌─────────────────────┐
              │ SLO breached NOW?   │──No──▶ Resolve alert, tune threshold
              └─────────┬───────────┘
                        │ Yes
                        ▼
              ┌─────────────────────┐
              │ Known runbook exists?│──No──▶ Sev-1, page TL, start generic runbook
              └─────────┬───────────┘
                        │ Yes
                        ▼
              ┌─────────────────────┐
              │ Can IC execute solo? │──No──▶ Escalate to TL, form war room
              └─────────┬───────────┘
                        │ Yes
                        ▼
              ┌─────────────────────┐
              │ Execute runbook steps│
              └─────────────────────┘

Phase 2: Investigation & Mitigation (2-10 Minutes)

2.1 Standard Investigation Commands (Kubernetes)

# Pod health
kubectl get pods -n {namespace} -l app={service} -o wide
kubectl describe pod {pod} -n {namespace}
kubectl logs {pod} -n {namespace} --tail=100 --since=10m

# Resource pressure
kubectl top pods -n {namespace} --sort-by=memory
kubectl top nodes
kubectl describe node {node} | grep -A 10 "Allocated resources"

# Network / DNS
kubectl exec -it {pod} -n {namespace} -- nslookup {dependency}
kubectl exec -it {pod} -n {namespace} -- curl -v {dependency}:{port}/health

# Recent events
kubectl get events -n {namespace} --sort-by='.lastTimestamp' | tail -20

# HPA / Scaling
kubectl get hpa -n {namespace}
kubectl describe hpa {service} -n {namespace}

2.2 Common Mitigation Patterns (Runbook Snippets)

2.3 Communication During Incident

• Every 5 min: IC posts update in Slack: "🔍 Investigating {hypothesis}. Current status: {mitigation attempted}. Next check: {time}"
• Every 15 min: TL posts stakeholder summary in #incident-leadership
• StatusPage: Update within 10 min of Sev-1 declaration
• No speculation: Only share confirmed facts. "We're investigating" > "It might be X"

Phase 3: Resolution & Recovery (10-30 Minutes)

3.1 Confirm Fix & Monitor

# Verify fix
watch -n 5 'curl -s {service}/health | jq .status'
kubectl get pods -n {ns} -l app={svc} -w

# Confirm SLO recovery
# Wait for 2 full error budget windows (e.g., 10 min for