Best Practice: Blameless Postmortems and Continuous Learning

# Postmortem Policy

## When is a postmortem mandatory?

**Trigger criteria (all require a postmortem):**
- Incidents with user impact > 15 minutes (SEV-1 or SEV-2)
- SLO violation (Error Budget Burn Rate > threshold)
- Data loss of any extent
- Security incidents with production impact
- Deployments that required manual rollback

**Timeline:**
- Postmortem meeting: within 48h of incident resolution
- Postmortem document: completed and shared within 5 business days
- Action items: owner and due date assigned within the meeting

# Postmortem: Payment Service Outage

**Incident ID:** INC-2025-034
**Date:** 2025-03-15
**Severity:** SEV-1
**Duration:** 47 minutes (14:23 – 15:10 UTC)
**Written by:** @payment-engineer-lead
**Reviewed by:** @platform-lead, @cto
**Status:** FINAL

---

## Impact

- **Users affected:** ~12,000 users were unable to make payments
- **Transactions failed:** ~340 payment attempts with HTTP 500
- **Revenue impact:** ~€8,500 (estimate based on average transaction value)
- **SLO impact:** Availability SLO violated, 2.3% of monthly error budget consumed

---

## Timeline

| Time (UTC) | Event |
|-----------|---------|
| 14:23 | Deployment of version 2.4.1 to production completed |
| 14:25 | CloudWatch Alarm `payment-service-5xx-error-rate` triggered |
| 14:27 | On-call engineer (Alex) paged, starts diagnosis |
| 14:31 | Correlation to deployment established via logs |
| 14:35 | Rollback decision made (Alex + Maria) |
| 14:38 | Canary rollback initiated |
| 14:45 | Error rate drops to normal |
| 15:10 | Full recovery confirmed, incident resolved |
| 15:15 | Incident communication to stakeholders |

---

## Root Cause

The new version 2.4.1 contained a database query without an index on the `payment_methods`
table. Under production load (>100 concurrent requests), this query caused
full table scans that exhausted database connections.

**Root Cause:** Missing database index (`payment_methods.user_id`)

---

## Contributing Factors

1. **No performance test in staging:** Staging environment had < 1% of production volume
   – the performance difference was not visible.
2. **No slow query warning:** AWS Performance Insights was not configured.
3. **No rollback during canary:** Canary was at 20% traffic (not 5%) – too much for a new,
   untested deployment step.

---

## What Worked Well

- Alerting detected the problem within 2 minutes of deployment
- Runbook was up to date and correct – rollback steps were executed without senior engineer assistance
- Canary deployment limited the impact to 20% instead of 100% of users
- Communication to stakeholders happened within 30 minutes

---

## Action Items

| # | Action | Owner | Due Date | Jira |
|---|--------|-------|----------|------|
| 1 | Add index on `payment_methods.user_id` | @db-engineer | 2025-03-18 | PAY-456 |
| 2 | Enable AWS Performance Insights (all RDS instances) | @platform-team | 2025-03-22 | PLAT-789 |
| 3 | Reduce canary start percentage to 5% | @payment-lead | 2025-03-20 | PAY-457 |
| 4 | Performance test with production data volume in staging | @qa-team | 2025-04-01 | QA-234 |
| 5 | Update runbook for "failed deployment" | @payment-engineer-lead | 2025-03-22 | PAY-458 |

---

## Notes on Blameless Culture

This incident was not caused by a mistake, but by:
- Insufficient test infrastructure (no performance test)
- Missing monitoring configuration (Performance Insights)
- Unclear canary percentage policy

The engineers involved acted correctly within the system available to them.
The action items address the system, not the people.

// Jira Automation: automatically create action items from postmortem
// Trigger: "POSTMORTEM-FINAL" label added to issue
// Action: create child issues for each row in the action items table

{
  "trigger": {
    "type": "ISSUE_UPDATED",
    "conditions": [
      {"type": "LABEL_ADDED", "value": "POSTMORTEM-FINAL"}
    ]
  },
  "actions": [
    {
      "type": "SEND_SLACK_MESSAGE",
      "channel": "#postmortems",
      "message": "Postmortem {{issue.key}} published. Action items: {{issue.customfield.actionItems.count}}. Please review and prioritize."
    }
  ]
}

# Incident Trend Report – February 2025

## Overview

| Metric | Feb 2025 | Jan 2025 | Trend |
|--------|----------|----------|-------|
| SEV-1 Incidents | 2 | 4 | ↓ 50% |
| SEV-2 Incidents | 5 | 6 | ↓ 17% |
| MTTR Average | 38 min | 55 min | ↓ 31% |
| Action items created | 11 | 16 | ↓ |
| Action items completed | 9/11 (82%) | 8/16 (50%) | ↑ |

## Top Incident Categories

1. **Database issues** – 3 incidents (new: missing indexes, connection pool)
2. **Deployment errors** – 2 incidents (both mitigated by canary)
3. **External API outages** – 2 incidents (payment provider)

## Recommendations

- Integrate database performance testing into CI/CD (addresses top category)
- Implement circuit breaker for external APIs (addresses category 3)