Controls (WAF-REL)

Severity: Critical | Category: Reliability Governance | Automatable: Medium

Every production workload MUST have documented SLOs (availability, latency, error rate). SLOs MUST be monitored in monitoring dashboards with alerting on error budget burn rate.

Requirement: SLO document versioned; error budget calculated; burn rate alerts configured; quarterly review evidenced.

Terraform Checks (Excerpt):

Evidence: SLO document per workload (required); monitoring dashboard (required)

Best Practice: SLO & SLA Definition →

Severity: High | Category: Health Monitoring | Automatable: High

All production services MUST expose health check endpoints and configure readiness/liveness probes. Load balancers MUST use health checks with explicit thresholds, intervals and timeouts.

Requirement: No deployment without health check; probes with measured initialDelaySeconds; health check paths validate real dependencies.

Terraform Checks (Excerpt):

Evidence: IaC with probe configuration (required); LB health check (required)

Best Practice: Health Checks & Probes →

Severity: High | Category: High Availability | Automatable: High

All production workloads MUST be distributed across at least 2 Availability Zones. Single-AZ deployments in production are not permitted without written risk acceptance.

Requirement: Min. 2 AZs for compute; Multi-AZ for all databases; LB across 2+ AZs; AZ failover tested at least quarterly.

Terraform Checks (Excerpt):

Evidence: IaC with Multi-AZ configuration (required); cloud console screenshot (required)

Best Practice: Multi-AZ & High Availability →

Severity: Critical | Category: Backup Recovery | Automatable: High

All production databases and stateful services MUST have automated backups with retention >= 7 days and PITR. Backups MUST be stored in a separate account/region. Recovery MUST be tested and documented at least quarterly.

Requirement: PITR enabled; cross-account backup; restore test with results documentation; backup failure alerts configured.

Terraform Checks (Excerpt):

Evidence: IaC backup configuration (required); restore test report (required)

Best Practice: Backup & Recovery →

Severity: High | Category: Resilience Patterns | Automatable: High

All inter-service and downstream HTTP calls MUST define explicit timeouts. Critical dependencies MUST implement circuit breakers. Retry logic MUST use exponential backoff with jitter.

Requirement: No default timeout for external calls; circuit breaker for critical dependencies; retry maximum 3 attempts with backoff; bulkhead isolation for different classes.

Terraform Checks (Excerpt):

Evidence: Terraform/service mesh configuration (required); app config files (required)

Best Practice: Circuit Breaker & Timeouts →

Severity: High | Category: Incident Response | Automatable: Medium

All production workloads MUST have a documented incident response plan with severity definitions, escalation paths and on-call rotation. Runbooks MUST be directly linked from alert notifications. Post-incident reviews within 5 business days.

Requirement: 4 severity levels defined; on-call configured; all critical alerts with runbook link; MTTR tracked; post-mortems for SEV1/SEV2 documented.

Terraform Checks (Excerpt):

Evidence: Incident response plan (required); post-incident review records (required)

Best Practice: Incident Response & Runbooks →

Severity: High | Category: Disaster Recovery | Automatable: Partial

All critical production workloads MUST conduct DR tests at least twice a year with documented RTO/RPO results. DR plans MUST be updated after significant architecture changes.

Requirement: DR plan with RTO/RPO targets; semi-annual test; results documentation; deviation ⇒ remediation plan within 30 days.

Terraform Checks (Excerpt):

Evidence: DR test reports from last 12 months (required); DR plan (required)

Best Practice: Disaster Recovery →

Severity: Medium | Category: Dependency Management | Automatable: Medium

All production workloads MUST maintain an inventoried and classified dependency register. Critical dependencies MUST have circuit breakers. Fallback behavior MUST be defined for all optional dependencies.

Requirement: Dependency register with criticality and SLA; CB for critical dependencies; fallback for optional dependencies; quarterly review.

Terraform Checks (Excerpt):

Evidence: Dependency register (required); circuit breaker configuration (required)

Severity: Medium | Category: Chaos Engineering | Automatable: Medium

Production and staging workloads MUST conduct quarterly structured chaos experiments with a hypothesis framework, stop conditions and results documentation.

Requirement: Hypothesis-driven tests; stop conditions configured; experiments in staging before production; blast radius limited; results converted into remediation.

Terraform Checks (Excerpt):

Evidence: Chaos experiment reports (required); chaos engineering charter (required)

Best Practice: Chaos Engineering →

Severity: Medium | Category: Reliability Governance | Automatable: Low–Medium

All known reliability risks and deferred improvements MUST be captured in a versioned Reliability Debt Register with owner, severity and target date. Quarterly review is mandatory.

Requirement: Register version-controlled; all entries with owner, P1–P4 priority and target date; quarterly review with minutes; P1 items addressed within one sprint.

Terraform Checks (Excerpt):

Evidence: Reliability Debt Register (required); quarterly review minutes (required)