stevenacoffman/sre

Use when someone sets, demands, or challenges an SLO target ("we need five nines", 99.999%) without stating its cost. Frames each extra nine as roughly 10x more reliability for roughly 10x more cost, to justify or push back on 3-nines vs 4-nines vs 5-nines mandates.

Use after a significant incident (user-visible downtime, data loss, rollback, slow resolution, missed alert) to run a postmortem that is blameless, written, and widely shared, turning the incident into organizational learning that prevents recurrence.

Use when sizing a canary deployment (traffic percentage, duration) or judging whether blue/green counts as a canary. Size by arithmetic: canary size x defect rate x duration = error budget consumed. Requires concurrent control and canary populations, not before/after.

Use when designing a release process, debating change freezes, approval gates, or deploy windows, or when frequent outages get blamed on bad luck. About 70% of outages come from changes; make them smaller, observable, and reversible via progressive rollout, canarying, and automatic rollback.

Use when deriving a system's overall availability or SLO from serial or parallel dependencies. Serial: multiply SLOs. Parallel: multiply error budgets (1 - SLO), the case most engineers get wrong.

Use when designing, evaluating, or debugging a configuration system: rollbacks fail intermittently, config renders differently at different times, or choosing a config tool (Helm, Jinja2, Ansible, CUE, Dhall). Not for auditing a single wrong config value.

Use when classifying an incident as degradation, disruption, or outage, deciding whether to keep a degraded service running or take it down, or reporting an incident's business impact. Not for computing SLO targets or designing fallback and failover.

Use when a team is stuck in chronic ops mode (70%+ tickets, pages, and incidents; no project progress; burnout) and adding headcount or process has not helped. Plans a multi-week intervention embedding one SRE. Not for a rough quarter or a launch spike.

Use when choosing or designing a resilience mechanism that keeps a service up when a primary system or dependency fails: failover vs fallback, homogeneous vs heterogeneous backups, a dependency with a lower SLO than required, or common-cause failure risk.

Use when SRE ops work (toil, on-call, interrupts) crowds out engineering projects, or management proposes hiring more SREs for on-call load. Enforce a 50% ops cap and send excess ops work back to the dev team instead of adding headcount.

Use when responding to a production incident, alert, or anomaly and finding its root cause, especially when tempted to "just roll back" or blame last week's problem. Triage, examine telemetry, rank hypotheses, then test them. Not when a known playbook applies.

Use when one SLO target is not enough: request types (query, mutation, subscription) need different targets, a monthly SLO would allow a multi-day outage, or an aspirational target and a guaranteed threshold must coexist (aim for 200ms, guarantee 850ms).

Use when configuring SLO-based alerting (Prometheus, Datadog, Grafana) or replacing threshold, duration, or single-window burn-rate alerts that page late, falsely, or not at all. Derives multiwindow, multi-burn-rate alerts from your own SLO. Not for low-traffic services.

Use when designing a large distributed system and the design is still boxes and arrows with no machine counts, RAM estimates, or cost, or someone says "we'll scale horizontally". Non-abstract large system design (NALSD): iterate with real resource estimates per scale tier.

Use when an SRE team is in sustained operational overload (work piling up, after-hours work, falling velocity, broken collaboration), before any triage, automation, or headcount fix. Restore psychological safety first, then fix structure. Not for one acute incident.

Use when diagnosing why a team cannot deliver reliable software or designing roles and accountability: a team owns a system it cannot fix, a manager decides without bearing consequences, an expert lacks authority, or on-call feels like punishment. Covers DevOps and you-build-it-you-own-it.

Use when designing an on-call program, cutting high MTTR on recurring alerts, or when an alert has no documented response and on-call is "winging it". Pre-written playbooks for known failure modes cut MTTR about 3x; decide where to spend documentation time.

Use when designing update and rollback for firmware, self-updating binaries, or kernels where rollback conflicts with permanently enforcing a security patch. A minimum acceptable security version number (MASVN) blocks attacker-forced rollback. Not for stateless deploys.

Use when assigning SLI and SLO ownership across teams, splitting a latency or availability budget across dependent teams, or diagnosing why a team is paged for incidents it cannot fix. Scope each metric to what the owning team actually controls.

Invoke when deciding whether a system is ready for chaos engineering, weighing high-availability hardware vs automated failover, choosing MTBF or MTTR as the key metric, or checking whether redundancy adds availability or only cost. Robustness vs resilience vs antifragility.

Use when deriving SLIs and SLOs from first principles: a service has no SLIs or only infrastructure vanity metrics, alerts are noisy and ignored, metric ownership is disputed, or you are running a service level workshop. Starts from architecture, consumers, and their tasks.

Use when judging an existing SLI's signal quality against its cost: synthetic health checks vs organic measurement, endpoint pings vs user journey monitoring, whether to invest in RUM, or an SLI that fires on noise or misses incidents. Plans an SLI investment roadmap.

Use when writing or reviewing an SLI formula: defining good and valid events (valid, not total, in the denominator), choosing where to measure (client, edge, load balancer, service), event-based vs time-based SLIs, and scoping to what one team controls.

Use when SLOs exist but reliability is not improving: priorities don't change when the error budget is exhausted, there is no error budget policy or it is never invoked, or you must tell real SRE practice from SRE vocabulary. SLOs need agreed consequences.

Use when SLO status alone does not say what to do: SLO met but users complain, met but toil burns out the team, missed but users don't notice. Combines SLO, toil, and user satisfaction to choose tighten, relax, automate, or hand back. Not for incident response.

Use when setting, revising, or validating an SLO number: checking it is appropriate and defensible, resolving disagreement over whether it is achievable, or when an error budget policy can't be ratified because a party won't sign. Not for actions after budget exhaustion.

Use when forming, diagnosing, or intervening in an SRE team that is stuck, resisting, or not reaching norming or performing, or when a manager applies generic advice that fits the wrong formation type. Applies Tuckman stages. Not for technical reliability issues.

Use when asking "is this work toil?", deciding what to automate, or whether work counts against the 50% ops cap. Classifies work as toil, engineering, or overhead by six properties: manual, repetitive, automatable, tactical, no enduring value, O(n) with service growth.

Use when picking starting SLO levels across many systems that serve different business purposes and change at different rates. Heuristic by evolution stage: systems of innovation about 95%, differentiation about 99.6%, record about 99.99%.