SLIs, SLOs, and Error Budgets in SRE Practice: Reliability Governance, OpenTelemetry, and Multi-Burn-Rate Alerting at Scale

The historic friction between Product squads (demanding rapid feature velocity) and Infrastructure teams (mandating unyielding operational stability) is resolved through a single mathematical arbiter: the Error Budget. Under modern Site Reliability Engineering (SRE) principles, 100% uptime is mathematically the wrong target: pursuing 100% reliability paralyzes product agility and inflates infrastructure costs exponentially without meaningful user satisfaction benefits.

1. The SRE Triad: SLIs, SLOs, and SLAs Decoded

Engineering teams frequently conflate these three core terms, inducing alert fatigue and misaligned business incentives:

• SLI (Service Level Indicator): A precise real-time metric reflecting current service health. Example: 'The percentage of successful HTTP checkout requests returning 2xx status codes in under 250ms measured at the ingress gateway'.
• SLO (Service Level Objective): The internal target agreed between Engineering, Product, and Business stakeholders. Example: '99.95% of requests must satisfy the SLI over a rolling 30-day window'.
• SLA (Service Level Agreement): The external contractual commitment with clients specifying financial penalties or credit refunds if performance drops below a looser legal threshold (e.g., 99.5%).

Architecture Comparison: Traditional Host Monitoring vs. Better2Better SRE Governance

2. Operationalizing the Error Budget: Governing the Release Lifecycle

If your checkout SLO is 99.9% over 30 days, your Error Budget is exactly 0.1% (approximately 43 minutes of permissible downtime each month). This budget acts as a shared currency:

1. Surplus Budget: Product teams hold complete autonomy to ship experiments, refactor services, and push frequent releases.
2. Budget Exhaustion: An automated Release Freeze takes immediate effect. Feature deployments halt; 100% of engineering bandwidth pivots toward stability, technical debt elimination, and test hardening until the budget recovers.

3. Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard)

Threshold alerts on error percentages either flood on-call engineers with noise or fail during slow budget drains. Better2Better deploys the mathematical Multi-Window Multi-Burn-Rate algorithm:

• 14.4x Burn Rate (2% budget consumed in 1 hour): Critical paging alert (1h and 5m concurrent windows).
• 6x Burn Rate (5% budget consumed in 6 hours): High-priority ticket (6h and 30m concurrent windows).
• 1x Burn Rate (10% budget consumed in 3 days): Non-urgent asynchronous notification for regular business-hour review.

4. Enterprise Observability with OpenTelemetry

To eradicate monitoring silos, we standardize on OpenTelemetry (OTel). We instrument microservices across Go, TypeScript, Java, and Python with W3C Trace Context propagation, connecting client frontend clicks directly to database query spans on a single unified timeline.

5. Production PromQL Multi-Burn-Rate Alert Configuration

Prometheus alerting rule calculating real-time Error Budget consumption:

groups:
  - name: slo_checkout_alerts
    rules:
      - alert: CheckoutErrorBudgetBurn14x
        expr: |
          (
            sum(rate(http_requests_total{job="checkout", status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{job="checkout"}[1h]))
          ) > (14.4 * (1 - 0.999))
          and
          (
            sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="checkout"}[5m]))
          ) > (14.4 * (1 - 0.999))
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Checkout API burning budget at 14.4x (2% consumed in 1 hour)"
          runbook_url: "https://wiki.better2better.com.br/sre/runbooks/checkout-burn-rate"

6. Blameless Post-Mortem Engineering Culture

Punishing developers for outages incentivizes secrecy and hides fragility. Human error is an operational given; if a single mistyped command took down production, the fault lies squarely with systemic design and lacking guardrails. Blameless reviews transform incidents into actionable automated safeguards.

7. Enterprise SRE Maturity Checklist

• [x] User-centric SLIs measured at edge ingress gateways across all critical paths.
• [x] SLOs formally endorsed by Product, Engineering, and Business executive leadership.
• [x] Host CPU threshold alerts decommissioned in favor of multi-burn-rate alerts.
• [x] End-to-end OpenTelemetry distributed tracing with unified W3C trace context.
• [x] Automated Release Freeze gate triggered when Error Budgets deplete.

8. Frequently Asked Questions about SRE and Observability (FAQ)