01 Why an SLO and not a threshold #
A threshold rule (“page if error rate > 1%”) fires on a thirty-second blip that nobody noticed and stays quiet through a week of 0.9% that burns your whole quarter's budget. It measures the wrong thing.
An SLO measures how fast you are consuming the failure you already agreed to tolerate. If your target is 99.9% over 30 days, you have budgeted 0.1% of requests to fail. Burning that steadily is fine by definition. Burning it 14 times faster than budgeted means you will be out in two days — that is worth waking someone.
Budget fraction = 1 - target/100. Bad fraction over a window = bad events / total events. Burn rate = bad fraction / budget fraction. A burn rate of 1.0 exhausts the budget exactly at window end; 14.4 exhausts a 30-day budget in about two days.
02 Defining an SLO #
APM → SLOs → New SLO.
- Service and environment
- What the SLO is scoped to. If the service has not registered yet, it is created on save — you can define an SLO the moment a service first reports.
- Operation (optional)
- Narrow to one route. Useful when a checkout endpoint deserves a stricter target than a search endpoint on the same service.
- SLI type
- availability and error rate count errored requests and are exact. latency counts requests slower than your threshold.
- Latency threshold
- Required for latency SLIs. “99% of requests under 500 ms” means target 99, threshold 500.
- Target
- A percentage above 0 and below 100. 99.9 is a common starting point; 99.99 is expensive and should be chosen deliberately.
- Window
- 7, 30 or 90 days. The budget is computed over this window.
- Burn alerts + channels
- Whether breaches notify, and where.
Every extra nine costs roughly ten times more engineering. 99.9% over 30 days allows about 43 minutes of full outage; 99.99% allows about 4. If you would not staff for four minutes, do not write 99.99.
03 How latency SLIs are estimated #
Availability and error-rate SLIs are exact — the rollups store error and request counts.
Latency is different. The rollup stores a t-digest sketch, not every duration, so “how many requests were slower than 500 ms” is estimated by interpolating the quantile curve. Resolution is roughly 0.1%.
A latency SLO of 99.95% is asking about a 0.05% tail, which is below the sketch's resolution — the estimate will not be reliable at that precision. For latency targets, 99–99.9% is the range this data supports. Availability SLOs have no such limit.
04 Burn alerting #
ZenPlus implements the Google SRE Workbook's multi-window multi-burn-rate configuration. Three tiers run continuously, each with a long and a short window:
| Tier | Burn factor | Long window | Short window | Severity | Means |
|---|---|---|---|---|---|
| Fast | 14.4× | 1 hour | 5 minutes | Critical | A 30-day budget gone in ~2 days — page now |
| Mid | 6× | 6 hours | 30 minutes | Critical | Budget gone in ~5 days — page |
| Slow | 1× | 3 days | 6 hours | Warning | On track to exhaust the budget — ticket |
A tier fires only when both its windows exceed the factor. The long window establishes that consumption is genuinely sustained; the short window is what lets the alert clear about five minutes after recovery instead of staying lit until the long window rolls off.
RED data is stored in five-minute buckets, and the newest bucket is always partially filled. Measuring a nominal five-minute window over a single partial bucket produced a burn rate that was mostly zero, which meant the both-windows gate could never be satisfied and burn alerts never fired at all. Windows are now snapped to bucket boundaries and floored at two buckets, and the alert text reports the window it actually measured. If you are upgrading from an earlier build, expect burn alerts to start working — and to check that your targets are ones you meant.
05 Reading the budget #
The error-budget cell on each row opens the full picture:
- Budget consumed
- Fraction of the budget used so far in the window. Above 1.0 means the budget is exhausted — you are over your target for this window and can only recover by rolling forward into the next one.
- Budget remaining
- How much headroom is left, floored at zero.
- Window requests
- Total requests the calculation was made over. A small number here means treat the percentage with suspicion — ten requests cannot tell you anything about 99.9%.
- Per-tier burn
- Long and short burn rate for each tier, the request counts behind them, and whether the tier is currently breaching.
An exhausted budget is a monthly conversation about priorities. A breaching fast tier is a right-now conversation about the current incident. They are different signals and both are shown.
06 Where burn alerts go #
Breaches raise alerts in the shared ZenPlus alerts table — the same place device and server alerts land — with severity critical for the fast and mid tiers and warning for the slow tier. Notifications go to the SLO's own channels. Alerts deduplicate per SLO and tier, so a sustained breach is one alert, not one per minute, and it resolves automatically when burn falls back under the factor.