01 What counts as a request #
A single traced request produces many spans: the inbound handler, outbound HTTP calls, database queries, internal work. If APM counted all of them, a service that makes five database calls per request would look like it serves six times its actual traffic.
So the golden signals are measured on entry spans only — OpenTelemetry spans of kind
SERVER or CONSUMER. Those are the spans representing work the service was
asked to do.
| Span kind | Typical source | Counted as a request? |
|---|---|---|
SERVER | Inbound HTTP/gRPC handler | Yes |
CONSUMER | Message-queue consumer | Yes |
CLIENT | Outbound HTTP call, DB query | No — but it draws service-map edges |
PRODUCER | Publishing to a queue | No |
INTERNAL | Function-level spans | No |
If a service shows spans in the trace explorer but zero throughput on this page, it is almost always emitting only INTERNAL spans. Its inbound handler is not instrumented, or hand-written spans omit the kind.
02 The four signals #
- Throughput (rate)
- Entry spans in the window divided by the seconds the window covers. Shown per second, or per minute below 1/s so low-traffic services stay readable.
- Error rate
- Entry spans whose OpenTelemetry status is
ERROR, over total entry spans. Note that an HTTP 500 is only an error if the instrumentation set the span status — most auto-instrumentations do. - Duration (p50 / p95 / p99)
- Percentiles of entry-span duration, computed by merging t-digest sketches from the rollups. Percentiles are approximate by construction, which is what makes them cheap enough to keep for a year.
- Apdex
(satisfied + tolerating/2) / totalwith a target threshold T of 500 ms. Requests under 500 ms are satisfied, under 2000 ms (4T) are tolerating, slower are frustrated. One number from 0 to 1 for “are users happy”.
There is no correct way to combine two services' p95 into a fleet p95, which is why the Overview page shows the worst service's p95 rather than an average. An averaged percentile is a number that describes nobody.
03 How health is decided #
The badge on each row is derived from the same window you are looking at — there is no separate health check.
Thresholds are evaluated worst-first: a service is Critical if either its error rate is at or above 5% or its p95 is at or above 1000 ms. A service is Degraded at 1% errors or 500 ms p95. This is deliberately blunt — health is a triage hint, not a contract. The contract is an SLO.
Once a service has reported, it is registered. If it then stops reporting for 10 minutes, a background sweep flips it to No data rather than leaving the last known-good health frozen in place. A dead service must never look healthy.
04 Reading the list #
The list defaults to worst-health-first, then busiest, so the service that needs you is at the top. Any column header re-sorts.
- Environment filter — scopes everything to one
deployment.environment. - Search — client-side name filter; the count next to it shows how much you hid.
- Time range — 15m through 7d, shared across every APM screen, so moving between Services and Traces does not silently change the window.
Windows up to 6 hours read the 5-minute rollup; longer windows read the hourly one. Numbers can shift very slightly when you cross that boundary — that is two different aggregations of the same spans, not a bug.
05 Service detail #
Clicking a service opens its detail page with three tabs.
- Overview
- The three RED charts over the selected window, plus the top 25 operations by request count with per-operation error rate and p95. Operations are the fastest way to see that one route is dragging a service's numbers down.
- Performance
- p50 and p95 side by side, plus throughput and error rate. Use it to separate “everything got slower” (p50 rises with p95) from “the tail got worse” (only p95 rises).
- Errors
- Error issues attributed to this service over the last 24 hours, linking into the errors inbox.
View traces in the header jumps to the trace explorer pre-filtered to this service and time range — the usual next step once a chart tells you when and you need which request.
06 A worked incident #
The Overview flags a service Degraded
Error rate 4.8%, p95 863 ms. Everything else is healthy.
Open it and check the shape
On Performance, p50 is flat and p95 has doubled: this is a tail problem, not a uniform slowdown. Something is slow for a subset of requests.
Find which operation
On Overview, the operations table shows one route carrying nearly all of the errors while the rest are clean. You now have a route, not just a service.
Look at the actual requests
View traces, filter to errors only, and open one. The waterfall shows where the time went and which downstream call failed. See Traces.
Confirm the blast radius
The service map shows who calls the failing service, so you know which user-facing paths are affected before anyone asks.