01 Start here: is anything arriving at all? #
Almost every APM problem resolves into one of three questions, in this order. Answer them in order and you will not waste time.
Is the key being used?
Settings → Ingest keys. If last used is empty, nothing has ever authenticated with it — the problem is entirely on the producer side. Skip to “No data at all”.
Is the pipeline accepting?
Settings → Data quality. Non-zero rejected, skewed or dropped counters name the problem directly.
Is the data the shape you expect?
Spans in the trace explorer but no throughput on Services means the spans are the wrong kind. See “Partial data”.
02 No data at all #
| Check | How | If it fails |
|---|---|---|
| Endpoint reachable | curl -I http://APPLIANCE/v1/traces from the app host | Firewall or routing between app and appliance |
| Route reaches the API | The response must be JSON, not HTML | The reverse proxy is returning the dashboard for /v1/ — its /v1/ route is missing |
| Key accepted | POST a test span (see Getting started) | 401 — wrong, revoked, or RUM-type key |
| Protocol | Response is 415 | SDK is exporting protobuf; set OTEL_EXPORTER_OTLP_PROTOCOL=http/json |
| SDK actually exporting | Enable SDK debug logging | Exporter never configured, or the process exits before the batch flushes |
This is the most common cause of “APM does not work”. Most OpenTelemetry SDKs default to http/protobuf. The appliance's built-in receiver takes JSON, answers 415 to protobuf, and the SDK drops the batch silently — your application logs nothing, and APM stays empty forever. Test with curl before debugging anything else.
03 Partial data #
| Symptom | Cause | Fix |
|---|---|---|
| Traces exist, service shows 0 throughput | Only INTERNAL spans are emitted | Instrument the inbound handler so it produces SERVER spans |
Service named unknown | service.name not set | Set OTEL_SERVICE_NAME |
| Traces split into fragments | Trace context not propagated across a boundary | Use compatible propagators (W3C by default); check proxies are not stripping traceparent |
| Service map missing edges | Same as above — edges need a parent span id crossing the wire | See Service map |
| Errors counted with no type or stack | Span marked ERROR with no exception event | Record exceptions properly in your error handler |
| Operations list explodes | Concrete URLs used as http.route | Emit templated routes |
| Unique users always 0 | No end-user attribute | Set enduser.id — see Usage |
04 Numbers that look wrong #
- Throughput lower than expected
- Only entry spans count. A service making five downstream calls per request shows one request, not six. This is correct.
- Error rate is 0 during a visible outage
- Error rate follows OpenTelemetry span status, not HTTP status. Instrumentation that returns 500 without setting span status to ERROR reports no errors. Check what your framework's auto-instrumentation does with handled exceptions.
- Numbers shift when changing time range
- Windows up to 6 hours read the 5-minute rollup; longer windows read the hourly one. Two different aggregations of the same spans differ slightly. Expected.
- Percentiles look approximate
- They are. Duration percentiles come from t-digest sketches, which is what makes a year of retention affordable. Counts and error rates are exact.
- A service is stuck at “No data”
- It has not reported in 10 minutes. That flag is deliberate — freezing the last known-good health would let a dead service look healthy forever.
- Usage analytics will not go past 7 days
- It reads raw spans, which expire at 7 days. Rollup-based screens are unaffected.
05 Alerting problems #
| Symptom | Cause | Fix |
|---|---|---|
| No APM alerts ever | No rule and no SLO exists | Nothing alerts by default. Create an alert rule on an apm_* metric, or an SLO |
| SLO burn never fires | Fixed in ZenPlus 1.6.0 | Earlier builds measured the short burn window against a single partially-filled rollup bucket, so it read zero and the both-windows gate could never be satisfied. Windows are now bucket-aligned. After upgrading, expect real burn alerts — verify your targets are the ones you meant |
| A throughput rule fires constantly | Fixed in ZenPlus 1.6.0 | The evaluator divided a partial bucket's count by a full window, under-reporting throughput by up to 5×. Now divided by the seconds actually covered |
| Alert fires on every blip | A threshold rule is the wrong instrument | Use an SLO. Burn-rate alerting is specifically designed to ignore spikes that do not consume meaningful budget |
| Alert raised but no notification | No channels on the rule/SLO, or quiet hours | Check the notification channels and the rule's schedule window |
APM alert rules can be written against: apm_latency_p50, apm_latency_p95,
apm_latency_p99 (milliseconds), apm_error_rate and apm_apdex
(fractions from 0 to 1), and apm_throughput (requests per second). Leave the target empty to
apply a rule to every reporting service.
06 Ingest under load #
| Symptom | Meaning | Action |
|---|---|---|
503 with Retry-After | Ingest queue is full | SDKs retry automatically. Persistent 503 means sustained overload — reduce span volume with sampling, or check storage health |
| Dropped (flush) climbing | Spans accepted then lost on write | Storage is unhealthy or out of disk. This is real data loss |
| Queue depth persistently high | Ingest outrunning storage | Same investigation as above |
| Clock-skewed climbing | A producer's clock is wrong | Fix NTP on that host. The producer is named in its span resource attributes |
A span stamped three hours in the future lands in a future rollup bucket and quietly corrupts every window that later includes it. Refusing it means the producer learns something is wrong; storing it means everyone else's dashboards are wrong instead.
07 Still stuck #
Collect these before opening a support case — they answer most of the first round of questions:
- The exact
curlcommand you used to test ingest, and its full response. - A screenshot of Settings → Data quality.
- The key's last used value.
- Your SDK's exporter configuration (with the key redacted).
- Service name, environment, and the time range where you expected data.