01 Ingest keys #
Every producer authenticates with an ingest key sent as
Authorization: Bearer <key>.
zpi_— SDK / Collector- Server-side telemetry. This is what the trace receiver accepts.
zpr_— Browser RUM- Reserved for browser telemetry. The trace receiver rejects RUM keys, on purpose — a key embedded in a web page is public, and must not be able to write server-side traces.
Keys are stored only as a SHA-256 hash and the plaintext is shown exactly once. Validation results are cached for 30 seconds, so a revoked key stops working within about half a minute rather than instantly.
- Environment
- Applied to spans that do not declare
deployment.environmentthemselves. An explicit resource attribute always wins. - Last used
- When telemetry last authenticated with this key. A key reading never used has never worked — which is the single fastest way to distinguish a broken exporter from a broken key.
- Revoke
- Immediate and permanent. There is no un-revoke; issue a new key.
A single fleet-wide key means a leak forces you to silence everything at once. One key per environment is the minimum; one per service is better if you can distribute them.
02 Enrollment tokens #
Baking a long-lived key into a machine image or config-management repository means the key lives as long as the image and leaks with it. An enrollment token is the alternative: a short-lived, use-capped credential a host redeems for its own ingest key at provisioning time.
- Max uses
- 1 to 100. Use 1 for a single host; a higher cap for an autoscaling group.
- Expiry
- 1 hour to a year. Short is better — a token only needs to survive from creation to first boot.
- State
- Available, used up, expired, or revoked. Revoking a token that has already been redeemed does not revoke the key it produced; revoke that key on the Ingest keys tab.
03 Data quality #
This tab monitors the telemetry pipeline itself. It exists because bad telemetry is worse than no telemetry: a dashboard built on silently rejected spans is confidently wrong.
| Counter | Meaning | What a non-zero value tells you |
|---|---|---|
| Spans accepted | Decoded and queued for storage | Your baseline — watch for step changes |
| Rejected | Failed to decode, or refused | Malformed payloads, or a producer with a broken clock |
| Clock-skewed | Timestamp more than 5 minutes in the future or 7 days in the past | A producer's clock is wrong. Fix NTP — skewed spans would corrupt every time-window query, so they are refused rather than stored |
| Dropped (flush) | Accepted, then lost writing to storage | Storage pressure or an outage. This is real data loss and worth investigating |
| Queue depth | Payloads waiting to be written right now | Persistently high means ingest is outrunning storage |
Counters are cluster-wide and survive restarts, and cover the last 24 hours.
Service reporting freshness
Every registered service with how long since it last reported. live means within two minutes. A service silent for more than 10 minutes is flagged — that is the difference between “quiet” and “dead”, and it is the check most monitoring setups forget to make.
Agent APM forwarders
For hosts forwarding through the ZenPlus agent: whether forwarding is on, spans per minute, export errors, spool depth, and the last error. A growing spool with export errors means the agent is collecting but cannot deliver — usually connectivity or a revoked key.
“Traffic dropped at 14:00” and “we started rejecting spans at 14:00” look identical on a service chart. This tab tells them apart in one glance.
04 Environments #
Environments (prod, staging, dev) scope ingest keys, filter every
screen, and scope SLOs. A span's own deployment.environment attribute always takes precedence
over its key's environment, so a shared key still produces correctly separated data as long as the SDK
declares where it is running.