Ir al contenido

Guías

Investigate latency

Follow a request through the code and distinguish waiting from execution.

En esta página
Esta página aún no está traducida, así que estás leyendo la versión en inglés.

A slow reply can spend its time waiting for work, waiting for a dependency, or executing. Find the boundary first. Adding more spans inside the engine will not explain a queue that is already backed up.

Prerequisites and outcome

Choose a slow operation, a time window and the affected service or lane. Access to the relevant logs and measurements is needed; distributed traces require monitoring to be configured. Without a license the application can still run, but do not expect a New Relic trace. Record the symptom and a baseline before changing instrumentation.

1. Check available instrumentation

Check whether the service has monitoring configured, then inspect its logs and available traces for the chosen operation. Go services start monitoring through pkg/svcboot. pkg/monitor.New returns no application when NEW_RELIC_LICENSE_KEY is absent, so local execution and tests do not require an observability account.

pkg/monitor attaches entity metadata to logs and trace/span IDs when a transaction exists. It disables agent log forwarding, excludes request/response attributes and HTTP URLs, and disables datastore raw query and parameter capture. ScrubError keeps a bounded error prefix and type. These controls reduce captured data; they are not a reason to put tokens or message bodies into log fields.

pkg/bus/telemetry.go carries tracing across bus boundaries. Preserve the handler context when calling dependencies so the next segment can belong to the same request.

2. Locate the slow boundary

Follow the event through the following boundaries and identify where elapsed time or backlog first grows.

flowchart TD
    Input["Accepted platform event"] --> Publish["Publish and acknowledgement"]
    Publish --> Delivery["Consumer delivery and backlog"]
    Delivery --> Engine["Decode and engine execution"]
    Engine --> State["Projection, RPC and database waits"]
    State --> Output["Output publication"]
    Output --> Effect["Outgress admission and platform request"]
Symptom Start in the code
Input is accepted but work arrives late Ingress dispatcher, bus publisher, consumer backlog
Cold requests are slower internal/projection, Projector hydration and owner RPC
Database work waits before a query pkg/db/gate.go, pool settings and pkg/db/stats.go
Output keeps returning for another attempt pkg/bus delivery handling and Outgress limiter decisions
HTTP effect is slow Outgress or Gossip transport, upstream status and response handling

Pool wait, process gate wait and query duration are different measurements. A cache miss is not automatically an error. A delivery acknowledgement is not proof that Twitch displayed a message.

Use bounded operation names, subjects and outcome classes. Broadcaster IDs, message bodies, tokens, arbitrary URLs and cache keys do not belong in metric names.

3. Verify and compare

Run the focused tests for the boundary you changed. pkg/bus/telemetry_test.go covers trace transport, and pkg/monitor/scrub_test.go covers scrubbing. Keep the no-license path working. If you claim a latency improvement, attach the workload and commit that produced it; a new span alone proves no improvement.

See performance work and testing.

Source: monitor, bus telemetry, database pool metrics.

Troubleshoot and report

If a trace is missing, check whether monitoring is enabled and whether the handler passes its context across the bus and dependency calls. Use logs and boundary measurements when no trace is available; do not infer a queue delay from total latency alone. Compare the same operation and window before and after a change, and report the commit, workload, sample size, errors and latency distribution. State which boundaries were measured and which remain unknown. Continue with Measure performance when a reproducible workload is available.