Go OpenTelemetry tracing for end-to-end API visibility

What end-to-end tracing means for a Go API

A trace is the timeline of one request as it moves through your system. It starts when an API call arrives and ends when you send the response.

Inside a trace are spans. A span is one timed step, like “parse request,” “run SQL,” or “call payment provider.” Spans can also carry useful details, such as an HTTP status code, a safe user identifier, or how many rows a query returned.

“End-to-end” means the trace doesn’t stop at your first handler. It follows the request through the places where issues usually hide: middleware, database queries, cache calls, background jobs, third-party APIs (payments, email, maps), and other internal services.

Tracing is most valuable when problems are intermittent. If one out of 200 requests is slow, logs often look identical for fast and slow cases. A trace makes the difference obvious: one request spent 800 ms waiting on an external call, retried twice, then kicked off a follow-up job.

Logs are also hard to connect across services. You might have one log line in the API, another in a worker, and nothing in between. With tracing, those events share the same trace ID, so you can follow the chain without guessing.

Traces, metrics, and logs: how they fit together

Traces, metrics, and logs answer different questions.

Traces show what happened for one real request. They tell you where the time went across your handler, database calls, cache lookups, and third-party requests.

Metrics show the trend. They’re the best tool for alerts because they’re stable and cheap to aggregate: latency percentiles, request rate, error rate, queue depth, and saturation.

Logs are the “why” in plain text: validation failures, unexpected inputs, edge cases, and decisions your code made.

The real win is correlation. When the same trace ID shows up in spans and structured logs, you can jump from an error log to the exact trace and immediately see which dependency slowed down or which step failed.

A simple mental model

Use each signal for what it’s best at:

• Metrics tell you something is wrong.
• Traces show where time went for one request.
• Logs explain what your code decided and why.

Example: your POST /checkout endpoint starts timing out. Metrics show p95 latency spiking. A trace shows most of the time is inside a payment provider call. A correlated log line inside that span shows retries due to a 502, which points you to backoff settings or an upstream incident.

Before you add code: naming, sampling, and what to track

A bit of planning up front makes traces searchable later. Without it, you’ll still collect data, but basic questions get hard: “Was this staging or prod?” “Which service started the problem?”

Start with consistent identity. Pick a clear service.name for each Go API (for example, checkout-api) and a single environment field such as deployment.environment=dev|staging|prod. Keep these stable. If names change mid-week, charts and searches look like different systems.

Next, decide sampling. Tracing every request is great in development, but often too expensive in production. A common approach is to sample a small percentage of normal traffic and keep traces for errors and slow requests. If you already know certain endpoints are high volume (health checks, polling), trace them less or not at all.

Finally, agree on what you will tag on spans and what you will never collect. Keep a short allowlist of attributes that help you connect events across services, and write simple privacy rules.

Good tags usually include stable IDs and coarse request info (route template, method, status code). Avoid sensitive payloads entirely: passwords, payment data, full emails, auth tokens, and raw request bodies. If you must include user-related values, hash or redact them before adding them.

Step-by-step: add OpenTelemetry tracing to a Go HTTP API

You’ll set up a tracer provider once at startup. This decides where spans go and which resource attributes are attached to every span.

1) Initialize OpenTelemetry

Make sure you set service.name. Without it, traces from different services get mixed together and charts become hard to read.

// main.go (startup)
exp, _ := stdouttrace.New(stdouttrace.WithPrettyPrint())

res, _ := resource.New(context.Background(),
	resource.WithAttributes(
		semconv.ServiceName("checkout-api"),
	),
)

tp := sdktrace.NewTracerProvider(
	sdktrace.WithBatcher(exp),
	sdktrace.WithResource(res),
)

otel.SetTracerProvider(tp)

That’s the foundation for Go OpenTelemetry tracing. Next, you need a span per incoming request.

2) Add HTTP middleware and capture key fields

Use HTTP middleware that starts a span automatically and records status code and duration. Set the span name using the route template (like /users/:id), not the raw URL, or you’ll end up with thousands of unique paths.

Aim for a clean baseline: one server span per request, route-based span names, HTTP status captured, handler failures reflected as span errors, and duration visible in your trace viewer.

3) Make failures obvious

When something goes wrong, return an error and mark the current span as failed. That makes the trace stand out even before you look at logs.

In handlers, you can do:

span := trace.SpanFromContext(r.Context())
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())

4) Verify trace IDs locally

Run the API and hit an endpoint. Log the trace ID from the request context once to confirm it changes per request. If it’s always empty, your middleware isn’t using the same context your handler receives.

Carry context through DB and third-party calls

End-to-end visibility breaks the moment you drop context.Context. The incoming request context should be the thread you pass to every DB call, HTTP call, and helper. If you replace it with context.Background() or forget to pass it down, your trace turns into separate, unrelated work.

For outgoing HTTP, use an instrumented transport so every Do(req) becomes a child span under the current request. Forward W3C trace headers on outbound requests so downstream services can attach their spans to the same trace.

Database calls need the same treatment. Use an instrumented driver or wrap calls with spans around QueryContext and ExecContext. Record only safe details. You want to find slow queries without leaking data.

Useful, low-risk attributes include an operation name (for example, SELECT user_by_id), table or model name, row count (count only), duration, retry count, and a coarse error type (timeout, canceled, constraint).

Timeouts are part of the story, not just failures. Set them with context.WithTimeout for DB and third-party calls, and let cancellations bubble up. When a call is canceled, mark the span as error and add a short reason like deadline_exceeded.

Tracing background jobs and queues

Background work is where traces often stop. An HTTP request ends, then a worker picks up a message later on a different machine with no shared context. If you do nothing, you get two stories: the API trace and a job trace that looks like it started from nowhere.

The fix is straightforward: when you enqueue a job, capture the current trace context and store it in job metadata (payload, headers, or attributes, depending on your queue). When the worker starts, extract that context and start a new span as a child of the original request.

Propagate context safely

Only copy trace context, not user data.

• Inject only trace identifiers and sampling flags (W3C traceparent style).
• Keep it separate from business fields (for example, a dedicated "otel" or "trace" field).
• Treat it as untrusted input when you read it back (validate format, handle missing data).
• Avoid putting tokens, emails, or request bodies into job metadata.

Spans to add (without turning traces into noise)

Readable traces usually have a few meaningful spans, not dozens of tiny ones. Create spans around boundaries and “wait points.” A good starting point is an enqueue span in the API handler and a job.run span in the worker.

Add a small amount of context: attempt number, queue name, job type, and payload size (not payload content). If retries happen, record them as separate spans or events so you can see backoff delays.

Scheduled tasks need a parent too. If there’s no incoming request, create a new root span for each run and tag it with a schedule name.

Correlate logs with traces (and keep logs safe)

Traces tell you where time went. Logs tell you what happened and why. The simplest way to connect them is to add trace_id and span_id to every log entry as structured fields.

In Go, grab the active span from context.Context and enrich your logger once per request (or job). Then every log line points to a specific trace.

span := trace.SpanFromContext(ctx)
sc := span.SpanContext()
logger := baseLogger.With(
  "trace_id", sc.TraceID().String(),
  "span_id",  sc.SpanID().String(),
)
logger.Info("charge_started", "order_id", orderID)

That’s enough to jump from a log entry to the exact span that was running when it happened. It also makes missing context obvious: trace_id will be empty.

Keep logs useful without leaking PII

Logs often live longer and travel further than traces, so be stricter. Prefer stable identifiers and outcomes: user_id, order_id, payment_provider, status, and error_code. If you must log user input, redact it first and cap lengths.

Make errors easy to group

Use consistent event names and error types so you can count and search them. If the wording changes every time, the same issue looks like many different ones.

Add metrics that actually help you find issues

Metrics are your early warning system. In a setup that already uses Go OpenTelemetry tracing, metrics should answer: how often, how bad, and since when.

Start with a small set that works for almost every API: request count, error count (by status class), latency percentiles (p50, p95, p99), in-flight requests, and dependency latency for your DB and key third-party calls.

To keep metrics aligned with traces, use the same route templates and names. If your spans use /users/{id}, your metrics should too. Then when a chart shows “p95 for /checkout jumped,” you can jump straight into traces filtered to that route.

Be careful with labels (attributes). One bad label can explode costs and make dashboards useless. Route template, method, status class, and service name are usually safe. User IDs, emails, full URLs, and raw error messages usually aren’t.

Add a few custom metrics for business-critical events (for example, checkout started/completed, payment failures by result code group, background job success vs retry). Keep the set small and remove what you never use.

Exporting telemetry and rolling it out safely

Exporting is where OpenTelemetry becomes real. Your service has to send spans, metrics, and logs somewhere reliable without slowing requests.

For local development, keep it simple. A console exporter (or OTLP to a local collector) lets you see traces quickly and validate span names and attributes. In production, prefer OTLP to an agent or OpenTelemetry Collector near the service. It gives you a single place to handle retries, routing, and filtering.

Batching matters. Send telemetry in batches on a short interval, with tight timeouts so a stuck network doesn’t block your app. Telemetry shouldn’t be on the critical path. If the exporter can’t keep up, it should drop data rather than build up memory.

Sampling keeps costs predictable. Start with head-based sampling (for example, 1-10% of requests), then add simple rules: always sample errors, and always sample slow requests above a threshold. If you have high-volume background jobs, sample those at lower rates.

Roll out in small steps: dev with 100% sampling, staging with realistic traffic and lower sampling, then production with conservative sampling and alerts on exporter failures.

Common mistakes that ruin end-to-end visibility

End-to-end visibility fails most often for simple reasons: the data exists, but it doesn’t connect.

The issues that break distributed tracing in Go are usually these:

• Dropping context between layers. A handler creates a span, but a DB call, HTTP client, or goroutine uses context.Background() instead of the request context.
• Returning errors without marking spans. If you don’t record the error and set span status, traces look “green” even when users see 500s.
• Instrumenting everything. If every helper becomes a span, traces turn into noise and cost more.
• Adding high-cardinality attributes. Full URLs with IDs, emails, raw SQL values, request bodies, or raw error strings can create millions of unique values.
• Judging performance by averages. Incidents show up in percentiles (p95/p99) and error rate, not mean latency.

A quick sanity check is to pick one real request and follow it across boundaries. If you can’t see one trace ID flowing through the inbound request, the DB query, the third-party call, and the async worker, you don’t have end-to-end visibility yet.

A practical “done” checklist

You’re close when you can go from a user report to the exact request, then follow it across every hop.

• Pick one API log line and locate the exact trace by trace_id. Confirm deeper logs from the same request (DB, HTTP client, worker) carry the same trace context.
• Open the trace and verify nesting: an HTTP server span at the top, with child spans for DB calls and third-party APIs. A flat list often means context was lost.
• Trigger a background job from an API request (like sending an email receipt) and confirm the worker span connects back to the request.
• Check metrics for the basics: request count, error rate, and latency percentiles. Confirm you can filter by route or operation.
• Scan attributes and logs for safety: no passwords, tokens, full credit card numbers, or raw personal data.

A simple reality test is to simulate a slow checkout where the payment provider is delayed. You should see one trace with a clearly labeled external call span, plus a metric spike in p95 latency for the checkout route.

If you’re generating Go backends (for example, with AppMaster), it helps to make this checklist part of your release routine so new endpoints and workers stay traceable as the app grows. AppMaster (appmaster.io) generates real Go services, so you can standardize one OpenTelemetry setup and carry it across services and background jobs.

Example: debugging a slow checkout across services

A customer message says: “Checkout hangs sometimes.” You can’t reproduce it on demand, which is exactly when Go OpenTelemetry tracing pays off.

Start with metrics to understand the shape of the problem. Look at request rate, error rate, and p95 or p99 latency for the checkout endpoint. If the slowdown happens in short bursts and only for a slice of requests, it usually points to a dependency, queuing, or retry behavior rather than CPU.

Next, open a slow trace from the same time window. One trace is often enough. A healthy checkout might be 300 to 600 ms end-to-end. A bad one might be 8 to 12 seconds, with most of the time inside a single span.

A common pattern looks like this: the API handler is quick, the DB work is mostly fine, then a payment provider span shows retries with backoff, and a downstream call waits behind a lock or queue. The response might still return 200, so alerts based only on errors never fire.

Correlated logs then tell you the exact path in plain language: “retrying Stripe charge: timeout,” followed by “db tx aborted: serialization failure,” followed by “retry checkout flow.” That’s a clear signal you’re dealing with a few small issues that combine into a bad user experience.

Once you’ve found the bottleneck, consistency is what keeps things readable over time. Standardize span names, attributes (safe user ID hash, order ID, dependency name), and sampling rules across services so everyone reads traces the same way.