From Black Box to Glass Box: Teaching Developers Tracing with New Relic Segments

Based on the talk delivered by Nisansala Hettiarachchi and Chamod Perera at KCD Sri Lanka 2025.

The chaos begins

Every microservices architecture starts as a clean diagram on a whiteboard. Order Service calls Payment Gateway. Payment Gateway writes to the User Profile DB. Inventory Service talks to a couple of partner APIs. Six boxes, seven arrows, everybody nods.

flowchart TD
    O[Order Service] --> P[Payment Gateway]
    O --> I[Inventory Service]
    P --> DB[(User Profile DB)]
    I --> A[Partner APIs]

Eighteen months later that diagram is a hairball. Services call services that call partner APIs that call back into services you thought were leaf nodes. Nobody has the full picture in their head anymore, because the full picture no longer fits in a head.

flowchart LR
    O[Order] --> P[Payment]
    P --> DB[(Profile DB)]
    P --> F[Fraud]
    F --> O
    O --> I[Inventory]
    I --> A[Partner A]
    A --> B[Partner B]
    B --> I
    F --> N[Notifications]
    I --> N
    DB --> O

This is fine — right up until the moment something gets slow.

The blame game

You know how this goes. The alert fires, the incident channel fills up, and within ninety seconds the conversation has become an argument:

"It's the database." "The API looks fine on our side." "Is it a downstream service problem?" "Did something change in CI/CD?"

Six engineers, six theories, zero evidence. Everyone is defending their own service rather than finding the actual problem, because defending is the only thing you can do when nobody has data. The debugging strategy is essentially social: whoever argues most confidently wins, and the real culprit stays hidden.

The uncomfortable truth is that this isn't a people problem. It's an instrumentation problem.

The black box problem

From the outside, a service in a distributed system is a black box. Requests go in, responses come out, and everything in between is invisible.

flowchart LR
    subgraph BB["Black box — took 10s, where?"]
        direction TB
        q1["?"] --> q2["?"] --> q3["?"]
    end
    subgraph GB["Glass box — 9.5s is the external call"]
        direction TB
        a["auth · 2ms"] --> d["db · 450ms"] --> e["ext-api · 9.5s"]
    end

That gives you a specific and painful set of challenges:

• Invisible operations. You can see that a request took 10 seconds. You cannot see where those 10 seconds went.
• Unknown behavior in dependent services. Your service is only as fast as the slowest thing it waits on, and you have no visibility into what that is.
• Hard to pinpoint root causes. You can narrow the problem to a service, maybe. Narrowing it to a function, a query, or a specific external call is guesswork.
• Lack of visibility overall. Logs tell you what happened. Metrics tell you how much and how often. Neither tells you how a single request moved through your system.

What you want is to turn that black box into a glass box — same architecture, same complexity, but now you can see the wiring inside it, with timings attached to every segment of the path.

That's what distributed tracing does.

What is distributed tracing?

A trace is the complete story of one request as it travels across your system. A span (or segment, in New Relic's vocabulary) is one step in that story — an HTTP call, a database query, a chunk of internal computation.

Each span records where it started, how long it took, and who its parent was. Stack them up and you get a waterfall — and the slow one is impossible to miss:

flowchart TD
    G["API Gateway · 20ms"] --> U["User Service · 30ms"]
    G --> P["Product Service · 10ms"]
    G --> D["Database · 450ms"]
    classDef slow fill:#fee2e2,stroke:#ef4444,color:#991b1b;
    class D slow;

You don't need to argue about which service is slow anymore. The waterfall just shows you.

The critical piece that makes this work across service boundaries is context propagation. Service A generates a trace ID and passes it in the request headers to Service B, which passes it to Service C. Every span, in every service, carries the same trace ID. That's what lets a tracing backend reassemble one coherent timeline out of spans emitted by a dozen independent processes.

sequenceDiagram
    participant A as Service A
    participant B as Service B
    participant C as Service C
    A->>B: request (trace-id abc123)
    B->>C: request (trace-id abc123)
    Note over A,C: one trace, reassembled by trace-id

How New Relic segments work

New Relic's model has two layers:

• Transactions — the top-level unit of work. An inbound HTTP request, or a background job.
• Segments — the individual timed operations inside a transaction. External calls and datastore queries get segments automatically; anything you care about that the agent doesn't know about, you create yourself as a custom segment.

Auto-instrumentation gets you a long way, but it stops at the boundaries of your own code. The agent can see that your handler called httpbin.org and took 18 seconds. It cannot see that 16 of those seconds were spent inside your own computeSomethingExpensive() function, because that's just... Go. It looks like nothing to the agent.

Custom segments are how you fill in that gap. In the Go agent, you pull the transaction off the request context and open a named segment:

package orders
 
import (
	"context"
 
	"github.com/newrelic/go-agent/v3/newrelic"
)
 
func processOrder(ctx context.Context, orderID string) (Order, error) {
	txn := newrelic.FromContext(ctx)
	defer txn.StartSegment("Custom/ChamodInternalProcessing").End()
 
	enriched, err := enrichOrder(ctx, orderID)
	if err != nil {
		return Order{}, err
	}
	return applyPricingRules(ctx, enriched)
}

StartSegment names a custom segment; the defer ... .End() closes it when the function returns, so its duration is exactly the time spent inside. Use a Custom/<domain>/<operation> name so it groups and reports cleanly as its own metric over time.

Named segments show up in the trace waterfall alongside the outbound calls they make — so in-process time stops being a mystery block and starts being a labelled, measurable thing:

flowchart TD
    T["POST /orders"] --> C["Custom/ChamodInternalProcessing · 16s"]
    T --> E["External/httpbin.org · 2s"]
    classDef custom fill:#ede9fe,stroke:#7043EC,color:#4c1d95;
    class C custom;

Make sure distributed tracing is turned on so those segments link up across services. In Go you configure it when you construct the application:

app, err := newrelic.NewApplication(
	newrelic.ConfigAppName("chamod-service-dev"),
	newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
	newrelic.ConfigDistributedTracerEnabled(true),
)
if err != nil {
	log.Fatalf("new relic: %v", err)
}

Once that's in place, a trace spanning chamod-service-dev → nisansala-service-2-dev → httpbin.org renders as a single connected picture: the service map at the top showing who called whom, and the waterfall underneath showing exactly how the time was distributed.

Segments connect the dots — and attach time to every one of them.

Our journey: instrumenting everything that matters

We started with one service and worked outward. The first pass was mostly auto-instrumentation and configuration: turn on distributed tracing, confirm trace headers are being propagated, verify traces actually stitch together across service boundaries. The second pass was the interesting one — going through the hot paths and adding custom segments around the operations that carried real business meaning.

The change in team behaviour was faster than we expected. Once a service was instrumented end to end, the finger-pointing stopped. Not because people became nicer, but because there was nothing left to argue about. Somebody would paste a trace link into the incident channel and the conversation would jump straight from "whose fault is it" to "okay, why is that call taking 10 seconds."

Evidence beats opinion. That's the whole trick.

A real production example

Here's a case from our own dashboards.

The P99 latency panel for Service ABC showed an overall 1.969 s. Broken down by endpoint, most APIs sat where you'd want them — API-02 at 1.961 s, API-03 at 1.786 s, API-04 at 0.348 s, API-05 at 0.303 s, API-06 at 0.257 s.

And then API-01 at 10.002 s, glowing red.

The old workflow at this point would be: open the API-01 code, read it, form a hypothesis, add some logs, deploy, wait, repeat. Maybe an hour of work if you're lucky and know the codebase well.

Instead we opened a trace for a slow API-01 request and expanded it. Four spans — and one of them owns the entire budget:

flowchart TD
    A["API-01 (Service ABC) · 10.00s"] --> X["Service x → Downstream x · 1.55ms"]
    A --> Y["Service y → Downstream y · 10.00s"]
    X --> X2["Downstream x · 0.07ms"]
    classDef culprit fill:#fee2e2,stroke:#ef4444,color:#991b1b;
    class Y culprit;

The entire 10 seconds lived in one downstream call. Service x — the service everyone assumed was the problem, because it was the one that had been touched recently — was answering in under two milliseconds. Service y's downstream API was consuming the whole budget, and the trace even flagged it with error and slow-span indicators.

Time to identify the culprit: about thirty seconds. No code reading, no hypothesis, no log deploy. Just look at the picture.

Lessons learned

Trace everything that matters, not everything that's possible. This is the single most important rule. It's tempting to wrap every function in a segment, but you'll drown yourself in noise and pay for the privilege. Instrument the operations that would change your diagnosis if they were slow.

Pass context to the segment properly. This is where most instrumentation quietly fails. If you kick off work on another goroutine without carrying the transaction, the agent loses the thread and your segments vanish or attach to the wrong parent. For a background consumer, start a transaction and put it on the context; when you fan out to a goroutine, use txn.NewGoroutine():

func (c *Consumer) onMessage(msg Message) {
	txn := c.app.StartTransaction("queue:processMessage")
	defer txn.End()
 
	// Carry the transaction across the async boundary.
	asyncTxn := txn.NewGoroutine()
	go func(txn *newrelic.Transaction) {
		defer txn.StartSegment("Custom/Queue/HandleMessage").End()
		ctx := newrelic.NewContext(context.Background(), txn)
		handleMessage(ctx, msg)
	}(asyncTxn)
}

Segment external calls and async functions. These are where latency actually hides. Synchronous in-process code is rarely your 10-second problem; a partner API without a timeout very much is. Wrap outbound HTTP with newrelic.StartExternalSegment so those calls always show up as their own bars.

Use naming conventions. Custom/OrderService/ValidatePayment is searchable, groupable, and comprehensible six months from now. Custom/segment1 is not. Pick a convention — Custom/<domain>/<operation> works well — and enforce it in code review.

Combine traces with logs. Traces tell you where time went. Logs tell you why. Logs-in-context links them, so you can click from a slow span straight to the log lines that particular request produced. Neither signal is complete on its own.

Common pitfalls

Missing context propagation. Symptom: traces that stop at a service boundary, or orphaned single-span traces. Usually a proxy stripping headers, a hand-rolled HTTP client that doesn't wrap its request with newrelic.StartExternalSegment, or work started on a goroutine without txn.NewGoroutine().

Over-instrumentation. Symptom: traces with hundreds of spans that nobody can read, and a monitoring bill that raises eyebrows. Instrumentation has runtime cost and cognitive cost. Both are real.

Forgetting error segments. Failure paths are where you most need visibility, and they're the paths least likely to be instrumented — because when you wrote the happy path you were thinking about the happy path. Call txn.NoticeError(err) and make sure failed operations still close their segments (a defer-ed End() handles that for you):

seg := txn.StartSegment("Custom/OrderService/ChargeCard")
defer seg.End()
 
if err := chargeCard(ctx, order); err != nil {
	txn.NoticeError(err)
	return err
}

The bigger picture

It's easy to file tracing under "incident response tooling" and stop there. That undersells it. Tracing isn't just for debugging — it's for understanding how your system actually behaves, as opposed to how you believe it behaves.

• Performance optimization. You stop optimizing on instinct. The trace tells you which 5% of the code owns 80% of the latency, and you go fix that instead of the thing that was merely annoying to read.
• Dependency mapping. The service map is generated from real traffic, not from a diagram somebody drew in 2023 and never updated. It shows the dependencies you have, including the ones you forgot about.
• Developer onboarding. This one surprised us most. A new engineer can open a trace for a single request and see the entire execution path of the system, in order, with timings. That's a better introduction to an architecture than any wiki page, and it's automatically current.

The black box isn't a fact of distributed systems. It's a choice you make by not instrumenting. Segments are how you unmake it.

Nisansala Hettiarachchi is a Principal Engineer at Circles. Chamod Perera is a Software Engineer II at Circles, a CNCF and CDF Ambassador, and lead of Kubernetes Sri Lanka & KCD Sri Lanka.