Threads & Tracing — keeping memory of, and watching, your agent

1 Threads — remembering across turns

First, recall: the model is stateless

Back in Session 1 we learned the big, surprising truth: an LLM is stateless. It has no memory between requests. Each time you call the model, it reads what you send, produces an answer, and then completely forgets the whole exchange. ChatGPT only feels like it remembers because the app secretly re-sends the entire conversation with every new message.

So if the model has no memory, who keeps the conversation? Something on the outside has to store the history and re-send it each turn. In the Agent SDK, that "something" is a thread.

What a thread actually is

A thread (sometimes called a session or conversation) is a persistent object that holds everything about one ongoing conversation across multiple turns. A turn is one round of "user says something → agent responds." A thread accumulates turns.

Concretely, a thread holds:

How the SDK uses a thread, step by step

When you send a new message on an existing thread, the SDK does roughly this:

Notice step 5: the reply is written back into the thread. That's what makes the next turn aware of this one. Without a thread, you'd be back to the forgetful friend.

2 What is tracing?

What tracing captures

Tracing means automatically capturing a structured, time-ordered record of everything that happened during one agent run (one full attempt to handle a request, which may involve many model calls and tool calls — the agent loop from Session 3). A good trace records:

• Every model call — the exact prompt sent, the response, the model used.
• Every tool call — which tool, what arguments went in, what came back.
• Inputs and outputs at each step, so you can read the data flowing through.
• Timing — how long each step took (latency).
• Token usage & cost — how many input/output tokens each model call burned.
• Errors — anything that failed, with the error message.

Two key words: span and trace

These two words are the whole vocabulary of tracing, so let's nail them:

So a trace is made of many spans, often nested like a tree. Reading a trace top-to-bottom is like replaying the agent's whole thought process in slow motion.

TRACE: "weather in Paris" run  (total 2.3s, 1,840 tokens)
├─ SPAN model_call #1            (0.6s)  LLM decides to call a tool
│    input:  user question + system prompt
│    output: tool_call get_weather(city="Paris")
├─ SPAN tool_call get_weather    (0.9s)
│    input:  {"city": "Paris"}
│    output: {"temp_c": 14, "rain": true}
└─ SPAN model_call #2            (0.8s)  LLM writes the final answer
     input:  history + tool result
     output: "It's 14°C and raining in Paris — yes, bring an umbrella!"

Why you can't debug agents with print statements alone

When you wrote normal programs, sprinkling print() statements was often enough. Agents break that approach for several reasons:

• They're non-deterministic. The same input can produce different steps each run (recall temperature, Session 1). A bug might appear only 1 time in 20 — a print you happened to add won't be there next time it strikes.
• They loop and branch. An agent might call a tool, reconsider, call another, loop back (Session 3's agent loop). Flat printed lines make it almost impossible to see which step led to which.
• The data is huge. Full prompts, tool outputs, and histories are enormous. Dumping them as raw prints turns your console into an unreadable wall of text.
• You need structure and timing. "Which step was slow? How many tokens did step 3 cost?" Prints don't measure that; spans do, automatically.
• It happens in production. You can't add prints to a bug a user hit yesterday. But if tracing was on, the trace is already saved and waiting for you.

3 Debugging agent workflows

The common ways agent runs go wrong

From Session 3 you already met agent failure modes. Traces are how you actually spot which one happened. The usual suspects:

A step-by-step debugging walkthrough

Here's the reliable routine for using a trace to find where a run went wrong. Always work from the symptom back toward the cause:

1. Reproduce or locate the run. Find the trace for the exact failed request (by thread_id, user, or timestamp). With tracing on, it's already saved.
2. Read the final answer span first. Confirm what actually went wrong — wrong fact? Cut off? Refused? This tells you what kind of failure to hunt for.
3. Walk backwards through the spans. Look at the step just before the final answer. Was the input to that step already wrong? If yes, the bug is earlier; keep going back. If the input was good but the output was bad, you've found the broken step.
4. Inspect inputs vs outputs at the suspect span. This is the heart of it: at each span, ask "given this input, is this output reasonable?" The first span where a good input produced a bad output is your culprit.
5. Check tool calls closely. Right tool? Right arguments? Did the tool return what you expected, or an error/empty result the agent then misused?
6. Check for loops. Scan for repeated spans. If the agent kept retrying, look at why each attempt "failed" in its eyes.
7. Form a fix and re-run. Adjust the prompt, tool description, or logic — then run the same input again and compare the new trace to the old one.

4 Observability practices

The three pillars: logs vs metrics vs traces

Observability is the practice of being able to understand what your system is doing from the outside, without guessing. It rests on three kinds of signal — people often confuse them, so here's the clean version:

They work together: a metric tells you something is wrong ("error rate jumped"), logs tell you roughly what ("weather tool timing out"), and a trace lets you see one bad run in full detail to confirm the cause.

The numbers that matter for agents

For LLM agents specifically, three metrics dominate everything:

Tools you'll hear about

You don't build observability from scratch. There are dedicated platforms — at a high level:

• SDK tracing dashboards — many agent frameworks (including the Agent SDK) ship a built-in trace viewer. You turn tracing on, run your agent, and view the span trees in a web UI.
• LangSmith and similar LLM-observability tools — specialised dashboards for inspecting traces, comparing prompt versions, tracking token cost and latency, and running evaluations over many runs.
• General observability stacks (the kind used for any software) can ingest logs and metrics too, but the LLM-specific tools understand prompts, tokens, and tool calls out of the box.

What to monitor in production

Once your agent is live and real users depend on it, watch at minimum:

• Latency (and percentiles — the slowest 5% of users, not just the average).
• Token cost per run and total daily spend (catch runaway loops before the bill does).
• Error rate and tool-failure rate (which tools fail most?).
• Loop / step-count — how many model+tool steps an average run takes; sudden jumps mean trouble.
• User feedback signals — thumbs up/down, retries, abandonment.
• A sample of full traces kept for spot-checking and for debugging reported bugs.

5 Failure investigation: a case study

The case: a confidently wrong answer

You run a customer-support agent for an online store. It has a tool lookup_order(order_id) that fetches order details, and it uses RAG retrieval over the store's policy documents. A customer complains:

The investigation process

Here's the repeatable flow a good investigation follows — the same skeleton every time:

Following the clues

We pull up the trace for that exact run and walk it backwards, applying the debugging routine from Topic 3:

★ Putting it all together

This session gave you the two things that turn an agent from a fragile demo into something you can run for real. Here's the one-paragraph story that ties it together:

Quick self-check

📚 References & Further Reading

Class material

• 📄 Original course notes / handout (source sheet) — open the shared GenAI class material for this session.
• Class handout: "Threads & Tracing in Agent SDK".

Papers, docs & deep dives

• 📘 LangSmith documentation — tracing & observability — inspecting traces, comparing prompt versions, and tracking cost/latency.
• 📘 OpenAI Agents SDK — Tracing — built-in traces and spans for agent runs (Topic 2).
• 📘 OpenTelemetry documentation — the open standard for spans, traces, logs, and metrics (Topic 4).