🎧 ▶ Listen: 5-minute briefing
▶ Play audiobook (Google Drive)
Locally synthesized AI audiobook (Qwen3-TTS)

If you plan to train on what your agents actually did, it is worth seeing how such a pipeline returns zero without saying so. Our exporter turned 486 runs into 468 episodes, every one of them schema-valid, and not one run that called a tool produced anything trainable.

Two kinds of defects: red dots on the surface, and a cube missing from the grid “Wrong” defects show up as red dots on the surface. “Absent” defects are the missing cube, invisible no matter how long you read the code.

Nothing failed

No error log. Exit code 0. The report said “468 written, 0 failed.”

The catch was that all 254 tool-using episodes came out carrying a truncated flag, and that flag makes the validator refuse the episode. The rule behind it is right: training on half an observation is worse than skipping the run. Here every observation was half.

Agent orchestration is tool execution by definition. A parent delegating to a child IS a tool call, one named agent. So the yield for orchestration trajectories was exactly zero.

Key-concept summary infographic 1 Infographic generated by NotebookLM from the sources.

Four defects that reading the code found

We read the code first, and four things came out.

The trace had no termination-reason field at all. The loop was producing typed values such as success, max_turns and timeout, and the function that finalises the trace was holding one in its hand, but the stored struct had nowhere to put it. Of 631 traces, zero contained that string.

The classifier searched for a different string than the one emitted. The loop emits max_turns with an underscore; the classifier looked for max turns with a space. A run stopped by the turn cap exported as “no failure”: a clean success. The exact pathology we wanted as a negative example, arriving with a positive label.

The outcome-label table was never joined. It held 402 rows and the assembler never read them.

The interface for fetching whole observations was declared, tested, and never injected in production. A full search of its callers returned four lines, all in a test file.

All four are real defects. Fixing all four and rerunning still produced zero.

Three defects that reading the code could not find

The same fact was written to two places and read from one

Tool-call records had two storage homes. The chat path writes them inside message metadata; the channel and connector path writes them to a dedicated column. The reader looked only at the column.

It took counting rows in both schemas to see it.

Storage home Written by Actual rows
unified_messages.tool_calls column channel path 0
metadata->'tool_calls' chat path 22

All traffic on this deployment is chat. So the assembler ran on the belief that no agent on this platform had ever used a tool, and that belief does not look wrong from outside. Runs without tool calls are perfectly ordinary. From the output alone you cannot tell a bug from a quiet day.

A display cap was deciding what the platform remembers

The chat surface truncates tool output at 1,000 characters. The tool-result store keeps a separate copy only for results above 16 KiB. The band between them was the problem: a result longer than 1,000 characters but under 16 KiB kept a truncation marker and no body anywhere.

Sorting the real stored results by size, 12 of the top 15 sat in that band.

Display truncation is a display concern, and it was deciding what the platform could remember.

Parallel calls collided on their identifiers

Tool-call ids were built as loop_<turn>_<tool>. Call the same tool twice in one turn and both records get the same id. When a result arrives, the handler scans backwards and attaches to the first match, so one record ends up holding another call’s output while its twin stays empty.

A reviewer raised this in the abstract and I filed it as a corner case. Then I ran an orchestrator for real and opened the message row.

loop_1_agent   no body
loop_1_agent   no body
loop_1_agent   850 bytes
loop_2_agent   no body
loop_2_agent   751 bytes

Delegate three investigations at once and all three are named agent. For an orchestrator that is not a corner case, it is the normal path.

The unique id was already in the model’s response. It was in scope at both dispatch sites. The callback simply never passed it along.

Results

Metric Before After
Episodes written 468 468
Episodes with a tool result 254 256
Of those, passing validation 0 5
Tool results carrying a body 0 73

Same command, same database, only the code differs. It is a deterministic replay over stored rows, so repeating it reproduces the numbers exactly. It is not a sampled estimate and carries no error bar.

It is 5 of 256, not 256 of 256. The rest are ephemeral evaluation runs that persist no session, or rows written before the display-cap fix whose bodies are unrecoverable. Nothing brings those back.

Passing is not the same as useful

With collection fixed we ran an orchestrator and took the first trajectory. The validator passed it. Here is what it contained.

tool_result  agent   [tool_use name=clarify id=chatcmpl-tool-9043d607de913a27]
tool_result  agent   [tool_use name=clarify]
tool_result  agent   [tool_use name="clarify" id="chatcmpl-tool-837c349fe54ce5cb"]

The subagent had no way to look anything up. The scenario asked for the current error rate and that deployment has no metrics tool. The 8B tried to call a tool that does not exist and emitted the markup as its final answer; the orchestrator concluded that it lacked the input to judge.

The validator checks whether an observation was cut off. It does not check whether the observation means anything. Schema checking and content checking are different jobs and you need both. Trained on as-is, this data would have taught a model to delegate into a void.

What generalises

Four were wrong defects: a label lands incorrectly, a value comes back empty. Three were absent defects: the data never arrives at all. Reading code finds the first kind. It does not find the second.

And the three each demanded a different kind of evidence. One needed row counts across two schemas. One needed the size distribution of real results. One needed running an orchestrator and opening the stored row by hand. None of them would have surfaced by reading the code longer.

Two of them were already marked done on the tracker. The fields existed. The store existed. The one adapter between them did not.

Key-concept summary infographic 2 Infographic generated by NotebookLM from the sources.

References

Tags: agent-platform, data-pipeline, debugging, observability

Categories:

Updated: