Debugging approach / Agent reliability
The agent worked yesterday. What changed?
Replay failing tasks, locate the first wrong decision, and separate model failures from context, retrieval and tool failures.
The problem
An agent starts repeating tool calls, losing constraints or failing tasks it used to complete. A final success rate tells you there is a regression. It does not tell you which change caused it.
How we investigate
- Reconstruct the failing run: model and prompt versions, retrieved context, tool responses, state transitions and retry decisions. Compare it with a successful trace for the same task.
- Replay a fixed task set with captured tool responses, then with live tools. This separates a harness or model regression from a changed API, index or external state.
- Find the first point where the run diverges. Check missing context, truncation, stale retrieval, tool-schema changes and recovery logic before changing the model.
- Change one suspected cause at a time. Check task completion alongside repeated calls, latency and cost; retain successful tasks as regression checks.
What the engagement produces
A reproducible failure, an explanation of the responsible component, and a tested harness change where the cause is established. The replay set becomes a regression gate for the next model, prompt or tool update.