Contents
AgentDebugX: Debug LLM Agents Like a Pro — Open-Source Toolkit for Failure Observability, Attribution & Recovery
1. Introduction
LLM agents fail in ways that are uniquely hard to debug. The error you see — a malformed tool call, a timeout, a crash — is rarely the root cause. The real culprit often lies several steps earlier: a bad plan, a hallucinated argument, a misread from memory. By the time you spot the crash, the agent’s chain of reasoning is already gone.
AgentDebugX is an open-source, MIT-licensed toolkit from the University of Illinois at Urbana-Champaign designed to close that gap. It gives you a structured loop — Detect → Attribute → Recover → Rerun — that turns opaque agent failures into auditable, fixable events.
pip install agentdebugxOne command and you have a debugger that works across LangGraph, CrewAI, OpenAI Agents SDK, OpenTelemetry, and any custom agent framework.
2. The Core Problem: Why Agent Failures Are Hard to Debug
Traditional observability tools — LangSmith, Langfuse, Arize — show you what happened: traces, spans, token counts, latency. But they don’t tell you why it happened or how to fix it.
- Error surface ≠ root cause. The agent plans wrong → a later tool call fails with a 404. You see the 404, but the real failure was the planning step 10 seconds earlier.
- Multi-agent cascades. Agent A hands off bad state to Agent B. Agent B crashes. Observability blames Agent B — but Agent A caused the problem.
- The attribution gap. Research benchmarks for blame assignment exist (e.g., Who&When), but no production-ready infrastructure connects diagnosis to recovery. Developers are left stitching together log scrapers and manual replay scripts.
AgentDebugX fills this gap by providing a closed-loop infrastructure that doesn’t stop at “what broke” — it answers “who broke it, why, and how do we fix it.”
3. AgentDebugX Closed-Loop Architecture
The entire system revolves around four stages, illustrated here with a booking agent that tries to reserve a flight but fails on checkout:
Stage 1: Detect
Rule packs catch mechanical failures — malformed tool calls, no-progress loops, invalid output schemas, infinite retries. An optional LLM judge handles ambiguous cases (e.g., “the response is technically valid but semantically wrong”).
AgentDebugX recognizes 19 failure modes spanning:
- Planning (duplicate steps, contradictory goals)
- Memory (context drift, hallucinated recall)
- Tool use (wrong arguments, missing permissions)
- Verification (self-check passes but output is wrong)
- Coordination (agent A waits for B who already terminated)
Stage 2: Attribute
Traces the symptom back to its root cause. AgentDebugX offers multiple attribution strategies, from cheap heuristics (fast, ~10ms) to DeepDebug (detailed, ~1.6x tokens). Crucially, it returns ranked hypotheses with confidence scores, not a single blame assignment.
Stage 3: Recover
Proposes concrete fixes. The native option is DeepDebug’s own evidence-backed correction. Alternatives include Reflexion, CRITIC, Self-Refine, and AutoManual. All suggestions are gate-kept — they require human approval or a policy check before any code or prompt is modified.
Stage 4: Rerun
Applies the fix from a checkpoint and produces a new trajectory. Both the original and repaired branches are preserved for side-by-side comparison. If the rerun fails, it re-enters the Detect stage — the loop continues until the agent succeeds or the human operator halts it.
4. DeepDebug: The Multi-Turn Root-Cause Agent
A single-pass attribution pass isn’t enough. A global read of the entire trajectory anchors on the loudest symptom (the crash) and misses the subtle planning error three steps before. A step-by-step walk loses the big picture.
DeepDebug solves this with a 4-stage investigation protocol:
- Global read — ingests the full trajectory and surfaces an initial candidate
- Structure-guided investigation — walks the handoff cascade (multi-agent mode) or bisects the timeline (single-agent mode)
- Cross-examination — when candidates disagree, runs side-by-side adjudication between them
- Diagnosis & suggestion — produces a structured report: responsible agent + step, evidence trail, explanation, and a concrete fix suggestion
Every inspection DeepDebug performs is recorded — you can audit exactly why it blamed Agent A over Agent B. This makes the output suitable for regulated environments where decisions must be explainable.
5. Developer Experience: API, CLI, Web Console
Python API (Context Manager)
The simplest way to use AgentDebugX is as a context manager wrapping your agent’s execution loop:
from agentdebug import AgentDebug, EventType
debugger = AgentDebug()
with debugger.trace(goal="Book flight", framework="my-agent") as trace:
trace.record(EventType.PLAN, agent_name="planner", step_index=1, output="Search fares")
trace.record(EventType.TOOL_RESULT, agent_name="browser", step_index=3, error="Checkout failed")
report = trace.analyze()
print(report.summary)CLI Workflow
Post-hoc debugging is just as easy. Ingest any existing trace (LangGraph, CrewAI, raw JSON), diagnose it, and rerun:
# Ingest a trace from any supported format
agentdebug ingest raw_trace.json --format auto --out trace.json
# Diagnose with DeepDebug
agentdebug diagnose trace.json --mode deepdebug --out report.json
# Rerun from checkpoint
agentdebug rerun report.json --trajectory trace.json --out rerun.jsonWeb Console
Running agentdebug serve launches a FastAPI dashboard at localhost:7777 where you can visually inspect traces, click through attribution reports, and approve recovery suggestions.
6. Error Hub: From Incidents to Organizational Learning
AgentDebugX’s Error Hub stores scrubbed trajectory-diagnosis-repair bundles. This turns every debugged incident into a learning artifact:
- CI regression fixtures — automatically add failing trajectories to your test suite
- Cross-team debugging memory — share anonymized failure patterns across teams
- Benchmark corpora — build datasets from real failures to benchmark new models or frameworks
The system is opt-in and local-first. Pattern-based PII redaction strips sensitive data (API keys, emails, PII) before any bundle leaves the local store. DeepDebug can also retrieve similar past cases to seed its hypotheses — if a planning bug looks like something the team fixed last week, it surfaces that fix.
7. Benchmark Results
AgentDebugX was evaluated on the Who&When attribution benchmark and the GAIA multi-step reasoning suite. DeepDebug consistently outperforms single-pass baselines:
| Metric | AgentDebugX (DeepDebug) | Best Baseline | Improvement |
|---|---|---|---|
| Who&When Agent Accuracy | 56.0% | 47.8% | +8.2% |
| Who&When Strict A+S | 28.8% | 21.7% | +7.1% |
| GAIA Repairs (of 73 failed) | 13 | 4-6 | 2x-3x |
| Overall GAIA Accuracy | 63.6% | 55.8% | +7.8% |
DeepDebug costs approximately 1.6x tokens vs. a single attribution pass, but its localization accuracy is substantially better on long traces (>40 events) — the exact case where single-pass methods fail hardest.
8. Getting Started
Install the core package and optionally add integrations:
pip install agentdebugx
# With optional integrations:
pip install "agentdebugx[ui]" # local web dashboard
pip install "agentdebugx[langgraph]" # LangGraph adapter
pip install "agentdebugx[crewai]" # CrewAI adapter
pip install "agentdebugx[all]" # everythingSupported frameworks: LangGraph, CrewAI, OpenAI Agents SDK, OpenTelemetry, raw ReAct, GAIA/Open Deep Research, OSWorld. If your agent produces traces, AgentDebugX can ingest them.
9. Conclusion
AgentDebugX closes the loop that other observability tools leave open: from observing a failure to attributing its root cause, recovering with a concrete fix, and verifying that the fix works. It is MIT-licensed, community-driven, and ready to integrate with your existing agent stack today.
Project: github.com/AgentDebugX/AgentDebugX
Paper: arxiv.org/abs/2607.18754
