Agent Bench · Author & Maintainer
Comparing agent frameworks on the same tasks.
A benchmark that runs the same tasks across agent frameworks and compares tool-calling reliability, orchestration control, and observability, with every run published.
- Approach
- Fixed task, varied framework
- Unit of measure
- End-to-end agent tasks
- Results
- Transparent & reproducible
- Task
- intake-triage · 9 runs
Methodology.
Benchmarks that vary the model tell you about the model. Agent Bench fixes the task and varies the framework, scoring what matters when choosing one: reliability of the tool loop, the control you get over orchestration, and what it costs to build, read, and extend. Each framework implements the same end-to-end workflow; every run is published with its configuration so results can be reproduced and challenged.
Current published runs.
| Framework | Runtime model | Task | Framework | Final | Outcome |
|---|---|---|---|---|---|
| Anthropic Agent SDK | sonnet-4.6 | 4.75 | 4.63 | 4.69 | Pass |
| OpenAI Agents SDK | gpt-5.4 | 4.75 | 4.16 | 4.46 | Pass |
| LangGraph | gemini-3.1-pro-preview | 4.25 | 3.85 | 4.05 | Pass |
| Pydantic AI | gpt-5.4 | 3.50 | 4.43 | 3.97 | Partial |
| Google ADK | gemini-3.1-pro-preview | 3.50 | 4.11 | 3.81 | Partial |
| LangGraph | gpt-5.4 | 2.48 | 4.20 | 3.34 | Partial |
| Pydantic AI | gemini-3.1-pro-preview | 2.48 | 4.20 | 3.34 | Partial |
| CrewAI | gpt-5.4 | 2.48 | 3.80 | 3.14 | Partial |
| AutoGen | gpt-5.4 | 2.48 | 3.80 | 3.14 | Partial |
Task: intake-triage, scores out of 5. Each row links to a run directory in the repository, with the full tool-call metrics.
Eight dimensions.
Tool-calling reliability
Whether the tool loop completes long, multi-step tasks without dropped or malformed calls.
Orchestration primitives
How much control the framework gives over sequencing, parallelism, and state.
Human review quality
How well do checkpoints and approvals fit real workflows?
Ease of implementation
Time and friction to a working end-to-end task.
Lines of code required
How much glue code the framework makes you write.
Readability
How easily a new engineer can follow what the agent will do.
Extensibility
Cost of adding tools, models, and custom behavior.
Observability
Tracing, debugging, and run inspection support.