Agent Bench · Author & Maintainer

Comparing agent frameworks on the same tasks.

A benchmark that runs the same tasks across agent frameworks and compares tool-calling reliability, orchestration control, and observability, with every run published.

Approach
Fixed task, varied framework
Unit of measure
End-to-end agent tasks
Results
Transparent & reproducible
Task
intake-triage · 9 runs
Methodology

Methodology.

Benchmarks that vary the model tell you about the model. Agent Bench fixes the task and varies the framework, scoring what matters when choosing one: reliability of the tool loop, the control you get over orchestration, and what it costs to build, read, and extend. Each framework implements the same end-to-end workflow; every run is published with its configuration so results can be reproduced and challenged.

Scoreboard

Current published runs.

FrameworkRuntime modelTaskFrameworkFinalOutcome
Anthropic Agent SDKsonnet-4.64.754.634.69Pass
OpenAI Agents SDKgpt-5.44.754.164.46Pass
LangGraphgemini-3.1-pro-preview4.253.854.05Pass
Pydantic AIgpt-5.43.504.433.97Partial
Google ADKgemini-3.1-pro-preview3.504.113.81Partial
LangGraphgpt-5.42.484.203.34Partial
Pydantic AIgemini-3.1-pro-preview2.484.203.34Partial
CrewAIgpt-5.42.483.803.14Partial
AutoGengpt-5.42.483.803.14Partial

Task: intake-triage, scores out of 5. Each row links to a run directory in the repository, with the full tool-call metrics.

Dimensions

Eight dimensions.

01

Tool-calling reliability

Whether the tool loop completes long, multi-step tasks without dropped or malformed calls.

02

Orchestration primitives

How much control the framework gives over sequencing, parallelism, and state.

03

Human review quality

How well do checkpoints and approvals fit real workflows?

04

Ease of implementation

Time and friction to a working end-to-end task.

05

Lines of code required

How much glue code the framework makes you write.

06

Readability

How easily a new engineer can follow what the agent will do.

07

Extensibility

Cost of adding tools, models, and custom behavior.

08

Observability

Tracing, debugging, and run inspection support.