Documentation / Benchmark

Benchmark

Task structure, seeded environments, difficulty tiers, version history, and how to run the current WebAgentBench stack.

Task Structure

Each task is defined as a YAML document with four top-level sections. Together they specify what the agent must do, what the environment looks like, and how success is measured.

metadata

Identifies the task: id, page, difficulty, and the list of primitives the task exercises. The primitives list links each task to the cognitive taxonomy and is used by the results dashboard to break down performance by skill area.

instruction

A natural-language template shown to the agent. Placeholders of the form {{target.field}} are resolved against the seed's target object at runtime. This keeps the instruction human-readable in the YAML while remaining parametric across fixture variations.

Open billing settings and add the card ending in
{{target.last4}} with expiry {{target.expiry}}.

seed

Drives the fixture generator. Contains three sub-sections:

Key	Purpose
actors	Named entities with task roles: customers, sellers, senders, patients, moderators, and other environment-specific identities.
steps	Ordered seeder operations that build the initial environment state: create messages, bookings, payment methods, products, posts, or portfolio records.
distractors	Additional records or UI elements injected to test attention and filtering. Controlled by count, similarity, and placement parameters.

eval

Defines the scoring criteria as two lists. Positive checks are assertions that must be true for the agent to receive credit — each contributes 1 / total to the base score. Negative checks are guard-rail assertions — behaviours the agent must avoid — each carrying an explicit penalty subtracted from the base score.

Seeded Environments

Current benchmark environments include Amazon, Booking, Gmail, LMS, Patient Portal, Reddit, and Robinhood. Each environment exposes a live task surface backed by a seeded session API rather than a static mock page.

The exact state model differs by environment, but the contract is consistent: a task materializes targets, a backend seeds canonical state, the frontend renders that state, and evaluation checks the resulting records and audit trail after the agent acts.

Fixture Generation Pipeline

Stage	What happens
1. Actor and target generation	Seed roles are resolved to concrete identities and targets. The same role maps consistently within a session so instructions and evals remain deterministic.
2. Seeder steps	Each step is executed against environment-specific state stores to create the starting world for the task.
3. Distractor and variant injection	Optional distractors and degradation variants are layered in to stress specific primitives without changing the core task objective.
4. Session handoff	The backend returns the start path, resolved instruction text, and metadata needed by the frontend, evaluator, and trajectory tools.

Difficulty Tiers

Tasks are grouped into four difficulty tiers based on expected step count and the number of cognitive primitives exercised simultaneously. The tiers are calibrated against human baselines, not model performance.

Tier	Steps	Primitives	Description
Easy	5 – 10	Single-primitive	Isolated skill test. Straightforward instruction, minimal distractors, short action chain.
Medium	10 – 20	2 – 3 primitives	Requires combining skills such as memory with attention, or patience with backtracking.
Hard	20 – 35	Complex reasoning	Multi-step plans, conditional logic, and adversarial distractors designed to mislead.
Expert	30 – 50	All primitives	Full-coverage gauntlet. Requires near-perfect planning, resistance to distraction, and recovery from dead ends.

Version History

WebAgentBench has gone through ten versioned releases. Major milestones are listed below; see webagentbench/CHANGELOG.md for the full per-release notes.

Version	Pages	Summary
v1	10	Initial release. Core page set with basic task definitions and programmatic scoring.
v2 – v4	10	Iterative scoring refinement: negative checks added in v2, trajectory modifier in v3, calibrated weights in v4.
v5	12	Major redesign. Two new pages, unified YAML task format, and the actor/seed/distractor fixture pipeline.
v6 – v8	15	Environment expansion and harder interaction patterns, including frontier tasks designed to expose weak planning and attention behavior.
v9	15	Hardening release. Tightened eval criteria, distractor density increased, seed stability test suite added.
v10	15	Unified runtime release. Indexed accessibility-tree observations, shared task metadata, and trajectory tooling were standardized across environments.

Running the Benchmark

WebAgentBench evaluations are driven by webagentbench/agent_eval.py. It spins up a local FastAPI server, initializes seeded sessions, then runs the agent against the live DOM via Playwright.

# Evaluate the full benchmark
python -m webagentbench.agent_eval --model gpt-4o --provider openai

# Restrict to specific environments
python -m webagentbench.agent_eval --model gpt-4o --provider openai \
    --environments amazon booking reddit

# Run specific tasks only
python -m webagentbench.agent_eval --model gpt-4o --provider openai \
    --tasks booking_add_payment gmail_thread_detective

# With visible browser (useful for debugging)
python -m webagentbench.agent_eval --model gpt-4o --provider openai --no-headless

Results are written to results/webagentbench/results.json and can be visualised with python -m webagentbench.visualize results/webagentbench/results.json.