Documentation / Architecture

Architecture

WebAgentBench is organized around a seeded-session runtime: YAML tasks define goals and checks, Python backends materialize environment state, and React frontends expose live UIs that agents operate through the browser.

Runtime Layers

The benchmark is split into four layers so task authoring, state management, browser interaction, and scoring can evolve independently.

Layer	Responsibility	Examples
Task registry	Defines instructions, seed targets, positive checks, and negative checks.	tasks/<env>/*.yaml
Session backend	Materializes seeded state, serves session APIs, and records audit events.	backend/routes/, backend/seeders/
Environment UI	Renders the live task surface and sends user actions to the backend.	environments/<env>/src/
Harness and scoring	Runs agents, captures trajectories, evaluates outcomes, and generates reports.	agent_eval.py, evaluator.py, visualize.py

Observation Contract

Agents act on indexed accessibility trees extracted from the live page. Each visible element receives a stable numeric reference within the current observation, and actions point back to those references.

Observations are rendered as indented trees with numeric reference indices:

[1] button "Settings"
[2] textbox "Search"
[3] option "Option A"

Actions are JSON-shaped commands that reference those elements by index:

// click

{"action": "click", "ref": 1}

// fill

{"action": "fill", "ref": 2, "value": "hello"}

Session Lifecycle

Each benchmark run follows the same seeded-session lifecycle, regardless of environment.

Materialize

Resolve seed targets

→

Create session

Seed backend state

→

Interact

Drive the live UI

→

Evaluate

Score outcomes

Reset reuses the same task contract to rebuild the session deterministically, while evaluate inspects the final server state, audit log, and any task-specific client-side evidence required by the YAML checks.

State And Audits

The benchmark grades outcomes against canonical backend state, not just what appears on screen. Backends also emit structured audit events so evals can detect harmful actions, ordering mistakes, and forbidden side effects.

Signal	Used for
Server state	Source of truth for emails, bookings, carts, payment methods, portfolio state, forum posts, and other environment-specific records.
Audit log	Verifies that required actions occurred and negative checks can penalize specific prohibited events.
Client benchmark state	Optional evidence for interaction-sensitive tasks, such as search terms, navigation state, or transient UI choices not stored in the backend model.

Trajectories And Visualization

Evaluation runs emit step-by-step trajectories that include observations, parsed actions, targets, and terminal outcomes. These artifacts power the launcher replay screen and the standalone visualization workflow.

results/*.json

Stores run summaries, task-level scores, and captured trajectories for each episode.

visualize.py

Generates replayable HTML visualizations and embedded trajectory views for debugging.

/launch

Surfaces the current benchmark, evaluation controls, saved trajectories, and interactive replay from the same backend session contract.

Model Providers

The evaluation harness supports multiple model backends behind the same task runtime.

openai--provider openai

OpenAI models through the standard API.

gemini--provider gemini

Google Gemini models through the Gemini API.

bedrock--provider bedrock

AWS Bedrock-hosted models for managed evaluation runs.

vllm--provider vllm

OpenAI-compatible self-hosted endpoints, including local or cluster-backed serving.