List what the site can do
When source code is available, we inspect it and the site’s features to establish what can be read or changed and what counts as correct. Without source access, we assess which tasks can be covered.
COMPUTER USE, WITH A REFERENCEWEBMCP GROUND TRUTH · PILOT
Did your agent get it right?
How many tokens did it use?
Compare success, token usage, and time to completion.
We first establish the correct answer, then use WebMCP to attempt the task in fewer steps. Compare your computer-use agent: did it answer correctly, which extra actions did it take, and how many more tokens and seconds did it use?
FOR AGENT & MODEL TEAMS · COMPARE VERSION RESULTS
BOOKS / 5 RUNSOBSERVED
SAME SUCCESS. DIFFERENT COST.
Both correct. Different usage.
65 books → the three cheapest
Per-run resultsCUWebMCP
Full success5/55/5
Token92,55944,440
Time46.807 s29.855 s
52.0% ↓
Fewer tokens
36.2% ↓
Less time
WebMCP vs CU · five-run medians for tokens and time; an observed reference, not a theoretical limit.
01
What can be completed
Check which tasks can be completed
02
Token overhead
Count the total tokens used for the task
03
Time & completion
Check completion, then compare elapsed time
WHY WEBMCP
When the model can call a function,
it spends less time finding buttons.
Suppose the task is to find the three cheapest books on a site. CU typically browses pages, finds pagination controls, and reads prices. We can provide book lookup and price-reading tools through WebMCP. Then we compare whether both methods answer correctly and how many tokens and how much time each uses.
Tell the model how to use them
WebMCP lists callable functions, their inputs, and their results. The model can choose a function without finding its button on the page each time.
Spend less time searching and retrying
Each extra page observation or attempted action can add tokens and waiting time. Calling a function directly can reduce these steps. We measure the savings by running the task.
Run the reference without screenshots
If the required information and actions are available as text and tools, a tool-capable text model can attempt the task. This measures the tokens and time needed without locating buttons or interpreting screenshots.
How can each method complete the task?
ILLUSTRATION, NOT ACTUAL ACTION COUNTSCOMPUTER USE
- Read the page
- Find the control
- Click or type
- Check the change and retry if needed
- Check the answer
WEBMCP
- Read the available functions
- Choose and call a function
- Check the result
This answers a specific question: how many resources does the task need when the model already knows the available functions and how to call them? Comparing that result with CU reveals the extra time and tokens.
We must build and check these tools against the site’s features. WebMCP does not automatically understand all source code or guarantee the fewest steps. Read about WebMCP ↗
GROUND TRUTH + REFERENCE EXECUTION
Check the answer first.
Then compare resource usage.
An evaluation needs two things: a definition of the correct result, called ground truth, and a tested way to reach it with recorded actions, tokens, and time. Comparing speed alone can make an unfinished task look faster.
What counts as correct
For example: were the right three books selected, are their prices correct, and are they in the requested order? Check each requirement rather than whether an answer looks plausible.
DELIVERY: CORRECT RESULTS AND CHECKSWhat completing the task used
Record whether WebMCP completes the task, how many tokens it uses, and the time from start to finish. Then compare those figures with CU.
DELIVERY: COMPLETION, TOKENS, AND TIMEA successful WebMCP run shows that the task can be completed under those conditions. Its resource usage is an observed comparison point; another method may still be faster or use fewer tokens.
SUCCESS × TOKENS × TIME
Compare success,
tokens, and time.
Switch between three completed experiments to compare full-task success, total tokens, and elapsed time. Inspect completion and execution cost together, with unequal-work runs explained separately.
Computer UseWebMCP reference run
Full-task success
Successful runs / total runs
CU5/5
WebMCP5/5
Token usage
Total tokens · median per run
CU92,559
WebMCP44,440
Elapsed time
Seconds · median per run
CU46.807 s
WebMCP29.855 s
Same completion. Different execution costs.
Both groups passed 5/5 runs. WebMCP used 52.0% fewer median tokens and 36.2% less median time per run, exposing an efficiency gap beyond success rate.
Task: scan four pages and 65 books to find the three cheapest. These compare five-run medians, not theoretical minimum costs.Completed experiments, not a live evaluation. Total tokens are accumulated run usage, including repeated context; they are neither output length nor monetary cost. Results are shown per experiment, without pooling success rates.
Download comparison data ↓ONE TASK. TWO EXECUTION PATHS.
Give both groups the same task.
Check the same requirements.
Agree on page data, task requirements, and run limits. WebMCP uses tools; CU uses the agreed interface controls. Independently check both answers and count tokens and elapsed time.
SHARED TASK STATESame snapshot · same task · same success criteria
Run WebMCP
Function list → text-model calls → independent answer checks
Ground truth · Tokens · TimeRun your agent
Declared screenshots / accessibility capabilities → UI actions → output
Completion · Tokens · Time↓Compare completion, tokens, time, and specific errors
Keep the evaluator separate
The CU agent cannot access reference answers or grading interfaces. Reference and CU tracks declare their different capabilities.
Compare costs at equal completion
Report success alongside tokens and time. Declare model, budget, environment, and retries; retain resource usage from failed runs.
Validate the reference itself
Validate evaluators with known correct, incorrect, and partial states. Separate environment failures from agent failures.
COMPLETION BEFORE EFFICIENCY
In this HN run,
what was correct and what was missing?
In one HN CU run, all 100 text entries were correct, but only 25 reply relationships were confirmed. Ground truth must check the complete deliverable for resource comparisons to mean anything.
COMPLETED EXPERIMENT · SCORING EXAMPLEHN / RUN 02
Reply relationships
VERIFIED 25/100Verified correctUnverified
- Requirement
- Every comment must have a verified parent, or be verified as top-level. An unknown relationship is incomplete.
- Observation
- 25 relationships were confirmed: 12 top-level and 13 with verified parents. The other 75 were explicitly unknown.
This criterion did not pass. With 75 relationships unverified, the complete task did not pass.
Data comes from a completed HN experiment verified independently against the page. This interactive example explains scoring; it does not run an agent. The WebMCP evaluator is the pilot design described below.
Download sample data ↓YOUR EVALUATION PACKAGE
What will you receive
from the evaluation?
Build a task suite for agent, model, or enterprise automation teams. Compare versions with consistent criteria and see whether capability improves—and what that improvement costs.
01
Tasks & ground truth
Site capability maps, task snapshots, target fields, business rules, and validated outcome evaluators.
02
WebMCP reference baseline
Validated short paths, text-model conditions, per-run completion, tokens, and time; explicit mapping and setup costs.
03
CU gap report
Success, resource gaps, failed-run usage, and task-level evidence for version comparisons.
04
Budget & regression evaluation
Repeat evaluation under agreed budgets; track success and resource usage with environment and version records.
Currently a browser-task pilot. Task scope, reference feasibility, delivery format, maintenance, and pricing are agreed per engagement.
READ THE METRICS CORRECTLY
Questions about
the evaluation
Does WebMCP always use the fewest steps?
No. Knowing the available functions and how to call them can reduce searching and retries. We record methods that have been tested, without claiming they are the only or best methods. The fewest steps may not mean the fewest tokens or the least time.
Why can a text model take part?
If information is returned as text and actions are available through tools, the model does not need to interpret screenshots. It must still understand the task, choose functions, and check the answer. Tasks that require judging images need a separate evaluation method.
Can this show where an agent can improve?
It provides a tested comparison. If WebMCP finishes and CU does not, a working method existed under those conditions. If both finish, compare actions, tokens, and time. Differences may come from available information, tools, planning, or interface actions—not vision alone.
What does ground truth mean?
It is the evidence used to judge whether an answer is correct: for example, which three books should be selected, their prices, and their order. We check these answers independently rather than assuming a tool result is correct.
What are tokens? Are they the same as cost?
Tokens are units used to count the content a model processes as input and output. We show accumulated run usage, including repeated content. Tokens do not directly equal cost: model prices, caching, and billing rules also affect the bill.
How do you compare the groups fairly?
Use the same page data, account permissions, and task requirements where possible. Record models, tools, retries, and other conditions. Report the resources spent preparing tools and documenting site functions separately. An unfinished task is not a faster completion; different page states or task sizes do not support a direct speedup claim.
What happens after I apply?
We will contact you at the email you provide to agree on tasks, versions, deliverables, pricing, and timing. Applying does not charge you or activate a service.
BENCHMARK AGAINST WHAT’S POSSIBLE
Want to evaluate your agent?
Tell us which tasks it should do.
Tell us the tasks and versions to compare, and whether you want higher success, fewer tokens, or less time. We will discuss the evaluation scope with you.
- Define tasks and success criteria
- Establish correct answers and run WebMCP for comparison
- Compare each version’s results, tokens, and time