How I use benchmarks to improve an agent

DeepDeck is a desktop client built on DeepSeek Harness. A harness is the environment around the model: what it can observe, which tools it can call, how errors are reported, and how it checks that an action worked. The model makes decisions; the harness gives it ways to carry them out.

The benchmark covered 49 tasks across eight local websites, with WebMCP on and off for each model. WebMCP exposes operations such as looking up an order or filtering a directory as callable tools. With it disabled, the agent completes tasks through other browser operations. The paired runs help show whether the tools actually help.

MiMo V2.6 Flash was inexpensive, but slower, and enabling WebMCP did not save as many steps as I expected. I initially attributed this to inefficient tool use, then checked the calls to identify the cause.

I examined tool discovery, selection, arguments, and result verification to locate repeated errors and delays.

MiMo was discovering and calling WebMCP tools. One recurring mistake was specific: it used the tool catalog digest as an individual tool revision. The digest identifies the catalog for caching; the revision identifies a tool version. They are not interchangeable.

The old interface required the model to copy the tool name, frameId, documentId, and revision where applicable, alongside the business inputs. It could understand that it needed to check an order and still have its call rejected for copying the wrong identifier. Discovery and retries then consumed more time and tokens.

The interface contributed to these errors by requiring models to supply identifiers that code could store and validate. Luna made fewer mistakes, but requiring it to copy those identifiers was still unnecessary.

DeepDeck now assigns each discovered tool a short reference called toolRef and saves its document, frame, and version. The model selects a tool from the list and sends its reference with the business inputs. DeepDeck retrieves and validates the saved identifiers instead of asking the model to copy them.

The model still decides which tool to use and what to ask it to do. It no longer has to transcribe all the routing identifiers for every call.

The shorter interface preserves the checks. A page or tool change invalidates the old reference; it never silently redirects an action to a new tool. Legacy explicit-identity calls remain strictly validated.

Errors also distinguish an action that was not dispatched from one whose outcome is unknown. The first calls for rediscovery. The second calls for checking business state: if submitting an order causes navigation, a missing return value must not trigger an automatic second submission.

The change lives in DeepDeck’s browser plugin, without modifying upstream Harness source. Unit tests and a real Electron check covered execution, navigation invalidation, and tool-version replacement before the model retest.

I selected six WebMCP ON tasks with frequent call errors: an access boundary, directory filtering, a service lookup, course enrollment, an event location, and an order lookup. Each model ran each task three times: 36 formal attempts across MiMo and Luna.

These tasks had ultimately passed in the old run too. The failures occurred during tool calls. This experiment tests whether the agent can complete the same task with fewer errors, steps, seconds, and dollars. It does not demonstrate a higher task success rate.

The formal comparison checked the corpus, model configuration, context window, maximum-output configuration, and available tool-name set against each model’s baseline. Timing includes agent execution, excluding environment startup, resets, and scoring.

The old baseline has one attempt per task, while the new run has three. This is a historical comparison, not a contemporaneous randomized A/B test. It provides evidence for an improvement, but cannot attribute every difference to the code change.

All 36 formal attempts passed. Time, steps, and cost below are totals for completing the six-task suite: one old run versus the average of three new runs. The WebMCP error rate uses all calls in each group, not the number of failed tasks.

MiMo went from 20.04 to 5.75 minutes, from 108 to 60.3 steps, and used about 53% less estimated cost. Luna also made fewer tool errors, but its total time barely changed.

For the Yoga Class existence check, old MiMo took 150.5 seconds and made six WebMCP call errors. The three new attempts had none of those errors and averaged 14.0 seconds.

The old MiMo order task took almost 590 seconds, making the aggregate time reduction look larger. Even excluding that task, the other five went from 612.5 to an average of 304.8 seconds, roughly half the time. The single-attempt historical baseline remains a limitation.

The access-boundary task remained difficult. MiMo’s average steps increased from 25 to 26.7, and cost rose. Luna went from 24.5 to an average of 55.5 seconds on that task, and also became slower on course enrollment. The change did not resolve repeated exploration or delayed stopping.

Across both models, 14 remaining WebMCP errors involved stale references and eight involved navigation during an operation, leaving the outcome unknown. Those traces identify concrete follow-ups: refreshing discovery after navigation and confirming business state after an action changes the page.

I also inspected tool-return evidence for the course and order tasks. Enrollment and the first lesson had to actually occur; the returned order reference had to be checked with the same email. A final answer saying “done” is not sufficient evidence.

Costs use recorded uncached input, cached input, output, and cache writes where applicable, at the original report’s fixed USD rates. Luna costs are API-equivalent estimates, not incremental ChatGPT subscription charges. Missing usage cannot be treated as zero.

MiMo’s input cache-hit share fell from 92.54% to 90.50%; Luna’s changed from 76.19% to 76.07%. The lower overall cost did not come with a higher aggregate cache-hit share. Repeated runs can still affect server-side caching, so I would not attribute every cost difference to the code.

Nine initial MiMo calibration attempts did not match the old model configuration and were archived separately. Three Luna order attempts hit expired evaluator credentials. Their original failures were retained, then the entire group was rerun after refreshing evaluator authentication, rather than selecting a successful attempt.

The formal comparison has 36 attempts; the experiment recorded 48. Excluded runs also consumed resources, so the ledger includes them. All recorded runs together cost approximately $0.244 on an API-equivalent basis.

The benchmark identified repeated argument errors and provided tasks for checking whether the interface change reduced them.

For subsequent iterations, I will use these steps:

Before merging the fix into main, I saved the previous version on deepdeck-bench. Future model comparisons can use the same harness, while harness improvements can be evaluated explicitly as version comparisons.

Changing the model and harness together makes their effects difficult to distinguish. Comparisons need recorded code versions, corpus, configuration, run conditions, and pricing definitions.

The fix has been merged into main. The retest supports fewer tool-call errors and lower MiMo execution cost on this selection. Its effect on the full task suite still needs evaluation.