Overview
Research indicates that current benchmarking practices for computer-use agents (CUA) frequently misrepresent their performance. The scoring mechanisms, often reliant on scripted oracles, exhibit vulnerabilities that lead to inaccurate performance assessments. These issues manifest across various stages of the evaluation pipeline, including task construction, trajectory observation, scoring, and the aggregation of reports, potentially obscuring the root causes of agent failures.
Research Context
Computer-use agents are increasingly deployed for tasks such as web browsing and desktop software operation. The reliability of benchmarks used to evaluate these agents is critical for understanding their capabilities and limitations. Existing benchmark scores for CUAs are commonly generated using scripted oracles. This approach can lead to several problems: tasks may become stale, recorded trajectories might lack crucial visual evidence, evaluators can incorrectly reject valid alternative actions, and aggregate reports may not clearly identify the reasons for failure. These issues collectively impact the accuracy and interpretability of CUA performance metrics.
Approach
The research employed a reliability framework to categorize problems within CUA evaluation. This framework encompassed four key areas: task construction, trajectory observation, scoring methodologies, and reporting practices. To assess the extent of these issues, an audit was conducted on 150 public failure-scored trajectories. These trajectories were drawn from five distinct benchmarks, covering web-based interactions, enterprise-workflow scenarios, and desktop-control tasks. The audit specifically focused on identifying instances where reported FAIL verdicts were inaccurate or misattributed.
Findings
- An audit of 150 public failure-scored trajectories from five CUA benchmarks revealed that 15.3% of the reported FAIL verdicts were incorrect.
- Within these incorrect FAIL verdicts, 10.7% were identified as evaluator false negatives.
- Additionally, 4.7% of the incorrect FAIL verdicts were attributed to broken tasks within the benchmarks.
- For genuine failures, a three-tier diagnostic taxonomy was applied. This taxonomy indicated that verification/feedback failures and planning failures were the dominant error types.
- Execution/grounding errors were less prevalent than verification/feedback and planning failures among genuine failures.
- The research concluded that a single scalar success rate is insufficient to explain the nuanced causes of agent failures.
Why This Matters
The identified issues with CUA benchmarking, specifically the high rate of incorrect failure verdicts and the limitations of scalar success rates, suggest that current evaluation methods may misrepresent the true capabilities and failure modes of these agents. This has implications for the development and deployment of CUAs, as it can hinder accurate diagnosis of performance bottlenecks. The findings offer a basis for deriving stage-specific design rules for CUA evaluation, particularly for newer long-horizon benchmarks.