Overview
The research introduces HxAgent, an iterative LLM-based planning agent designed for end-to-end web application testing. This agent is engineered to generate test cases and perform testing functions using natural-language descriptions of functionality. Its operational framework integrates proactive correction strategies, enabling it to reassess web states and determine subsequent actions after each step. HxAgent's methodology incorporates current observations, a short-term memory of prior actions, and long-term experience derived from both correct and incorrect action sequences.
Research Context
Automated web testing critically relies on the ability to generate test cases and execute testing processes using natural language descriptions of functionalities. This requirement necessitates testing agents that can autonomously carry out tasks on target applications and independently generate tests. The study addresses this context by proposing an agent designed to improve efficacy in these areas.
Approach
HxAgent functions as an iterative Large Language Model (LLM)-based planning agent. A core component of its design is a proactive correction strategy. This strategy involves HxAgent reassessing the web state following each executed step. The determination of the next action is informed by a combination of three distinct information sources:
- Current observations from the web environment.
- Short-term memory, which retains a record of past actions.
- Long-term experience, which is extracted from previous sequences of actions, including both correct and incorrect executions.
Findings
HxAgent demonstrated specific performance metrics across multiple testing benchmarks:
- On the MiniWoB++ dataset, HxAgent achieved an Exact-Match accuracy of 97.4%. This performance was described as comparable to leading baselines that do not utilize human demonstrations. It also surpassed the performance of the recent WALT model by 10.5%.
- When evaluated on a dataset comprising 350 web tasks, HxAgent attained an 83.8% Exact-Match accuracy. Additionally, it achieved a 91.8% Prefix-Match accuracy on this same dataset. In this context, HxAgent exceeded WALT's performance by 13.4%.
- On the OnlineMind2Web benchmark, HxAgent showed further improvement over WALT, with a performance increase of 4.6%.
Why This Matters
Generating test cases and performing testing via natural-language functionality descriptions is crucial for improving efficacy in automated web testing. The ability of a testing agent to autonomously execute tasks on target applications and generate tests is a key factor in achieving these improvements.