Overview
This research investigates the efficacy of AI agents powered by open-weight large language models (LLMs) for data preparation tasks, specifically within longitudinal population studies. The study examines the potential for deploying such models locally to address governance requirements that typically restrict the transmission of sensitive data to external, cloud-based services. The work introduces an open-source framework designed for the evaluation of these models on specific data preparation challenges.
Research Context
Large language models and AI agents are increasingly utilized in code development. However, their use in research involving personal data is often constrained by governance regulations that prohibit sending data to third-party cloud-based models. Open-weight LLMs, deployable locally, offer an alternative by ensuring sensitive data remain within the local environment. Data preparation is identified as a persistent bottleneck in research on longitudinal population studies.
Approach
The researchers developed an open-source framework to evaluate open-weight LLMs for agentic coding. This framework consists of three primary components:
- Curated Ground-Truth Dataset: This dataset includes cleaning scripts used in preparing six sweeps of data from a British cohort study.
- Task Definitions: The framework defines specific data preparation tasks, such as category harmonization and multi-wave merging.
- Automated Evaluation Routines: These routines are designed to assess the quality of the R code produced by the LLMs and the resulting outputted data.
Using this framework, LLMs were benchmarked across the consumer-grade deployment spectrum. The evaluation focused on their efficacy in 20 distinct data preparation tasks, which involved the creation of 102 variables.
Findings
- Current state-of-the-art open-weight LLMs, specifically those with 31-35 billion parameters, demonstrated high performance in the benchmark tasks.
- These models achieved an 'average task completion' rate of up to 87.9%.
- The performance was observed on consumer-grade hardware.
Why This Matters
The observed performance of open-weight LLMs running on consumer-grade hardware suggests a viable pathway for AI-assisted data preparation within research settings that are subject to governance restrictions. This offers an alternative to cloud-based solutions, which are often incompatible with regulations surrounding sensitive personal data.
Potential Applications
The open-source framework and benchmark are publicly available at https://github.com/UCL-ARC/RRBench. This tool can be used to further evaluate and develop AI-assisted data preparation methods in contexts requiring local data processing.