Overview
This research evaluates the investigatory intelligence of Large Language Models (LLMs) in the context of data science. The concept of investigatory intelligence is distinguished from executional intelligence, with the former defined as the autonomy to set goals and decide what to explore, going beyond merely answering correctly. The latter, executional intelligence, is described as the ability to complete assigned tasks. The study introduces a novel open-ended task called Deep Data Research (DDR), designed to assess the autonomous extraction of key insights from databases by LLMs. To facilitate verifiable evaluation, a large-scale, checklist-based benchmark named DDR-Bench was developed.
Research Context
Agentic Large Language Models are expected to exhibit agency that extends beyond providing correct answers, requiring autonomous goal-setting and exploratory decision-making. Data Science is identified as a suitable domain for testing such capabilities, given that real-world data analysis typically originates from raw data rather rather than explicit queries. Despite this, existing benchmarks are noted as having a limited focus on this aspect of data science.
Approach
The research established Deep Data Research (DDR) as an open-ended task. In this task, LLMs are required to autonomously extract key insights directly from databases. To enable a verifiable assessment of performance in DDR, the study developed DDR-Bench. This benchmark is described as large-scale and checklist-based.
Findings
The evaluation using DDR-Bench indicated that frontier models exhibit emerging agency. However, the study also found that long-horizon exploration remains challenging for these models. Analysis from the research suggests that effective investigatory intelligence is not solely dependent on agent scaffolding or merely scaling. Instead, it is highlighted that intrinsic strategies of agentic models play a significant role in their investigatory intelligence.
Why This Matters
The distinction between investigatory and executional intelligence offers a framework for understanding and evaluating the capabilities of agentic LLMs beyond task completion. The development of DDR and DDR-Bench provides a method for assessing LLM autonomy in real-world data science scenarios, which start from raw data rather than defined queries.