Overview
Recent advancements in large language models (LLMs) and their integration into software engineering tasks have positioned Artificial Intelligence (AI) coding agents as a component of modern software development workflows. This study provides an empirical analysis of agentic pull requests (PRs) compared to human-generated PRs. The research focuses on understanding how the characteristics of agentic contributions evolve across different stages of the software development lifecycle.
Research Context
While developers are increasingly utilizing AI coding agents, their impact on software quality has yet to be fully understood. Specifically, the evolution of agentic contributions throughout the software development lifecycle lacks thorough investigation. This study addresses this gap by characterizing agentic PRs and examining how their properties change over time in real-world practices.
Approach
The study employed the AIDev dataset to analyze agentic pull requests. The analytical approach involved three main steps:
- First, the researchers analyzed how differences in merge rates between agentic and human-generated PRs changed over time.
- Second, they identified the types of development tasks where AI coding agents are primarily applied. This step also investigated how the distribution of these tasks evolved across different development quarters.
- Finally, the study compared a set of key characteristics of agentic and human-generated PRs. This comparison focused on the implications of these characteristics for software quality and their temporal dynamics.
Findings
The study's findings offer an empirical and longitudinal perspective on the role of AI coding agents in software development:
- Differences in merge rates between agentic and human-generated PRs were analyzed for their temporal variation.
- Specific types of development tasks where AI coding agents are predominantly applied were identified.
- The distribution of these AI agent application tasks was observed for its evolution across development quarters.
- Key characteristics of agentic PRs were compared against those of human-generated PRs, focusing on their implications for software quality.
- The temporal dynamics of these key characteristics were also examined.
Why This Matters
The findings offer a more nuanced understanding of the benefits and limitations of AI coding agents in real-world software development practices. This characterization helps to clarify their role and impact on software quality within contemporary workflows.