ICANEWS

Engineering & Technology

Impact of Instruction Files on AI-Agent Pull Request Performance in Software Engineering

This research investigates the relationship between developer-created instruction files for AI-agents and the resulting performance of AI-agents in generating p

Overview

This research investigates the relationship between developer-created instruction files for AI-agents and the resulting performance of AI-agents in generating pull requests (Agentic-PRs). The study assesses performance across three dimensions: success rate (merge rate), task complexity (code churn), and effort required for merging (time to merge).

Research Context

AI-agents, such as GitHub Copilot, function as collaborators in various software engineering tasks, including the proposal of code via pull requests. To enhance agent efficiency, developers are creating instruction files. These files are designed to guide AI-agents in tasks such as project navigation, component location, test execution, and adherence to best practices. The investigation focuses on understanding whether these instructions contribute to the creation of 'better' pull requests, defined by higher merge rates, the ability to address more complex tasks, and reduced merging effort.

Approach

The study analyzed 15,549 agentic pull requests originating from 148 distinct projects within the AIDev dataset. The methodology involved comparing each project's performance before and after the implementation of instruction files. This comparison utilized the three aforementioned dimensions: merge rate, code churn, and merge effort (including merge time and number of comments).

Findings

  • Specifying instructions for AI-agents does not consistently lead to improved outcomes.
  • Among the projects studied, 27.7% experienced an increase in their merge rate of at least 20% after instruction file creation.
  • Conversely, 26.35% of projects observed a decrease in their merge rate following the introduction of instruction files.
  • Similar varied observations were noted across other performance indicators, including the amount of changes (e.g., code churn, number of modified files) and the effort required to merge an agentic PR (e.g., merge time, number of comments).
  • A preliminary exploration indicated that projects which achieved an increase in their merge rate tended to have substantially longer instruction files.
  • These more effective instruction files were also characterized by being well-structured, featuring a higher number of sections and sub-sections.

Why This Matters

The findings highlight a need for further research to support practitioners in approaching the development of instruction files as a structured software engineering activity, termed «Instructions-as-Code.»

Key Findings