Overview
RAV is a lightweight and modular framework designed to enhance functional correctness in code generation by large language models (LLMs). This framework operates through three coordinated stages: Route, Align, and Verify. It aims to address challenges in achieving strong functional correctness, particularly for heterogeneous programming tasks where single prompting strategies or direct single outputs are often insufficient. The framework's design focuses on optimizing how tasks are prompted, how the model is adapted, and how final outputs are selected, rather than modifying the backbone LLM architecture.
Research Context
Current large language models have advanced code generation capabilities, but consistently achieving high functional correctness remains an area of difficulty. This issue is particularly pronounced in heterogeneous programming tasks, which may not be adequately addressed by a singular prompting approach or by relying solely on a single directly generated output. The research investigates methods to improve this aspect of LLM performance without necessitating changes to the underlying backbone model.
Approach
The RAV framework integrates three distinct stages to improve code generation functional correctness:
- Route: This initial stage involves applying task-aware prompt routing prior to code generation.
- Align: This stage focuses on reducing the discrepancy between prompts used during fine-tuning and those used during inference. This is achieved through aligned LoRA (Low-Rank Adaptation) adaptation.
- Verify: The final stage involves selecting the ultimate output from multiple generated candidates. This selection process is performed by executing the candidates against visible public tests to ensure functional correctness.
Findings
The complete RAV pipeline demonstrated improved performance on the MBPP benchmark. When evaluated, the pipeline achieved 0.8911 on MBPP Sanitized and 0.8520 on MBPP Full. These results represent improvements of 6.35 percentage points on MBPP Sanitized and 9.92 percentage points on MBPP Full when compared to the base model's performance.
Component-wise ablation experiments indicated that the effectiveness of task-aware routing and aligned adaptation is substantially increased when combined with execution-based verification. Robustness and contamination analyses were also conducted, supporting the reliability of the observed performance improvements. The overall findings suggest that functional correctness in code generation can be significantly enhanced by jointly optimizing task prompting, model adaptation, and output selection processes, without requiring modifications to the backbone architecture of the LLM.
Why This Matters
The findings indicate that functional correctness in code generation can be meaningfully improved without altering the backbone LLM architecture, by optimizing prompting, adaptation, and output selection. This approach offers a pathway to enhance the reliability of code generated by LLMs, particularly for complex and varied programming tasks.