Overview
An extensive empirical study investigates the comparative strengths and limitations of different code translation techniques. The research specifically focuses on methods categorized as learning-based, LLM-based, and general-purpose Large Language Models (LLMs). This investigation aims to enhance understanding of their performance across varying translation granularities and programming language contexts.
Research Context
Automated code translation is recognized as an increasingly significant component of software evolution. Despite its growing importance, a sufficient understanding of the relative strengths and limitations inherent in learning-based and LLM-based translation techniques has been identified as a gap in current knowledge. The study was designed to address this deficit by providing empirical evidence regarding these methodologies.
Approach
The study employed a large-scale empirical approach to compare representative code translation techniques. The comparison encompassed different methodological paradigms and translation granularities. Specific types of methods evaluated included learning-based methods, LLM-based methods, and general-purpose LLMs.
The evaluation utilized multilingual benchmarks, testing translation at both method-level and class-level granularities. These benchmarks involved multiple programming languages. The analytical framework considered several performance metrics and factors:
- Executable correctness
- Code similarity
- Translation direction
- Translation granularity
- Failure patterns
An error analysis component was included to identify primary challenges within existing code translation systems.
Findings
The empirical analysis yielded several key observations regarding code translation performance:
Method-Level Performance
- LLMs and LLM-based methods generally exhibited superior performance compared to learning-based methods in terms of method-level correctness.
- Code similarity metrics, when used alone, were found not to reliably reflect functional correctness.
Impact of Translation Direction
- Translation direction significantly influenced performance. This impact was particularly notable when translating between languages possessing different type-system characteristics.
Translation Granularity Challenges
- Class-level translation proved considerably more difficult than method-level translation.
- The increased difficulty in class-level translation stemmed from the requirement to preserve global semantics, interfaces, member relationships, and cross-method dependencies.
Failure Patterns and Error Analysis
- The error analysis identified static semantic errors and logical errors as the primary challenges confronting existing code translation systems.
Why This Matters
These findings contribute empirical evidence that can inform the development of future code translation technologies. The insights provide practical guidance for designing more robust, type-aware, structure-aware, and context-aware code translation techniques, addressing identified challenges in current systems.