Overview
Research addresses the challenge of building reward models optimized for contextual grounding and abstention in large language models (LLMs) used in high-stakes domains like law. Traditional reward models are often tailored for general preferences, limiting their efficacy in evaluating critical behaviors such as grounding reasoning in retrieved evidence and abstaining when evidence is insufficient within Retrieval-Augmented Generation (RAG) settings. This work introduces a framework designed to convert existing legal Question Answering (QA) datasets into contextual preference data. Utilizing this framework, the LegalRewardBench (LRB) benchmark was constructed to assess the capabilities of grounded legal generation, specifically under conditions of noisy and insufficient retrieval.
Research Context
Large language models are increasingly deployed in high-stakes applications, including legal contexts. In these environments, it is crucial for systems to ground their reasoning in provided evidence and to abstain from generating responses when the available evidence is inadequate. However, the prevailing reward models are largely optimized for general preferences. This optimization paradigm restricts their utility in accurately evaluating whether LLMs effectively ground their outputs in retrieval and when they appropriately abstain, particularly within RAG frameworks.
Approach
The research employed a specific framework to generate contextual preference data from existing legal QA datasets. This framework facilitated the creation of LegalRewardBench (LRB), which serves as a benchmark for evaluating grounded legal generation. The evaluation conditions explicitly included noisy and insufficient retrieval scenarios. The methodology involved assessing reward models across both general and legal contextual evaluation settings. A key aspect of the approach was the investigation of contextual DPO (Direct Preference Optimization) and the impact of various preference-data construction methods. Specifically, the study examined the role of length-balanced augmentation in improving grounded legal evaluation.
Findings
- Contextual DPO enhances grounded evaluation across both general and legal contextual settings.
- The performance of contextual DPO is sensitive to the specific methods used for preference-data construction.
- Length-balanced augmentation substantially improves grounded legal evaluation.
- The strongest configuration for performance improvement combines length-balanced legal and general contextual preference data. This configuration resulted in a performance improvement of up to +25.6 percentage points (pp) over baseline models.
- Evidence indicates cross-jurisdiction transfer: models primarily refined using Victorian criminal-law data demonstrated improved grounded evaluation on external US legal benchmarks.
- A specific instance of cross-jurisdiction transfer includes a +16.2pp improvement on the \(\text{Housing Statute QA}\) benchmark.
Why This Matters
The developed framework and benchmark provide a reproducible foundation for the construction and evaluation of reward models specifically designed to enhance grounded legal generation in retrieval-augmented settings. This addresses a critical need for LLMs operating in high-stakes legal domains, where accurate grounding in evidence and appropriate abstention are paramount for reliability and trustworthiness.