Overview
Evaluating large language models (LLMs) currently relies on fixed benchmarks. This approach, which applies a consistent set of items to all models, can lead to ceiling and floor effects, potentially obscuring differences in model capabilities. The most informative evaluation signal is posited to exist at the model's boundary, specifically where the per-prompt pass probability approaches $0.5$ under a random-sampling decoding strategy.
Dynamic Boundary Evaluation (DBE) is proposed as a method to actively locate each model's boundary and position it on a globally comparable difficulty scale. DBE generates three primary artifacts: a calibrated item bank, a search algorithm known as Skill-Guided Boundary Search (SGBS), and an evaluation protocol.
Research Context
Existing evaluation practices for LLMs frequently employ fixed benchmarks. Such benchmarks present the same set of items across various models. A drawback of this methodology is the potential for ceiling effects, where high-performing models achieve near-perfect scores, and floor effects, where low-performing models achieve near-zero scores. These effects can mask nuanced differences and capability gaps among models.
Approach
DBE's approach involves three distinct components:
Calibrated Item Bank: This artifact comprises items covering categories such as safety, capability, and truthfulness. Each item within the bank is associated with a difficulty label. These difficulty labels were validated across nine reference LLMs.
Skill-Guided Boundary Search (SGBS): SGBS is described as a search algorithm engineered to identify boundary items for a specific target LLM. Its operation requires only API-level query access to the target LLM.
Evaluation Protocol: This protocol is designed to situate a new LLM onto a unified ability scale. The protocol can adaptively expand the evaluation set in instances where the target LLM's performance falls outside the initial coverage of the item bank.
The instantiation of DBE focused on four categories. These categories encompassed safety (including harmful request refusal and over-refusal), capability (specifically constrained instruction following), and truthfulness (addressing multi-turn sycophancy resistance).
Findings
The application of DBE resulted in an evaluation that covered a broader spectrum of models. This approach allowed for assessment without encountering saturation effects. Furthermore, the resulting evaluation maintained compatibility with existing datasets.
Why This Matters
Current LLM evaluation methods using fixed benchmarks can create ceiling and floor effects that obscure model differences. DBE offers a refined approach by pinpointing a model's operational boundary, where performance is most indicative, facilitating a more granular understanding of model capabilities across various domains without encountering saturation.