Overview
CoEval is an open framework designed to rank language models (LMs) for specific applications. It addresses challenges associated with traditional public benchmarks, particularly their suitability for sub-domains or sub-tasks and the potential for item leakage into pretraining. CoEval operates without human labels or external raters.
Research Context
Selecting or evaluating a pretrained or fine-tuned language model for a particular application is a high-value decision. Existing public benchmarks may not accurately reflect the requirements of a specific sub-domain or sub-task. Additionally, the scores from these benchmarks can be compromised if their items have been encountered during pretraining, leading to recall rather than problem-solving. This necessitates a method to generate a trustworthy, task-specific signal for LM evaluation.
Approach
CoEval utilizes an ensemble self-evaluation process. Given a task or domain description, a pool of language models assumes three rotating roles: teacher, student, and judge. In this process, the models collaboratively generate a fresh, contamination-free benchmark, answer the created questions, and score each other's responses. Crucially, no human labels or raters are involved. Each model also functions as a student, generating responses that serve as data points. This data is used to weight each question based on its discriminative power and each judge based on its consensus within the panel. The framework employs this label-free weighting to mitigate the impact of unreliable judges and saturated questions, thereby preserving the integrity of the ranking.
- Roles: Models rotate through teacher, student, and judge roles.
- Benchmark Generation: A fresh, contamination-free benchmark is generated from task or domain descriptions.
- Scoring: Models score one another, with no human intervention.
- Weighting Mechanism: Responses are used to weight questions by their discriminative power and judges by their panel consensus.
Findings
CoEval demonstrates several key characteristics:
- True Ranking Recovery: Where ground truth data is available, CoEval recovers the true ranking of models.
- Objective Correctness Tracking: The method tracks objective correctness with a Spearman's $\rho$ value of 0.86.
- Gold Ranking Recovery with Weighting: The integrated weighting mechanism recovered the gold ranking of thirteen models with a Spearman correlation of 0.95.
- Reliability from Panel Composition: The reliability of the framework stems from the composition of the model panel, rather than its size. The label-free weighting system effectively zeroes out the influence of "broken judges" and down-weights questions that are "saturated," preventing them from distorting the overall ranking.
- Zero Verbatim Overlap: Generated benchmark items exhibited zero verbatim overlap with five public benchmarks.
- Bias Mitigation: The panel structure is observed to cancel verbosity bias and prevent same-family self-preference among models.
- Domain-Specific Rankings: Rankings generated by CoEval are domain-specific. For four de-novo domains, three different models emerged as top performers, indicating that a generic leaderboard may not accurately guide practitioners for specific applications.
Why This Matters
CoEval offers a self-contained method for evaluating and ranking language models against custom tasks without relying on potentially outdated or contaminated external benchmarks. This allows for the generation of application-specific, contamination-free leaderboards, providing teams with a way to independently assess their models upon each new release.
Potential Applications
The CoEval pipeline can be rerun with each new model release, offering any team a contamination-free leaderboard tailored to its specific application. This capability can support continuous evaluation and selection of language models for custom tasks.