Overview
Peer review in machine learning (ML) conferences increasingly relies on reviewer scores as a primary decision instrument. This analysis posits that acceptance outcomes are influenced by factors beyond reviewer scores, stemming from a measurement design failure rather than individual bias.
Research Context
With ML conference submissions escalating to tens of thousands annually, a systematic examination of the uniformity of the reviewer scoring instrument across diverse research areas had not been conducted. The paper argues that when a fixed numerical scale aggregates quality judgments across communities with structurally non-uniform reviewer pools, absolute scores become incomparable across areas. Consequently, area chairs are compelled to substitute community priors for score-based decisions.
Approach
The study utilized data from ICLR 2021 through 2026, encompassing 50,289 papers across 219 distinct research topics. This dataset was analyzed to determine how reviewer scores correlate with acceptance probabilities across different topics.
Findings
- A paper's acceptance probability, for any given reviewer score, can vary by a factor of up to eight, contingent on its specific research topic.
- The analysis systematically ruled out several alternative explanations for this variance. These excluded factors include: scoring culture differences, varying expert reviewer standards, rational reweighting by area chairs, and quality dilution effects.
- The observed inconsistency in acceptance probabilities across topics, despite identical reviewer scores, is attributed to a fundamental flaw in the measurement design, rather than biases of individual reviewers. The fixed numerical scale for aggregating quality judgments is deemed structurally non-uniform across diverse reviewer pools for different communities.
Why This Matters
The findings suggest that the current peer review system, which relies on a fixed numerical scale for reviewer scores, does not function uniformly across different ML research areas. This non-uniformity implies that the acceptance outcomes of papers are significantly influenced by their research topic, independent of their assigned reviewer score, potentially affecting fairness in the review process. The paper advocates for program committees to adopt inherently calibrated review signals and to publish topic-stratified, score-conditional acceptance rates as a foundational fairness metric.