Overview
This research investigates the extent to which formal semantic structure accounts for human label variation within Natural Language Inference (NLI). While human label variation is progressively recognized as meaningful data, the direct measurement of its explanation by formal semantic structure has been underexplored. The study establishes quantitative bounds on this explanatory power across group-level and item-level analyses, utilizing a dataset specifically chosen for low original agreement.
Research Context
The study operates within the domain of Natural Language Inference, where human labeling exhibits variation. This variation is increasingly considered a signal rather than a source of noise. The central research question concerns the direct measurement of how much of this observed human label variation can be explained by formal semantic structure.
Approach
The investigation utilized 3,113 items from the SNLI and MNLI datasets, drawn specifically from ChaosNLI, a collection characterized by low original agreement. A rule-based operator and a monotonicity tagger were employed. This tagger was validated against the Monotonicity Edit Distance (MED) metric, achieving an agreement of 0.883 at the edit site and 0.807 on the sentence-level summary consumed by the analyses. The study incorporated three preregistered analysis blocks, and all negative results were fully reported. Analyses were preregistered in a version-controlled research log, documenting the audit trail, including one corrected interpretation rule.
Findings
The research established three distinct bounds regarding the explanatory power of formal semantic structure on human label variation in NLI:
First: Group-Level Boundary
- Hypotheses that were not purely upward monotone consistently exhibited higher label entropy. This effect was quantified with Cliff's delta at -0.284.
- Rank-based tests corroborated this effect, defending it against potential influences of operator-presence and length reductions.
- A bounded-outcome sensitivity check, however, attenuated the regression form of the length defense.
Second: Item-Level Ceiling
- The formal profiles investigated explained a limited proportion of entropy variance, specifically ranging from 3.3% to 3.6%.
- These profiles achieved a median-split Area Under the Curve (AUC) of 0.606.
- This level of performance was deemed insufficient to effectively identify high-disagreement items.
Third: Composition Invariance
- Across the identified group-level boundary, the study conducted three high-powered, preregistered contrasts. These contrasts focused on validated error shares (VariErr) and explanation-type shares (LiTEx).
- All three contrasts yielded null results, indicating an invariance in composition.
- In this specific sample, formal semantic structure was observed to subtly alter the degree of annotator disagreement by a small margin, but it did not detectably change the substance of what annotators disagreed about.
It is crucial that all claims presented are conditioned on the scope of the ChaosNLI-S/M dataset, which comprises items selected for their low original agreement.
Why This Matters
The findings indicate that while formal semantic structure has a measurable, albeit small, impact on the quantity of annotator disagreement, it does not significantly influence the qualitative aspects of that disagreement. This suggests inherent limits to understanding human label variation solely through the lens of formal semantic structure within NLI contexts, particularly for datasets with inherently low agreement.