Benchmarking LLM Competence on Logical Inference Over Probability Operators

Overview

This research introduces a benchmark designed to evaluate Large Language Model (LLM) competence in logical inference involving probability operators. The benchmark, comprising 14,320 procedurally-generated English prompts, focuses on inferences over sentences containing gradable epistemic modals such as 'probably', 'might', and 'must'. The evaluation encompassed 29 distinct models.

Research Context

Uncertainty and inferences are common elements in natural language. The ability to perform valid inferences over natural-language expressions of uncertainty is critical for both daily communication and high-stakes fields, including medicine and law. While LLMs are increasingly assessed on logical reasoning tasks, a challenge exists in distinguishing between principled, symbolic reasoning and surface-level pattern matching.

Approach

The benchmark employs 14,320 procedurally-generated English prompts. These prompts are structured across fifteen inference templates. Systematic variations were introduced across several parameters: question form, negation strategy, and surface content. The evaluation process involved assessing 29 different models against this benchmark. A 'competence floor' metric was established, defined as the lower accuracy achieved by a model on either 'Yes-correct' or 'No-correct' items, intended to summarize answer biases.

Findings

• Most of the 29 evaluated models displayed answer biases that were independent of the logical form of the prompt. These biases were characterized by a systematic preference for either 'Yes' or 'No' answers.
• Only 9 out of the 29 models tested exceeded random chance performance on the benchmark.
• The research identified a 'competence floor' for each model, reflecting the worse accuracy between 'Yes-correct' and 'No-correct' items, which served to quantify the answer biases.
• Biases were also observed across variations in question form, verb phrases/activity, and in relation to both the gender and origin of names utilized within the prompts. These biases were found to exist across every axis tested.