Overview
Research indicates an increasing trend among developers and consumers to select reasoning models (RMs) based on their publicly available API prices. However, a study investigated the accuracy with which these listed prices reflect actual inference costs. This systematic evaluation, involving 8 frontier reasoning models across 12 diverse tasks, uncovered a phenomenon termed "pricing reversal." This phenomenon signifies instances where a model with a lower listed API price ultimately incurs a higher total actual cost, with reversals observed reaching up to 28 times the anticipated magnitude.
Research Context
The selection of reasoning models often hinges on their listed API pricing, influencing consumer and developer choices. This study sought to systematically assess whether these listed prices accurately represent the true inference costs associated with using these models across various applications. The research aimed to quantify any discrepancies and identify underlying factors contributing to potential cost misalignments.
Approach
The study conducted a systematic evaluation of 8 frontier reasoning models (RMs), including Gemini 3 Flash and GPT-5.4, across 12 distinct tasks. These tasks encompassed various domains such as competition mathematics, science question answering, code generation, and multi-domain agent operations. To analyze the contributions to cost heterogeneity, a formal cost attribution framework was constructed, leveraging Shapley value. This framework facilitated tracing the primary drivers behind variations in thinking token consumption and the number of interaction turns.
Findings
- The study identified a "pricing reversal phenomenon": in 32% of model-pair comparisons, the model with a lower listed price resulted in a higher total actual cost.
- The magnitude of these pricing reversals reached up to 28 times.
- For example, Gemini 3 Flash, despite having a listed price 80% cheaper than GPT-5.4, incurred an actual cost 38% higher across all evaluated tasks.
- The formal cost attribution framework, based on Shapley value, indicated that the dominant contributors to cost heterogeneity were significant variations in thinking token consumption and the number of interaction turns.
- For the same query, one model was observed to use 900% more thinking tokens than another, or exhibit 10 times more environment interactions.
- The study also revealed that per-query cost prediction is inherently challenging due to variability. Repeated runs of the same query showed thinking token variation up to 9.7 times, establishing an irreducible noise floor for predictive models.
- This variability suggests that predicting cost distributions is an open challenge.
Why This Matters
The findings demonstrate that listed API pricing serves as an unreliable proxy for actual inference costs. This unreliability necessitates a shift towards cost-aware model selection strategies and advocates for transparent per-request cost monitoring mechanisms to accurately assess and manage expenditures associated with reasoning model usage.