Overview
The advancement of Large Reasoning Models (LRMs) has led to a shift from reactive text generation to a systematic, step-by-step reasoning approach, achieving state-of-the-art results in complex mathematical and logical tasks. A fundamental gap exists in understanding the internal reasoning mechanisms of these models, distinct from token-level behavioral analysis, and in addressing the instability of reinforcement learning (RL) for reasoning optimization due to its reliance on external verifiers.
This work identifies and formally defines “Entropy-Gradient Inversion” as a robust negative correlation between token entropy and logit gradients. This inversion serves as a specific geometric fingerprint for the reasoning capability of LRMs. Based on this identification, the researchers propose “Correlation-Regularized Group Policy Optimization (CorR-PO),” a method that embeds this inversion signature into RL reward regularization.
Experiments on various reasoning benchmarks, tested across multiple model scales, indicate that CorR-PO consistently outperforms existing baseline methods. These results further suggest a direct correlation between the strength of the observed inversion and superior reasoning performance.
Research Context
Large Reasoning Models (LRMs) have demonstrated an ability to move beyond simple 'fast thinking' text generation to 'slow thinking' reasoning processes. This capability has enabled them to achieve high performance in tasks requiring complex mathematical and logical operations. Despite these advancements, the field faces two primary challenges:
- A gap between analyzing LRM behavior at the token level and understanding their internal reasoning mechanisms.
- The inherent instability of reinforcement learning methods when used for optimizing reasoning, largely due to their dependence on costly external verifiers for feedback.
Approach
The research first centered on identifying a quantifiable internal mechanism related to LRM reasoning. This led to the formal definition of “Entropy-Gradient Inversion,” characterized as a negative correlation between token entropy and logit gradients. This phenomenon was posited as a definitive geometric fingerprint indicative of an LRM's reasoning capability.
Building upon this identified signature, the researchers developed “Correlation-Regularized Group Policy Optimization (CorR-PO).” This method integrates the Entropy-Gradient Inversion signature directly into the reward regularization mechanisms of reinforcement learning. The goal of embedding this signature is to guide the optimization process internally, potentially reducing reliance on external verifiers and enhancing stability.
To validate CorR-PO, extensive experiments were conducted. These experiments involved diverse reasoning benchmarks and were performed across multiple scales of Large Reasoning Models. The methodological approach sought to evaluate whether the proposed CorR-PO could consistently outperform established baselines and to determine if a stronger observed Entropy-Gradient Inversion directly correlated with improved reasoning performance.
Findings
- Entropy-Gradient Inversion is formally defined as a robust negative correlation between token entropy and logit gradients.
- This inversion acts as a definitive geometric fingerprint for the reasoning capability of Large Reasoning Models (LRMs).
- Correlation-Regularized Group Policy Optimization (CorR-PO) was developed, which integrates the Entropy-Gradient Inversion signature into reinforcement learning (RL) reward regularization.
- Extensive experiments on various reasoning benchmarks across multiple model scales showed CorR-PO consistently outperformed state-of-the-art baselines.
- A direct correlation exists where stronger Entropy-Gradient Inversion leads to superior reasoning performance.