Overview
HRBench is introduced as a unified evaluation framework designed for studying thinking-mode switching mechanisms within hybrid-reasoning large language models (LLMs). This framework aims to facilitate comparative analysis of adaptive thinking-mode selection methods, which are typically evaluated under disparate models, datasets, and implementation assumptions. The objective is to provide a controlled environment for researching efficient reasoning in hybrid-reasoning LLMs.
Research Context
Hybrid-reasoning LLMs enable explicit control over reasoning effort, allowing for a trade-off between answer quality and inference cost. The challenge addressed by HRBench is the current difficulty in comparing the practical behavior of existing adaptive thinking-mode selection methods due to variations in evaluation settings, including models, datasets, and implementation specifics. Prior methods are reimplemented to ensure consistency across comparisons within the HRBench framework.
Approach
HRBench organizes the design space for thinking-mode switching strategies along two primary axes: strategy families and training regimes. There are three identified switching strategy families: prompt-based selection, external routing, and speculative execution. These are evaluated across four training regimes: training-free, supervised fine-tuning (SFT), offline reinforcement learning (RL), and online RL. This combination yields 12 controlled evaluation settings.
The evaluation involved 6 LLMs, ranging in scale from Qwen3.5-2B to Kimi-K2.5-1.1T. Performance was assessed across 5 reasoning benchmarks, which covered domains such as mathematics, science, and code. Over 12 representative prior methods were reimplemented and integrated into the same evaluation pipeline to ensure comparability.
Findings
- Different switching strategies occupy distinct effectiveness-efficiency trade-off regions.
- Prompt-based methods often provide favorable token-accuracy trade-offs.
- Routing methods offer more stable cost reduction.
- Speculative methods tend to improve accuracy, typically at a higher token cost.
- Training affects strategies differently, indicating that the choice of training regime can influence strategy performance.
- The preferred strategy varies with both model scale and task domain, suggesting that a single optimal strategy may not exist for all contexts.
Why This Matters
HRBench provides reference implementations and a unified evaluation platform, supporting controlled research on efficient reasoning in hybrid-reasoning LLMs. This framework enables a more structured comparison of thinking-mode switching strategies, which is crucial for advancing the understanding and development of LLMs that can balance performance and computational cost effectively.