Overview
This research presents a rigor-matched, three-seed audit evaluating layer-skipping methods designed for efficient Large Language Model (LLM) inference. The audit focuses on two periodic-step, search-based methods that operate online during inference and re-evaluate decisions every few generation steps: ConfLayers, a confidence-gated early-exit baseline, and SWIFT (self-speculative decoding), introduced by Xia et al. (2024). These methods were compared against vanilla autoregressive decoding. Additionally, the study included a supplemental analysis of two trained routing methods, LayerRoute (Sikdar, 2026) and LayerDrop (Fan et al. 2020), which employ coarser decision granularities.
Research Context
Efficient LLM inference is a critical challenge. Layer-skipping techniques aim to address this by selectively executing transformer layers for a given input. This decision-making process can occur at various granularities. The audit specifically investigated methods that make these decisions online at inference time, continually adjusting execution based on generation steps. The comparison included methods that are dynamic and those that are pre-determined or input-conditioned at a coarser level.
Approach
The audit utilized a rigor-matched protocol, employing three distinct seeds for reproducibility and robustness. The primary comparison involved ConfLayers and SWIFT. ConfLayers is characterized as a confidence-gated early-exit baseline, while SWIFT is genuine self-speculative decoding. These methods were evaluated alongside vanilla autoregressive decoding.
Experiments were conducted using two LLM models: Qwen2.5-0.5B and Qwen2.5-1.5B (Yang et al. 2024). Performance was assessed on two distinct datasets: GSM8K (Cobbe et al. 2021) and CNN/DailyMail (Nallapati et al. 2016; See et al. 2017). These datasets represent different types of natural language processing tasks.
A supplemental analysis was performed on two trained routing methods: LayerRoute and LayerDrop. LayerRoute (Sikdar, 2026) is described as a per-sequence, input-conditioned hard gate. LayerDrop (Fan et al. 2020) is characterized as a fixed, input-independent pruning pattern. This supplemental analysis was not a head-to-head comparison with ConfLayers and SWIFT due to the differing decision granularities of the methods.
The evaluation protocol for the trained routing methods ensured genuine per-input gating, a genuine full-model baseline, and genuine inference-time compute skipping.
Findings
- Accuracy Comparison (Periodic-Step Methods): SWIFT demonstrated superior accuracy, emerging as the strongest in three out of four cells in the primary comparison. Conversely, ConfLayers was consistently dominated across all evaluation cells. This deficit was particularly pronounced on the GSM8K dataset when using the Qwen2.5-1.5B model.
- Inference Speed and Overhead (Periodic-Step Methods): When separating online-search overhead from pure inference cost, SWIFT exhibited faster true inference speed than ConfLayers in all four evaluation cells, ranging from 5% to 21%. This finding reversed the naive wall-clock ranking in three of the cells. ConfLayers incurred a small and stable search overhead, approximately 1% to 2% of the total cost. In contrast, SWIFT's search overhead was larger and displayed greater variability across the three seeds, reaching up to 28.7%.
- Performance of Trained Routing Methods (Supplemental Analysis): LayerRoute and LayerDrop, under the verified protocol, showed modest real speedups ranging from 1.08x to 1.33x. However, their accuracy was significantly lower than that of the periodic-step methods. LayerRoute experienced a near-total collapse on the GSM8K dataset with the Qwen2.5-1.5B model, achieving a mean exact-match accuracy of 0.003 across the three seeds.
Why This Matters
The audit provides a rigorous, multi-seed comparison of online, periodic-step layer-skipping methods for LLM inference, identifying performance differences in both accuracy and inference speed. It also offers insights into the trade-offs associated with search overhead in dynamic skipping strategies. The release of the full audit protocol aims to provide a template for future rigor-matched efficiency comparisons in the field.