Overview
Research addressed the reproducible evaluation of expert caching in Mixture-of-Experts (MoE) models, particularly concerning their interaction with accelerator and host memory. The study investigated the fragility of measurement in assessing expert cache management policies, which aim to reduce expert traffic per token by increasing hit rates. It focused on isolating three critical evaluation axes that were found to alter research conclusions and quantitative results.
Research Context
MoE models have grown to a scale where their expert weights frequently exceed the capacity of accelerator memory. Consequently, offloading these expert weights to host memory has become a standard practice. This architectural shift positions expert cache management as a significant leverage point for performance optimization. Effective cache policies that can improve the hit rate are expected to decrease the volume of expert traffic per token. The core challenge addressed is the robust and accurate evaluation of such policies.
Approach
The research employed a trace-driven, event-atomic simulator. This simulator was utilized across three distinct MoE models, specifically configured with 40, 64, and 128 experts. Through this simulation framework, the study systematically isolated and analyzed three specific evaluation axes: replay semantics, workload contamination, and operating regimes.
Findings
Replay Semantics
The study identified that replay semantics significantly influence evaluation outcomes. Under a fused-event traffic contract, an inconsistent per-access replay methodology inflated the performance of recency-based policies by 27-29%. In contrast, frequency-based and static policies showed performance variations within 4% under the same conditions. This disparity in sensitivity led to an inversion of the policy ranking when compared to evaluations using consistent replay semantics.
Workload Contamination
Workload contamination was found to distort evaluation results. Probe sets constructed with a single instruction template per category generated verbatim-identical generation prefixes. The implementation of a matched-pair rendering intervention caused the measured early-window effect to shift by 19.4-31.9 percentage points. This intervention also reversed which workloads were perceived as more cache-friendly during evaluation.
Operating Regimes
The research established that normalized miss fractions do not directly transfer across different MoE models. This necessitated the reporting of the per-step expert union relative to per-layer capacity. Furthermore, even when the identical event stream was subjected to only a permutation of its temporal order, the offline-optimal gap shifted from 44.9% to 30.8%, indicating that reporting per-step expert union alone is insufficient for robust evaluation.
Gap to Offline Optimum and Causal Prediction
Despite corrections applied to the evaluation methodology, a stable gap to the offline optimum persisted. This gap ranged from 44.2% to 45.9% across 13 frozen workload compositions. A forced-admission oracle attributed 84.3-96.6% of this persistent gap to the ability to know which resident expert would be utilized furthest in the future. A causal next-use predictor, implemented as an eviction rule, recovered -11.4% of this gap. This predictor selected an optimal victim in 3.4% of cases. For comparison, a random resident block achieved optimal victim selection 2.4% of the time, while LRU and LFRU policies achieved 20.6-22.1%.
Why This Matters
The findings indicate that the large offline-optimal gap observed in the evaluated settings substantially overstates the potential gains recoverable by lightweight causal mechanisms currently represented. This underscores the need for careful consideration of evaluation methodologies in MoE expert caching.