Overview
Streaming systems that manage a pool of expert models face the recurring challenge of deciding whether to continue using an existing expert for new data, to initiate a new expert, or to defer a decision. This research introduces a decision layer designed to provide statistical meaning to these three potential outcomes: reuse, spawn, or defer. The methodology frames reuse and spawn as one-sided sequential hypotheses operating on a conditional discrepancy, specifically a mechanism-level discrepancy, with an indifference zone separating these hypotheses. The defer state precisely corresponds to a scenario where neither betting e-process has accumulated sufficient evidence to support either reuse or spawn.
Research Context
The operational efficiency of streaming systems heavily relies on their ability to adapt to changes in data distribution or underlying processes. A key challenge is discerning whether observed data shifts represent a change in input features (covariate shift) or a fundamental alteration in the data-generating mechanism (mechanism change). Such distinctions are critical for managing pools of expert models, where an incorrect attribution of shift can lead to suboptimal model performance or inefficient resource allocation. The proposed approach, termed CJSD (Conditional Joint Statistical Discrepancy), aims to provide a robust statistical framework for making these distinctions within dynamic streaming environments.
Approach
The core of the approach involves the use of two discriminators to separate covariate shift from mechanism change. The decision layer models reuse and spawn as one-sided sequential hypotheses on a conditional discrepancy. This conditional discrepancy is a mechanism-level metric. An indifference zone is established between the conditions for reuse and spawn. A state of 'defer' is defined when neither betting e-process, associated with these hypotheses, has gathered adequate evidence. The research establishes finite-time anytime validity for an observable surrogate discrepancy derived from a predictable discriminator sequence. An unconditional one-sided transfer to the population quantity is proven, where the slack on each side is quantified by the excess risk of a single discriminator. An empirically observed downward-bias regularity contributes to the spawn side being exactly conservative.
To maintain recency without compromising guarantees, the system employs a restarted e-detector. This detector consists of a bank of unwindowed betting supermartingales initiated at geometrically spaced restart times. This design requires $O(\log t)$ memory. The error budget is managed by distributing it across restart instances, which preserves lifetime anytime validity. Similarly, controlling multiplicity for an unbounded number of experts is achieved by spending the error budget over expert-creation order.
Findings
- The instance-accounted restarted bank achieved zero false spawns and zero false reuses following switches and matches on synthetic multi-concept streams.
- On the Electricity, Covertype, and the recurrence-heavy INSECTS benchmark datasets, the instance-accounted restarted bank either matched or exceeded the performance of the retired windowed heuristic.
- Specifically, on the INSECTS benchmark, the reoccurring accuracy achieved was 0.675.
- The deployed algorithm and the guaranteed algorithm were found to be identical.
Why This Matters
The capability to statistically distinguish between covariate shift and mechanism change in streaming systems is fundamental for the efficient management of expert model pools. By providing a decision layer that robustly guides whether to reuse an existing model, spawn a new one, or defer action, this research directly addresses a critical operational challenge. The demonstrated anytime validity and empirical performance, including zero false positives for spawns and reuses and competitive accuracy on benchmarks, suggest a reliable framework for adaptive model deployment in dynamic data environments.