Overview
This research introduces Online Conformal Prediction with Queries (OCPQ), a method designed for uncertainty quantification in machine learning models operating in scenarios where direct feedback on deployed predictions is not provided. OCPQ addresses a setting where, on each round, a learner must choose between outputting a prediction set or querying the correct label, precluding the direct evaluation of deployed predictions. This problem is framed as a partial monitoring game, where prediction actions yield no observation, and a separate query action reveals the label. The reward function encourages the generation of small prediction sets while ensuring the correct label is covered with a user-specified probability.
Research Context
Uncertainty quantification is a critical aspect for the deployment of machine learning models, particularly in safety-critical applications. Online conformal prediction (OCP) is an established framework providing theoretically principled uncertainty quantification. It enables the construction of prediction sets for arbitrary black-box classifiers and non-i.i.d. data streams, guaranteeing that these sets contain the true label at a user-specified frequency. Traditional OCP approaches typically rely on feedback from previously deployed predictions to update these prediction sets. The current work explores an OCP setting that deviates from this feedback-dependent paradigm, termed "beyond feedback," where deployed predictions are not directly evaluated.
Approach
The core problem involves a learner who, at each round, has two mutually exclusive actions: either output a prediction set or query the correct label. This setup means that no deployed prediction is directly assessed for its accuracy or coverage. The researchers modeled this as a partial monitoring game. In this game, issuing a prediction set action provides no observation, while executing a query action reveals the true label. The reward structure for this game is engineered to simultaneously incentivize the learner to produce compact prediction sets and to maintain a high probability of covering the true label. To address this partial monitoring game, OCPQ was developed. This involved adapting the label efficient forecaster, a technique initially proposed by Cesa-Bianchi, Lugosi, and Stoltz (2004), to fit the specific requirements of this beyond-feedback online conformal prediction setting.
Findings
For any black-box classifier and any (non-i.i.d.) oblivious data stream of length $T$, OCPQ demonstrated specific performance characteristics:
- It achieved an expected regret of $O(T^{2/3})$.
- It maintained an expected coverage of at least $\beta - O(T^{-1/3})$, where $\beta$ is a user-defined parameter.
- The method queries labels only for an expected $T^{-1/3}$ fraction of rounds.
These findings suggest that OCPQ can provide coverage comparable to bandit-based OCP methods, yet it operates without requiring feedback from deployed prediction sets. Experimental evaluations conducted on real-world datasets further indicated the effectiveness of the OCPQ approach.
Why This Matters
The ability to quantify uncertainty without direct feedback on predictions addresses a practical limitation in certain deployment scenarios for machine learning models. This is particularly relevant for safety-critical applications where direct evaluation of every prediction might be impractical or impossible, but reliable uncertainty bounds are still essential. The method's capacity to achieve coverage guarantees while minimizing label queries offers a resource-efficient solution for maintaining model reliability in such environments.
Potential Applications
While the source does not detail specific applications, the context of "safety-critical applications" and "beyond feedback" settings implies relevance for areas where acquiring ground truth for every prediction is costly, delayed, or impossible. Examples could include medical diagnostics where expert verification is slow, or autonomous systems where real-time feedback loops are constrained.