Overview
ProxyGuard is a methodology developed for directly inferring the reliability of randomized data release mechanisms, particularly when researchers select a proxy dataset from multiple releases, transformations, or seeds. Its design addresses the issue that search processes can render an invalid release seemingly adequate, and that a single adequate release does not confirm the reliability of its generator. The system controls for both types of errors through the application of prespecified bounded risks and a sealed target set.
Research Context
The evaluation of randomized data release mechanisms often involves selecting a 'proxy dataset' from a pool of potential candidates, which can include various releases, transformations, or seeds. A challenge in this context is the potential for selection bias: an extensive search might make an invalid release appear sufficient. Conversely, the adequacy of one specific release does not automatically establish the inherent reliability of the underlying mechanism that generated it. The development of ProxyGuard aims to provide a structured approach to address these evaluation complexities.
Approach
ProxyGuard operates in two primary configurations:
- Named-Release Mode: This mode is designed to correct for multiplicity issues and certify specific individual releases.
- Direct Shared-Target Mode: This mode evaluates independent draws of a mechanism against a common target. It functions by calculating a lower bound for the favorable-score rate and subsequently subtracting a bound on favorable scores that might originate from invalid releases.
A key aspect of the direct shared-target mode is that, conditional on the target, the release scores are independent. This independence facilitates a finite-sample mechanism-reliability guarantee. This guarantee is achieved without the necessity for independent target batches or assumptions regarding the dependence of release-level $p$-values. The methodology also established that the mean-only penalty employed is sharp, and derived a smooth-score certificate that includes additive target concentration.
Findings
In a registered study involving three requirements, the application of ProxyGuard's direct mode demonstrated a notable increase in power. Specifically, at a reliability level of 0.95, the power was raised from 5.6% to 64.2%. The named mode of ProxyGuard was observed to maintain a stronger performance under conditions characterized by high-signal evidence.
Potential Applications
Prospective audits leveraging ProxyGuard can encompass full-pipeline systems such as Rice–TVAE, which involves retraining on every draw. The methodology is also applicable to non-tabular text mechanisms.