Overview
This research addresses the objective of demonstrating positive effects within adaptive experiments, a common goal in various fields including online platforms, healthcare, biotechnology, and the social sciences. The core aim is not precise treatment effect estimation but rather to show that at least one candidate intervention generates a positive effect for some subpopulation on a measured outcome. The study formalizes this objective as testing the global null within a threshold bandit framework.
Two distinct inference procedures were developed that maintain validity under general adaptive sampling conditions. One procedure involves pooling information from promising arms, while the other relies on time-uniform multiple testing of individual arm means. To support the latter procedure, a moderate-deviations principle for the sequential $t$-statistic was established, which provides justification for asymptotic confidence sequences in scenarios where the number of arms is large relative to the sample size.
Furthermore, the research illustrates how adaptive designs can target the proposed statistics by reformulating experimental design as bandit optimization. In this reformulation, an arm's reward is defined by its signal-to-noise ratio. An allocation rule was analyzed for which a logarithmic regret bound was established.
Research Context
Adaptive experiments are widely employed across diverse domains. In many of these applications, the primary research question deviates from precisely quantifying a treatment effect. instead, the goal is often to establish the existence of a positive impact from at least one intervention within a given subpopulation and for a specific measured outcome. This objective necessitates statistical methods capable of handling adaptive sampling strategies while providing valid inferential guarantees.
Approach
The research formalized the objective of demonstrating a positive effect as testing the global null in a threshold bandit framework. This formalization provided the foundation for developing the inference procedures.
Two specific inference procedures were developed:
- A procedure that pools information from promising arms.
- A procedure based on time-uniform multiple testing of individual arm means.
For the second procedure, a theoretical advancement was made by establishing a moderate-deviations principle for the sequential $t$-statistic. This principle is crucial for justifying the use of asymptotic confidence sequences, particularly in contexts where the number of experimental arms is substantial compared to the available sample size.
To demonstrate the application of adaptive designs in targeting the proposed statistics, the research reframed experimental design as a bandit optimization problem. In this revised framework, the reward for an arm is defined by its signal-to-noise ratio. An allocation rule was specifically analyzed, and a logarithmic regret bound was established for this rule.
The developed methods were subsequently applied and evaluated in a simulation study. This simulation study focused on targeting unconditional cash transfer programs, providing an illustrative example of the practical utility of the proposed inference procedures and adaptive design strategies.
Findings
- The objective of demonstrating at least one positive intervention effect can be formalized as testing the global null in a threshold bandit framework.
- Two inference procedures were developed: one pooling information across promising arms and another based on time-uniform multiple testing of individual arm means, both valid under general adaptive sampling.
- A moderate-deviations principle for the sequential $t$-statistic was established, supporting asymptotic confidence sequences when the number of arms is large relative to the sample size.
- Recasting experimental design as bandit optimization with an arm's reward as its signal-to-noise ratio facilitates targeting the proposed statistics.
- An analyzed allocation rule demonstrated a logarithmic regret bound.
Potential Applications
The methods developed were applied in a simulation study focused on targeting unconditional cash transfer programs. This suggests potential applicability in social science interventions where identifying effective programs for specific subpopulations is a key objective.