Overview
The study introduces a novel approach to Off-Policy Evaluation and Learning (OPE/L) within the domain of contextual bandits, termed Cross-Domain OPE/L. This formulation addresses limitations of existing OPE/L methods, particularly when confronting scenarios characterized by few-shot data, deterministic logging policies, or the presence of new actions. The proposed framework integrates logged datasets from both the target domain and multiple source domains to facilitate more robust policy evaluation and learning.
Research Context
OPE/L in contextual bandits is utilized in real systems for evaluating and learning new policies using historical logged data securely. However, current OPE/L methods encounter challenges in several prevalent scenarios. These include situations where data scarcity, such as few-shot data, exists; where logging policies are deterministic; or where new actions are introduced. Applications such as personalized medicine, content recommendations, education, and advertising frequently present these conditions. Existing methods often struggle in these contexts due to issues like high variance or limited exploration within the logged data, failing to effectively evaluate and optimize policies. The proposed Cross-Domain OPE/L setup aims to enable OPE/L even under these previously unsolved challenges.
The formulation of Cross-Domain OPE/L recognizes that historical data often exists not only from the specific target environment (e.g., a particular hospital, country, device, or user segment) but also from comparable environments (other hospitals, countries, devices, or segments). This availability of data from diverse domains underpins the utility of the Cross-Domain OPE/L framework.
Approach
The research proposes a Cross-Domain OPE/L problem setup. This setup relies on access to logged data from the target domain, where a new policy will be implemented, and additionally to logged datasets collected from other domains. To solve OPE/L within this new formulation, the researchers developed a new estimator. Concurrently, a novel policy gradient method was developed. Both the estimator and the policy gradient method are designed to leverage information from both the target and source datasets. This combined utilization of data from multiple domains is intended to address the identified limitations of traditional OPE/L methods.
Findings
Empirical evaluations indicated that the developed estimator and policy gradient method resulted in substantially enhanced OPE/L capabilities. This enhancement was observed specifically in situations that were previously unsolved by existing methods, including scenarios with few-shot data, deterministic logging policies, and new actions. The integration of logged data from both target and source datasets contributed to these improved performance metrics.
Why This Matters
The ability to accurately evaluate and learn new policies using historical data is crucial in many real-world applications where direct experimentation might be impractical or unethical. By enabling effective OPE/L in challenging scenarios such as limited data or new actions, this research extends the applicability of secure policy evaluation and learning. This could impact fields such as personalized medicine, content recommendations, education, and advertising by providing more robust methods for policy optimization.