Overview
This research presents a case study on multi-task ranking within industrial-scale heterogeneous feeds, specifically focusing on Google Discover. The study introduces a novel architecture, HA-MoE, designed to manage content heterogeneity, and an observability framework, LENS, for diagnostics. Evaluation, conducted through both offline and online methods, indicates performance improvements in recommender system metrics.
Research Context
Recommender systems operating in highly homogeneous environments (e.g., music-only or video-only platforms) differ significantly from those dealing with heterogeneous recommendation feeds. Google Discover serves as an example of the latter, integrating diverse content types from the open web, including web articles, long-form and short-form videos, and user-generated content (UGC). A key challenge in such systems is that different content types possess distinct feature densities and user interaction patterns. The objective is to construct a unified ranking model that maintains high performance across this heterogeneity without succumbing to negative transfer or majority bias, which is identified as a significant industrial challenge.
Approach
The research introduces two primary components: HA-MoE and LENS.
HA-MoE Architecture
HA-MoE (Heterogeneity-Adaptive Multi-gated Mixture-of-Experts) is an architecture designed to incorporate explicit heterogeneity context into both its gating networks and expert representations. This integration aims to facilitate effective specialization among experts while limiting increases in operational overhead.
LENS Observability Framework
LENS is a lightweight observability framework developed to support reliable deployment of the ranking system. Its purpose is to provide interpretable diagnostics of expert specialization. Additionally, LENS tracks the functional heterogeneity across continuous retraining cycles of the model.
Evaluation Methodology
The evaluation of the proposed method utilized Dual-Level AUC (DL-AUC). DL-AUC is described as a heterogeneity-aware evaluation metric that combines global ranking performance with cross-segment ranking correctness. This metric was applied during offline evaluations using a large-scale industrial dataset. Further validation involved online A/B testing.
Findings
- Offline evaluations on a large-scale industrial dataset demonstrated consistent improvements over baseline models.
- Online A/B testing confirmed gains in feed activity metrics.
- Online A/B testing also confirmed gains in exploration metrics.
- Collectively, both offline and online results validate the effectiveness of the HA-MoE and LENS approach for managing heterogeneity within industrial-scale recommender systems.
Why This Matters
The development and validation of the HA-MoE architecture and LENS framework address the complex issue of ranking diverse content in platforms like Google Discover. By managing heterogeneity effectively, the system aims to improve user experience through enhanced feed activity and content exploration, thereby supporting the performance of large-scale industrial recommender systems.