Overview
Scaling recommendation models represents a central challenge in recommender systems research. Recent work has introduced RankMixer as a solution for scalable performance, utilizing a unified token representation and alternating token mixing with per-token feedforward networks (P-FFNs). However, RankMixer models exhibit embedding collapse, characterized by low effective rank in learned representations, which consequently limits expressivity and underutilizes the expanded representation space.
Research Context
The issue of embedding collapse in RankMixer models was identified through empirical analysis and theoretical insights. This collapse manifests as a damped oscillatory trajectory in the effective-rank evolution across model layers. The investigation pinpoints rigid token mixing and P-FFN modules as the primary contributing factors to this phenomenon.
The RankMixer architecture operates by processing a unified token representation through sequential layers. Each layer involves a token mixing step, which facilitates interaction across different tokens, followed by a P-FFN application to each token independently. This alternating structure is designed to achieve scalable performance in recommendation tasks.
Approach
To address the identified embedding collapse, the researchers proposed a novel architecture named RankElastor. This architecture is designed to generate spectrum-robust representations and provide provable mitigation of collapse. RankElastor integrates two primary components:
- Parameterized Full Mixing: This component aims to enhance token mixing by introducing a parameterized approach. The objective is to enable more expressive token mixing while simultaneously improving spectral robustness within the model.
- GLU-Improved P-FFNs: This component focuses on stabilizing representation spectra. It incorporates GLU-style (Gated Linear Unit) FFN modules within the per-token feedforward networks. The use of GLU-style FFNs is intended to contribute to the overall stability of the representation spectra, thereby countering collapse.
The design of RankElastor explicitly targets the mechanisms implicated in embedding collapse within RankMixer models. By modifying both the token mixing and the P-FFN stages, it seeks to reshape the effective-rank dynamics throughout the network layers.
Findings
Extensive experiments were conducted using large-scale industrial datasets to evaluate the performance of RankElastor. The findings indicate that RankElastor consistently improved recommendation performance when compared to existing methods. Furthermore, the architecture demonstrably mitigated embedding collapse, addressing the core issue identified in RankMixer models.
The experiments also revealed that RankElastor exhibited robust scaling behavior. This suggests that the proposed modifications not only resolve the collapse issue but also maintain or enhance the model's ability to scale effectively for larger datasets or more complex recommendation tasks.
Why This Matters
The mitigation of embedding collapse in recommendation models improves the expressivity of learned representations, which can lead to better predictive accuracy and more granular understanding of user preferences or item characteristics in large-scale recommender systems. The robust scaling behavior of RankElastor suggests potential for maintaining performance benefits as datasets and models grow in size.