Overview
Mixtures of Experts (MoE) are conditional mixture models characterized by both mixing proportions and component densities that depend on predictors. They find application in regression, classification, and model-based clustering for heterogeneous data. A challenge in MoE fitting arises from numerous or correlated predictors, leading to unstable or infeasible maximum likelihood estimation.
A regularized maximum likelihood framework has been proposed to address these challenges, enabling simultaneous parameter estimation and feature selection within MoE models. This framework specifically targets MoE where experts belong to the generalized linear model family, encompassing Gaussian, Poisson, and multinomial responses within a unified formulation.
Research Context
The utility of Mixtures of Experts (MoE) models spans several statistical learning tasks, including regression, classification, and model-based clustering, particularly for datasets exhibiting heterogeneity. The effectiveness of traditional maximum likelihood fitting in MoE models is compromised when the number of predictors is high or when predictors display strong correlations. Such conditions can lead to instability or render the estimation process computationally intractable.
Approach
The proposed framework introduces sparsity into both the gating network and the expert components of the MoE model. This sparsity is induced through the application of $\ell_1$ penalties. The objective function, a penalized log-likelihood, is maximized using a proximal Newton-EM algorithm.
Key aspects of the algorithm's design and operation include:
- The M-step of the algorithm reduces to solving weighted Lasso problems.
- These weighted Lasso problems incorporate closed-form coordinate-ascent updates.
- The algorithm does not necessitate a local quadratic approximation of the penalty function.
- It avoids any matrix inversion procedures.
- The method is designed to return exactly sparse estimates, eliminating the need for subsequent thresholding operations.
- A proximal Newton-type variant is incorporated to ensure a monotone increase of the penalized objective function during every iteration.
Findings
Evaluation of the proposed method was conducted on simulated data and five distinct real datasets. The observed outcomes indicated specific performance characteristics and advantages:
- The method successfully recovers the actual sparsity support present in the data.
- It delivers prediction accuracy that is competitive with existing state-of-the-art regularized MoE procedures.
- In many instances, the prediction accuracy achieved by the new method was superior to other regularized MoE approaches.
- Clustering accuracy demonstrated similar competitive, and often superior, performance compared to established methods.
Why This Matters
The development of this regularized maximum likelihood framework addresses a significant challenge in the application of Mixtures of Experts (MoE) models: the instability and infeasibility of fitting when dealing with high-dimensional or correlated predictors. By simultaneously performing parameter estimation and feature selection, the method provides a more robust and efficient approach for modeling heterogeneous data. The observed ability to recover actual sparsity support suggests an improved capacity for identifying relevant features, potentially leading to more interpretable and parsimonious models. The competitive, and often superior, performance in prediction and clustering accuracy underscores its practical utility for diverse tasks including regression, classification, and model-based clustering. Furthermore, the algorithmic design, which avoids local quadratic approximations and matrix inversions while ensuring exactly sparse estimates, contributes to computational efficiency and methodological clarity.
The source codes for the developed algorithms and their documentation are publicly available on Github at https://github.com/nv-thin/GLM-RMoE.