Overview
This study investigates the mechanisms of generalization, memorization, and overfitting within modern score-based generative models, particularly focusing on their operation in high-dimensional tasks such as image, audio, and video synthesis. It proposes a theoretical framework for generative models that parallels existing theories of benign overfitting and algorithmic regularization previously developed for overparameterized neural networks within the supervised lazy-training regime. The research specifically examines denoising score matching implemented in a vector-valued reproducing kernel Hilbert space, utilizing an inner-product kernel.
Research Context
Modern score-based generative models have demonstrated empirical success in complex, high-dimensional generative tasks. These models inherently approach distribution learning by translating it into a series of regression problems. If these regression problems were solved precisely on a finite dataset, the models would theoretically reproduce the original training samples, implying a need for implicit or explicit regularization during training to achieve generalization beyond the training data. The study positions itself within the broader understanding of how such models, often overparameterized, manage to generalize despite their capacity to memorize.
Approach
The research methodology involves a theoretical analysis of denoising score matching within a vector-valued reproducing kernel Hilbert space defined by an inner-product kernel. The study focuses on the proportional high-dimensional regime, characterized by $n \asymp d$, where $n$ represents the number of samples and $d$ represents the dimensionality. Within this regime, the researchers derive exact risk trajectories for models undergoing gradient flow training.
Findings
The derived exact risk trajectories under gradient flow training reveal the presence of three distinct phases. Each phase is governed by qualitatively different estimators:
- A spectral estimator, which is associated with the model's ability to generalize.
- A pure-noise score, characterized by localized peaks that interpolate the training objective.
- An empirical Bayes estimator, which is responsible for memorizing the training data.
The analysis also extends to how these identified estimators combine along the reverse-time stochastic differential equation (SDE) intrinsic to generative processes. This characterization helps to understand the distribution of the samples generated by the model. The study indicates that while familiar mechanisms from supervised learning, such as kernel linearization and self-induced regularization originating from the nonlinear components of the kernel, are present, there is also a distinct phenomenology specific to generative modeling.
Why This Matters
Understanding the interplay between generalization, memorization, and overfitting in high-dimensional generative models is central to improving their robustness and predictive capabilities. This research offers a theoretical underpinning for how these complex models operate, revealing specific phases and estimators that dictate their performance. The identified mechanisms provide insights into how score-based generative models achieve their empirical success, moving beyond a black-box understanding towards a more mechanistic view.