Overview
Stochastic Gradient Descent (SGD) performance can degrade in the later stages of training due to anisotropic curvature and gradient noise. This work investigates preconditioned SGD within a geometry defined by a symmetric positive definite matrix $\mathbf{M}$. It develops bounds that connect the convergence rate and the stochastic noise floor to quantities dependent on $\mathbf{M}$. For nonconvex objectives, a guarantee for basin stability, contingent on the preconditioner, is established.
Research Context
The late-stage slowing of SGD is a recognized issue, attributed to two primary factors: anisotropic curvature and gradient noise. Preconditioning is a technique used to address these challenges by transforming the optimization problem's geometry. This research focuses on understanding the design principles for effective SGD preconditioners, particularly in the context of Scientific Machine Learning (SciML), where achieving low training loss with stochastic updates is linked to physical fidelity, numerical stability, and constraint satisfaction.
Approach
The research analyzes preconditioned SGD by introducing a geometry induced by a symmetric positive definite matrix $\mathbf{M}$. This framework allows for the derivation of bounds on both the convergence rate and the stochastic noise floor. These bounds are expressed in terms of $\mathbf{M}$-dependent quantities.
- Convergence Rate: The convergence rate is governed by an effective condition number within the $\mathbf{M}$-metric.
- Stochastic Noise Floor: The noise floor is determined by the product of that effective condition number and the preconditioned noise level.
For nonconvex objectives, the study establishes a basin-stability guarantee that depends on the chosen preconditioner. This guarantee states that when smoothness and basin size are measured using the $\mathbf{M}$-norm, the probability of iterates remaining within a well-behaved local region possesses an explicit lower bound.
The framework encompasses both diagonal/adaptive and curvature-aware preconditioners.
Findings
- The convergence rate of preconditioned SGD is controlled by an effective condition number in the $\mathbf{M}$-metric.
- The stochastic noise floor of preconditioned SGD is controlled by the product of the effective condition number in the $\mathbf{M}$-metric and the preconditioned noise level.
- For nonconvex objectives, a preconditioner-dependent basin-stability guarantee exists, providing an explicit lower bound on the probability of iterates remaining in a well-behaved local region when smoothness and basin size are measured in the $\mathbf{M}$-norm.
- A design principle is proposed: choose $\mathbf{M}$ to improve local conditioning while simultaneously attenuating noise.
- Experiments conducted on a quadratic diagnostic and three SciML benchmarks supported the predicted rate-floor behavior.
Why This Matters
In Scientific Machine Learning (SciML), achieving small training loss using stochastic updates is directly related to ensuring physical fidelity, numerical stability, and the satisfaction of constraints. The proposed framework and design principles offer a way to address challenges associated with SGD in these contexts by guiding the selection of preconditioners to improve training performance and stability.