Overview
This paper presents a method for scalable uncertainty quantification (UQ) in deep learning weather models using empirical Neural Tangent Kernels (NTK-UQ). The framework leverages last-layer empirical features to generate uncertainty estimates for deterministic forecasts, a capability noted as critical for high-stakes decisions concerning extreme weather events. The approach avoids model retraining and requires only a single matrix-vector product per sample for inference-time uncertainty.
Research Context
Deep learning models for weather forecasting have advanced to match the accuracy of numerical weather prediction (NWP) systems, while operating with significantly reduced computational times. However, these deep learning models typically produce deterministic forecasts, lacking the uncertainty estimates that are essential for informing critical decisions, especially during extreme weather phenomena. Addressing this gap by providing robust and scalable uncertainty quantification without compromising the speed advantages is the focus of this research.
Approach
The proposed NTK-UQ framework integrates with deep learning weather models by utilizing last-layer empirical features to derive uncertainty estimates. The theoretical analysis supporting NTK-UQ identifies two primary mechanisms influencing the quality of uncertainty quantification, both dependent on the model architecture:
Variance Collapse Mechanism
This mechanism explains conditions under which UQ can fail. Specifically, when the eigenvalue truncation rank approaches the effective rank of the feature space, the Gaussian Process (GP) correction term can consume nearly all prior variance. This process diminishes the ability to discriminate between distinct conditions, such as tropical cyclones and routine weather patterns. Architectures characterized by concentrated spectra, such as those employing spectral operators, demand aggressive truncation (e.g., $k \leq 10$). In contrast, attention-based models are observed to tolerate full-rank computation without encountering this issue.
Decomposition Performance
The effectiveness of feature decomposition is linked to the non-Gaussian, heavy-tailed statistical properties characteristic of extreme weather events. Independent Component Analysis (ICA) is employed to take advantage of higher-order statistics, including kurtosis and negentropy, to specifically isolate heavy-tailed features associated with extreme events. This approach is noted to achieve higher discrimination compared to Singular Value Decomposition (SVD), which primarily captures second-order variance. A data-driven selection rule, based on the feature eigenspectrum concentration ratio, determines whether ICA or SVD is the superior decomposition method for a given architecture. This rule correctly assigned the more effective decomposition for all four evaluated architectures.
The NTK-UQ framework functions without requiring retraining of the underlying deep learning model. The computation of uncertainty during inference is efficient, involving a single matrix-vector product per sample.
Findings
- NTK-UQ, when compared to split conformal prediction (identified as a natural post-hoc baseline), achieved sharper prediction intervals, specifically by 31% to 37% at a 90% coverage level.
- The framework uniquely produces adaptive prediction intervals. These intervals scale in magnitude according to the severity of the extreme weather event being forecasted. This adaptive characteristic is noted as a capability that conformal prediction methods, by their construction, cannot achieve.
- The theoretical analysis indicates that UQ quality is dependent on architectural characteristics through the mechanisms of variance collapse and decomposition performance.
- Architectures with concentrated spectra, such as those employing spectral operators, required aggressive eigenvalue truncation ($k \leq 10$) to avoid variance collapse. Attention-based models, however, were found to tolerate full-rank computation.
- ICA was found to be more effective than SVD for isolating heavy-tailed extreme-event features due to its exploitation of higher-order statistics (kurtosis, negentropy).
- A data-driven selection rule, based on the feature eigenspectrum concentration ratio, successfully prescribed the superior decomposition (ICA or SVD) for all four architectures evaluated.
- Inference-time uncertainty estimation with NTK-UQ requires only a single matrix-vector product per sample and does not necessitate model retraining.
Why This Matters
The ability to provide uncertainty estimates in deep learning weather models, particularly for extreme events, addresses a critical gap in high-stakes decision-making processes. The documented gains in prediction interval sharpness and the generation of adaptive intervals that reflect event severity offer a more robust tool for weather forecasting analysts and decision-makers.