Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Models

Overview

Deep models designed for irregularly-sampled time series commonly address queries at continuous timestamps but typically do not provide an assessment of prediction trustworthiness. A novel approach, termed Lévy Attention, integrates predictive uncertainty directly into the attention layer itself. This method utilizes a stochastic formulation that allows for the simultaneous generation of predictions and their associated uncertainty during a single computational pass, eliminating the need for extra computational cost or dedicated trained heads.

Research Context

Conventional deep models for irregularly-sampled time series deliver point predictions for queries at arbitrary continuous timestamps. However, these models generally lack mechanisms to quantify the reliability or confidence associated with each prediction. This gap necessitates methods that can provide an indication of how far each answer should be trusted, which is critical for applications where prediction certainty is as important as the prediction itself.

Approach

Lévy Attention operates as a cross-attention operator, conceptualized as a stochastic integral against an inhomogeneous Poisson random measure. Its construction involves query-key compatibilities that form an intensity function across a continuous (time x channel) index space. The Poisson measure then scatters 'atoms' under this intensity, and the operator's output is derived by averaging an interpolated value field at these atom locations.

In terms of its expected behavior, Lévy Attention reduces to a mollified cosine-kernel attention. This characteristic enables it to serve as a replacement for a softmax layer and facilitates training with exact gradients. A key differentiation from softmax layers is its capacity to preserve specific information: the evidence (total compatibility mass), denoted $\Lambda_q$, and the disagreement (value spread), denoted $\mathrm{tr}\,\Sigma_V(q)$.

The method leverages an exact variance identity to combine these elements into a single root-mean-square deviation, $\hat\sigma(q)=\sqrt{\mathrm{tr}\,\Sigma_V(q)\,\varphi(\Lambda_q)}$. This $\hat\sigma(q)$ is emitted by the deterministic pass alongside the prediction, requiring no separate trained head for its generation. This constitutes the mechanism for reporting predictive uncertainty.

Findings

• The Poisson construction within Lévy Attention preserves two specific metrics in closed form: the evidence $\Lambda_q$ (total compatibility mass) and the disagreement $\mathrm{tr}\,\Sigma_V(q)$ (value spread).
• An exact variance identity allows for the combination of these metrics into $\hat\sigma(q)=\sqrt{\mathrm{tr}\,\Sigma_V(q)\,\varphi(\Lambda_q)}$, which represents the root-mean-square deviation of the sampled operator. This $\hat\sigma(q)$ is generated directly by the deterministic prediction pass without requiring a separate trained head.
• Empirical evaluation indicated that disagreement carries the primary signal for uncertainty.
• The evidence factor's informativeness varied; it was largely uninformative on dense datasets but became strongly informative on sparse datasets.
• When the Lévy Attention operator replaced components in a t-PatchGNN architecture, the accuracy cost was at most 5.6% compared to a matched control. On the sparsest dataset, no accuracy cost was observed.
• The free disagreement signal provided by Lévy Attention demonstrated improvement over 20-pass Monte Carlo dropout across matched five-seed suites.
• The $\hat\sigma$ metric effectively scaled a calibrated Gaussian distribution. Its zero-sample Continuous Ranked Probability Score (CRPS) outperformed a fifty-draw sampler.
• A split-conformal wrapper, when applied, allowed Lévy Attention to achieve nominal coverage at every tested level.
• The system was capable of ranking 3,383 unseen patients by trust in approximately 1.4 seconds using a single pass.

Why This Matters

The capacity to provide predictive uncertainty in a single pass addresses a critical limitation in deep models for irregularly-sampled time series, particularly where the reliability of a prediction is crucial. The efficiency of generating uncertainty metrics without additional computational passes or specialized network components suggests practical advantages for real-time or resource-constrained applications. The observed performance gains in uncertainty quantification, especially on sparse data and over multi-pass methods, indicates improved robustness and trustworthiness of model outputs.