cs.LGAug 19, 2026
SaveLévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
Organizations: GenML Laboratory, Department of Electrical Engineering and Computer Science and Engineering, Cyprus University of Technology, Limassol, Cyprus · Ethical AI Novelties, Cyprus
Abstract
Deep models for irregularly-sampled time series answer queries at arbitrary continuous timestamps, yet report nothing about how far each answer should be trusted. We show the attention layer itself can close that gap: with the right stochastic formulation, the pass that makes each prediction also reports, in closed form and at no extra cost, how far it should be trusted. We introduce Lévy Attention, a cross-attention operator whose output is a stochastic integral against an inhomogeneous Poisson random measure: query-key compatibilities assemble an intensity over a continuous (time x channel) index space, the measure scatters atoms under it, and the output averages an interpolated value field at those atoms. In expectation it reduces to a mollified cosine-kernel attention, so it replaces a softmax layer and trains with exact gradients. What softmax discards, the Poisson construction preserves in closed form: the evidence (total compatibility mass) and the disagreement (value spread). An exact variance identity makes their combination the root-mean-square deviation of the sampled operator, emitted by the deterministic pass with no trained head. Empirically, disagreement carries the signal, while the evidence factor swings from uninformative on dense data to strongly informative on sparse. On t-PatchGNN the operator swap costs at most 5.6% accuracy against a matched control and nothing on the sparsest dataset. The free disagreement signal improves on 20-pass MC dropout across matched five-seed suites, and scales a calibrated Gaussian whose zero-sample CRPS beats a fifty-draw sampler; a split-conformal wrapper reaches nominal coverage at every level, and one pass ranks 3,383 unseen patients by trust in 1.4 seconds.
Figures & tables
Figure 1: The Lévy Attention pipeline on a time series. Observation tokens sit at positions (orange stars; the channel coordinate is suppressed in the sketch). The intensity is the flashlight beam of § 1 ; the Poisson measure scatters atoms under its peaks (red dots) and none where the beam is dark, and the output averages the value field there. The same pass meters the beam: total mass ( evidence ), value spread ( disagreement ), and their combination ( theorem 2 ).
Figure 2: The uncertainty interface in use: three variables from three held-out PhysioNet records. One deterministic forward pass of the seed-0 interpolation checkpoint produces the continuous prediction curve and the shaded band at every point of the timeline (the validation calibration of § 5.1 ). The band tightens where observations are dense and opens in the gaps. The dashed outline is the 20-pass MC-dropout band under its own validation-calibrated scale. The band is a one-sigma interval, not an envelope; table 6 measures coverage at the , and levels.
| Model | PhysioNet ( ) | Activity ( ) | USHCN ( ) |
|---|---|---|---|
| DLinear ( Zeng et al., 2023 ) † | 41.86 0.05 | 4.03 0.01 | 6.21 0.00 |
| TimesNet ( Wu et al., 2023 ) † | 16.48 0.11 | 3.12 0.01 | 5.58 0.05 |
| Crossformer ( Zhang & Yan, 2023 ) † | 6.66 0.11 | 4.29 0.20 | 5.25 0.04 |
| Graph WaveNet ( Wu et al., 2019 ) † | 6.04 0.28 | 2.89 0.03 | 5.29 0.04 |
| GRU-D ( Che et al., 2018 ) † | 5.59 0.09 | 2.94 0.05 | 5.54 0.38 |
| SeFT ( Horn et al., 2020 ) † | 9.22 0.18 | 12.20 0.17 | 5.80 0.19 |
Table 1: IMTS forecasting under the t-PatchGNN protocol: MSE , mean std. † Quoted from Zhang et al. (2024) (Table 1; 5 seeds). ‡ Quoted from the same-protocol reruns of Luo et al. (2025) (5 seeds). Anchor the benchmark’s model run in our environment; all rows measured by us use 5 seeds, matching the benchmark’s seed protocol. Our rows share one untuned configuration across datasets. Percentages quoted against this table are computed from the unrounded means, not from the two-decimal cells. Bold: best per column; underline: second.
| Output block | PhysioNet ( ) | Activity ( ) | USHCN ( ) |
|---|---|---|---|
| Original (time-blind linear MLP) | 5.11 | 2.70 | 5.01 |
| Softmax decode layer MLP | 5.22 0.02 | 2.74 0.13 | 5.02 0.04 |
| Lévy decode layer MLP | 5.17 0.12 | 2.73 0.06 | 5.02 0.12 |
Table 2: Transplanting the decode layer into t-PatchGNN: test MSE under its own protocol (mean std over 3 seeds for the two transplants; the Original row is a single seed and carries no spread). The Original row is the unmodified model retrained in the same batch as the transplants. All three rows come from one batch, so the comparison is within-run throughout. Scales as in table 1 .
| Signal (1 free pass) | PhysioNet | Activity | USHCN |
|---|---|---|---|
| (evidence only) | / | / | / |
| (disagreement only) | / | / | / |
| (ours, combined) | / | / | / |
Table 3: Error ranking on the forecasting test sets (Spearman / AUSE ; mean std over 5 seeds), from the same single forward pass as the predictions of table 1 ; all three signals are free and available before the head has run.
| Signal | Passes | Spearman | AUSE | MAE |
| time gap, same variable § | 0 | – | ||
| time gap, any variable § | 0 | – | ||
| same-variable count § | 0 | – | ||
| windowed density § | 0 | – | ||
| softmax partition function (control) | 1 | – | ||
| softmax attention entropy (control) | 1 | – |
Table 4: PhysioNet 2012 interpolation: per-query error ranking. Every row is mean std over the same 5 seeds, one training per seed. The upper block holds estimators whose uncertainty is read out without any fit to labelled targets; the lower block holds estimators that fit theirs on labels, whether by changing the model, its training, or by fitting something new on a frozen one, each scored on its own errors. Bold: best AUSE in the upper block. “Passes”: forward passes at inference. MAE: each row’s test-MAE change against the point-head model (negative more accurate). § Data-only heuristics: observation pattern alone, no model and no forward pass. ∥ Attention-weighted value dispersion, measured on the matched softmax control. ¶ Last-layer Laplace ( Daxberger et al., 2021 ) . ∗∗ TNP-D ( Nguyen & Grover, 2022 ) , a Transformer Neural Process matched to our tokenisation, width, depth, optimiser and epoch budget. Recipes for all are in appendix B .
| Signal | Passes | Spearman | AUSE | MAE |
| time gap, same variable § | 0 | – | ||
| time gap, any variable § | 0 | – | ||
| same-variable count § | 0 | – | ||
| windowed density § | 0 | – | ||
| (evidence only) | 1 | – | ||
| (ours, combined) | 1 | – |
Table 5: USHCN interpolation in the sparse regime (30% masking; 5 seeds, one training per seed; same protocol and metrics as table 4 , over the same-model signals and the trained and sampling baselines). Spearman / AUSE , mean std. No AUSE entry is bolded: the block minimum is a five-training ensemble, and among the one-pass rows the closest comparison (DDU) splits the seeds – . MAE is each row’s test-MAE change against the point-head model, and (–) marks the upper-block rows that read that model’s own predictions. § Data-only heuristics as in table 4 : no model, no forward pass.
| Predictive distribution, 30% masked | val-calibrated | raw |
|---|---|---|
| deep ensemble ( models) | ||
| post-hoc variance head (frozen backbone) | ||
| ours : Gaussian on , 0 samples | – | |
| -NLL head | ||
| Gaussian-NLL head | ||
| TNP-D, natively probabilistic |
Table 6: Predictive distributions on PhysioNet 2012 interpolation. Top: CRPS at 30% masking (mean std over five seeds) for the distributional baselines of table 4 ; the last two rows are the same model read as a point prediction and by sampling (§ 2.4 ). Every row receives the same affine validation calibration; the raw column drops it and is empty for , which carries no units of the prediction. Bold: lowest calibrated CRPS. Bottom: empirical coverage and median interval width of the Gaussian and of its split-conformal wrapper, one checkpoint.
| Method | Samples per target | CRPS |
|---|---|---|
| CSDI (conditional diffusion) | 100 | |
| ours : Gaussian on | 0 | |
| unconditional diffusion | 100 | |
| GP-based imputers | – | and up |
Table 7: Our model under the unmodified CSDI imputation protocol (PhysioNet, missing, five folds). All rows use the quantile-CRPS estimator of Tashiro et al. (2021) , imported unmodified; for our Gaussian predictive its nineteen quantiles are available in closed form, so our row needs no sample. Our row is measured; the others are the values Tashiro et al. (2021) report under the same protocol.
Figure 3: Coverage-degradation stress test (PhysioNet interpolation, fixed targets). Both curves are normalised to their full-coverage value. Grey numbers give the mean evidence at each level. The model’s self-reported deviation tracks its true degradation without ever seeing an error.
| Screening: patient-level triage quality | ||||
| Trust signal | Passes | Wall-clock | Spearman | Flagged decile |
| (ours) | 1 | 1.35 s | ||
| (ours) | 1 | 1.35 s | ||
| (ours) | 1 | 1.35 s | ||
| MC dropout spread | 20 | 18.5 s | ||
| Deep ensemble spread ¶ | 5 | 5.3 s | ||
Table 8: Cohort screening: the set-a-trained model on the untouched set-b partition (3,383 records; 30% hidden targets per record, 373,713 queries), one RTX 3090. Upper block: patient-level triage quality, mean std over five same-recipe trainings; wall-clock is the full screening pass on the reported checkpoint; bold marks the best wall-clock and the best mean of each quality column. Flagged decile: mean per-patient MAE of the 10% least-trusted patients, as % change against the cohort mean; larger means better ranking. ¶ The ensemble is scored on its own averaged predictions (cohort MAE against the single model’s ). ¶¶ Post-hoc head: fitted per seed on frozen set-a features ( MAE ). ∥ Softmax value dispersion: matched control, single run (cohort MAE ). Neither is separately timed. Lower block: wall-clock for the dense continuum (36 variables 288 times per record).
| Index space | Grid | MAE |
|---|---|---|
| time only | 8 | 0.386 |
| time only | 32 | 0.371 |
| time only | 64 | 0.365 |
| time learned channel (ours) | 0.302 |
Table 9: Ablations, all on PhysioNet interpolation. Top — index-space geometry (MAE ; each row is compared against the matched softmax control of its own run, in row order , , , ); bold: best MAE. Percentages quoted in § 5.7 are computed from the unrounded values, not from the three-decimal cells. Middle — backbone size (seed 0, one recipe): is Spearman between signal and error, PICP the fraction of targets inside the nominal-80% interval under the per-size validation-calibrated scale; bold marks the best per row for MAE, and AUSE, while PICP is scored against the nominal so no entry is bolded. Bottom — grid resolution (seed 0), time cells channel cells; bold: best AUSE.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Signal | PhysioNet | Activity | USHCN |
|---|---|---|---|
| Lévy model (free, 1 pass) | |||
| MC dropout (20 passes) | |||
| AUSE on the same runs | |||
Table 10: Free against paid under the forecasting protocol, all three datasets. Superseded on PhysioNet and USHCN by the five-seed suites of tables 4 and 5 ; on Activity it is the paper’s only comparison against a paid estimator. Spearman except in the block labelled AUSE ; mean std over 3 seeds. MC dropout ranks the Lévy run’s own error stream; the softmax rows rank the matched control’s. § 5.4 reads the numbers.