Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
Authors: Sotirios P. Chatzis, Loukas Papadoulas
Organizations: GenML Laboratory, Department of Electrical Engineering and Computer Science and Engineering, Cyprus University of Technology, Limassol, Cyprus · Ethical AI Novelties, Cyprus
Deep models for irregularly-sampled time series answer queries at arbitrary continuous timestamps, yet report nothing about how far each answer should be trusted. We show the attention layer itself can close that gap: with the right stochastic formulation, the pass that makes each prediction also reports, in closed form and at no extra cost, how far it should be trusted. We introduce Lévy Attention, a cross-attention operator whose output is a stochastic integral against an inhomogeneous Poisson random measure: query-key compatibilities assemble an intensity over a continuous (time x channel) index space, the measure scatters atoms under it, and the output averages an interpolated value field at those atoms. In expectation it reduces to a mollified cosine-kernel attention, so it replaces a softmax layer and trains with exact gradients. What softmax discards, the Poisson construction preserves in closed form: the evidence Λq (total compatibility mass) and the disagreement trΣV(q) (value spread). An exact variance identity makes their combination σ^(q)=trΣV(q)φ(Λq) the root-mean-square deviation of the sampled operator, emitted by the deterministic pass with no trained head. Empirically, disagreement carries the signal, while the evidence factor swings from uninformative on dense data to strongly informative on sparse. On t-PatchGNN the operator swap costs at most 5.6% accuracy against a matched control and nothing on the sparsest dataset. The free disagreement signal improves on 20-pass MC dropout across matched five-seed suites, and σ^ scales a calibrated Gaussian whose zero-sample CRPS beats a fifty-draw sampler; a split-conformal wrapper reaches nominal coverage at every level, and one pass ranks 3,383 unseen patients by trust in 1.4 seconds.
Figures & tables
Figure 1: The Lévy Attention pipeline on a time series. Observation tokens sit at positions si=(ti,ci) (orange stars; the channel coordinate is suppressed in the sketch). The intensity λq(s) is the flashlight beam of § 1 ; the Poisson measure scatters atoms under its peaks (red dots) and none where the beam is dark, and the output averages the value field there. The same pass meters the beam: total mass Λq ( evidence ), value spread \trΣV(q) ( disagreement ), and their combination σ^(q) ( theorem 2 ).
Figure 2: The uncertainty interface in use: three variables from three held-out PhysioNet records. One deterministic forward pass of the seed-0 interpolation checkpoint produces the continuous prediction curve and the shaded band aσ^(q)+b at every point of the timeline (the validation calibration of § 5.1 ). The band tightens where observations are dense and opens in the gaps. The dashed outline is the 20-pass MC-dropout band under its own validation-calibrated scale. The band is a one-sigma interval, not an envelope; table 6 measures coverage at the 80 , 90 and 95% levels.
Model
PhysioNet ( ×10−3 )
Activity ( ×10−3 )
USHCN ( ×10−1 )
DLinear ( Zeng et al., 2023 ) †
41.86 ± 0.05
4.03 ± 0.01
6.21 ± 0.00
TimesNet ( Wu et al., 2023 ) †
16.48 ± 0.11
3.12 ± 0.01
5.58 ± 0.05
Crossformer ( Zhang & Yan, 2023 ) †
6.66 ± 0.11
4.29 ± 0.20
5.25 ± 0.04
Graph WaveNet ( Wu et al., 2019 ) †
6.04 ± 0.28
2.89 ± 0.03
5.29 ± 0.04
GRU-D ( Che et al., 2018 ) †
5.59 ± 0.09
2.94 ± 0.05
5.54 ± 0.38
SeFT ( Horn et al., 2020 ) †
9.22 ± 0.18
12.20 ± 0.17
5.80 ± 0.19
Table 1: IMTS forecasting under the t-PatchGNN protocol: MSE ↓ , mean ± std. † Quoted from Zhang et al. (2024) (Table 1; 5 seeds). ‡ Quoted from the same-protocol reruns of Luo et al. (2025) (5 seeds). Anchor = the benchmark’s model run in our environment; all rows measured by us use 5 seeds, matching the benchmark’s seed protocol. Our rows share one untuned configuration across datasets. Percentages quoted against this table are computed from the unrounded means, not from the two-decimal cells. Bold: best per column; underline: second.
Output block
PhysioNet ( ×10−3 )
Activity ( ×10−3 )
USHCN ( ×10−1 )
Original (time-blind linear + MLP)
5.11
2.70
5.01
Softmax decode layer + MLP
5.22 ± 0.02
2.74 ± 0.13
5.02 ± 0.04
Lévy decode layer + MLP
5.17 ± 0.12
2.73 ± 0.06
5.02 ± 0.12
Table 2: Transplanting the decode layer into t-PatchGNN: test MSE ↓ under its own protocol (mean ± std over 3 seeds for the two transplants; the Original row is a single seed and carries no spread). The Original row is the unmodified model retrained in the same batch as the transplants. All three rows come from one batch, so the comparison is within-run throughout. Scales as in table 1 .
Signal (1 free pass)
PhysioNet
Activity
USHCN
1/Λq (evidence only)
0.01±0.04 / 0.59±0.03
0.03±0.05 / 0.61±0.05
0.40±0.10 / 0.43±0.06
\trΣV(q) (disagreement only)
0.22±0.10 / 0.49±0.09
0.09±0.06 / 0.55±0.06
0.47±0.11 / 0.40±0.07
σ^(q) (ours, combined)
0.12±0.08 / 0.51±0.05
0.07±0.06 / 0.56±0.06
0.49±0.10 / 0.37±0.06
Table 3: Error ranking on the forecasting test sets (Spearman ρ↑ / AUSE ↓ ; mean ± std over 5 seeds), from the same single forward pass as the predictions of table 1 ; all three signals are free and available before the head has run.
Signal
Passes
Spearman ρ↑
AUSE ↓
Δ MAE
time gap, same variable §
0
0.019±.016
0.698±.024
–
time gap, any variable §
0
0.009±.002
0.658±.014
–
same-variable count §
0
0.043±.016
0.606±.015
–
windowed density §
0
0.038±.018
0.621±.022
–
softmax partition function (control)
1
0.060±.019
0.605±.013
–
softmax attention entropy (control)
1
0.009±.022
0.629±.016
–
Table 4: PhysioNet 2012 interpolation: per-query error ranking. Every row is mean ± std over the same 5 seeds, one training per seed. The upper block holds estimators whose uncertainty is read out without any fit to labelled targets; the lower block holds estimators that fit theirs on labels, whether by changing the model, its training, or by fitting something new on a frozen one, each scored on its own errors. Bold: best AUSE in the upper block. “Passes”: forward passes at inference. Δ MAE: each row’s test-MAE change against the point-head model (negative = more accurate). § Data-only heuristics: observation pattern alone, no model and no forward pass. ∥ Attention-weighted value dispersion, measured on the matched softmax control. ¶ Last-layer Laplace ( Daxberger et al., 2021 ) . ∗∗ TNP-D ( Nguyen & Grover, 2022 ) , a Transformer Neural Process matched to our tokenisation, width, depth, optimiser and epoch budget. Recipes for all are in appendix B .
Signal
Passes
Spearman ρ↑
AUSE ↓
Δ MAE
time gap, same variable §
0
0.009±.007
0.728±.026
–
time gap, any variable §
0
0.005±.006
0.735±.034
–
same-variable count §
0
−0.018±.025
0.759±.027
–
windowed density §
0
0.030±.011
0.700±.015
–
1/Λq (evidence only)
1
0.341±.239
0.506±.214
–
σ^(q) (ours, combined)
1
0.415±.241
0.412±.153
–
Table 5: USHCN interpolation in the sparse regime (30% masking; 5 seeds, one training per seed; same protocol and metrics as table 4 , over the same-model signals and the trained and sampling baselines). Spearman ρ↑ / AUSE ↓ , mean ± std. No AUSE entry is bolded: the block minimum is a five-training ensemble, and among the one-pass rows the closest comparison (DDU) splits the seeds 3 – 2 . Δ MAE is each row’s test-MAE change against the point-head model, and (–) marks the upper-block rows that read that model’s own predictions. § Data-only heuristics as in table 4 : no model, no forward pass.
Predictive distribution, 30% masked
val-calibrated
raw
deep ensemble ( 5× models)
0.229±.005
0.258±.005
post-hoc variance head (frozen backbone)
0.241±.005
0.248±.006
ours : Gaussian on σ^ , 0 samples
0.244±.006
–
β -NLL head
0.255±.012
0.256±.011
Gaussian-NLL head
0.269±.011
0.271±.011
TNP-D, natively probabilistic
0.295±.020
0.298±.020
Table 6: Predictive distributions on PhysioNet 2012 interpolation. Top: CRPS ↓ at 30% masking (mean ± std over five seeds) for the distributional baselines of table 4 ; the last two rows are the same model read as a point prediction and by sampling (§ 2.4 ). Every row receives the same affine validation calibration; the raw column drops it and is empty for σ^ , which carries no units of the prediction. Bold: lowest calibrated CRPS. Bottom: empirical coverage and median interval width of the Gaussian and of its split-conformal wrapper, one checkpoint.
Method
Samples per target
CRPS ↓
CSDI (conditional diffusion)
100
0.238±0.001
ours : Gaussian on σ^
0
0.327±0.005
unconditional diffusion
100
0.360
GP-based imputers
–
0.489 and up
Table 7: Our model under the unmodified CSDI imputation protocol (PhysioNet, 10% missing, five folds). All rows use the quantile-CRPS estimator of Tashiro et al. (2021) , imported unmodified; for our Gaussian predictive its nineteen quantiles are available in closed form, so our row needs no sample. Our row is measured; the others are the values Tashiro et al. (2021) report under the same protocol.
Figure 3: Coverage-degradation stress test (PhysioNet interpolation, fixed targets). Both curves are normalised to their full-coverage value. Grey numbers give the mean evidence Λq at each level. The model’s self-reported deviation tracks its true degradation without ever seeing an error.
Screening: patient-level triage quality
Trust signal
Passes
Wall-clock
Spearman ρ↑
Flagged decile
\trΣV (ours)
1
1.35 s
0.43±.08
+29.9±2.8%
σ^ (ours)
1
1.35 s
0.18±.05
+18.6±1.4%
1/Λq (ours)
1
1.35 s
0.05±.03
+9.4±0.7%
MC dropout spread
20
18.5 s
0.49±.01
+29.6±0.7%
Deep ensemble spread ¶
5
5.3 s
0.52±.01
+32.9±0.9%
Table 8: Cohort screening: the set-a-trained model on the untouched set-b partition (3,383 records; 30% hidden targets per record, 373,713 queries), one RTX 3090. Upper block: patient-level triage quality, mean ± std over five same-recipe trainings; wall-clock is the full screening pass on the reported checkpoint; bold marks the best wall-clock and the best mean of each quality column. Flagged decile: mean per-patient MAE of the 10% least-trusted patients, as % change against the cohort mean; larger means better ranking. ¶ The ensemble is scored on its own averaged predictions (cohort MAE 0.305±0.002 against the single model’s 0.322±0.003 ). ¶¶ Post-hoc head: fitted per seed on frozen set-a features ( Δ MAE =0 ). ∥ Softmax value dispersion: matched control, single run (cohort MAE 0.306 ). Neither is separately timed. Lower block: wall-clock for the dense continuum (36 variables × 288 times per record).
Index space S
Grid
MAE
time only
8
0.386
time only
32
0.371
time only
64
0.365
time × learned channel (ours)
16×8
0.302
Table 9: Ablations, all on PhysioNet interpolation. Top — index-space geometry (MAE ↓ ; each row is compared against the matched softmax control of its own run, in row order 0.289 , 0.288 , 0.286 , 0.298 ); bold: best MAE. Percentages quoted in § 5.7 are computed from the unrounded values, not from the three-decimal cells. Middle — backbone size (seed 0, one recipe): ρ is Spearman between signal and error, PICP the fraction of targets inside the nominal-80% interval under the per-size validation-calibrated scale; bold marks the best per row for MAE, ρ and AUSE, while PICP is scored against the nominal 0.80 so no entry is bolded. Bottom — grid resolution (seed 0), Lt×Lv time cells × channel cells; bold: best AUSE.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Signal
PhysioNet
Activity
USHCN
Lévy model (free, 1 pass)
σ^(q)
0.04±0.03
0.07±0.05
0.49±0.15
\trΣV(q)
0.21±0.05
0.08±0.06
0.55±0.12
1/Λq
−0.09±0.06
0.04±0.03
0.26±0.23
MC dropout (20 passes)
0.24±0.05
−0.02±0.01
0.52±0.09
AUSE ↓ on the same runs
Appendix
Table 10: Free against paid under the forecasting protocol, all three datasets. Superseded on PhysioNet and USHCN by the five-seed suites of tables 4 and 5 ; on Activity it is the paper’s only comparison against a paid estimator. Spearman ρ↑ except in the block labelled AUSE ↓ ; mean ± std over 3 seeds. MC dropout ranks the Lévy run’s own error stream; the softmax rows rank the matched control’s. § 5.4 reads the numbers.
We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.
Transformers provide a state-of-the-art modeling framework, yet poor calibration limits their reliability in safety-critical applications. A promising direction addresses this issue by interpreting attention as a Gaussian process (GP) posterior, which enables principled uncertainty calibration but incurs cubic complexity in sequence length due to the inversion of the kernel; although decoupled GP variants reduced the cost to quadratic, the computation remains prohibitive in practice. In this paper, we propose the plug-and-play random Fourier feature Gaussian process attention (RFF-GPA) module, which represents the attention as a GP with a stationary kernel approximated by random Fourier features. This low-rank approximation results in linear-time complexity for approximating the posterior mean and variance, making it far more scalable compared to previous work. Empirical results on multiple real-world datasets show that our attention module improves calibration while maintaining predictive accuracy, and simultaneously reduces computational complexity to linear in the sequence length.
Amir Mohammad Mahfoozi, Zi Yang, Ying Li +1
Department of Computer Engineering, Sharif University of Technology · School of Computing and Data Science, The University of Hong Kong
We study efficient reasoning under tight compute. We ask how to make structured, correct decisions without increasing test time cost. We add two training only components to small and medium Transformers that also transfer to broader differentiable optimizers. First, a length aware attention prior built via fuzzy regime position alignment, RPA, yields a normalized pre softmax bias that guides attention like a structured regularizer while adding no new inference parameters. Second, a minimal gain aware controller, Guardian, nudges attention sharpness only when validation improvements warrant it, following a two timescale policy gradient view of nonconvex optimization. It is disabled at inference. A KL perspective shows softmax of z plus log pi as MAP with KL regularization, grounding the prior in a principled objective. Under strict compute parity on WikiText 2, we reduce validation cross entropy while matching baseline latency and memory. At inference, we add a precomputed, cached prior B of T as a single additive bias per head. The controller does not run. In practice, this incurs negligible overhead, a cached bias add per head, with no measurable p50 latency shift. Our results suggest that length aware priors and late phase gain control preserve scarce improvements, especially in long span, noisy logit regimes, while keeping test time costs effectively unchanged.