Neural Processes (NPs), and specifically Transformer Neural Processes (TNPs), have demonstrated remarkable performance across tasks ranging from spatiotemporal forecasting to tabular data modelling. However, many of these applications are inherently sequential, involving continuous data streams such as real-time sensor readings or database updates. In such settings, models should support cheap, incremental updates rather than recomputing internal representations from scratch for every new observation -- a capability existing TNP variants lack. Drawing inspiration from Large Language Models, we introduce the Incremental TNP (incTNP). By leveraging causal masking, Key-Value (KV) caching, and a data-efficient autoregressive training strategy, incTNP matches the predictive performance of standard TNPs while reducing the computational cost of updates from quadratic to linear time complexity. We empirically evaluate our model on a range of synthetic and real-world tasks, including tabular regression and temperature prediction. Our results show that, surprisingly, incTNP delivers performance comparable to -- or better than -- non-causal TNPs while unlocking orders-of-magnitude speedups for sequential inference. Finally, we assess the consistency of the model's updates -- by adapting a metric of "implicit Bayesianness", we show that under a one-at-a-time streaming protocol, incTNP retains a prediction rule as implicitly Bayesian as standard non-causal TNPs, demonstrating that incTNP achieves the computational benefits of causal masking without sacrificing the consistency required for streaming inference.
Figures & tables
Figure 1: Computational complexity analysis in the streaming setting (see theoretical analysis in Appendix F ). Top : As the data stream grows (steps 1,…N ), the model updates its context C and predicts targets T . In AR mode, the model must perform Nt forward passes (FP) at every step, amplifying the computational burden. Bottom Right : A comparison of the FP mechanics. Standard TNP must re-encode the entire context history at every step (cost O(N2) ), whereas incTNP leverages KV caching to process only the incremental update (cost O(N) ). Bottom Left : The cumulative cost over a stream of length N . Because TNP recomputes the full attention matrix repeatedly, its total cost scales cubically ( O(N3) ). incTNP reduces this to quadratic scaling ( O(N2) ), making it the only viable option for long streams, particularly when combined with the expensive AR decoding loop.
Table 1: Average Test Log-Likelihoods ( ↑ ) for (a) Factorised and (b) AR deployment. We report the absolute performance of the reference non-incremental TNP-D. For all the other methods we report the difference ( Δ ) relative to the reference performance. Held-out task are shown in purple, while transfer scenarios, such as Sim-to-Real (Tabular) and Forecasting (Temperature) are coloured in orange. Best results (including ties within one SEM) are bolded, with standard errors provided in Table 2 .
Figure 2: Test log-likelihoods ( ↑ ) on synthetic GP (RBF kernel, Nc up to 64 ) and Tabular datasets across multiple training configurations. incTNP-Seq exhibits reduced variance compared to TNP-D and incTNP , demonstrating the robustness offered by its training strategy.
Figure 3: Performance (Joint NLL gap relative to optimal) versus implicit Bayesianness (KL Gap). incTNP-Seq achieves a KL gap similar to the non-causal TNP-D , whereas the streaming GP baseline ( WISKI ) performs worse in both metrics. Diamond markers ( ⋄ ) denote the exchangeable versions of the models.
Figure 4: Test log-likelihood ( ↑ ) on Tabular (real-world datasets). Performance generally increases with stream length for all models. While incTNP and TNP-D remain competitive on smaller datasets, incTNP-Seq demonstrates greater robustness on Protein.
Figure 5: Test log-likelihood ( ↑ ) on the Tabular (Synthetic) dataset. (Left) Performance improves with more data, with incTNP-Seq consistently outperforming TNP-D and incTNP —even when evaluating beyond the training limit of 1024 points. (Right) While achieving similar accuracy, the incremental variants demonstrate significantly better scaling efficiency as context size grows.
Figure 6: Test log-likelihood ( ↑ ) on the Temperature Prediction task (streaming AR mode).
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Architecture of the incremental Transformer Neural Process ( incTNP ), which uses interwoven M-MHSA and MHCA layers.
Figure 8: Schematic of incTNP-Seq during training. All M-MHSA and M-MHCA layers use causal masking.
Figure 9: Distribution of the 2,805 stations across the restricted latitude and longitude used for our station temperature prediction task, using 0.75∘×0.75∘ patches.
Table 2: Average Test Log-Likelihoods ( ↑ ) for (a) Factorised and (b) AR deployment. We report the absolute performance of the reference non-incremental TNP-D. For all the other methods we report the difference ( Δ ) relative to the reference performance. Held-out task are shown in purple, while transfer scenarios, such as Sim-to-Real (Tabular) and Forecasting (Temperature) are coloured in orange. The values indicate absolute or relative performance ± the standard error of the mean (SEM) of each method.
Figure 10: Mean Log-Likelihood boxplots for RBF and Mixed kernels and for different context sizes ( Nc ).
Figure 11: Mean Log-Likelihood boxplots for tabular data prediction with Nc up to 1024 .
Figure 12: Test log-likelihoods ( ↑ ) on synthetic GP (RBF kernel, Nc up to 64 ) and Tabular datasets across multiple training configurations without any outlier removal. The highest learning rate we used led to failure of some models ( TNP-D and incTNP in the GP case, and incTNP in the tabular case) to converge, resulting in outliers.
Figure 13: The gradient noise-to-signal ratio (GNSR) for incTNP and incTNP-Seq as training progresses on the RBF kernel with Nc(max)=64 . For each checkpoint, we compute gradients over a fixed collection of mini-batches, estimate the signal as the squared norm of the mean gradient, and estimate the noise as the variance of mini-batch gradients around this mean. We estimate this using 80 k samples with batch size 16 .
Table 3: Average Test Log-Likelihoods ( ↑ ) for (a) Factorised and (b) AR deployment. We report the absolute performance of the reference non-incremental TNP-D. For all the other methods we report the difference ( Δ ) relative to the reference performance. All tasks are considered out-of-distribution and coloured in orange. The values indicate absolute or relative performance ± the standard error of the mean (SEM) of each method.
Figure 14: Cost of each AR mode deployment step as weather observations arrive sequentially on the HadISD task. At each step, models condition on all streamed data Ns and perform AR prediction on a fixed target set of Nt=250 using S=50 sample unrolls. The dotted orange line represents the cost of updating incTNP ’s context representation. Both the original unoptimised TNP-A model from Nguyen and Grover (2022) , TNP-A (Original) , and our optimised version with KV caching implemented, TNP-A (Cached) , are compared. We only run TNP-A (Original) over 2,000 context points and AR TNP-D over 5,000 context points due to their expensive scaling, but fit quadratic functions to them (with dotted black lines) to show their expected scaling. We also fit cost curves to TNP-A (Original) and incTNP with dotted black lines to highlight their respective quadratic and linear per-step AR scaling. At Ns=100,000 , TNP-A (Cached) is 3.5× slower than incTNP despite having approximately half as many parameters.
Figure 15: Performance (Joint NLL gap relative to optimal) versus implicit Bayesianness (KL Gap). Similarly to Figure 3 , incTNP-Seq achieves a KL gap similar to the non-causal TNP-D and a better performance. In contrast, incTNP ’s performance is poorer, and the KL gap is slightly larger, but within errors. Diamond markers ( ⋄ ) denote the permutation-invariant versions of the models.
Figure 16: Test log-likelihood ( ↑ ) on the Tabular (Synthetic) dataset including baselines ( LBANP and WISKI ). (Left) The TNP variants show significantly better performance than the baselines. (Right) Although the TNP-based models show better performance, the TNP-D is more costly than the baselines at large streaming lengths. In contrast, the incTNP variants have both the best performance and lowest computational cost.
Figure 17: Test log-likelihood ( ↑ ) on Tabular (real-world datasets). The TNP-based variants show better performance on the real-world datasets too.
Figure 18: Test log-likelihood ( ↑ ) on the temperature prediction task using AR predictions, reporting streamed performance for both spatiotemporal interpolation (left) and forecasting (right). Results are averaged over 2,000 samples, with the shaded areas representing the standard error of the mean (SEM).
Figure 19: Standard deviation of the log-likelihood over different target orderings (following Figure 9 in Bruinsma et al. (2023) ). Empty markers indicate small, fixed-size contexts (1 or 10 tabular datapoints), while filled markers represent larger, variable contexts sampled uniformly per the legend. We observe similar behaviour between TNP-D and incTNP-Seq , while incTNP tends to show slightly higher sensitivity to target ordering.
Querying Cost (Step)
Model
Update Cost (Step)
Factorised
Autoregressive
Training Cost
incTNP
O(Ns)
O(NsNt)
O(SNt(Ns+Nt))
O(Nc2+NcNt)
incTNP-Seq
O(Ns)
O(NsNt)
O(SNt(Ns+Nt))
O((Nc+Nt)2)
TNP-D
O(Ns2)
O(NsNt)
O(SNt(Ns+Nt)2)
O(Nc2+NcNt)
Appendix
Table 4: Summary of per-step streamed conditioning and querying cost for NP models, using both factorised and AR predictions. The training step cost for Nc context points and Nt targets is also included.
Figure 20: Cumulative runtime cost (measured in seconds) on the HadISD dataset of deploying in AR mode as the context set grows ( Ns ). Runtime measured as station observations arrive in a streamed fashion, resulting in updated predictions at Nt=250 stations using S=50 sample unrolls across 32 batches. We fit quadratic and cubic cumulative cost functions to incTNP and TNP-D (plotted with dotted black lines), to highlight the cumulative AR scaling cost of both approaches.
Querying Cost (Cumulative)
Model
Update Cost (Cumulative)
Factorised
Autoregressive
incTNP ( -Seq )
O(Ns2)
O(Ns2Nt)
O(SNtNs(Ns+Nt))
TNP-D
O(Ns3)
O(Ns2Nt)
O(SNtNs(Ns+Nt)2)
Appendix
Table 5: Summary of cumulative streamed conditioning and querying cost for NP models, using both factorised and AR predictions. Both incTNP and incTNP-Seq share the same streamed cumulative scaling and are presented together.
Querying Cost
Model
Update Cost
Factorised
Autoregressive
Attention
incTNP
O(DzNs)
O(Dz(Nt+Ns))
O(SDz(Nt+Ns))
Mem-Eff
TNP-D
—
O(Dz(Nt+Ns))
O(SDz(Nt+Ns))
Mem-Eff
incTNP
O(Ns(Dz+H))
O(Dz(Nt+Ns)+HNsNt)
O(S(Nt+Ns)(Dz+H))
Standard
TNP-D
—
O(Dz(Nt+Ns)+H(Ns2+NsNt))
O(S(Nt+Ns)(Dz+H(Nt+Ns)))
Standard
Appendix
Table 6: Summary of per-step streamed conditioning and querying transient memory costs for NP models, using both factorised and AR predictions. Cost is shown for the memory-efficient kernel Mtransformerflash ( Mem-Eff ) and standard attention kernel Mtransformernaive ( Standard ).
Cost
Model
Factorised
Autoregressive
Attention
incTNP
O(Dz(Nt+LNs)+W)
O(SDzL(Nt+Ns)+W)
Mem-Eff
TNP-D
O(Dz(Nt+Ns)+W)
O(SDz(Nt+Ns)+W)
Mem-Eff
incTNP
O(Dz(Nt+LNs)+H(NsNt)+W)
O(S(Nt+Ns)(LDz+H)+W)
Standard
TNP-D
O(Dz(Nt+Ns)+H(Ns2+NsNt)+W)
O(S(Nt+Ns)(Dz+H(Nt+Ns))+W)
Standard
Appendix
Table 7: Summary of per-step streamed maximum memory costs for NP models, using both factorised and AR predictions. Cost is shown for the memory-efficient kernels Mtransformerflash ( Mem-Eff ) and standard attention kernels Mtransformernaive ( Standard ).
Figure 21: Measured memory scaling of TNP-D and incTNP models on the HadISD temperature prediction task as the streamed context set Ns grows for factorised predictions on Nt=250 locations. Peak memory use is reported when using FlashAttention (left - Mtransformerflash ) and when using standard quadratic memory attention (right - Mtransformernaive ). A dotted orange line is used to indicate the persistent memory costs of incTNP (i.e. the memory cost of the KV cache and model weights), which contribute to the peak memory. Memory measured on a single NVIDIA A100 80GB GPU, using 16 bit floats and PyTorch’s torch.nn.attention.SDPBackend FlashAttention and Math backends. Dotted black lines are fit to peak memory plots to highlight O(N) linear or O(N2) quadratic memory scaling.
Figure 22: Performance (NLL gap relative to optimal for factorised predictions) versus context permutation sensitivity (context KL gap) for Nc=128 context points and Nt=128 target points. Diamond markers ( ⋄ ) denote context permutation invariant model constructions. TNP-D is context permutation invariant (up to numerical noise) and thus is represented as a dotted line. We plot the standard error of the mean (SEM) of all models. WISKI is pretrained on the first 64 context points, and updated in a streaming manner for the remaining 64 context points. All NP models are trained with Nc(max)=512 .
Figure 23: Performance (NLL gap relative to optimal for factorised predictions) versus context permutation sensitivity (context KL gap) for Nc=8 context points and Nt=10 target points. Diamond markers ( ⋄ ) denote context permutation invariant model constructions. TNP-D is context permutation invariant (up to numerical noise) and thus is represented as a dotted line. We plot the standard error of the mean (SEM) of all models. We plot the test time characteristics of TNP-A when unrolling over S=50 and S=100 autoregressive sample unrolls. All NP models are trained with Nc(max)=64 .
Figure 24: Distribution of test set log-likelihoods across random context permutations for TNP-D , incTNP and incTNP-Seq . We consider the distribution for both Nc(max)=64 (left) and Nc(max)=512 (right) RBF GP regression tasks. We approximate over 1,000,000 test set permutations per model (processing ∼4 billion samples per model) for both tasks. TNP-D is context permutation invariant and thus represented with a dotted line. For incremental NPs, we plot distribution density with a violin plot, using internal dashed lines to signify the distribution quartiles.
Figure 25: Distribution of test set log-likelihoods across random context permutations for TNP-D , incTNP and incTNP-Seq on synthetic tabular data. We approximate over 1,000,000 test set permutations per model. TNP-D is context permutation invariant and thus represented with a dotted line. For incremental NPs, we plot distribution density with a violin plot, using internal dashed lines to signify the distribution quartiles.
Figure 26: Distribution of test set log-likelihoods across random context permutations for an incTNP-Seq variant trained to condition on the null context set that we term incTNP-Prior . We measure performance over 1,000,000 test set permutations for 1D GP regression using the RBF kernel and Nc(max)=64 . Our test set consists of 4096 samples. incTNP-Seq ’s distribution density is shown using a violin plot, with internal dashed lines signifying distribution quartiles. We also benchmark incTNP-Prior ’s modelling accuracy when using variance-based greedy contextual ordering as outlined in Algorithm 2 . We consider all selection strategies s∈{max,median,min} , corresponding to Var (Highest), Var (Median) and Var (Lowest) respectively.
Reference
Ours ( Δ ( ↑ ) relative to Ref.)
Dataset
Model
TNP-D
K=1
K=2
K=10
K=100
Gain ( 1→100 )
1D GP (64)
incTNP
0.4309±0.0070
−0.0133±0.0070
−0.0102±0.0070
−0.0077±0.0070
−0.0071±0.0069
+0.0062
incTNP-Seq
−0.0014±0.0070
+0.0012±0.0069
+0.0034±0.0069
+0.0039±0.0069
+0.0053
1D GP (512)
incTNP
0.7930±0.0033
−0.0037±0.0033
−0.0028±0.0033
−0.0022±0.0033
−0.0020±0.0033
+0.0017
incTNP-Seq
+0.0027±0.0032
+0.0033±0.0032
+0.0038±0.0032
+0.0039±0.0032
+0.0012
Tabular
incTNP
0.1541±0.0054
−0.0202±0.0054
−0.0165±0.0054
−0.0137±0.0053
−0.0129±0.0053
+0.0073
Appendix
Table 8: Average Test Log-Likelihoods ( ↑ ) for factorised deployment. We report the absolute performance of the non incremental TNP-D . For incTNP and incTNP-Seq we report the difference ( Δ ) relative to the reference performance when averaging model predictions over K∈{1,2,10,100} permutations of the context set. We also report the performance gain for K=100 context permutations over a single permutation K=1 . Values indicate absolute or relative performance ± the standard error of the mean (SEM) of each method. We measure performance on 1D GP regression with an RBF kernel for both Nc(max)=64 and Nc(max)=512 , as well as on the synthetic tabular dataset.
Figure 27: Test log-likelihood ( ↑ ) on the distribution shift experiment (transitioning from kernel k1 to k2 ) using models trained on the mixed kernel GP task. The red line indicates the crossover point t0=20 , where both kernels k1 and k2 are mixed equally. The shaded purple region represents the transition from k1 to k2 . Shaded region starts at 5% mixing interval (distribution is drawn mostly from k1 ) and ends at 95% mixing interval (distribution is drawn mostly from k2 ), using τ=10 . The dotted grey line indicates the ground truth log-likelihood (GT LL). Results are averaged over 4,096 tasks.
Figure 28: Test log-likelihood ( ↑ ) on the distribution shift task for the RBF kernel as the transition rate τ between kernels changes. Smaller τ values result in a much more abrupt distribution shift, whilst larger values lead to a smoother transition. t0=20 is used throughout. Results are averaged over 4,096 tasks.
Although deep neural networks perform extremely well in controlled environments, they fail in real-world scenarios where data isn't available all at once, and the model must adapt to a new data distribution that may or may not follow the initial distribution. Previously acquired knowledge is lost during subsequent updates based on new data. a phenomenon commonly known as catastrophic forgetting. In contrast, the brain can learn without such catastrophic forgetting, irrespective of the number of tasks it encounters. Existing spiking neural networks (SNNs) for class-incremental learning (CIL) suffer a sharp performance drop as tasks accumulate. We here introduce CATFormer (Context Adaptive Threshold Transformer), a scalable framework that overcomes this limitation. We observe that the key to preventing forgetting in SNNs lies not only in synaptic plasticity but also in modulating neuronal excitability. At the core of CATFormer is the Dynamic Threshold Leaky Integrate-and-Fire (DTLIF) neuron model, which leverages context-adaptive thresholds as the primary mechanism for knowledge retention. This is paired with a Gated Dynamic Head Selection (G-DHS) mechanism for task-agnostic inference. Extensive evaluation on both static (CIFAR-10/100/Tiny-ImageNet) and neuromorphic (CIFAR10-DVS/SHD) datasets reveals that CATFormer outperforms existing rehearsal-free CIL algorithms across various task splits, establishing it as an ideal architecture for energy-efficient, true-class incremental learning.
Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon Borthakur
Autoregressive language models execute Transformer layers sequentially, creating a latency bottleneck that is not removed by conventional tensor or pipeline parallelism. We study whether this layerwise dependency can be relaxed by treating the hidden-state trace across layers as the solution of a nonlinear residual equation and solving it with parallel Newton-style updates. While this view is principled, exact Newton corrections require expensive Jacobian-vector products and naive fixed-point iterations are unstable on trained Transformers. We introduce Structured Newton Layer Parallelism (SNLP), a training and inference framework that replaces exact layer Jacobians with cheap architecture-induced surrogate dynamics. In residual Transformers, this yields Identity Newton (IDN), where the correction reduces to a prefix-sum-like update; in mHC-style architectures, HC Newton (HCN) uses the model's residual mixing matrix. We also study SNLP-aware training, including pretraining regularization and direct SNLP-forward SFT. Experiments on Nanochat-scale Transformers show that SNLP exposes a practical speed-quality frontier: on 0.5B models, it reaches up to 2.58x wall-clock speedup, and a less aggressive configuration reaches 1.40x speedup without increasing PPL. The useful tradeoff comes from the biased finite-iteration computation induced by IDN/HCN rather than exact recovery of the sequential trace. We further show that SNLP-forward SFT can preserve downstream task accuracy, and that SNLP can serve as a drafter for self-speculative decoding while a sequential verifier preserves output correctness.
Ligong Han, Kai Xu, Hao Wang +1
1Red Hat AI Innovation · 2MIT-IBM Watson AI Lab · 3Core AI, IBM
The ongoing digitization has led to a proliferation of time-series data streams that monitor a variety of processes, from which valuable insights may be obtained. Further, the emergence of successful foundational language models begs the question of whether it is possible to achieve time-series models with the foundational properties of handling multiple tasks, while being sufficiently lightweight to allow real-time data stream processing. Existing foundational time-series models are often large and only effective in offline settings without stringent time and computational constraints, and where repeated model calibration is not needed. However, when applied to data streams, these models are ineffective due to their size and lack of support for continual calibration, which compromise their ability to deliver accurate real-time responses, their durability, and their deployability in hardware-limited settings. We propose TimeBlocks to enable versatile time-series processing by facilitating the efficient building of lightweight models suitable for multiple tasks under variable conditions. In particular, the method maintains a pool of interchangeable and modular model blocks that can be used to construct new time-series models. When presented with specific time-series data, a routing strategy iteratively selects the most suitable blocks to construct a lightweight and accurate model for the data. We equip TimeBlocks with a method called StreamCore to build a representative small subset of the data stream, which preserves a guaranteed approximation of the stream over time, enabling continual model calibration. An experimental study on multiple data sets and covering multiple tasks shows that TimeBlocks enables to build models capable of outperforming existing baselines.
David Campos, Bin Yang, Tung Kieu +3
Aalborg University, Denmark · University of Amsterdam Netherlands · East China Normal University China +1