Test-Time Training with KV Binding Is Secretly Linear Attention

Junchen Liu; Sven Elflein; Or Litany; Zan Gojcic; Ruilong Li

cs.LGFeb 24, 2026

Test-Time Training with KV Binding Is Secretly Linear Attention

Authors: Junchen Liu, Sven Elflein, Or Litany, Zan Gojcic, Ruilong Li

Organizations: NVIDIA, Toronto, Ontario, Canada · University of Toronto, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada · Technion – Israel Institute of Technology, Haifa, Israel

Abstract

Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity. Project page: https://research.nvidia.com/labs/sil/projects/tttla/.

Figures & tables

Figure 1: Inner-Loop Optimization vs. Performance . Increasing inner-loop iterations improves inner-loop loss but degrades task performance, contradicting the memorization-based interpretation of TTT. Experiments are based on LaCT ( Zhang et al., 2025 ) .

Model	Perplexity $\downarrow$ (LaCT-LLM)	PSNR $\uparrow$ (LaCT-NVS)	Top-1 Acc $\uparrow$ (ViTTT)
Baseline	16.43	25.94	79.34
Gradient Ascent	16.19	25.85	79.61
Replace $Q$ with $K$	16.18	25.95	79.18

Table 1: Observations Contradicting the Storage-and-Retrieval Interpretation of TTT. Replacing gradient descent with ascent breaks the storage interpretation, and replacing queries with keys breaks the retrieval interpretation, yet task performance remains mostly unchanged. Experiments are base on LaCT ( Zhang et al., 2025 ) and ViTTT ( Han et al., 2025 ) .

Figure 2: Distributional Asymmetry Between $Q$ and $K$ . t-SNE visualizations of $(Q,K)$ and $(V,O)$ features in a pretrained LaCT ( Zhang et al., 2025 ) model on the NVS task, showing that the TTT inner loop is evaluated out of distribution and thus does not perform reliable retrieval.

Alias	Description of TTT Inner-loop Designs	Perplexity $\downarrow$ (LaCT-LLM)	PSNR $\uparrow$ (LaCT-NVS)	Top-1 Acc $\uparrow$ (ViTTT)	Tokens Per Sec $\uparrow$ (Recurrent Impl.)	Tokens Per Sec $\uparrow$ (Parallel Impl.)
Baseline	LaCT ( Zhang et al., 2025 ) / ViTTT ( Han et al., 2025 )	16.43	25.94	79.34%	4.30M	N/A
Variant 1	Baseline w/ only update the last layer parameters	15.93	25.97	79.63%	10.60M	N/A
Variant 2	Variant 1 w/ remove weight normalization	16.31	25.93	79.63% †	11.02M	30.18M
Variant 3	Variant 2 w/ multi-layer MLP $\rightarrow$ singel linear layer	16.23	25.71	79.39%	12.95M	49.69M
Variant 4	Variant 3 w/ remove per-token learnable lr	16.12	25.70	79.39% †	13.31M	53.99M
Variant 5	Variant 4 w/ remove momentum in SGD	15.97	25.70 †	79.39% †	14.40M	57.28M

Table 2: Ablation Trajectory Reducing TTT to Standard Linear Attention. By progressively simplifying complex TTT formulations into standard linear attention (Variant 6), we quantify the contribution of each design component in TTT. Variant 1 achieves the best performance across all three tasks. Variants 2–6 admit parallel implementations. We additionally report the inference throughput of each variant’s TTT layer on the LLM task. † denotes ablations that do not apply, in which case performance matches the preceding variant. * indicates that ViTTT does not use gradient orthogonalization, so we ablate gradient normalization instead.

Figure 3: Perplexity Metric for Ablation on LaCT-LLM. Evaluated on 2.5B tokens from the Book-3 dataset.

Figure 4: Training loss vs. wall-clock time on LaCT-LLM. We compare the original LaCT-TTT with both parallel and recurrent form of Variant 2. The parallel form achieves a

1.19\times

end-to-end speedup while maintaining comparable convergence.

Test-Time Training

Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.

See all 104 papers

Modular TTT: Rethinking Test-Time Training as Composable Modules
Aug 7, 2026Bohao Tang, Zhen Qin, Yuqi Pan +3Test-Time Training Fast Weights
Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed
Universal Test-Time Training
Oct 4, 2026Zefan Cai, Qinzhe Hu, Ziqiao Ma +2Memory-Augmented Language Models Recurrent Transformers
University of Wisconsin–Madison · University of Michigan–Ann Arbor · Adobe
Linearizing Vision Transformer with Test-Time Training
May 4, 2026Yining Li, Dongchen Han, Zeyu Liu +3Efficient Transformer Inference Self-Attention
Tsinghua University, Beijing, China

Aug 7, 2026J/K move · Enter open · S save

Modular TTT: Rethinking Test-Time Training as Composable Modules

Bohao Tang, Zhen Qin, Yuqi Pan, Zheng Li+2

Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed

Test-Time Training Fast Weights

Open paper AskPDF arXiv

Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Modular TTT automatically composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition. Using Modular TTT, we systematically ablate the components of TTT and find that small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections and gating provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.

Abstract

Test-Time Training

Explore similar work

Modular TTT: Rethinking Test-Time Training as Composable Modules

Universal Test-Time Training

Linearizing Vision Transformer with Test-Time Training