Error Correction in Trained TTT Layers

Yuhe Hu

cs.LGOct 12, 2026

Error Correction in Trained TTT Layers

Authors: Yuhe Hu

Organizations: Duke University

Abstract

Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself establish how the memory revises an existing association. We study the structure and empirical role of prediction-derived updates in released TTT implementations. For dot-product objectives, we characterize the absence of a prediction-error residual and establish conditions under which inner-loop ascent and descent are equivalent after reparameterization. For the LayerNorm-based squared-error objective used by TTT-Linear and TTT-MLP, we derive the inner gradient and identify a prediction-derived component that cancels at the released initialization in the zero-epsilon limit. We then measure the full normalized prediction's contribution on three trained checkpoints. Removing this contribution while retaining the LayerNorm Jacobian gives a mean gradient norm ratio of 0.25 to 0.32 across the models. On 100,303 tokens from War and Peace, the same intervention with local per-token norm matching raises perplexity by 2.8-2.9% on TTT-Linear-760M and 1.3B, whereas freezing all fast-weight updates raises it by factors of 4.2-4.5. Separate synthetic tests illustrate the dependence of plain delta-rule overwriting on step size. The results distinguish a prediction-derived update from its behavioral value and show why implementation analysis and live interventions are both needed to assess test-time memory.

Figures & tables

Checkpoint	Chunks	Mean $R$	Median layer $\mathrm{CV}(\gamma^{2})$	$\rho$ [ $95\%$ interval]
TTT-Linear-760M	128	0.265	0.504	$0.490$ [ $0.015,0.794$ ]
TTT-Linear-1.3B	512	0.251	0.584	$0.315$ [ $-0.128,0.694$ ]
TTT-MLP-760M	512	0.318	0.536	$0.512$ [ $0.124,0.769$ ]

Table 1: Live probes of the prediction-derived gradient. The ratio is defined in Eq. ( 10 ). Chunks are counted per layer. Correlation intervals are exploratory paired layer-bootstrap intervals with average ranks for ties.

Figure 1: Gradient probes from the saved runs. Each point is a layer. The left panel shows the mean norm ratio by depth. The center compares it with gain variation. The right shows per-layer medians of

\eta\|\mathbf{k}\|^{2}

for the two linear models. This last quantity is a reference scale from the plain delta rule, not a contraction factor for the LayerNorm objective.

	TTT-Linear-760M		TTT-Linear-1.3B
Arm	Perplexity	$\Delta$ NLL	Perplexity	$\Delta$ NLL
Baseline	12.77	0	11.34	0
Prediction removed	13.15	0.02891	11.68	0.02943
Prediction removed, norm matched	13.13	0.02789	11.66	0.02811
Ascent	2595.69	5.31423	1530.27	4.90509
Frozen	54.18	1.44484	50.56	1.49505

Table 2: Inference-time interventions on

100{,}303

paired token positions from War and Peace . The prediction-removed arm deletes both

g_{z}

and

g_{\beta}

. Norm matching is local to each token and head. NLL differences are in nats per token.

Figure 2: Five inference-time interventions on the two linear checkpoints. Left, perplexity by arm. Right, the paired NLL difference by context position on 760M. Curves summarize the fixed evaluation text, with no independence assumption about adjacent tokens.

Figure 3: Plain-rule overwrite behavior, separate from the trained language models. Panels (a) and (b) show the saved two-write simulation, independently checked using a closed-form calculation. Bars denote

95\%

normal-approximation intervals over independent trials. The shaded range in (b) marks the measured linear-760M median step range only as a numerical reference. Panel (c) is an exact analytical calculation for identical unit keys and orthogonal values with

r

old writes followed by

r

new writes. It is not the noisy-key simulation or a trained-model measurement.

Test-Time Training

Test-time training adapts model parameters or memory during deployment using each input or stream to improve performance under changing tasks and data.

See all 103 papers

Aug 7, 2026

Modular TTT: Rethinking Test-Time Training as Composable Modules

Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Modular TTT automatically composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition. Using Modular TTT, we systematically ablate the components of TTT and find that small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections and gating provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.

Bohao Tang, Zhen Qin, Yuqi Pan +3

Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed

Test-Time Training Fast Weights

Ask PDF

Oct 6, 2026

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel TTT baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines.

Yining Li, Dongchen Han, Jie Fu +1

Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research

Deep Learning Optimization Neural Network Optimization Test-Time Training Language Modeling

Ask PDF

Sep 28, 2026

GradLev: Token-Parallel Test-Time Training Via Costate Prediction

Test-time training (TTT) allows a model to improve its predictions at inference time by updating weights after every observed token. However, sequential gra- dient writes make parallel training difficult. We observe that, given layer inputs and activation gradients (costates), online gradient descent admits exact parallel scans for both forward evaluation and reverse backpropagation. GradLev lever- ages this duality: a causal auxiliary network predicts costates across all tokens in parallel; associative scans compute the adapted weights and forward activations and propagate gradients backward; and the resulting gradient targets supervise the predictor via a consistency loss. Exact consistency guarantees exact recovery of the sequential online learner. At deployment, the auxiliary predictor is discarded, and the model updates natively via token-by-token forward and backward passes.

Bo Liu, Qiang Liu

The University of Texas at Austin

Gradient Descent Test-Time Training

Ask PDF

Abstract

Test-Time Training

Explore similar work

Modular TTT: Rethinking Test-Time Training as Composable Modules

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

GradLev: Token-Parallel Test-Time Training Via Costate Prediction