Self-Generated Feedback Destabilizes Test-Time Training: A Causal Decomposition of Long-Horizon Adaptation

Cheng Luo; Bing Li; Bernard Ghanem

cs.CLOct 4, 2026

Self-Generated Feedback Destabilizes Test-Time Training: A Causal Decomposition of Long-Horizon Adaptation

Authors: Cheng Luo, Bing Li, Bernard Ghanem

Organizations: King Abdullah University of Science and Technology (KAUST)

Abstract

Test-time training (TTT) lets a model store information in its weights during inference. When the model learns from its own output, however, each update also changes the model that generates the next training example. Across 128K-token streams, retaining generated-text updates worsens prediction on independent human-written text with three TTT-E2E model configurations (labeled 125M, 760M, and 3B). The same failure occurs when Adam updates Qwen3-4B's existing weights. The same update mechanisms can improve on real text, so writing itself is not the failure. Three matched comparisons trace the causal pathway. Fixed Generation removes over 98% of the damage at 125M and 760M by using a frozen model to generate training chunks. Recorded Replay separates the loss caused by reading degraded text from the additional loss stored by updating on it. A paired one-update comparison then shows the local conflict: an update predicts its source better but new real text worse. This cost grows after Closed Loop adaptation, with a few trajectories accounting for most large failures. Finally, Settlement evaluates the candidate state on independent real text before commitment. It leaves mean endpoint gaps of 0.07 and -0.02 nats at 125M and 760M while retaining real-text adaptation. These results motivate checking prediction on independent evidence before retaining an update.

Figures & tables

Model	$H$ ( $\uparrow$ worse)	95% Interval
125M TTT-E2E	$+3.01$	$[+1.86,+4.20]$
760M TTT-E2E	$+6.00$	$[+4.27,+7.73]$
3B TTT-E2E	$+0.40$	$[+0.30,+0.51]$

Table 1: Generated-text writes worsen real-text prediction. The same six books, five seeds, and 128K horizon are used at every scale.

Figure 1: Long horizons and sparse real-text exposure expose the failure. (a) Real-text NLL across a 128K stream; shading marks the earlier 8K+8K horizon ( Tandon et al., 2025 ) . (b) Harm relative to a separate long-book suite’s 0% baseline. Even spacing interrupts the loop; grouping the same 31% real-text budget into bursts restores much of the harm. Panel (b) is normalized within a separate matched suite; error bars scale its 95% intervals by the fixed 0%-exposure estimate (Appendix B.3 ).

Adam LR	Persistent self-writing $H$ 95% CI; $\uparrow$ worse	Real-text $B$ state range; $\uparrow$ better
$10^{-5}$	$.026\;[-.071,.122]$	$.139\;[.115,.163]$
$10^{-4}$	$\mathbf{1.231\;[.580,2.350]}$	$.168\;[.132,.204]$

Table 2: Qwen3-4B with Adam. The same weights benefit from real text but fail under persistent self-writing. Sampling units and complete settings: Appendix C.5 .

Figure 2: Three controlled comparisons separate generation, content, and updating. (a) The initial state

W_{0}

generates while a separate

W_{t}

learns, breaking feedback. (b) Saved tokens are processed read-only or with updates, holding content fixed. (c) Current

W_{t}

both generates and learns, closing the loop. Every write-enabled condition evaluates the adapted

W_{t}

; Appendix A.1 verifies generator isolation.

Model	Fixed Generation $H$	Closed Loop harm removed
125M	$.052$	$98.3\%$
760M	$.073$	$98.8\%$

Table 3: Breaking feedback removes most of the excess change at both evaluated scales. Fixed Generation residuals and removal fractions use matched within-scale comparisons on the same six canonical books. Complete endpoints are in Appendix B .

Scale	Receiver operation on the same recorded text	Real-text NLL above control
125M	Read only	$1.606$
125M	Read and write	$3.874$
125M	Additional cost of writing	$\mathbf{2.268}$
3B	Additional cost of writing	$\mathbf{.193\;[.104,.327]}$

Table 4: Degraded content harms through both attention and weights. The 125M read-only excess shows that the recording is harmful in attention; the paired read-and-write difference measures the additional persistent cost.

Figure 3: Feedback, not parameter drift, amplifies the cost of writing. (a) Matched 125M control: the same four policies are ordered across relative drift and final clean-text NLL; horizontal bars are 95% book intervals. (b) Identical recorded tokens with updates disabled or retained; only this panel holds text fixed. Both panels use eight books and five seeds. (c) In a separate paired one-update comparison, the real-text cost of retaining one update rises sharply over a Closed Loop history, but stays near zero when earlier generated writes were discarded or came from Fixed Generation.

Preceding History	Training-Chunk NLL ( $\downarrow$ better fit)	Next Real-Text NLL ( $\uparrow$ worse)	Future Repeated-4 ( $\uparrow$ more repetition)
Writes Off History	$-.1374\;[-.1453,-.1285]$	$+.0045\;[.0037,.0053]$	$+.0049\;[-.0079,.0248]$
Closed Loop	$-.0354\;[-.0384,-.0317]$	$+.0381\;[.0098,.0718]$	$+.0086\;[.0020,.0184]$
Fixed Generation	$-.1449\;[-.1482,-.1407]$	$+.0018\;[.0014,.0023]$	$+.0017\;[-.0034,.0063]$

Table 5: An update can improve source fit while harming new text. 125M; keep minus discard, with 95% book intervals.

History	125M $\rho$	760M $\rho$
Closed Loop	$-.502\;[-.642,-.305]$	$-.633\;[-.808,-.383]$
Writes Off	$-.788\;[-.826,-.720]$	$-.731\;[-.810,-.666]$

Table 6: Two gradient signals rank harmful updates. Entries are Spearman correlations with the measured increase in next-real-text NLL after keeping rather than discarding one update. (a) Negative

\rho

means that more-negative gradient cosine predicts greater harm. (b) Positive

\rho

means that a larger first-order predicted loss increase matches greater measured harm. Each model-history setting contains 96 paired updates.

Figure 4: Damage concentrates in a few late passages. (a) Mean and median passage-level write cost; (b) number of costs above

0.5

nats among 72 paired comparisons at each position.

Figure 5: Independent evidence evaluates transfer. (a) Settlement leaves little endpoint residual. (b) Source Masking deteriorates under corrupted labels; Settlement remains stable. Lower is better in (a–b). (c) Task-metric Settlement uses WebShop reward to select updates; the plot shows exact success (mean

\pm

SD; five paired seeds; shared Writes Off). Higher is better. Details are in Appendices E – F and G .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Question	Shared configuration	Controlled comparison
Long horizon	125M, 760M, 3B; canonical six books; five seeds; 128K; width 8	Closed Loop versus Writes Off
Feedback path	125M and 760M; canonical six books; five seeds; 128K; Fixed Generation added	Closed Loop versus Fixed Generation
Read vs. write	Fixed recorded tokens; 125M: eight receiver books, five seeds; 3B: six source and six disjoint receiver books	Read Only versus Read + Write
Single update	Eight books and four stream positions; identical state and source text in each branch	Keep versus discard one update
Decoder/exposure	Decoder: canonical setup; exposure: one eight-book long-stream set, five seeds	Decoder endpoints versus a common Writes Off reference; evenly spaced versus bursty real text
Validation	Scale-specific long-book sets; each scale has its own paired Writes Off trajectory	Closed Loop versus Settlement

Appendix

Table 7: Main experiment configurations and why they differ. Each row compares policies under one shared configuration; books are the independent units for language-model intervals.

Policy	Final clean NLL	Final relative drift	Tail Distinct-2	Tail Repeated-4
Closed Loop	$6.5084$	$.00857$	$.0084$	$.9902$
Writes Off	$3.6776$	$.00500$	$.8493$	$.0217$
Fixed Generation	$3.7483$	$.01387$	$.8779$	$.0082$
Real-Text Learning	$3.3232$	$.01759$	—	—

Appendix

Table 8: Matched 125M feedback and displacement control. Drift is

\|W_{t}-W_{0}\|/\|W_{0}\|

. Real-Text Learning contains no generated tail, so token-diversity statistics are not applicable.

Model	0	1	2	3	4	5	6	7
760M, DCLM	3.69	1.44	1.15	1.23	1.18	1.19	1.14	1.18
3B, DCLM	10.35	4.12	1.30	1.33	1.29	1.32	1.30	1.28
3B, Books finetune	8.83	2.67	1.44	1.69	1.44	1.41	1.37	1.46

Appendix

Table 9: Unadapted loss ratio

L_{125M}/L_{\mathrm{model}}

on the same book. A large ratio means unusually low loss under the larger model.

Model	Condition	Initial NLL	Final NLL	Final excess over Writes Off [95% CI]
125M	Writes Off	3.3469	3.3845	0
	Fixed Generation	3.3469	3.4361	.0516 [.0248,.0772]
	Closed Loop	3.3469	6.3942	3.0097 [1.8575,4.1972]
760M	Writes Off	2.7913	2.9383	0
	Fixed Generation	2.7913	3.0116	.0733 [.0538,.0939]
	Closed Loop	2.7913	8.9405	6.0021 [4.2695,7.7347]

Appendix

Table 10: Fixed Generation confirmation on the same six receiver books. Intervals resample books after averaging seeds.

Control	Endpoint gap to canonical Writes Off	Late slope (nats/evaluation)
top- $p=.95$	$3.0097$	$+.1339$
top- $p=.99$	$3.3531$	$+.1497$
typical- $p=.95$	$2.9292$	$+.0948$
full support	$1.4520$	$+.1979$
$T=.7$ , top- $p=.95$	$2.2246$	$+.1704$
repetition penalty $1.1$	$\mathbf{.8794}$	$\mathbf{+.0027}$

Appendix

Table 11: Canonical decoder and reset controls. The endpoint gap uses the same canonical Writes Off endpoint for every row; late slope is the mean change per evaluation over the final four intervals. Lower is better in both columns.

Real fraction	Schedule	Real slots	Max self burst	$H$ [95% CI]	$B$ [95% CI]
0%	Even	0	105	$1.1858;[.5326,1.9430]$	$.0000;[.0000,.0000]$
5%	Even	5	17	$1.0275;[.5679,1.6195]$	$.1350;[.0557,.2578]$
10%	Even	11	8	$.4389;[.2517,.6465]$	$.0933;[.0729,.1155]$
20%	Even	21	4	$.0806;[.0657,.0981]$	$.0802;[.0585,.1052]$
31%	Even	33	3	$.0694;[.0534,.0843]$	$.1051;[.0668,.1431]$
31%	Bursty	33	12	$.7155;[.3730,1.1578]$	$.1040;[.0502,.1527]$

Appendix

Table 12: Real-text cadence controls excess NLL change.

H

is the excess NLL change of retaining generated writes;

B

is the benefit of retaining real-text writes. Intervals use a paired book bootstrap after averaging seeds.

Source book	Mean $J$	Leave-one-out $J$
12	.1862	.1940
13	.1953	.1922
14	.0617	.2189
15	.1331	.2046
16	.1602	.1992
17	.4196	.1473

Appendix

Table 13: The 3B passage-level write cost is positive across every source and receiver aggregate. The statistical interval is clustered by source and receiver, not by individual trajectory.

History	$\Delta$ source NLL	$\Delta$ real NLL	$\Delta$ future repeated-4
Closed Loop	$-.0354\;[-.0384,-.0317]$	$.0381\;[.0098,.0718]$	$.0086\;[.0020,.0184]$
Writes Off	$-.1374\;[-.1453,-.1285]$	$.0045\;[.0037,.0053]$	$.0049\;[-.0079,.0248]$
Fixed Generation	$-.1449\;[-.1482,-.1407]$	$.0018\;[.0014,.0023]$	$.0017\;[-.0034,.0063]$

Appendix

Table 14: Keep minus discard for one update: negative source NLL improves fitting; positive real NLL harms independent prediction. Intervals are 95% book-bootstrap intervals.

History	Position 1	33	65	97
Closed Loop	.0062	.0087	.0250	.1126
Writes Off	.0063	.0044	.0038	.0037
Fixed Generation	.0042	.0008	.0011	.0012

Appendix

Table 15: Mean one-update real-text NLL effects by position, 125M. Positions differ in both current text and receiver state.

Model / history	$\rho(\cos,\Delta L_{\rm real})$	$\rho(g_{\rm real}^{\top}\Delta W,\Delta L_{\rm real})$	$\Delta L_{\rm self}$	Mean $\cos$
125M / Closed Loop	$-.502\;[-.642,-.305]$	$.704\;[.521,.829]$	$-.033$	$-.093$
125M / Writes Off	$-.788\;[-.826,-.720]$	$.794\;[.721,.838]$	$-.136$	$-.005$
760M / Closed Loop	$-.633\;[-.808,-.383]$	$.632\;[.449,.781]$	$-.051$	$-.103$
760M / Writes Off	$-.731\;[-.810,-.666]$	$.751\;[.685,.828]$	$-.166$	$-.004$

Appendix

Table 16: Gradient conflict predicts the measured cost of retaining one generated-text update.

\Delta L_{\rm real}

is keep minus discard on the next real passage; negative cosine means the source and real-text gradients oppose one another. Negative mean

\Delta L_{\rm self}

indicates improved source fit on average.

Condition	NLL change $D$	Relative drift	Distinct-2	Repeated-4
Writes Off	-.060	.00000	.140	.831
Adam $10^{-5}$	-.034	.00627	.187	.775
Adam $10^{-4}$	1.171	.02399	.008	.991

Appendix

Table 17: In-place Adam long-horizon control. The larger-rate excess is

1.231

[.580,2.350]; the smaller-rate excess is

.026

[-.071,.122].

Adam LR	$B$ [range of two state means]	State 0	State 1
$10^{-5}$	$.1392\;[.1150,.1634]$	$.1634$	$.1150$
$10^{-4}$	$.1680\;[.1317,.2043]$	$.2043$	$.1317$

Appendix

Table 18: The Qwen3-4B update configuration also learns useful real text. Two independently adapted states each contain four books; the bracket is their range, not a book-level confidence interval.

Source stage	Median	Mean	Standard deviation	Cases above .5 nats
1	.0422	.0439	.025	0/72
17	.0040	.0898	.385	3/72
33	.0039	.1399	.620	3/72
49	.0042	.3135	1.009	6/72
65	.0047	.4855	1.274	9/72
81	.0105	.4858	1.267	9/72

Appendix

Table 19: Fixed-receiver stage replay. The increasing mean is produced by a growing right tail; the median remains near zero.

Policy	Final NLL	$E$ [95% interval]	Distinct-2
Writes Off	3.5767	0	.8425
Closed Loop	6.4620	2.8853 [2.0072,3.8310]	.0082
Repetition weighting	5.1404	1.5637 [1.1858,1.9825]	.0469
Random dose	5.6325	2.0558 [1.6339,2.4946]	.0086
Uniform dose	6.0628	2.4861 [1.9510,3.0939]	.0155
Read-only real context	4.9746	1.3979 [1.0988,1.7140]	.0730

Appendix

Table 20: Repetition weighting and real context reduce the endpoint gap but do not reach Writes Off. Lower

E

is better.

$\alpha$	Excess NLL change $H$ [95% CI]	Real-text benefit $B$ [95% CI]
$0$	$.0000;[.0000,.0000]$	$.0000;[.0000,.0000]$
$1/16$	$.3856;[.1737,.6432]$	$.0412;[.0249,.0579]$
$1/8$	$.8583;[.5480,1.2276]$	$.0543;[.0328,.0758]$
$1/4$	$.9248;[.6525,1.3374]$	$.0682;[.0413,.0942]$
$1/2$	$1.0319;[.6424,1.4540]$	$.0812;[.0507,.1106]$
$1$	$1.1867;[.8512,1.5056]$	$.0922;[.0579,.1248]$

Appendix

Table 21: Uniformly smaller writes reduce both harm and useful adaptation. Every evaluated nonzero dose has positive values for both quantities.

Read-only content	Final NLL	Difference from no-op [95% CI]
No-op	6.4042	0
Self-generated	6.7214	.3172 [-.4775,1.1963]
Random tokens	7.3969	.9927 [-.2699,2.0992]
Shuffled real	5.5680	-.8362 [-1.7643,.0324]
Fixed Generation text	5.7912	-.6130 [-1.5127,.2200]
Real text	4.8349	-1.5693 [-2.3011,-.8433]

Appendix

Table 22: Read-only content at a matched write schedule. Negative differences improve prediction relative to skipping the slot.

Model / books	Closed Loop minus Writes Off	Settlement minus Writes Off	Generated admitted
125M / 12	3.183 [2.106,4.253]	.067 [-.013,.169]	22/936
760M / 4	1.263 [.799,1.618]	-.021 [-.058,.014]	18/312

Appendix

Table 23: Completed Settlement outcomes and later admission-log extraction. The two scales use separate matched suites and Writes Off trajectories.

Without validation	Excess [95% interval]	With Settlement	Excess [95% interval]
Closed Loop	3.06 [.67,7.02]	Settlement	.04 [.00,.09]
Quarter Dose	.90 [.32,1.97]	+ Quarter Dose	.02 [-.01,.04]
Read-Only Real Context	1.04 [.19,2.59]	+ Real Context	-.02 [-.03,.00]
Both controls	.28 [.13,.53]	+ Both controls	.00 [-.01,.02]

Appendix

Table 24: Completed component ablation, 125M, four books and three seeds. Rounded interval endpoints at zero are not equivalence statements.

Policy	Real admitted	Benefit [95% interval]
Reject All	$0/113$	$0$
Write All	$113/113$	$.0450\;[.0235,.0609]$
Settlement	$\mathbf{89/113}$	$\mathbf{.0534\;[.0299,.0721]}$

Appendix

Table 25: Settlement admits most real-text candidates and preserves their adaptation benefit in the same matched experiment. Positive benefit is better; intervals resample books after averaging seeds.

Policy	Candidate judged against	State committed	$H$ ( $\downarrow$ )	$B$ ( $\uparrow$ )
Ordinary writing	—	$W+\delta_{t}$	$+1.56$	$+.034$
Joint commit	$W$ (each $\delta$ )	$W+\sum_{i}\delta_{i}$	$\approx 0$	$-1.093$
Sequential	$S_{i-1}$	$S_{i}$	$+.10$	$+.019$
Settlement	$S_{i-1}$ on next $q$	$S_{i}$	$\approx 0$	$+.037$

Appendix

Table 26: Validate the state that will be committed. Joint commit checks each candidate at

W

but deploys their unevaluated sum; sequential policies judge the accumulated state

S_{i}

Figure 6: Four ways to apply persistent updates. (a) Ordinary writing immediately changes the live state. (b) Joint commit validates candidates separately at

W

but deploys their unevaluated sum. (c) Sequential validation judges each candidate against the accumulated accepted state. (d) Settlement keeps a candidate pending until arriving external text

q

can evaluate the state that would be committed.

Observed error	Generated kept	Real discarded	Source Masking NLL	Settlement NLL
.000	0	0	3.8072	3.8196
.053	4	2	3.8116	3.8230
.124	9	5	3.8129	3.8248
.212	15	9	3.8377	3.8286
.389	26	18	3.8584	3.8215
.619	41	29	3.9242	3.8270

Appendix

Table 27: Corrupting source labels harms Source Masking but has little effect on Settlement. Lower final NLL is better. The middle columns show the incorrect decisions made by Source Masking. Bold marks comparisons whose paired 95% interval excludes zero in favor of Settlement.

Policy	Reward	Exact success	Retained blocks
Writes Off	$.3309$	$.1600$	$0$
Closed Loop	$.4356$	$.1053\pm.0597$	$180/180$
Fixed Generation	$.3724$	$.1800\pm.0089$	$180/180$
Settlement	$.3823$	$\mathbf{.1853\pm.0202}$	$10/180$

Appendix

Table 28: WebShop outcomes across five paired seeds. Exact success is mean

\pm

SD; reward is the mean score. Writes Off is a shared deterministic reference. Retained counts pool 25-episode blocks across seeds.

Figure 7: ALFWorld unseen-task success by update policy. Each open circle shows one seed’s success over 134 tasks; filled diamonds and the right-hand numbers show the three-seed mean. Points are vertically offset to separate overlapping values. Writes Off is shared across update scales.

		Success rate (%)			Accepted
Scale	Policy	Overall (274)	Seen (140)	Unseen (134)	blocks
–	Writes Off	$82.1$ $\pm$ $0.4$	$\mathbf{77.6}$ $\pm$ $0.4$	$86.8$ $\pm$ $0.9$	–
8	Closed Loop	$43.9$ $\pm$ $36.3$	$59.5$ $\pm$ $25.6$	$27.6$ $\pm$ $47.8$	All updates
8	Settlement	$\mathbf{83.8}$ $\pm$ $5.0$	$76.9$ $\pm$ $6.9$	$\mathbf{91.0}$ $\pm$ $3.0$	$12/36$
16	Closed Loop	$30.7$ $\pm$ $35.3$	$42.1$ $\pm$ $38.6$	$18.7$ $\pm$ $32.3$	All updates
16	Settlement	$\mathbf{83.2}$ $\pm$ $2.2$	$76.7$ $\pm$ $4.6$	$\mathbf{90.0}$ $\pm$ $1.1$	$7/36$

Appendix

Table 29: ALFWorld success rates and update acceptance. Success rates are percentages (mean

\pm

sample SD across three seeds). Bold marks the highest mean per scale, including shared Writes Off, not statistical significance. Block counts pool seeds.

Explore similar work

Test-Time Training with Next-Token Prediction
Jun 19, 2026Xuan Ouyang, Zefan Cai, Junjie HuTest-Time Training Next-Token Prediction
University of Wisconsin–Madison
A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt
Jun 14, 2026Tomoya WakayamaStable Test-Time Adaptation Test-Time Training
RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan.
TTSR: Test-Time Self-Evolving via Reflection
Feb 6, 2026Haoyang He, Zihua Rong, Yunjia Zhao +3Test-Time Training Test Time
Beijing University of Posts and Telecommunications · Southwestern University of Finance and Economics · China Unicom Online Information Technology Co., Ltd.

Jun 19, 2026, cs.CLJ/K move · Enter open · S save

Test-Time Training with Next-Token Prediction

Xuan Ouyang, Zefan Cai, Junjie Hu

University of Wisconsin–Madison

Test-Time Training Next-Token Prediction Autoregressive Token Prediction Test-Time Adaptation Model Checkpoints

Open paper AskPDF arXiv

Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test time. We study whether this signal can define the inner-loop objective for test-time training (TTT) in pretrained long-context language models. Many TTT architectures require models to be trained with test-time adaptation in mind, limiting their direct applicability to released LLM checkpoints. While recent in-place TTT methods make fast-weight adaptation possible for pretrained LLMs without redesigning the backbone, they leave a central question unresolved: what should each test-time write store? Existing recipes train the fast weight to match a learned local value proxy but they are not directly tied to the self-supervised next-token prediction signal. We introduce Test-Time Training with Next-Token Prediction (TTT-NTP), a drop-in fast-weight adaptation method for pretrained LLMs that instead supervises updates using the model's own next contextual hidden state. This makes each local write follow the same causal computation that supports next-token prediction: the value target is a pointwise linear projection of a single next-position contextual state. On RULER Full-13, averaged over 4k to 32k contexts, TTT-NTP is the only method that consistently improves the released backbone across four models spanning three families and a 0.6-8B size range, by 3.9 points on Llama-3.1-8B, 3.0 on Mistral-7B-v0.3, 4.1 on Qwen3-4B, and 2.9 on Qwen3-0.6B. On the real-world LongBench-v2 long-document QA benchmark, TTT-NTP improves over the base model by 5.6 points on Llama-3.1-8B and 3.7 on Mistral-7B-v0.3, while preserving commonsense and knowledge performance. Our code is publicly available at https://github.com/yancyou/TTT-NTP.

Abstract

Explore similar work

Test-Time Training with Next-Token Prediction

A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt

TTSR: Test-Time Self-Evolving via Reflection