One Model, Two Paths: Balancing Single-Pass and Looped Performance in Language Models

Jiarun Shen; Lei Zhao; Xiao Li

cs.CLOct 24, 2026

One Model, Two Paths: Balancing Single-Pass and Looped Performance in Language Models

Authors: Jiarun Shen, Lei Zhao, Xiao Li

Organizations: The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University

Abstract

Repeating Transformer layers can improve language-model performance without adding parameters. A pretrained model can be adapted to support both single-pass and looped inference by reusing middle-layer weights for additional computation. Because both paths share weights, training to improve looped predictions can degrade single-pass quality. We study joint training strategies using weighted losses on both outputs, with guidance from the frozen original model. Joint supervision substantially reduces single-pass degradation compared to looped-only training, while preserving looped gains over ordinary continued training. Most of this benefit comes from adjusting the single-pass loss weight, with guidance from the frozen original model further improving the looped output. With validation-selected weights, our method preserves single-pass quality within a predefined tolerance of ordinary continued training while improving looped predictions.

Figures & tables

Figure 1: Prediction quality relative to ordinary continued training on the supervision-study confirmation text at the forward-block cap. Axes show single-pass and looped NLL minus ordinary single-pass NLL, in nats per target. Lower left is better. Joint CE uses labels on both paths, Matched CE increases the single-pass label weight, and Online KD adds a teacher signal using the current looped output. Panel (b) enlarges the joint methods boxed in panel (a). Points average six paired continuations. Arrows compare endpoints; they are not training trajectories or a causal decomposition. The star marks the reference origin, not a measured ordinary looped output. The horizontal line marks equal looped and ordinary NLL; the vertical dashed line marks the 0.005 single-pass tolerance. Other methods and Loop

\to

schedules are defined in § 3.2 and § 4.2 . Frozen-original KD (ours) is evaluated in Figure 2 .

Figure 2: Both outputs after validation-based label-weight selection. Bars show four-order mean NLL differences from fixed-time ordinary continuation on the new confirmation text, including teacher computation in the budget. Within each method, the upper bar is single-pass and the lower bar is looped. Lower is better. The dashed line marks the single-pass retention tolerance; it is not a looped-gain threshold. Bars are descriptive means; Table 2 reports the retention criteria and key uncertainty intervals, with full results in Appendix H .

Method	Updates	Single	Looped
Ordinary	5120	3.638339	–
Loop-only	3413	3.809579	3.629291
Joint CE 5%	2925	3.674221	3.632287
Joint CE	2925	3.661175	3.633369
Online KD	2925	3.650679	3.634773
Matched CE	2925	3.651328	3.635767

Table 1: Confirmation NLL at the forward-block cap, averaged over three new data orders for each of two fixed parents. Lower is better. Staged update counts include the 2,048-update loop-only prefix. The ordinary reference uses its single pass; the unused ordinary looped output is omitted.

Recipe	$D_{\rm start}$	$x$	$y$	$d$	All four
Label-only CE, $m=0.8$	$-6.291$	$+1.929$	$+3.571$	$+1.642$	0/4
Online KD, $m=0.8$	$-5.657$	$+2.563$	$+4.641$	$+2.078$	0/4
Frozen-original KD, $m=0.4$	$-8.953$	$-0.733$	$-8.327$	$-7.595$	4/4

Table 2: Validation-selected recipes at the complete-update time cap, including teacher computation. All values are in

10^{-3}

nats per target on the new confirmation text. The upper panel gives four-order means for the criteria in Eq. 7 :

D_{\rm start}\leq 5

x\leq 5

y<0

, and

d<0

in these units. All four counts point-estimate successes, not simultaneous per-order guarantees. The lower panel highlights paired Frozen-original KD contrasts. The first two use the twelve-effect text family (99.5833%); the third uses the four-effect direct-comparison family (98.75%). Order intervals are separate conditional 95% intervals. Full results appear in Appendix H .

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value	Setting	Value
Parameters	133,830,912	Physical blocks	8
Hidden / MLP size	768 / 2,048	Attention / KV heads	12 / 12
Vocabulary	50,257	Repeated block indices	2–5 (zero-based)
Sequence / batch	1,024 / 32	Microbatch / accumulation	4 / 8
Arithmetic / weights	BF16 / FP32	Gradient norm limit	1
Optimizer	Muon NS5 + AdamW	Muon / AdamW learning rates	0.004 / 0.00006

Appendix

Table 3: Main-study model and training configuration. Method-specific continuation counts are in Table 1 . Learning rates and optimizer states are inherited and retained through stage switches. Appendix F reports GPU hours. Related architectures use depth sharing ( Dehghani et al., 2019 ; Lan et al., 2020 ) , block recurrence ( Hutchins et al., 2022 ) , recurrent depth ( Geiping et al., 2025 ) , or layer dropping ( Fan et al., 2020 ) .

Dataset / split	Documents or items	Targets	Blocks	Role
FineWeb-Edu training	1,935,463	2,000,000,435 tokens	–	Shared training pool
Supervision validation	551	524,288	16	Checkpoint selection
Supervision confirmation	2,190	2,097,152	32	Primary text evaluation
WikiText test	62 articles	282,624	18	Secondary text evaluation
ARC-Easy test	2,376	–	–	Task evaluation
PIQA validation	1,838	–	–	Task evaluation

Appendix

Table 4: Supervision-study data and evaluated task splits. Target-study documents are listed separately in Appendix G . Public task validation splits serve as final evaluations and do not select checkpoints.

Comparison	Block mean	Text interval	Time mean	Text interval
Static KD $-$ CE, single	$-0.010496$	$[-0.010766,\,-0.010229]$	$-0.010704$	$[-0.011431,\,-0.009958]$
Static KD $-$ CE, looped	$+0.001404$	$[+0.001166,\,+0.001648]$	$+0.002166$	$[+0.001409,\,+0.002942]$
Staged KD $-$ CE, single	$-0.012754$	$[-0.012976,\,-0.012528]$	$-0.011724$	$[-0.012606,\,-0.010809]$
Staged KD $-$ CE, looped	$+0.000545$	$[+0.000406,\,+0.000686]$	$+0.001023$	$[+0.000082,\,+0.001982]$
CE $-$ ordinary suffix, single	$+0.028091$	$[+0.027243,\,+0.028919]$	$+0.028047$	$[+0.027165,\,+0.028953]$
CE $-$ ordinary suffix, looped	$-0.213603$	$[-0.215512,\,-0.211710]$	$-0.210361$	$[-0.211949,\,-0.208645]$

Appendix

Table 5: All twelve confirmation-text effects under both budgets. Negative favors the first method. Adjusted intervals are 98.75% for the four output-guidance and four matched-control comparisons, and 97.5% for the two recovery and two loop-utility comparisons. Intervals describe paired text uncertainty conditional on fitted checkpoints. Static comparisons start at the parent; staged comparisons use the respective joint suffixes.

Comparison	C mean $\pm$ SD	D mean $\pm$ SD	Order interval (95%)
Static KD $-$ CE, single	$-0.010778\pm 0.000162$	$-0.010213\pm 0.000516$	$[-0.011073,\,-0.009918]$
Static KD $-$ CE, looped	$+0.001354\pm 0.000254$	$+0.001454\pm 0.000368$	$[+0.001027,\,+0.001781]$
Staged KD $-$ CE, single	$-0.012991\pm 0.000440$	$-0.012517\pm 0.000593$	$[-0.013366,\,-0.012142]$
Staged KD $-$ CE, looped	$+0.000537\pm 0.000271$	$+0.000554\pm 0.000070$	$[+0.000234,\,+0.000857]$
CE $-$ ordinary suffix, single	$+0.028809\pm 0.001039$	$+0.027372\pm 0.000813$	$[+0.027009,\,+0.029173]$
CE $-$ ordinary suffix, looped	$-0.208401\pm 0.006421$	$-0.218805\pm 0.011257$	$[-0.225142,\,-0.202064]$

Appendix

Table 6: Data-order variation at the forward-block cap. Each parent mean and sample SD uses three paired differences. The last column is a small-sample 95% interval for the equal-parent mean, conditional on C/D. It does not include text uncertainty or pretraining-seed variability.

Figure 3: Online KD minus coefficient-matched CE on the supervision-study confirmation text. Negative values favor KD. Columns apply the objectives from the start or after 2,048 loop-only updates; rows report the single-pass or looped output. Blue and orange denote forward-block and training-time budgets. Circles and triangles above each interval are the three paired effects within parents C and D, with orders 1–3 arranged from top to bottom within each budget. Diamonds mark the six-unit mean. Solid bars are 98.75% adjusted text-bootstrap intervals; dashed bars are data-order 95% intervals conditional on C/D. The intervals describe separate uncertainty sources. Values use

10^{-3}

nats per target. Appendix B retains all comparisons with the initial Joint CE control.

Comparison	Block mean	Text interval	Time mean	Text interval
Static KD $-$ CE, single	$-0.006274$	$[-0.007624,\,-0.004926]$	$+0.011566$	$[+0.007802,\,+0.015226]$
Static KD $-$ CE, looped	$+0.011492$	$[+0.010020,\,+0.012922]$	$+0.027138$	$[+0.023097,\,+0.031089]$
Staged KD $-$ CE, single	$-0.012736$	$[-0.013829,\,-0.011727]$	$-0.013408$	$[-0.016308,\,-0.010636]$
Staged KD $-$ CE, looped	$+0.000514$	$[-0.000877,\,+0.001668]$	$+0.000249$	$[-0.003560,\,+0.003763]$
CE $-$ ordinary suffix, single	$+0.026507$	$[+0.023868,\,+0.029364]$	$+0.029739$	$[+0.027105,\,+0.032528]$
CE $-$ ordinary suffix, looped	$-0.325245$	$[-0.333092,\,-0.316099]$	$-0.338551$	$[-0.346352,\,-0.330073]$

Appendix

Table 7: The same twelve contrasts on previously inspected WikiText. Family definitions match Table B , but these intervals use eighteen WikiText clusters. Results retain their corpus and budget identities.

Comparison	C mean $\pm$ SD	D mean $\pm$ SD	Order interval (95%)
Static KD $-$ CE, single	$-0.010861\pm 0.000601$	$-0.010547\pm 0.001709$	$[-0.012581,\,-0.008827]$
Static KD $-$ CE, looped	$+0.002451\pm 0.000930$	$+0.001880\pm 0.001388$	$[+0.000746,\,+0.003585]$
Staged KD $-$ CE, single	$-0.012895\pm 0.000679$	$-0.010554\pm 0.001606$	$[-0.013434,\,-0.010014]$
Staged KD $-$ CE, looped	$+0.000571\pm 0.000578$	$+0.001476\pm 0.003136$	$[-0.002706,\,+0.004753]$
CE $-$ ordinary suffix, single	$+0.029050\pm 0.001833$	$+0.027043\pm 0.001572$	$[+0.026093,\,+0.030000]$
CE $-$ ordinary suffix, looped	$-0.200435\pm 0.008434$	$-0.220287\pm 0.010471$	$[-0.221333,\,-0.199389]$

Appendix

Table 8: Confirmation-text paired data-order variation at the complete-update time cap. Means and sample SDs use three orders within each fixed parent. The final interval has the same conditional interpretation as Table B .

Method	Updates	Single	Looped
Ordinary	4404–4484	3.639523	–
Loop-only	3417–3461	3.807901	3.628643
Joint CE 5%	2622–2658	3.673922	3.633616
Joint CE	2633–2655	3.661637	3.635265
Online KD	2169–2188	3.650933	3.637430
Matched CE	2502–2520	3.651958	3.637436

Appendix

Table 9: Absolute confirmation NLL at the complete-update time cap. Values average six paired units. Update ranges include the prefix for staged methods. Every trajectory covers the cap.

Role	Budget	Order	Method	$D_{\rm start}$	$x$	$y$	$d$
Fixed	Block	C2	Matched CE	$-0.041201$	$-0.002845$	$-0.016076$	$-0.013232$
Fixed	Block	C2	Loop $\to$ CE	$-0.041696$	$-0.003339$	$-0.036249$	$-0.032910$
Fixed	Block	C2	Loop $\to$ KD	$-0.050431$	$-0.012075$	$-0.029137$	$-0.017063$
Fixed	Block	C2	Loop $\to$ matched CE	$-0.053064$	$-0.014708$	$-0.032329$	$-0.017621$
Fixed	Block	D2	Joint CE	$-0.026447$	$-0.002912$	$-0.031487$	$-0.028575$
Fixed	Block	D2	Matched CE	$-0.029374$	$-0.005839$	$-0.017509$	$-0.011669$

Appendix

Table 10: Every WikiText endpoint satisfying all four separate pointwise 95% criteria. The displayed differences are point estimates; inclusion requires all four interval criteria. Diagnostic checkpoints remain infeasible on validation even when their WikiText scores satisfy the criteria. Some fixed and diagnostic rows identify the same checkpoint.

Method	$D_{\rm start}$	$x$	$y$	$d$
Ordinary	6/0/0	6/0/0	0/6/0	0/6/0
Loop-only	0/6/0	0/6/0	6/0/0	6/0/0
Joint CE 5%	0/6/0	0/6/0	6/0/0	6/0/0
Joint CE	0/6/0	0/6/0	6/0/0	6/0/0
Online KD	5/0/1	0/6/0	6/0/0	6/0/0
Matched CE	5/0/1	0/6/0	4/0/2	6/0/0

Appendix

Table 11: Confirmation-text criteria at fixed forward-block endpoints. Each cell gives supported/violated/unresolved counts among six paired units. No method satisfies all four criteria. A failed conjunction does not imply that every criterion is violated.

Text	Budget	Fixed	Diagnostic
Confirmation	Block	0/60	0/60
Confirmation	Time	0/60	0/60
WikiText	Block	6/60	4/60
WikiText	Time	2/60	3/60

Appendix

Table 12: Number satisfying all four separate pointwise 95% criteria. Each cell evaluates 60 correlated method–unit endpoints. Diagnostic checkpoints were infeasible on validation and are not successful selections. The four intervals do not provide joint 95% coverage.

	ARC-Easy		PIQA		HellaSwag
Method	Single	Looped	Single	Looped	Single	Looped
Ordinary	42.88 / 42.97	39.97 / 40.41	58.41 / 58.66	56.96 / 57.68	27.18 / 27.24	27.13 / 26.89
Loop-only	41.76 / 41.74	43.16 / 43.09	57.90 / 58.57	57.98 / 58.51	27.28 / 27.35	27.16 / 27.18
Joint CE 5%	43.19 / 43.89	43.32 / 43.55	58.49 / 58.29	58.78 / 58.41	27.23 / 27.32	27.24 / 27.35
Joint CE	43.36 / 43.78	43.18 / 43.78	58.46 / 58.34	58.56 / 58.61	27.36 / 27.25	27.29 / 27.32
Online KD	43.11 / 43.85	43.06 / 43.54	58.35 / 58.20	58.36 / 58.08	27.30 / 27.24	27.25 / 27.26

Appendix

Table 13: Task accuracy (%) in the supervision study. Each cell gives the forward-block / training-time result. Values are descriptive means of six paired continuations. ARC-Easy and PIQA use accuracy; HellaSwag uses normalized accuracy. The ordinary looped output is an unadapted diagnostic. These means do not provide pooled confidence intervals.

	ARC-Easy		PIQA		HellaSwag
Method	Single	Looped	Single	Looped	Single	Looped
Ordinary	0.00 / 0.00	7.15 / 5.90	0.00 / 0.00	3.28 / 2.72	0.00 / 0.00	4.45 / 5.13
Loop-only	2.52 / 2.67	-0.89 / -0.93	1.35 / 1.33	0.43 / 0.36	1.94 / 2.06	0.43 / 0.22
Joint CE 5%	-1.06 / -1.99	-1.25 / -1.76	-0.12 / 0.38	-0.38 / 0.26	0.24 / 0.03	0.09 / 0.02
Joint CE	-1.24 / -2.16	-1.11 / -1.84	-0.04 / 0.26	-0.18 / 0.06	-0.11 / -0.05	0.05 / -0.16
Online KD	-1.50 / -2.48	-1.19 / -2.22	0.10 / 0.27	0.30 / 0.51	-0.18 / -0.01	0.27 / 0.22

Appendix

Table 14: Excess loss

E

of the parent’s correct examples, in percentage points. Each cell gives the forward-block / training-time result against ordinary continuation under the same budget. Negative favors the candidate. Each task and parent uses the same set

S_{0}

for every method and both paths. Values are descriptive six-unit means.

Method	Single NLL	Looped NLL	Forward KL	Fix (%)	Break (%)
Ordinary	3.6259	3.8698	0.2111	2.6402	4.6581
Loop-only	3.7974	3.6163	0.1809	4.2093	2.4029
Joint CE 5%	3.6603	3.6182	0.0489	2.0884	1.6266
Joint CE	3.6467	3.6189	0.0366	1.7648	1.4287
Online KD	3.6367	3.6207	0.0263	1.4517	1.2664
Matched CE	3.6379	3.6226	0.0264	1.4531	1.2704

Appendix

Table 15: Diagnostic means at forward-block endpoints on eight fixed supervision-study validation blocks, comprising 262,144 targets. Fix is the fraction of tokens where looped top-1 is correct and single-pass top-1 is wrong. Break is the opposite event. These are token prediction rates, not task accuracy or sequence-level reasoning measures.

Method	Seconds/update	Allocated GiB	Reserved GiB
Ordinary	0.50234	5.76	6.41
Loop-only	0.64863	6.52	7.19
Joint CE 5%	0.84690	8.51	8.81
Joint CE	0.84702	8.51	8.81
Online KD	1.02425	8.90	9.58
Matched CE	0.88865	8.51	8.81

Appendix

Table 16: Complete-update timing in the isolated RTX 5090 benchmark for the supervision study. Each method uses twenty warmup updates and five blocks of one hundred timed updates on the same device. Memory is peak allocated/reserved GiB. Measurements use default frequency scaling and the existing 575 W limit.

Supervision-study process category	Observed GPU-process hours
Training and engineering	36.978048
Validation NLL and diagnostics	4.256758
Confirmation and WikiText	2.096609
Task evaluation	24.670267
Task-interface checks	0.004445
Total observed	68.006127

Appendix

Table 17: Approximate compute for the supervision study, including exploratory diagnostics and engineering work. Process lifetime includes overhead and is distinct from active update time. The subtotal excludes inherited parent pretraining and historical experiments.

Implementation	Matched / Joint CE	Conditional 95%
Common weighted-CE entry	1.000229	$[0.998948,1.001511]$
Historical separate entries	1.051945	$[1.050699,1.053194]$

Appendix

Table 18: Balanced follow-up CE timing control: four fixed GPUs, eight interleaved rounds per card. Ratios use paired card/round measurements with conditional approximate 95% intervals. Kernel counts are not independent repetitions.

	Single-pass output			Looped output
Comparison	Mean	Text interval	Order interval	Mean	Text interval	Order interval
KD $-$ flat	$-453.124$	$[-456.717,-449.147]$	$[-498.394,-407.854]$	$-5.319$	$[-5.897,-4.783]$	$[-6.750,-3.888]$
KD $-$ permuted online	$-489.305$	$[-492.476,-485.731]$	$[-523.462,-455.149]$	$-8.159$	$[-8.638,-7.699]$	$[-8.678,-7.639]$
Frozen looped $-$ permuted	$-463.495$	$[-466.594,-460.052]$	$[-504.144,-422.846]$	$-10.262$	$[-10.777,-9.785]$	$[-10.711,-9.813]$
Frozen looped $-$ matched CE	$+28.096$	$[+27.244,+28.968]$	$[+22.671,+33.520]$	$-10.111$	$[-10.448,-9.786]$	$[-11.314,-8.908]$
Frozen single-pass $-$ matched CE	$+0.627$	$[+0.163,+1.106]$	$[-0.367,+1.622]$	$-13.449$	$[-13.789,-13.088]$	$[-14.204,-12.693]$

Appendix

Table 19: All predefined target-study comparisons at 2,925 student updates. Values are in

10^{-3}

nats per target. Negative favors the first method. Text intervals retain the original content/source/timing family corrections of 99.1667%/99.375%/99.5%; the re-evaluated KD-minus-matched-CE row retains 98.75%. Order intervals are conditional Welch 95% intervals for the two fixed parents. KD denotes the current online looped target; frozen methods use the corresponding initial predictors. Horizontal rules separate the three intervention families and the re-evaluated control.

	Single-pass output			Looped output
Comparison	Mean	Text interval	Order interval	Mean	Text interval	Order interval
KD $-$ flat	$-453.124$	$[-456.717,-449.147]$	$[-498.394,-407.854]$	$-5.319$	$[-5.897,-4.783]$	$[-6.750,-3.888]$
KD $-$ permuted online	$-489.305$	$[-492.476,-485.731]$	$[-523.462,-455.149]$	$-8.159$	$[-8.638,-7.699]$	$[-8.678,-7.639]$
Frozen looped $-$ permuted	$-484.749$	$[-487.809,-481.232]$	$[-539.010,-430.487]$	$-9.417$	$[-9.879,-8.977]$	$[-10.189,-8.644]$
Frozen looped $-$ matched CE	$+30.733$	$[+29.266,+32.212]$	$[+25.920,+35.546]$	$-3.204$	$[-4.322,-2.025]$	$[-5.012,-1.396]$
Frozen single-pass $-$ matched CE	$+1.862$	$[+0.797,+3.006]$	$[-0.160,+3.884]$	$-7.424$	$[-8.541,-6.192]$	$[-9.454,-5.393]$

Appendix

Table 20: Target-study comparisons at the 40,960-block cap including teacher computation. Units, contrasts, and intervals match Table G . Frozen looped and single-pass endpoints have 1,575 and 1,861 student updates; online joint endpoints have 2,925. Fixed-update and fixed-resource comparisons retain separate identities.

	Single-pass output			Looped output
Comparison	Mean	Text interval	Order interval	Mean	Text interval	Order interval
KD $-$ flat	$-479.619$	$[-483.104,-475.730]$	$[-527.769,-431.470]$	$-5.479$	$[-6.590,-4.530]$	$[-6.563,-4.394]$
KD $-$ permuted online	$-498.099$	$[-501.262,-494.650]$	$[-538.250,-457.947]$	$-9.290$	$[-10.254,-8.415]$	$[-10.383,-8.197]$
Frozen looped $-$ permuted	$-499.475$	$[-502.889,-495.529]$	$[-542.431,-456.519]$	$-10.347$	$[-11.328,-9.385]$	$[-12.906,-7.788]$
Frozen looped $-$ matched CE	$+27.279$	$[+25.876,+28.644]$	$[+25.548,+29.010]$	$-6.196$	$[-7.229,-5.244]$	$[-7.662,-4.729]$
Frozen single-pass $-$ matched CE	$+0.537$	$[-0.634,+1.595]$	$[-0.704,+1.778]$	$-9.884$	$[-10.908,-8.967]$	$[-10.866,-8.903]$

Appendix

Table 21: Target-study comparisons at 2,227.587 seconds of complete model updates, including teacher forward passes, target construction, and reused prefixes. Units and intervals match Table G . The time endpoints cover the cap, with the last saved state within it used for evaluation.

Method	$D_{\rm start}$	$x$	$y$	$d$
Online KD	$+0.002040$	$+0.011216$	$-0.001728$	$-0.012945$
Matched CE	$+0.003058$	$+0.012235$	$-0.001806$	$-0.014041$
Frozen looped	$+0.030337$	$+0.039514$	$-0.008002$	$-0.047515$
Frozen single-pass	$+0.003595$	$+0.012772$	$-0.011691$	$-0.024462$

Appendix

Table 22: Target-study mean NLL differences at the time cap against validation-selected ordinary continuation. Definitions follow § 4 . Each entry averages six paired units; the means are not new significance tests.

NLL increase	0.0005	0.001	0.002	0.005	0.010	0.020
PPL increase (%)	0.0500	0.1001	0.2002	0.5013	1.0050	2.0201

Appendix

Table 23: Algebraic conversion of NLL increases to relative perplexity increases,

100(\exp(\Delta\mathrm{NLL})-1)

. The conversion uses the same tokenization and evaluation protocol. It is neither a task-accuracy change nor a comparison of word-level perplexity across tokenizers.

Study / reference	Budget	$0$	$.001$	$.002$	$.005$	$.01$	$.02$
Supervision / fixed	Block	0/60/0	0/60/0	0/60/0	0/60/0	0/57/3	17/34/9
Supervision / fixed	Time	0/60/0	0/60/0	0/60/0	0/60/0	1/55/4	18/35/7
Target / fixed	Block	0/102/0	0/102/0	0/102/0	0/102/0	0/91/11	35/58/9
Target / fixed	Time	0/102/0	0/102/0	0/102/0	0/102/0	0/89/13	28/56/18
Target / selected	Block	0/102/0	0/102/0	0/102/0	0/102/0	0/91/11	35/58/9
Target / selected	Time	0/102/0	0/102/0	0/102/0	0/102/0	0/89/13	28/56/18

Appendix

Table 24: Supplementary retention sensitivity on each study’s primary confirmation text. Cells count supported/violated/unresolved conjunctions among fixed endpoints at the column’s tolerance

\tau

. The requirements are

D_{\rm start}\leq\tau

x\leq\tau

y<0

, and

d<0

. Supervision rows contain 60 method–unit endpoints; target rows contain 102. Fixed and selected denote the ordinary reference, not candidate selection. Selected ordinary states minimize validation NLL within the budget. Each conjunction uses the four separate pointwise 95% intervals for the same checkpoint and budget. The counts are correlated and do not provide simultaneous coverage.

Budget	Output	Interval	$.0005$	$.001$	$.002$	$.005$
Equal updates	Single	Text family	U	U	S	S
Equal updates	Single	Order 95%	U	U	S	S
Block	Single	Text family	V	U	U	S
Block	Single	Order 95%	U	U	U	S
Time	Single	Text family	U	U	S	S
Time	Single	Order 95%	U	U	S	S

Appendix

Table 25: Single-pass equivalence sensitivity for guidance from the frozen original single pass minus matched CE. Columns give the half-width of the equivalence range in nats per target. S means the entire interval is inside the range, V means it is entirely outside, and U means it overlaps a boundary. Looped intervals lie below every tested equivalence range, reflecting improvements beyond these scales. Text-family and data-order intervals remain separate.

Figure 4: Distillation effects beyond stronger single-pass labels. Online KD leaves a small residual, whereas Frozen-original KD primarily improves the looped output. Both methods start at the original checkpoint. All KL terms guide the current single pass; both label losses remain active. Each column compares with Matched CE on its own study’s confirmation text, so the columns do not rank methods across datasets. Columns use different horizontal ranges. Negative values favor distillation. Blue Block and orange Time rows use forward-block and complete-update time budgets, including teacher computation. Circles and triangles show three paired effects within parents C and D, with orders 1–3 from top to bottom. Diamonds mark the six-unit mean. Solid bars are adjusted text-bootstrap intervals, 98.75% for Online KD and 99.375% for Frozen-original KD. Dashed bars are data-order 95% intervals conditional on C/D. The two interval types describe separate uncertainty sources. Values use

10^{-3}

nats per target. Appendix Figure 3 retains both online-KD schedules.

Figure 5: Initial fixed-coefficient target study. Guidance to the single pass mainly improves the looped output with Frozen-original KD (ours). Bars show NLL differences from Matched CE at matched training time, including teacher computation, on the same target-study confirmation text. Lower is better. Both methods guide only the current single pass and retain label losses on both outputs. Error bars are 95% data-order intervals for six paired continuations, conditional on two fixed pretrained models. Adjusted text intervals, block-budget results, and individual effects are reported in Appendix G .

	Single-pass output		Looped output
Teacher / method	$\Delta$ NLL	Order 95% interval	$\Delta$ NLL	Order 95% interval
Online KD	$-1.018$	$[-3.044,+1.008]$	$+0.078$	$[-1.700,+1.856]$
Frozen initial looped output	$+27.279$	$[+25.548,+29.010]$	$-6.196$	$[-7.662,-4.729]$
Frozen-original KD (ours)	$+0.537$	$[-0.704,+1.778]$	$\boldsymbol{-9.884}$	$\boldsymbol{[-10.866,-8.903]}$

Appendix

Table 26: Initial fixed-coefficient target study. Teacher-source effects at the complete-update time cap, in

10^{-3}

nats per target. Every distillation objective guides the current single pass while retaining label losses on both current outputs. Differences are candidate minus coefficient-matched CE on the target-study confirmation text; negative favors the candidate. Teacher computation is included. Means summarize six paired continuations, and 95% intervals describe data-order variation conditional on the two pretrained parents. The online row re-evaluates existing trajectories. Equal-update and text-bootstrap results appear in Appendix G .

Tolerance $\tau$	CE	Online KD	Frozen-original KD
0.001	None	None	0.4
0.005	0.8	0.8	0.4
0.01	0.4	0.4	0.4
0.02	0.4	0.2	0.4

Appendix

Table 27: Weights selected from validation data on C1 and D1. The primary tolerance is

\tau=0.005

; other rows are the prespecified development-set sensitivity analysis. None means no admissible weight, not a missing evaluation or a substituted recipe. The frozen-teacher family selects

m=0.4

throughout.

Family	$m$	$D_{\rm start}$	$x$	$y$	$d$	All four
Label-only CE	0.2	$+0.003687$	$+0.011903$	$-0.002327$	$-0.014230$	0/6
	0.4	$-0.002705$	$+0.005511$	$+0.000808$	$-0.004703$	2/6
	0.8	$-0.006054$	$+0.002162$	$+0.003877$	$+0.001715$	0/6
	1.6	$-0.006226$	$+0.001990$	$+0.008381$	$+0.006392$	0/6
Online KD	0.2	$+0.002618$	$+0.010833$	$-0.002341$	$-0.013174$	0/6
	0.4	$-0.003185$	$+0.005030$	$+0.001023$	$-0.004008$	1/6

Appendix

Table 28: All twelve measured weight-grid points at the complete-update time cap on the new confirmation text. Values are descriptive six-order means in nats per target and include the development orders C1 and D1. They are separate from the four-order selected-recipe confirmation in Table 2 . All four counts orders meeting the four point-estimate criteria with the primary tolerance. No untrained intermediate weights or continuous frontier are inferred.

CE weight $m$	Output	Frozen $m=0.4$ $-$ CE	Text 99.9621%
0.2	Single	$-12.428$	$[-13.738,-11.187]$
	Loop	$-6.071$	$[-7.523,-4.772]$
0.4	Single	$-6.036$	$[-7.604,-4.616]$
	Loop	$-9.206$	$[-10.846,-7.817]$
0.8	Single	$-2.687$	$[-3.917,-1.376]$
	Loop	$-12.274$	$[-13.609,-10.994]$

Appendix

Table 29: Supplementary paired comparisons of frozen

m=0.4

with every tested CE weight, at the time cap on confirmation text. Units are

10^{-3}

nats per target; negative favors the frozen teacher. These six-order means include the development orders and are distinct from the four-order primary confirmation. Text intervals retain the 132-effect correction across all measured-point pairs and both coordinates in this budget/text/summary view.

Recipe	Criterion	Mean	Text 99.5833%	Order 95%
Label-only CE	$D_{\rm start}$	$-0.006291$	$[-0.007798,-0.004773]$	$[-0.010153,-0.002429]$
	$x$	$+0.001929$	$[+0.001013,+0.002827]$	$[-0.001274,+0.005132]$
	$y$	$+0.003571$	$[+0.002640,+0.004477]$	$[+0.000503,+0.006639]$
	$d$	$+0.001642$	$[+0.001281,+0.002090]$	$[+0.000980,+0.002305]$
Online KD	$D_{\rm start}$	$-0.005657$	$[-0.007281,-0.003673]$	$[-0.017427,+0.006112]$
	$x$	$+0.002563$	$[+0.001586,+0.003588]$	$[-0.009883,+0.015009]$

Appendix

Table 30: Selected-recipe criteria on the new confirmation text, in nats per target. Each estimate uses the same four held-out orders. Text intervals use the prespecified twelve-effect family (three recipes times four criteria), giving 99.5833% marginal coverage. Order intervals are separate conditional Welch 95% intervals for the two fixed parents, each with two orders. The thresholds are 0.005 for

D_{\rm start}

and

x

, and strictly zero for

y

and

d

. All four adjusted text upper bounds for Frozen-original KD meet their thresholds. The interval for

x

crosses zero, so the conclusion is retention within the margin.

Comparison	Output	Mean	Text 98.75%	Order 95%
Online KD $-$ selected CE	Single	$+0.000634$	$[-0.000196,+0.001512]$	$[-0.015180,+0.016448]$
Online KD $-$ selected CE	Looped	$+0.001070$	$[+0.000273,+0.001867]$	$[-0.010922,+0.013061]$
Frozen-original KD (ours) $-$ selected CE	Single	$-0.002662$	$[-0.003764,-0.001667]$	$[-0.011338,+0.006015]$
Frozen-original KD (ours) $-$ selected CE	Looped	$-0.011899$	$[-0.013084,-0.010875]$	$[-0.020206,-0.003592]$

Appendix

Table 31: Direct effects of selected KD recipes relative to selected CE, using the same four orders and fixed-time endpoints. Negative favors the KD recipe. The four comparisons form a separate family with adjusted 98.75% text intervals. Fixed-parent order intervals remain pointwise 95%. No interval combines text variation with variation across continuation orders or pretrained initializations.

Recipe	Order	$D_{\rm start}$	$x$	$y$	$d$	All four
Label-only CE	C2	$-0.004975$	$+0.002416$	$+0.004632$	$+0.002216$	No
	C3	$-0.006405$	$+0.000781$	$+0.002625$	$+0.001844$	No
	D2	$-0.005671$	$+0.003361$	$+0.004528$	$+0.001166$	No
	D3	$-0.008114$	$+0.001157$	$+0.002501$	$+0.001344$	No
Online KD	C2	$-0.005499$	$+0.001892$	$+0.004260$	$+0.002368$	No
	C3	$-0.005456$	$+0.001731$	$+0.004315$	$+0.002585$	No

Appendix

Table 32: All individual selected-recipe effects on the new primary confirmation text. Values are in nats per target. C2/C3 and D2/D3 identify the two retained orders for each fixed parent. Each row uses one candidate checkpoint for all four criteria and its paired fixed-time ordinary reference. Yes denotes joint satisfaction of the point-estimate inequalities, without an interval-based per-run claim.

Configuration	Split	Documents	Targets	Scoring blocks
Main / deeper, GPT-2 tokenizer	Validation	524	524,288	16
Main / deeper, GPT-2 tokenizer	Confirmation	2,175	2,097,152	32
Main / deeper, GPT-2 tokenizer	WikiText	–	282,624	1
Native, Parcae tokenizer	Validation	524	524,288	16
Native, Parcae tokenizer	Confirmation	2,175	2,125,824	32

Appendix

Table 33: New-study text scoring views. Native evaluation retokenizes the same document blocks, producing different token counts. Absolute NLL is not compared across tokenizers. WikiText is a previously inspected secondary corpus with one scoring block in this study; the earlier eighteen-cluster view and its intervals are not reused.

KL receiving path	Matched CE $(\mathcal{L}_{s},\mathcal{L}_{c})$	Single difference	Looped difference
Single pass	$(0.2,0.9)$	$+0.000519$	$-0.010163$
Looped output	$(0.1,1.0)$	$-0.013423$	$-0.009639$
Both outputs	$(0.15,0.95)$	$-0.005932$	$-0.008324$

Appendix

Table 34: Frozen-teacher guidance minus its respective coefficient-matched label control, in nats per target. Values are descriptive six-order means at matched training time on the new confirmation text. Every condition uses the frozen original single-pass teacher and total KL weight 0.1. Guidance to both outputs splits the weight into 0.05 per output. The controls differ across rows, so the improvement magnitudes do not rank receiving paths.

Configuration and recipe	$n$	$x$	$y$	Looped $-$ CE	CE reference
Main $8\to 12$ , selected $m=0.4$	4	$-0.000733$	$-0.008327$	$-0.011899$	Selected CE
Main selected recipe, WikiText	4	$+0.016779$	$+0.008051$	–	–
Deeper $8\to 16$ , fixed coefficients	4	$+0.015635$	$-0.010416$	$-0.010057$	Fixed CE
Native R1 $\to$ R4, fixed coefficients	6	$+0.006058$	$+0.003877$	$-0.002413$	Fixed CE

Appendix

Table 35: Frozen-original KD in separate configurations at the time cap.

x

and

y

are differences from each configuration’s own ordinary single-pass reference; the final numerical column compares looped outputs with the stated CE recipe. Values are descriptive means in nats per target;

n

counts parent–order units. The WikiText row does not reconstruct a paired CE effect from differently scoped averages. The deeper and native rows use the original fixed coefficients, not the selected main recipe. These within-configuration differences do not compare absolute NLL across vocabularies.

Receivers A $-$ B	Contrast	Output	Mean	Text 99.5833%	Order 95%
Single $-$ loop receiver	Direct	Single	$+2.802$	$[+1.953,+3.635]$	$[+0.871,+4.733]$
	Direct	Loop	$+1.283$	$[+0.519,+2.082]$	$[+0.130,+2.436]$
	Interaction	Single	$+13.941$	$[+12.895,+14.892]$	$[+12.327,+15.556]$
	Interaction	Loop	$-0.524$	$[-1.389,+0.398]$	$[-2.400,+1.352]$
Single $-$ both receivers	Direct	Single	$+2.241$	$[+1.392,+3.093]$	$[+0.002,+4.479]$
	Direct	Loop	$-0.632$	$[-1.529,+0.282]$	$[-2.432,+1.167]$

Appendix

Table 36: Completed receiving-path comparisons, in

10^{-3}

nats per target, for six orders at the time cap on confirmation text. Direct is KD

{}_{A}-

KD B ; interaction is (KD

{}_{A}-

CE A )

-

(KD

{}_{B}-

CE B ). CE controls are receiver-specific. All twelve comparisons retain their 99.5833% text-family intervals; conditional order 95% intervals are separate. The original label coefficients remain fixed.

Fixed-recipe configuration	Output	Frozen $-$ CE	Text 99.1667%	Order 95%
HF $8\to 16$	Single	$+3.189$	$[+2.120,+4.350]$	$[-1.514,+7.892]$
	Loop	$-10.057$	$[-11.202,-8.869]$	$[-17.355,-2.759]$
Native Parcae	Single	$-2.250$	$[-2.719,-1.815]$	$[-2.495,-2.005]$
	Loop	$-2.413$	$[-2.891,-1.959]$	$[-2.990,-1.836]$

Appendix

Table 37: Fixed-recipe extension intervals on each configuration’s confirmation text, in

10^{-3}

nats per target. HF uses four orders; native uses six, with primary evaluation noise view 0. Text intervals retain each configuration’s six-effect family. These are not tests of the selected main coefficient

\alpha=0.3

, nor comparisons of absolute NLL across tokenizers.

Fixed recipe	Noise view	$x$	$y$
Label-only CE	0	$+8.307$	$+6.290$
	1	$+8.316$	$+6.302$
	2	$+8.306$	$+6.316$
Online KD	0	$+9.084$	$+7.371$
	1	$+9.083$	$+7.383$
	2	$+9.080$	$+7.381$

Appendix

Table 38: Native fixed-recipe sensitivity to evaluation noise, in

10^{-3}

nats per target. Each row averages the same six fitted models at the time cap; views 1 and 2 are not new training repetitions. All displayed

x

and

y

means are positive relative to native ordinary continuation.

Recipe	Output	ARC-Easy	PIQA	HellaSwag
Ordinary	Single	43.08	58.65	27.32
Selected Label-only CE	Single	44.41	58.71	27.48
Selected Label-only CE	Looped	43.70	58.60	27.42
Selected Online KD	Single	44.03	58.31	27.20
Selected Online KD	Looped	43.82	58.30	27.22
Selected Frozen-original KD (ours)	Single	44.14	58.72	27.06

Appendix

Table 39: Task accuracy (%) for the selected recipes and paired ordinary continuation. Every row averages the same four held-out orders at the time cap. ARC-Easy and PIQA use accuracy; HellaSwag uses normalized accuracy. The ordinary row is recomputed from the four matching orders, not the six-order reference in the initial studies. These means are descriptive and have no pooled task interval.

Task	Selected recipe	Output	$E$ (pp)	Item 95% (pp)	Order 95% (pp)
ARC-Easy	Label-only CE, $m=0.8$	Single	$-2.39$	$[-3.78,-1.03]$	$[-5.72,+0.94]$
	Label-only CE, $m=0.8$	Looped	$-1.67$	$[-3.03,-0.30]$	$[-5.05,+1.71]$
	Online KD, $m=0.8$	Single	$-2.25$	$[-3.66,-0.87]$	$[-9.77,+5.26]$
	Online KD, $m=0.8$	Looped	$-1.75$	$[-3.16,-0.36]$	$[-13.64,+10.15]$
	Frozen-original KD, $m=0.4$	Single	$-3.53$	$[-4.84,-2.26]$	$[-18.12,+11.06]$
	Frozen-original KD, $m=0.4$	Looped	$-2.83$	$[-4.19,-1.52]$	$[-8.93,+3.27]$

Appendix

Table 40: Original-correct-set retention for all three selected recipes and both outputs on all tasks. For each parent and task,

S_{0}

contains examples its original single pass answers correctly.

F

is the error proportion within

S_{0}

;

E=F_{\rm candidate}-F_{\rm ordinary}

, so negative values mean less loss than ordinary, not an accuracy gain or zero forgetting. Each row averages C2/C3/D2/D3 at the fixed-time endpoints. Item intervals are paired, pointwise 95%, conditional on fitted models; order intervals are separate fixed-parent Welch 95% intervals. Neither gives a cross-task simultaneous guarantee.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 280 papers

Sep 30, 2026

Closing the Loop: Practical Training Recipes for Looped Language Models

Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain difficult to separate from differences in data and training. In this work, we establish practical training recipes for looped language models, with three main results. (1) We develop a compute-efficient from-scratch pipeline that reduces the training budget from 7.7T tokens in Ouro to 310B tokens while retaining strong reasoning performance. Pretraining followed by high-quality mid-training, together with learning-rate warmup and stronger exit-gate regularization, enables stable recurrent training without prior multi-stage schedules. (2) Under controlled comparisons, our 1.4B LoopLM outperforms a parameter-matched dense model trained on the same data and token budget on all 12 evaluated benchmarks, including +14 points on GSM8K, +10 on MATH, and +22 on DROP. At matched inference compute, it approaches a 3.9B dense model on mathematical reasoning and reading comprehension while using only 36% as many parameters. (3) We introduce a minimal recipe for converting pretrained dense models into looped ones: a single learned input-mixing scalar and a smoothed exit loss, with no step-specific parameters. Applied to Qwen3-1.7B-Base, Looped Qwen improves over an identically continued dense baseline on every evaluated benchmark across two data regimes, with statistically clear gains on GSM8K, MATH, and MMLU-Pro on the curated mixture. Together, these results make looped language models substantially cheaper to train from scratch and practical to introduce into existing pretrained checkpoints, while isolating the gains due to recurrence itself.

Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5

RND NLP, DAIMLD, Russian Federation

Efficient Language Model Training LLM Training Recursive Language Models Recurrent Language Models

Ask PDF

May 9, 2026

Sparse Layers are Critical to Scaling Looped Language Models

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

Ryan Lee, Jacob Biloki, Edward J. Hu +1

1USC Information Sciences Institute · 2Netflix · 3Independent Researcher

Mixture-of-Experts Language Models Recurrent Transformers Language Model Scaling LLM Inference Acceleration Language Modeling Recursive Language Models

Ask PDF

May 22, 2026

Training-Free Looped Transformers

We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning, continued training, or architectural changes. Unlike prior looped transformer methods that train with the looped structure end-to-end, we retrofit recurrence onto pretrained models at test time. We show that naive block reapplication usually degrades performance, highlighting the importance of the loop application strategy. Motivated by viewing a pre-norm transformer block as a forward Euler step on an ODE, we instead treat looping as a refinement of the same approximation, replacing one large update with smaller damped sub-steps. Across seven dense, sparse MoE, and MLA+MoE model families, our method improves Qwen3-4B-Instruct by +2.64 pp on MMLU-Pro, Qwen3-30B-A3B-Instruct by +1.14 pp on CommonsenseQA, and Moonlight-16B-A3B-Instruct by +1.20 pp on OpenBookQA.

Lizhang Chen, Jonathan Li, Chen Liang +2

University of Texas at Austin

Transformer Inference Recurrent Transformers

Ask PDF

Abstract

LLM Inference

Explore similar work

Closing the Loop: Practical Training Recipes for Looped Language Models

Sparse Layers are Critical to Scaling Looped Language Models

Training-Free Looped Transformers