TraceRelay: Attention-Aligned Recurrence over Rolling Traces

Sungwoo Goo; Hwi-yeol Yun; Sangkeun Jung

cs.LGOct 8, 2026

TraceRelay: Attention-Aligned Recurrence over Rolling Traces

Authors: Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

Organizations: College of Pharmacy, Chungnam National University · Department of Computer Science & Engineering, Chungnam National University

Abstract

We present TraceRelay, an attention-aligned recurrent architecture that distributes persistent representations over a rolling sequence of low-dimensional traces. Local right looking attention forms increments from lower-layer representations; delivery is delayed until all attended inputs are in the causal past. A fixed additive phase recurrence accumulates the delayed increments, and left-looking attention reads the resulting residual augmented stream. A stride-wise prefix sum supports parallel prefill and bounded-buffer continuation. We study 36 small-model runs on Equal Repeats, bounded Dyck closing-type prediction, and causal Most-Freq generation, using three seeds per setting. At trained length 256, Equal Repeats models with recurrent phase inheritance reach 98.81-99.69% accuracy versus 50.73-51.63% for separately trained variants without inheritance, despite the latter receiving more updates. Accuracy drops sharply at lengths beyond the training range. At the longest evaluated lengths, models with more dimensions in the middle layer's recurrent traces perform better on Dyck (76.34% versus 55.49% close accuracy at length 4096), whereas models with fewer trace dimensions perform better on five-symbol Most-Freq (70.74% versus 55.60% exact generation at length 1024). These contrasting cases motivate further study of how the size of recurrent representations should be chosen for different tasks, without establishing a general rule across tasks or model configurations.

Figures & tables

Figure 1: Reference computation graph. Right attention observes lower-layer positions

i

through

i+R

; its increment is delivered at

i+W

, with

W>R

. The stride-wise phase scan and cos/sin readout precede the residual merge and Left attention. The displayed readout applies to initialized lineages; until a valid delayed update arrives, the readout is zero. Left Q/K/V consume the merged stream. The orange branches pass the merged representation

m_{t}

to the attention residual and the post-attention state

\widetilde{h}_{t}

to the FFN residual. Right does not read this layer’s phase. Normalization is implicit, as in the equations.

Figure 2: Equal Repeats: with and without phase inheritance. Accuracy on 10,000 common examples per length, mean

\pm

sample SD over three training seeds. Shading marks lengths beyond the 32–256 training range. Models without inheritance are separately trained with the same task and geometry; all use 20,000 updates, while models with inheritance stop earlier. The strong separation within the trained range largely disappears in OOD evaluation. The dotted line is the

1/3

chance reference.

Figure 3: Bounded Dyck closing-type prediction. Three-seed mean

\pm

sample SD for

k=8,m=10

, using 2,000 common sequences per length. (a) Close accuracy versus length; shaded lengths exceed the training maximum 256. (b,c) Matching-distance and pre-close-depth slices at

T=4096

, with 4,096,000 close targets in total. The starred distance bin 1025–2048 has only 43 targets; the empty

>2048

bin is omitted. The 513–1024 bin has 1,639 targets. Dotted horizontal lines in (b,c) mark 12.5% chance. Full slice counts are in Table 3 .

Figure 4: Most-Freq causal generation. Five symbols, separated counts with minimum gap four, training lengths 128–256, and 1,024 common evaluation examples per length. Curves show mean

\pm

sample SD over three training seeds; shading denotes length OOD. (a) Exact greedy ranking including predicted EOS, alongside same-input shortcut baselines. (b) Top-1 accuracy. Suffix frequency (32) ranks symbols present in the last 32 input tokens by descending frequency there, breaking ties by first occurrence within that suffix. First appearance lists symbols seen in the full input in order of first occurrence, ignoring their counts. Fixed symbol order always lists

0,1,2,3,4

. All three heuristics append EOS and use the same evaluation inputs. No gold output is fed back during model generation. Narrower centers have higher OOD means in this sweep, with unequal early-stopped training exposure.

Task	Phase inheritance	$d_{c}$	ID: 256	OOD: 1024
Equal Repeats	On	64	$98.81\pm 0.58$	$33.76\pm 1.82$
Equal Repeats	On	32	$99.24\pm 0.56$	$35.63\pm 2.99$
Equal Repeats	On	16	$99.69\pm 0.13$	$32.84\pm 1.66$
Equal Repeats	Off	64	$51.03\pm 0.94$	$37.48\pm 0.14$
Equal Repeats	Off	32	$51.63\pm 1.24$	$37.65\pm 0.23$
Equal Repeats	Off	16	$50.73\pm 0.52$	$37.47\pm 0.09$

Table 1: Best-checkpoint test results, mean

\pm

sample SD over seeds 42–44 (percent). ID is

T=256

and OOD is

T=1024

for every task. Metrics are sequence classification (Equal Repeats), token-weighted close type (Dyck), and exact generated ranking including EOS (Most-Freq).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Equal Repeats	Dyck	Most-Freq
Phase inheritance	On and off; 18 runs	On; 9 runs	On; 9 runs
Task parameters	Binary $0^{a}1^{b}0^{c}$ ; three relation classes	$k=8$ bracket types; maximum depth $m=10$	$V=5$ ; decreasing count, first-appearance ties
Sampler	Inclusive repeated length $r\in[1,\lfloor T/2\rfloor]$ ; equal-all maps to class 0	Completion-count DP over bounded paths; uniform opening types	Separated positive counts, gap $\geq 4$ ; randomized symbol assignment and order
Online source length	Uniform integer $T\in[32,256]$	Uniform even $T\in[32,256]$	Uniform integer $T\in[128,256]$
ID validation lengths	32, 64, 128, 256	32, 64, 128, 256	128, 192, 256
Final ID lengths	32, 64, 128, 256	32, 64, 128, 256	128, 192, 256

Appendix

Table 2: Executed protocols. All tasks sweep

d_{c}\in\{64,32,16\}

with three training seeds per variant and width. ID validation occurs every 250 updates; two consecutive mean ID scores

\geq 99\%

trigger stopping. OOD results never control training. Table 5 gives individual stopping/selected steps and token exposure.

Slice	Closes	$d_{c}=64$	$d_{c}=32$	$d_{c}=16$
1–8	3,194,038	$87.27\pm 2.62$	$72.57\pm 10.69$	$62.65\pm 14.31$
9–16	360,329	$48.41\pm 6.04$	$38.07\pm 1.98$	$34.52\pm 2.93$
17–32	257,308	$30.90\pm 3.45$	$27.24\pm 1.32$	$27.23\pm 3.41$
33–64	159,952	$28.85\pm 4.23$	$26.04\pm 2.23$	$26.83\pm 3.21$
65–128	80,458	$31.20\pm 4.76$	$26.90\pm 2.19$	$27.89\pm 3.08$
129–256	32,358	$32.93\pm 5.24$	$26.64\pm 2.36$	$27.94\pm 3.24$

Appendix

Table 3: Dyck diagnostics at

T=4096

: percent close accuracy, mean

\pm

sample SD. Counts are targets in the common 2,000-sequence evaluation set, not independent replications across models. An empty bin is unavailable, not zero accuracy.

Metric	$d_{c}=64$	$d_{c}=32$	$d_{c}=16$
Full exact	$55.60\pm 12.82$	$65.40\pm 2.39$	$70.74\pm 4.95$
Top-1	$86.82\pm 2.05$	$89.10\pm 3.00$	$90.76\pm 7.91$
Top-2 prefix	$74.41\pm 7.43$	$78.71\pm 2.03$	$84.05\pm 6.15$
Set exact	$81.84\pm 7.26$	$87.96\pm 3.97$	$89.55\pm 4.67$
Termination	$82.91\pm 7.32$	$96.22\pm 5.36$	$95.64\pm 5.61$

Appendix

Table 4: Most-Freq diagnostics at

T=1024

(percent; three-seed mean

\pm

SD). Top-2 requires the first two outputs in the correct order. Set exact requires each source symbol exactly once but ignores rank; termination records a predicted EOS within the decode limit.

Figure 5: Optimization histories. Every logged ID validation mean for all 36 runs; colors indicate width and line styles indicate seed. Stars mark the checkpoint selected by ID accuracy with a cross-entropy tie-break. The dashed horizontal line is 99%; stopping requires two consecutive passes. Curves end at their actual stopping updates. Nine Equal Repeats runs without phase inheritance and one Dyck run exhaust the budget. Metrics and horizontal scales are task-specific, and the curves are not smoothed. Table 5 gives steps, tokens, and wall time.

Task / inheritance	$d_{c}$	Seed	Best	Final	Tokens (M)	Min.	Params
ER / On	64	42	2,500	4,250	39.10	6.7	295,875
ER / On	64	43	5,750	7,000	65.10	10.7	295,875
ER / On	64	44	7,750	7,750	71.52	12.3	295,875
ER / On	32	42	6,000	9,250	85.51	14.6	268,227
ER / On	32	43	8,250	8,250	76.43	13.0	268,227
ER / On	32	44	10,250	10,250	94.80	15.9	268,227

Appendix

Table 5: Actual training exposure for all 36 runs. Best is the selected checkpoint update; final is the stopping update. Tokens are millions of physical training tokens; Most-Freq includes separator and teacher-forced output tokens. Minutes are logged training-loop wall time, including periodic ID validation but excluding final evaluation. On/off indicates phase inheritance; † denotes budget exhaustion without satisfying the two-pass threshold.

Explore similar work

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
Sep 3, 2026Yuxiang Wang, Kunyu Feng, Yingda Shen +3Memory-Augmented Language Models Recurrent Transformers
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2
Temporal Recurrence Favors Fewer Layers
Sep 14, 2026Ivan Anokhin, Johan Obando-Ceron, Irina Rish +1Recurrent Neural Networks
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Sakana AI
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model Inference Recurrent Transformers
Department of Computer Science University of Virginia Charlottesville, VA, USA

Sep 3, 2026, cs.LGJ/K move · Enter open · S save

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

Yuxiang Wang, Kunyu Feng, Yingda Shen, Haoning Xu+2

The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University+2

Memory-Augmented Language Models Recurrent Transformers LLM Inference Acceleration Adaptive Computation

Open paper AskPDF arXiv

Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.

Abstract

Explore similar work

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

Temporal Recurrence Favors Fewer Layers

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models