Language models rely on finite token contexts, limiting access to earlier interactions. We introduce Recursive State-Augmented Language Models (Recursive SALM), a framework in which a persistent latent state evolves with each observation and generates adaptations of a pretrained language model. A modular read-transition-write architecture separates observation encoding, latent dynamics, and predictor adaptation. We develop deterministic and stochastic formulations. The experiments evaluate only deterministic SALM with frozen GPT-2 small and exclude earlier evidence from its decoder context. On a four-turn hypothesis-inference task, latent-gated LoRA reaches 89.1% accuracy, comparable to 88.0% for a trained full-context LoRA baseline, with earlier evidence available solely through a 64-dimensional state. Zeroing or shuffling the state substantially reduces accuracy. In a two-turn task with codewords unseen in training, several interfaces improve candidate ranking over chance, but rarely generate the exact codeword under the evaluated training schedule. These experiments show that persistent latent-conditioned adaptation can support sequential inference when earlier evidence is absent from the decoder context.
Figures & tables
Arm
Encoder
Transition
Adapter / LoRA
Total
Δ vs. anchor
Generated LoRA
591,360
160,128
1,604,032
2,355,520
+0.00%
Gated LoRA
591,360
160,128
1,618,692
2,370,180
+0.62%
Soft prompt
591,360
160,128
1,602,942
2,354,430
-0.05%
FiLM
591,360
160,128
1,603,893
2,355,381
-0.01%
Prefix tuning
591,360
160,128
1,597,952
2,349,440
-0.26%
Deep prompt
591,360
160,128
1,604,224
2,355,712
+0.01%
Table 1: Trainable parameters by component, from the optimizer’s parameter groups. The frozen GPT-2 (124M) is excluded. Δ is relative to the Generated LoRA anchor.
Arm
Params
Acc.
Bal. acc.
Gen.
KL
Zero acc.
Shuf. acc.
Δ KL shuf
Generated LoRA
2.36M
79.2 ± 7.0
81.4 ± 4.7
79.2 ± 7.0
0.71 ± 0.28
28.6 ± 4.5
31.8 ± 3.9
5.72 ± 2.67
Gated LoRA
2.37M
89.1 ± 4.1
89.6 ± 2.1
89.1 ± 4.1
0.41 ± 0.17
31.2 ± 0.0
33.9 ± 2.4
5.52 ± 2.16
Soft prompt
2.35M
87.5 ± 5.6
88.0 ± 6.4
87.5 ± 5.6
0.66 ± 0.22
31.2 ± 0.0
35.9 ± 0.0
7.36 ± 1.88
FiLM
2.36M
45.8 ± 15.7
46.1 ± 20.3
45.8 ± 15.7
1.03 ± 0.29
31.2 ± 0.0
29.2 ± 3.6
0.96 ± 0.85
Prefix tuning
2.35M
52.6 ± 3.6
54.2 ± 7.4
–
0.98 ± 0.07
31.2 ± 0.0
27.6 ± 1.8
0.87 ± 0.39
Deep prompt
2.36M
81.2 ± 3.1
83.1 ± 1.8
–
0.51 ± 0.20
31.2 ± 0.0
30.7 ± 1.8
2.42 ± 0.42
Table 2: Four-turn HCM, test split, mean ± s.d. over model seeds 0, 1, 2 (complete three-seed groups only). Zero/shuffled: state interventions for SALM; no-history/shuffled-history context interventions for full-context LoRA. Chance 25.0%; constant prediction of the majority code scores 34.4%.
Arm
Params
Acc.
Gen.
KL
Zero acc.
Shuf. acc.
Δ KL shuf
Generated LoRA
2.36M
40.6 ± 3.1
0.0 ± 0.0
2.73 ± 0.30
1.6 ± 0.0
1.0 ± 0.9
12.50 ± 1.95
Gated LoRA
2.37M
36.5 ± 6.3
0.0 ± 0.0
3.23 ± 0.29
1.6 ± 0.0
1.6 ± 0.0
11.17 ± 0.63
Soft prompt
2.35M
44.8 ± 13.0
2.6 ± 2.4
2.78 ± 0.72
1.6 ± 0.0
0.5 ± 0.9
11.52 ± 1.20
FiLM
2.36M
5.2 ± 3.9
0.0 ± 0.0
4.74 ± 0.45
1.6 ± 0.0
1.6 ± 0.0
1.93 ± 1.82
Prefix tuning
2.35M
3.6 ± 2.4
–
5.08 ± 0.27
1.6 ± 0.0
1.0 ± 0.9
1.17 ± 0.42
Deep prompt
2.36M
35.4 ± 8.6
–
3.20 ± 0.32
1.6 ± 0.0
2.1 ± 0.9
11.18 ± 1.17
Table 3: Two-turn recall of codes never seen in training, test split, mean ± s.d. over seeds 0, 1, 2. 64 candidates; chance 1.6%. Acc. is candidate-ranking accuracy and Gen. is exact greedy match. Each code occurs in one episode, so class-balanced accuracy equals accuracy.
Figure 1: Memory dependence on the test split. Panels (a) and (b) show candidate-ranking accuracy and panel (c) shows KL divergence to the HCM oracle. Bars show the normal model, its initial state, and a shuffled state. For full-context LoRA, the interventions remove or shuffle the history. Dots show individual seeds and error bars show ± s.d. over three seeds. Vertical lines mark the uniform predictor, frozen GPT-2 with or without history, and the exact posterior. Some reference values coincide.
Arm
Selected
Epoch
Min-NLL ckpt
Epoch
Final ckpt
Generated LoRA
40.6 ± 3.1
22.0 ± 7.2
9.4 ± 8.7
4.7 ± 1.2
51.6 ± 9.5
Gated LoRA
36.5 ± 6.3
27.3 ± 3.1
10.4 ± 8.6
3.3 ± 2.3
38.0 ± 8.6
Soft prompt
44.8 ± 13.0
20.7 ± 3.1
28.6 ± 12.1
8.7 ± 2.3
48.4 ± 9.8
FiLM
5.2 ± 3.9
20.7 ± 16.2
1.6 ± 0.0
2.0 ± 0.0
5.2 ± 3.9
Prefix tuning
3.6 ± 2.4
26.7 ± 3.1
1.6 ± 0.0
5.3 ± 2.3
5.7 ± 3.9
Deep prompt
35.4 ± 8.6
28.7 ± 2.3
21.4 ± 6.3
14.7 ± 1.2
37.5 ± 1.6
Table 4: Two-turn test candidate-ranking accuracy (%) at the validation-selected, minimum-validation-NLL, and final checkpoints. Checkpoint epochs are shown as mean ± s.d.
Figure 2: Validation learning curves (thick: mean over seeds; band: ± s.d.; thin: individual seeds). (a, b) Predictive NLL of the answer token over the full vocabulary. (c, d) Oracle KL of the candidate-normalized prediction. On two-turn, token NLL and candidate KL diverge as training proceeds. HCM means are drawn only while every seed of an arm is still training.