Where does the novelty a base language model produces with no task come from, and what can an LLM judge of a long stream actually see? We dismantle a cognitively inspired generation loop over 24 conditions on three base models. Most of its effect lives in one operation: a new subject injected every few hundred tokens (an interruption) into a stream whose literal repetition is damped (habituation). We judge windows of generated text only, with the premise as the unit (n=10) and a judge measured for repeatability, against a second judge family and against human readers. Under that protocol the interruption raises judged surprise by 1.2 to 1.4 points and connection by 0.8 over habituation alone. A connective that asks for continuity hurts; a bare paragraph break adds nothing detectable on fresh text; a reset context does at least as well as a kept one; and a pre-registered replication on new premises confirms the primary contrast. Three things the window judge could not see changed the first version of this study, and we think they are of general use. The judge scores the experimenter's injected sentence as the model's own. A fixed rotation of injected sentences makes the model replay its earlier segments from beyond the judge's horizon, and the judge scores the replay as surprise and connection (65-80% of post-interruption windows at periods 150-300). And the local gains do not compose: no arm produces an integrated document. The salience monitor, the in-loop judge, memory across interruptions and a judge-gated Review run with a gate that opens add nothing. On a problem with a verifier (online bin packing), the interruption multiplies valid, distinct candidate heuristics three- to fourfold without raising the quality of the best. We report an evaluation protocol for long generation and a controlled characterization of a simple intervention, not a mechanism of creativity.
Figures & tables
arm
habituation
interruption
bare
off
none: continuous generation, EOS masked
bare + habituation
on
none
habituation, EOS allowed
on
none; the model may emit end-of-text and start a new document
habituation 1.3
on (stronger)
none
interruption, no habituation
off
neutral subject change every 150 or 300 tokens, context preserved
habituation + interruption 150
on
neutral subject change every 150 tokens, context preserved
Table 1: Conditions: the habituation × interruption factorial with its baselines (top), the content and timing arms, the boundary and context controls, and the judge-gated arm (bottom). Ten premises each, 4,500 tokens per cell, main generator Qwen3-30B-A3B-Base; the ladder bare / habituation / habituation + interruption 150 / scaffold is replicated on Qwen3-8B-Base and OLMo-2-13B.
Figure 2: Habituation and interruption: bare, bare + habituation, interruption without habituation, habituation + interruption 150, the full scaffold, and the two habituation baselines (stronger penalty; EOS allowed). One point per premise, grey lines join the same premise across arms, bars are cell means with 95% bootstrap CIs over cells (Qwen3-30B-A3B, generated-only windows).
Qwen3-30B-A3B
Qwen3-8B
OLMo-2-13B
arm
S
C
H
S
C
H
S
C
H
bare
0.45
0.30
2.52
0.43
0.40
3.88
1.28
0.93
3.03
bare + habituation
1.58
1.28
4.45
1.37
1.00
4.35
1.96
1.26
3.99
habituation + interruption 150
3.02
3.68
6.12
2.76
3.17
5.15
2.77
2.87
4.05
DREAM scaffold
2.70
1.85
6.02
2.70
2.35
5.38
3.60
2.23
6.23
Table 2: The ladder on three generator models from two families: judged surprise (S), connection (C) and coherence (H), means over cells (generated-only windows, Opus 5, k=5 ; ten premises per arm). Habituation buys part of the surprise; the interruption buys the rest and most of the additional connection; the scaffold adds nothing detectable over the plain interruption on the Qwen models and, on OLMo, adds surprise and coherence (its selective forgetting lifts the stream out of a web-boilerplate mode) while giving back connection.
Figure 3: Where the stream goes: trajectories in sentence-embedding space (64-token windows, one PCA per seed), seed 6. ⋆ premise, ∘ start, × reseed, ⋄ judged event sized by surprise. Bare makes one jump and freezes; the scaffold orbits and returns; the clock reseed jumps across the space keeping its context.
Figure 4: Content of the interruption at period 150: neutral change, re-encounter stitch, the premise itself, the stream’s own past. Injecting a return is as bad as not interrupting.
Figure 5: When to interrupt: the re-encounter stitch on the clock (150 and 900), the neutral change every 900, the stitch on salience events, and salience only (nothing injected). Post-interruption windows; one point per premise.
Figure 6: Period of the neutral change: the post-interruption window (32–128 generated tokens after the injection) at periods 150, 300, 600 and 900; one point per premise.
Figure 7: Decay: cell-mean surprise of generated-only windows by tokens since the injection, per period.
Figure 8: What carries the effect at period 300: no interruption, a paragraph break (sham), a continuity connective (sham), the subject change on the preserved context, the subject change without habituation, the subject change on a reset context, and a break on a reset context; one point per premise.
Figure 9: The whole document, injected sentences removed: integration, development, coherence and surprise of the 4,500-token stream (Opus 5, k=3 ; one point per premise). No arm builds a whole; the interrupted stream over the preserved context reads as a sequence of restarts; the reset arm’s clean parts, each returning to the premise, read best.
Figure 10: The ladder on Qwen3-8B-Base (top) and OLMo-2-13B (bottom); one point per premise.
Figure 11: Spread of five judgments of the same window (delta rubric, 89 windows). Sonnet 5 concentrates at zero spread by scoring zero; Opus 5 gives continuous scores at ±0.7 noise.
Figure 12: N1: residual-stream trajectory geometry per layer and condition (Qwen3-30B-A3B, ablation battery): mean step, explored radius, and logit-lens commitment. Bare generation moves least at every sampled layer, most so at the surface.
Figure 13: N2: Spearman correlation between the judged window’s movement (novelty vs everything before it, left; local step, right) at each layer and judged surprise, pooled and within condition.
Figure 14: N3: how deep an interruption reaches (Qwen3-30B-A3B). Left: state change across an injection per layer, minus random-position control. Right: change in similarity to the premise’s own state. The forgetting reseed moves the deep state and returns it toward the premise; the clock reseed over preserved context barely touches it, and scores as high on the judged dimensions.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 15: Objective novelty against the OLMo-2 training corpus: novel 4-grams per cell, 3 prompts × 5 seeds. The anti-probable push roughly doubles surface novelty over min- p sampling.
Figure 16: Judged novel delta of the developed idea against the input’s semantic distance from 10,000 real prompts, per arm. No relationship.
The output of a Language Model can be tampered with while the model is writing it. A simple test can thus be constructed by evaluating the model's perception of this external perturbation. In this spirit, a simple benchmark is built in which a single word is consistently substituted with another in the generation process. We call this method Sleight of Word. Two distinct axes are measured: metrics that relate to the model's surprise, as well as an evaluation of the textual reaction for 19 different open-weight language models.
We present an automated, contrastive evaluation pipeline for auditing the behavioral impact of interventions on large language models. Given a base model M1 and an intervention model M2, our method compares their free-form, multi-token generations across aligned prompt contexts and produces human-readable, statistically validated natural-language hypotheses describing how the models differ, along with recurring themes that summarize patterns across validated hypotheses. We evaluate the approach in synthetic setting by injecting known behavioral changes and showing that the pipeline reliably recovers them. We then apply it to three real-world interventions, reasoning distillation, knowledge editing and unlearning, demonstrating that the method surfaces both intended and unexpected behavioral shifts, distinguishes large from subtle interventions, and does not hallucinate differences when effects are absent or misaligned with the prompt bank. Overall, the pipeline provides a statistically grounded and interpretable tool for post-hoc auditing of intervention-induced changes in model behavior.
Quintin Pope, Ajay Hayagreeve Balaji, Jacques Thibodeau +1
Looped language models (LMs) reuse one block of layers several times per token generation, promising more latent computation at no additional storage cost. Despite their rising popularity, the analysis of looped LMs is still in its infancy: little is known about whether the iterative transformations actually improve their capabilities, or whether these models remain stable under prompt perturbation. We construct an inspection protocol rooted in Mechanistic Interpretability comprising five diverse lenses. We apply it to examine eight looped LMs from four families. Models differ in many aspects but the key one is whether the prompt is transmitted only within the looped state (carry loops: Ouro, LoopUS) or re-supplied at every iteration (re-read loops: Parcae, and Rec-OLMo & Huginn from the Raven family). The results of our study provide valuable insights concerning safety and efficient use of looped language models.
Pawel Olszowiec, Paulina Tomaszewska, Maciej Żelaszczyk +2
Samsung AI Center, Warsaw, Poland · IFTR, Polish Academy of Sciences, Warsaw, Poland · University of Warsaw, Poland