That a prompt's effect is not a property of the prompt is established: prompts optimised for one model degrade on another, and rankings reorder under neutral reformatting. That evidence is about task accuracy, which cannot say whether the interaction is a fact about task machinery or about the conditional distribution itself. We ask on a readout with no task in it: the fixed-point structure of the short-window argmax map x_{t+1} = argmax_x p(x | x_{t-1}, x_t), censused from 96 starts. It is deterministic, so nothing can be helped or hurt, and it exists only at short windows -- four of six models lose it entirely by window 16 -- so everything here concerns how a model reads a fragment. Two results. First, the interaction reaches this readout at full magnitude: nine tokens of conditioning move the fixed-point fraction across most of its range, change a four-way structural class, and reorder models, while instruction tuning worth 60.5 IFEval points moves the class by zero. Second, nothing we proposed carries it. Prefix length fails: the effect is not monotone. Four phenomenological factors -- prose-versus-markup, a universal direction, bidirectionality, instruct-resistance -- were each withdrawn within one run of being proposed, dissolved by widening the sample. And the nearest mechanistic account, attention-sink dominance of early tokens, predicts the sign of the shift on 2 of 5 models -- chance -- while a length-by-content cross shows it holds on real text and fails on our probe's uniformly random input, so we are outside its regime, not against it. One fixed nine-token prefix drives four models toward 0 and two toward 1; the bidirectionality survives in-distribution starts. On this readout the unit of explanation is the prompt-model pair. The recurring error it caught in us has a name: a criterion with a shape applied to a quantity with no room to vary.
Figures & tables
claim
died on
source
“prose and markup differ, and the sign flips by model”
mid-range models
F147 → F151
“the effect is unidirectional: 18 of 18 arms down”
a wider model set
F151 → F152
“no text raises two models”
a wider corpus
F153 → F154
“bidirectionality is a model property”
a wider text class
F154 → F155
“instruct models resist raising”
ONE additional text
F156
Table 1: The factor record, and the paper’s roadmap. Each proposed factor was withdrawn within one run of being proposed, always because widening the sample dissolved it into an interaction that had been undersampled. § 4 –§ 6 walk the ladder; findings.md carries the banners.
readout
domain change
effect
source
structural CLASS
9-token chat template
ϕ:0.948→0.000 , class changes
F144
class, fragmented model
every domain tested
changes under all of them
F151
model RANKING
chat template
ρ=+0.400/+0.300 vs a 0.6 gate
F145
scalar VALUE
chat scaffold
0.1327 shift vs a 0.0406 gate ( 3.3× )
F135
ϕ
9 tokens of prose
0.714→0.005
F151
Table 2: Conditioning moves a task-free structural readout at magnitudes comparable to those the literature reports for accuracy. Rows 3–4 are measured on the share, the stochastic sibling readout (Setup); the ranking row is an instance of known ranking instability (§ 1.1 ); the class rows have no accuracy analogue.
model
raw ϕ
under the prefix
direction
Falcon3-1B-Base
0.214
0.979
up
Minerva-3B-base
0.328
1.000
up
pythia-410m-deduped
0.427
0.016
down
Qwen1.5-1.8B
0.510
0.000
down
SmolLM-1.7B
0.562
0.000
down
Qwen2.5-1.5B-Instruct
0.573
0.005
down
Table 3: One fixed nine-token prefix (a table-of-contents fragment) applied to six models screened to have headroom in both directions. The prefix is the same source text at the same index, truncated to nine tokens under each model’s own tokenizer — so the length is held fixed and the character span varies slightly between models; per-cell token counts are in the results file. The bidirectionality survives the in-distribution control of § 8 (F160). Source: F154, results/text_interaction.json .
model
Δ sink
Δϕ
agree
Falcon3-1B-Base
+0.0403
+0.693
yes
Minerva-3B-base
+0.0675
−0.323
no
pythia-410m-deduped
−0.0197
−0.422
yes
Qwen1.5-1.8B
+0.0298
−0.510
no
SmolLM-1.7B
+0.1484
−0.562
no
Table 4: One BOS token, against a length-matched ordinary prefix. The mechanism and the structural consequence agree on 2 of 5 models. Source: F158, results/sink_vs_fixedpoint.json .
In this paper, we define the quantity of prompting complexity: for a fixed instruction-tuned language model, what is the shortest plausible prompt that makes deterministic decoding produce a target text? It is an LM-relative analogue of resource-bounded Kolmogorov complexity: the prompt is a program, the model interface is the interpreter, and information omitted from the prompt is supplied by the model's weights, training distribution, tokenizer, template, and decoding rule. Unlike classical Kolmogorov complexity, this measure is intentionally non-universal. In the finite-context setting it is computable by enumeration, but there is no model-independent invariance theorem; the same text may be cheap for one model and inaccessible or expensive for another. To keep the search space aligned with prompt engineering, we restrict programs to plausible human-readable texts rather than arbitrary token strings. We extend the exact definition to soft prompting complexity for approximate outputs, yielding a lossy notion of model-relative text compression and a formal target for prompt optimization. We also define prompting distance by comparing shortest generating prompts, and behavioral prompting complexity for reaching any output satisfying a specification. Based on these formulations, we define a research agenda for empirically studying which texts and behaviors are accessible from short plausible prompts under a fixed LM interface.
Adrian Cosma
Dalle Molle Institute for Artificial Intelligence (IDSIA)
Two accounts of neural text degeneration coexist. One locates the cause in the training data -- repetition in the corpus produces repetition in the output, established by training on repetition-sorted data -- the other in the trained network, in copying circuits and repetition features. Neither has been arbitrated across a broad cohort of pretrained models: the causal work trains its own. We report an observational measurement in a different currency: the fixed-point structure of a model's own short-window argmax map, censused from 96 random two-token starts over 17 off-the-shelf models, always unprompted -- a companion paper shows nine tokens of conditioning move this readout across most of its range. The four-way class is stable across census seeds on 17 of 17. Three exhibits. At fixed corpus (The Pile), fixed scale and that fixed domain, the class is not determined: across two size-matched tiers, pythia is a funnel while RWKV, Mamba and a second transformer family are not, and both hold their class across an order of magnitude of scale. Six of seven models in that ladder reach the same endpoint token, and those concentrating on it most strongly are among those that never stay there -- what varies is not where trajectories go but whether the destination self-continues. The deduplicated Pythia suite does not change the class. And the corpus-side inflow term proposed for this phenomenon does not select our endpoints once frequency is controlled, in English and three other languages. This is observational and cannot refute a training intervention. Funnels are common: eight of seventeen models, seven families, five corpora -- so the limit is not that the phenomenon is one model's peculiarity, but that within the one corpus where training data can be held fixed only one available family funnels; that subset cannot show the split is corpus-independent.
Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.
Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov +6