Reversing the text is not reversing the path: a fixed-text measure without a zero, and an assembly-order comparison that vanishes identically
Authors: Y. -H. Taguchi
Organizations: Bioinformatics Center, Institute for Chemical Research, Kyoto University, Gokasho, Uji, Kyoto 611-0011, Japan · Department of Physics, Chuo University, Tokyo, Japan
For an ensemble of trajectories the fluctuation theorems give the entropy production its meaning and zero. The same expression has recently been evaluated on one fixed text, Δ=[logP(x)−logP(x~)]/L with x~ the text in reversed order, read as an entropy production separating causally from arbitrarily ordered prose. A single text is neither a distribution nor a dynamics, and the texts at issue were produced by other models. No theorem fixes where the zero of a fixed-text quantity lies: the expression carries over from an ensemble, its zero does not. We supply a zero from an exact antisymmetry. Scoring a text's n sentences in all n! orders, reversal carries π to π∘rev, a measure-preserving involution, so Δ(π)=−Δ(π∘rev) identically and the values are centred at zero for any score. The authored arrangement's midrank is then unitless, bounded, and 0.50 for a score blind to order. Across five autoregressive evaluators from 124M to 4B parameters, and taking the prompt-defined labels at face value, 8% to 15% of the median midrank's distance from the null is attributable to causal order and the rest to the arrangement being the order the passage was produced in. The thermodynamic reading needs a comparison whose zero does hold on the text read, and a masked model supplies one: the any-order chain rule gives a genuine log-probability, and its left-to-right and right-to-left orders for one text give σ, a comparison of two assembly orders. Both run from the empty sequence to the same text, so it is not a path against its time reverse. It vanishes identically when the evaluator's conditionals come from a joint, and a non-zero reading reports the evaluator's incompatibility, not the text. Neither is a fixed-text entropy production, for different reasons.
Figures & tables
evaluator
corpus
n
median Δtoken
median Δblock
r
gpt2 (124M, AR)
Opus46
200
3.4674
0.3182
−0.279
GPT54
200
3.0195
0.4216
+0.050
GPT56
200
2.8213
0.3468
−0.522
Gemini31
200
3.0016
0.3676
+0.078
Fable5
200
3.1874
0.3469
−0.214
Table 1: Token reversal and block reversal side by side on the same texts, which is the comparison this section makes: the medians of Δtoken and Δblock , and the rank-biserial effect size r of Eq. ( 11 ) under the token reversal. It is shown first because the choice of block reversal, on which everything after it depends, rests on it. Reversing the token order destroys syntax, so Δtoken is dominated by grammaticality: its median is larger than that of Δblock in 5 of 5 cells, and r is negative — the non-causal texts scoring higher — in 3 of the 5 cells of this table, and small and positive in the rest. Nothing else in this paper reads Δtoken .
midrank
z
evaluator
direction
authored
causal
share
authored
causal
share
gpt2
AR
+0.354
+0.042
11%
+1.254
+0.228
15%
gpt2-medium
AR
+0.354
+0.083
19%
+1.342
+0.122
8%
gpt2-large
AR
+0.354
+0.042
11%
+1.330
+0.151
10%
gpt2-xl
AR
+0.396
+0.042
10%
+1.334
+0.199
13%
Qwen3-4B-Base
AR
+0.396
+0.042
10%
+1.330
+0.208
14%
Table 2: Decomposition of the within-text distance from the 0.50 null into the part any authored arrangement contributes (Eq. 12 ; computed on the texts labelled non-causal, which uses the labels to choose its reference set and not to measure anything, in the sense Section 3.4 sets out) and the part attributable to causal order alone (Eq. 13 ). Each entry is the median over all 5 corpora of the per-corpus quantity, so the two components are paired within a corpus and need not sum to the median total. Reported in midrank units and in z=Δ(id)/sd{Δ(π)} , which has no ceiling. The two forms put the share in overlapping ranges of one order — 10 – 19% in midrank units and 8 – 15% in z — which is what shows the midrank’s bound is not what makes it small. The z column is the one the paper quotes as a size, for the reason Section 5.3 gives. They are not interchangeable evaluator by evaluator: one differs between the forms by a factor of 2.3 . Across the 5 of them the two columns rank-correlate at ρ=−0.53 ( p=0.467 ), which settles nothing either way at that n . Section 5.3 is where that belongs: a quantity measured in the room a bounded statistic leaves does not have a share that survives renormalisation.
Figure 1: Where the authored arrangement sits among all orders of the same text, by evaluator in AR ladder order. Causal texts are the blue circles, non-causal the vermillion squares; the marker shapes differ so the two series remain distinguishable in greyscale and under the common colour-vision deficiencies. The dashed line at 0.50 is the null: a score that cannot distinguish the arrangements of a text lands there exactly. The non-causal series is the one Section 4 reads: it is already near the ceiling at 124M , and over the whole AR ladder its range is 2 steps of the 1/2n! grid. The vertical separation between the two series is the causal component of Eq. ( 13 ), and it does not close as the AR ladder is climbed — it is +0.042 at gpt2 and +0.042 at Qwen3-4B-Base , 2–4 steps of that same grid at every evaluator (Section 5.3 ). The gap between the non-causal series and the ceiling of 1.00 goes from 0.146 to 0.104 , which is 2 steps of the same grid, so it does not measurably close either (Section 5.4 ). Bars are bootstrap 95% intervals of the median.
evaluator
corpus
median Δblock
r
p
gpt2
Opus46
+0.3182
+0.244
0.003
gpt2
GPT54
+0.4216
−0.027
0.742
gpt2
GPT56
+0.3468
+0.043
0.590
gpt2
Gemini31
+0.3676
+0.311
1×10−4∗
gpt2
Fable5
+0.3469
+0.278
6×10−4∗
gpt2-medium
Opus46
+0.5573
+0.367
5×10−6∗
Table 3: Every (evaluator, corpus) cell of the AR ladder. Columns from the left: the evaluator; the corpus; the median over that cell’s texts of Δblock , the raw value of Eq. ( 1 ) for the authored arrangement, per token; the rank-biserial correlation r of Eq. ( 11 ) between the two label groups, positive when the causal group ranks higher; and its two-sided Mann–Whitney p . A star marks the cells that clear the Bonferroni threshold of 2.0×10−3 , so the counts this section quotes can be read off rather than recomputed from a p printed at the threshold. Those counts, and the ones in Section 4.2 , are taken over these rows; the Δblock column is the quantity Section 4.2 reports a range of, and the denominators of the factor quoted in Section 2.1 .
evaluator
>0.50
>0.60
>0.70
>0.80
>0.90
>0.95
strict max
gpt2
0
2
3
6
17
13
+13
gpt2-medium
0
0
0
6
21
18
+18
gpt2-large
0
0
0
11
17
16
+16
gpt2-xl
0
0
1
12
25
19
+19
Qwen3-4B-Base
0
0
2
11
28
22
+22
Table 4: The causal/non-causal contrast at each midrank threshold: the points by which more causal than non-causal texts have a midrank above t , median over the 5 corpora. 0.80 is the first threshold at which every evaluator separates the groups by more than a point; below it single evaluators reach 3 . The >0.95 column is the strict-maximum column: the only midrank above 0.95 that n! arrangements admit is the largest one, so the two cuts coincide by construction and not by measurement. The last column is the strict-maximum contrast of Table 5 , which is the cut Section 5.4 follows; it is not the widest, and this table is printed so that choice can be seen.
evaluator
Opus46
GPT54
GPT56
Gemini31
Fable5
median
midrank: points by which more causal than non-causal texts reach the strict maximum
gpt2
+19
−2
+6
+14
+13
+13
gpt2-medium
+22∗
+17
+18
+18
+8
+18
gpt2-large
+26∗
+8
+14
+22∗
+16
+16
gpt2-xl
+18
+18
+21∗
+27∗
+19
+19
Qwen3-4B-Base
+34∗
+27∗
+12
+20
+22∗
+22
Table 5: The causal/non-causal contrast in the tail, for every (evaluator, corpus) cell. Upper block : the points by which more causal than non-causal texts have an authored arrangement that is the strict maximum of the n! arrangements of its own text, on a grid of 1.0 or finer: 100 is the smallest number of causally labelled texts in any cell, so 1.0 bounds the coarsest grid rather than naming one. Lower block : the same contrast with no upper bound to sit on — the percentage of causal texts whose z=Δ(id)/sd{Δ(π)} exceeds the non-causal group’s own 90 th percentile in that cell, whose null value is 10% by construction. A star marks a cell below 0.05/25=2.0×10−3 : in the upper block a two-sided Fisher exact test of the two groups’ counts against each other, and in the lower a two-sided binomial test of the causal count against the 10% the cut defines, which is a one-sample question and not the same test. 8 cells clear it in the upper block and 17 in the lower, against 0.05 expected by chance. The per-cell p values themselves are summarised rather than printed, because what the section uses from them is the count.
at ceiling
gap / room
evaluator
direction
params
causal
non-causal
gap
(95% CI)
gpt2
AR
124M
32%
20%
+0.042
0.29 [0.00, 0.57]
gpt2-medium
AR
355M
32%
15%
+0.083
0.57 [0.29, 0.57]
gpt2-large
AR
774M
33%
16%
+0.042
0.40 [0.29, 0.57]
gpt2-xl
AR
1.6B
36%
16%
+0.042
0.40 [0.40, 0.57]
Qwen3-4B-Base
AR
4B
40%
15%
+0.042
0.40 [0.40, 0.57]
Table 6: How far each evaluator has already driven an authored arrangement towards the top of its own permutation set, and what is left above it. The two percentage columns are the fraction of texts whose authored arrangement is the strict maximum of the n! arrangements, counted separately on the causal and the non-causal texts. Eq. ( 12 ) is defined on the latter, and it is the non-causal column that bears on saturation: it does not rise, which is why this section withdraws that claim. The causal column rises; that is a level read on the causal group alone, which Section 5.4 separates from the contrast of Section 5.3 , and it is not a property of the component. gap is the causal component of Eq. ( 13 ), which Figure 1 shows as the vertical separation between the two series; the last column is that gap as a fraction of the distance remaining between the non-causal median and the ceiling of 1.00 , with a 95% interval from resampling texts within each corpus and taking the median across the 5 corpora. That median is one of 5 per-corpus ratios, so this column has a grid of its own: it takes 3 distinct values over 5 evaluators and its intervals have 4 distinct endpoints, with 3 of the point estimates sitting on one. We read its level and not its width, for the reason Section 5.3 gives about the midrank. It is also not the ratio of the two medians printed beside it; those are medians of different per-corpus quantities.
evaluator
corpus
n
degen
causal
non-causal
pc
pnc
r
p
bert-base-uncased
Opus46
200
0
0.458
0.521
0.366
0.968
−0.060
0.462
bert-base-uncased
GPT54
200
0
0.479
0.521
0.676
0.514
−0.061
0.454
bert-base-uncased
GPT56
200
0
0.479
0.583
0.989
0.037
−0.126
0.122
bert-base-uncased
Gemini31
200
0
0.292
0.479
6×10−6
0.759
−0.287
4×10−4
bert-base-uncased
Fable5
200
0
0.417
0.521
0.050
0.736
−0.130
0.112
LLaDA-8B-Base
Opus46
200
0
0.292
0.229
6×10−6
1×10−11
+0.107
0.192
Table 7: The assembly-order quantity of Eq. ( 3 ), read as the midrank of the authored arrangement among the n! arrangements of the same text’s own sentences. Columns from the left: the masked evaluator; the corpus; n , the texts in that cell; how many were degenerate (all n! values tied, hence reported at the null and excluded from r ); the median midrank of the causal group and of the non-causal group; pc and pnc , the two-sided Wilcoxon signed-rank p of each group against the 0.50 null; r , the rank-biserial correlation of Eq. ( 11 ) between the two groups, negative when the causal group ranks lower; and its two-sided Mann–Whitney p . The two signed-rank columns answer a different question from the last one: whether each group is displaced from the null at all, as against whether the groups differ from each other. Masked evaluators only (Section 6 ).
evaluator
authored
steps
causal
steps
r
bert-base-uncased
+0.0208
+1.0
−0.1042
−5.0
−0.126
LLaDA-8B-Base
−0.2708
−13.0
−0.0208
−1.0
−0.111
Table 8: The same measurement as Table 7 , summarised per masked evaluator. Columns from the left: the masked evaluator; the authored component med(pctnon-causal)−0.50 , as a median over the 5 corpora; that component in steps of the 1/48=0.0208 grid; the causal component of Eq. ( 13 ), likewise a median over corpora; that component in grid steps; and r , the median over the 5 corpora of the rank-biserial correlation Table 7 prints cell by cell, negative when the causal group ranks lower. These are the numbers the text quotes; Table 7 is where they come from, one corpus at a time.
corpus
bert-base-uncased
LLaDA-8B-Base
Opus46
−0.1235
+0.0574
GPT54
−0.0169
+0.0753
GPT56
+0.0864
+0.1088
Gemini31
−0.1191
+0.0161
Fable5
−0.0384
+0.0552
causal component
Table 9: The unrearranged reading of Eq. ( 3 ): each text scored once in each direction, in the arrangement it came in, with no permutation set and no midrank. Upper block : the median σ over all of a cell’s texts, both label groups together, per token. It is not the authored component of Eq. ( 12 ), which is a midrank against a null of 0.50 ; the null here is σ=0 . Lower block : the causal component of this reading — the median σ of the causally labelled texts less that of the non-causal ones — which shares its name with Eq. ( 13 ) but is a difference of raw σ values rather than of midranks. For a masked evaluator whose conditionals derive from a joint every entry of both blocks would be exactly zero by Eq. ( 4 ). None is, and on 4 of the 5 corpora the two masked evaluators give the upper block’s median σ opposite signs.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: σEP of Eq. ( 16 ), the entropy production of the chain induced by a model’s own conditionals, in the exactly solvable system. An autoregressive model sits on zero by construction, to within 4.0×10−20 , whatever its accuracy; the masked values are what the instrument returns when no joint accounts for its conditionals, and the widest model’s value is 3.5 times the narrowest’s. No Δ readings for masked evaluators are shown beside these: what was measured for them was not Δ (Section 3.2 ), and it is withdrawn with the comparison it belonged to.
model
d
params
loss
σEP
rms∣A∣
MLM
32
13317
1.1148
0.0074
0.3377
MLM
64
101189
1.1286
0.0063
0.3618
MLM
96
337349
1.1360
0.0144
0.4960
MLM
128
597253
1.1430
0.0253
0.6496
MLM
160
930885
1.1570
0.0259
0.6392
AR
32
13317
1.1955
3.99×10−20
0.0000
Appendix
Table 10: Capacity sweep in the exactly solvable system ( L=6 , V=4 , 4096 states, N=100,000 ). Model size is the transformer width d ; params counts the model’s parameters, loss is its final training loss, and rms∣A∣ is the root-mean-square cycle affinity over the sampled cycles. The column σEP is Eq. ( 16 ), the entropy production of the chain induced by each model’s conditionals — not Δ of Eq. ( 1 ). An autoregressive model’s conditionals come from a joint, so its chain is reversible and σEP vanishes identically, whatever the loss it reaches; the loss column is not comparable between the two rows, the two objectives differing in what they condition on; the masked rows are what conditionals belonging to no joint produce instead.
σEP
d
params
loss
emitted
recovered
J(emitted∥recovered)
⟨∣σ∣⟩
32
13317
1.1148
0.0074
4.21×10−19
0.19–0.20
0.2261
64
101189
1.1286
0.0063
4.95×10−19
0.12–0.26
0.2169
96
337349
1.1360
0.0144
5.30×10−19
0.20–0.36
0.2130
128
597253
1.1430
0.0253
3.01×10−19
0.22–0.55
0.2508
160
930885
1.1570
0.0259
2.23×10−19
0.21–0.46
0.2169
Appendix
Table 11: The route, controlled for. One masked model at each width of Table 10 , with its Glauber rates built both ways: from the conditionals it emits, and from conditionals recovered out of a joint. Three joints are used — the exponentiated pseudo-log-likelihood normalised over the state space, and the any-order chain rule taken left to right and right to left — and the recovered column is the largest ∣σEP∣ of the three. Normalising it is possible here and not for text, because it needs a sum over every state and this system has 4096 of them; it enters as a candidate joint to recover conditionals from, and not as a score. J is the mean symmetrised Kullback–Leibler divergence between the emitted conditionals and the recovered ones, smallest and largest over the three joints. The last column is the quantity of Section 6.1 , σ=[logPl2r−logPr2l]/L , averaged in magnitude over all 4096 states. The emitted column reproduces Table 10 exactly, same seed and same training call, so the comparison is within one model rather than between two.
Pairwise model comparisons drawn from foundation-model benchmarks ("A is safer than B") are read as quantitative verdicts but hinge on harness choices benchmark papers under-specify. We close one theory-benchmark loop on this primitive: a finite-envelope proposition tying a measurable pairwise-disagreement rate to whether the strict ordering admits a configuration-pair reversal, paired with a commit-stamped evaluation protocol that operationalises it on widely cited alignment benchmarks. On every benchmark we test, configuration choice alone can flip the pairwise verdict; the proposition isolates this strict-reversal failure mode.
Yanhang Li, Zhichao Fan, Zexin Zhuang
Northeastern University, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA · Southern Methodist University, Dallas, TX, USA
Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional PMI decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings. The code is released at: https://github.com/wenlai-lavine/DiffScore.
Wen Lai, Yingli Shen, Dingnan Jin +4
1Ant Group · 2Tsinghua University · 3Technical University of Munich
That a prompt's effect is not a property of the prompt is established: prompts optimised for one model degrade on another, and rankings reorder under neutral reformatting. That evidence is about task accuracy, which cannot say whether the interaction is a fact about task machinery or about the conditional distribution itself. We ask on a readout with no task in it: the fixed-point structure of the short-window argmax map x_{t+1} = argmax_x p(x | x_{t-1}, x_t), censused from 96 starts. It is deterministic, so nothing can be helped or hurt, and it exists only at short windows -- four of six models lose it entirely by window 16 -- so everything here concerns how a model reads a fragment. Two results. First, the interaction reaches this readout at full magnitude: nine tokens of conditioning move the fixed-point fraction across most of its range, change a four-way structural class, and reorder models, while instruction tuning worth 60.5 IFEval points moves the class by zero. Second, nothing we proposed carries it. Prefix length fails: the effect is not monotone. Four phenomenological factors -- prose-versus-markup, a universal direction, bidirectionality, instruct-resistance -- were each withdrawn within one run of being proposed, dissolved by widening the sample. And the nearest mechanistic account, attention-sink dominance of early tokens, predicts the sign of the shift on 2 of 5 models -- chance -- while a length-by-content cross shows it holds on real text and fails on our probe's uniformly random input, so we are outside its regime, not against it. One fixed nine-token prefix drives four models toward 0 and two toward 1; the bidirectionality survives in-distribution starts. On this readout the unit of explanation is the prompt-model pair. The recurring error it caught in us has a name: a criterion with a shape applied to a quantity with no room to vary.