Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit

Shuyi Fan; Boyuan Deng; Mengyu Xu; Xinhong Xie; Chenyang Li; Hongyang Zhang

cs.CLAug 27, 2026

Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit

Authors: Shuyi Fan, Boyuan Deng, Mengyu Xu, Xinhong Xie, Chenyang Li, Hongyang Zhang

Organizations: The University of Chicago · Johns Hopkins University · The Hong Kong Polytechnic University, Hong Kong

Abstract

LLM-judge audits assess bias by comparing ratings across matched conditions. Difference-in-differences designs compare two candidate responses within each item and then compare that contrast across a manipulated attribute. We show in closed form that this endpoint need not identify differential preference on the latent scale when ratings are bounded. A severity shift common to both responses produces an observed interaction whenever the scale bounds attenuate it unequally. We examine this problem in a pre-registered audit comprising 990 calls to a frozen pedagogy judge. The registered primary analysis did not detect an effect of the stated learner profile on scaffolding preference. The only nominally significant secondary interaction concerned productive struggle ( $+0.378$ points; $p = 0.002$ ). A post-hoc construction with zero differential preference reproduced 79 to 85% of this interaction using the observed severity shift and the scale floor alone. The observed interaction therefore does not identify differential preference on the latent scale.

Figures & tables

	Weak		Strong
Arm	$\Delta$	95% BCa	$\Delta$	95% BCa
$D$	$+2.582$	$[+2.272,+2.917]$	$+3.147$	$[+2.763,+3.509]$
$P_{\mathrm{nov}}$	$+2.719$	$[+2.380,+3.037]$	$+3.194$	$[+2.922,+3.550]$
$P_{\mathrm{adv}}$	$+2.634$	$[+2.125,+2.981]$	$+3.089$	$[+2.700,+3.467]$

Table 1: Mean preference for high scaffolding,

\Delta=S(R_{H})-S(R_{L})

, by arm and blind-labeled demonstrated competence (pre-registration §6.1). Intervals are 95% BCa over source-run means; the inference unit is the source tutoring run (weak: 18 runs, 30 stimuli; strong: 10 runs, 25 stimuli; some runs contribute to both strata).

Figure 1: Censoring reproduces most of the audit’s one nominally significant interaction. Exploratory; weak stratum throughout. (a) Per-unit overall ratings by arm and pole; the black tick is that row’s unclustered mean. Counts at right are units, of 30, resting exactly on that row’s own bound – the ceiling for

R_{H}

, the floor for

R_{L}

. The ceiling binds harder than the floor, while (b) and (c) argue from the floor; § 6.4 takes up this tension. (b) The two terms of ( 1 ) on one axis. Both are negative – the advanced profile scores both poles lower – so

\mathrm{PAG}

is the gap between them rather than an opposition between them; per-field results are in Appendix C . Grey figures give the high pole’s share of the two terms’ total movement,

|a_{H}|/(|a_{H}|+|a_{L}|)

; the overall rating’s

61\%

sits outside the

72

–

96\%

the four sub-scores span. The low pole is pinned in all three arms on

4

,

0

,

17

,

10

and

22

of the 30 weak stimuli in panel order, so scaffolding shows the same one-sided pattern with none floored. (c) Productive struggle: the observed

\mathrm{PAG}

; the zero-differential-preference construction of § 6.3 , filled where the shifted ratings are clipped and hollow where they are first rounded to the integers the judge emits, together reproducing

79

–

85\%

of the observed gap; and the residual, which is that construction’s low-pole prediction error and bounds nothing.

k

is the nonzero-cluster count: at

k=5

the exact test’s attainable floor of

0.0625

exceeds

\alpha

, so no data could have rejected and no

p

-value is shown. The dashed rule is what a full low-pole clip would manufacture,

104.5\%

of the observed gap. Intervals are 95% BCa over source-run means.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Value	Breakdown
Stimuli	55	30 weak / 25 strong
Source tutoring runs	23	18 weak / 10 strong / 5 shared
Word problems, all acid-mixture	6
Single-message contexts, no tutor turn	23	7/30 weak, 16/25 strong
Stimuli by source-run tutor type	31 / 24	ped. / conv., from 13 / 10 runs
Real turn is the high pole	43/55	29/31 ped., 14/24 conv.

Appendix

Table 2: Composition of the 55 stimuli and the pole-construction imbalances referenced in § 3 , recomputed from the frozen stimulus record. A fraction

x/y

counts within the stated subgroup.

Figure 2: The registered endpoint is null –

\mathrm{PAG}_{\text{weak}}=+0.085

scale points, 95% BCa

[-0.167,+0.353]

– against a test that reaches

80\%

power only past a shift of

0.40

–

0.42

. Registered analysis (pre-registration §7). Error bars in both panels are 95% BCa over source-run means, the registered inference unit: weak, 18 clusters over 30 stimuli; strong, 10 over 25; 5 runs contribute to both, so the weak-to-strong contrast is not paired and is not tested (§ 6.1 ). (a) Preference for high scaffolding,

\Delta=S(R_{H})-S(R_{L})

, against blind-labeled demonstrated competence, by arm; each label is joined to its arm’s own mean by a leader. The

y

-axis is truncated at

2.05

; no plotted value falls below it. (b) The registered estimand itself.

\mathrm{PAG}

is the vertical gap between the two profile arms in (a); read off two overlapping intervals it is invisible, so it is drawn here against zero on its own axis, filled on the weak stratum and hollow on the strong. Its half-width,

0.260

, is narrower than any arm’s in (a) because clustering the difference within source run removes the between-run level variation. Shading (exploratory) marks

|\mathrm{PAG}|

against which the registered weak-stratum test has under

80\%

power: solid to

0.40

, fringed to the dashed rule at

0.42

, the two simulated shifts that bracket the crossing (Fig. 3 b). It is drawn under the weak column alone, the stratum it was simulated on. The weak interval lies inside it; the strong stratum’s upper limit,

+0.572

, would not, and the simulation does not cover that test.

Quantity	Role in the paper	Location
Registered; computed by the frozen, hashed analysis code
Preference table $\Delta$ (arm $\times$ stratum)	Baseline scaffolding preference, both strata	Table 1
Primary endpoint $\mathrm{PAG}_{\text{weak}}$	The registered result; null	§ 6.1
Strong-stratum companion	Secondary; agrees with the primary	§ 6.1
Pure profile effect	Shows the profile reached the rating	§ 6.2
Evidence-gradient check	Tests the anchoring reading; null	§ 6.2

Appendix

Table 3: Specification status of every reported estimand. Registered quantities were fixed before any score existed: some by the analysis code frozen and hashed with the pre-registration, the remaining registered diagnostics by code written afterwards that is a deterministic function of the same ratings. A registered quantity can sit inside an exploratory analysis (e.g. the per-field decomposition), so the status is per quantity, not per section.

Figure 3: The registered test first exceeds

80\%

power between location shifts of

+0.40

and

+0.42

scale points, past the

+0.353

its own interval reaches. Exploratory; weak stratum throughout,

\alpha=0.05

. (a) The 18 source-run means the test sees, stacked where they coincide, on the sparse rational lattice that thirds and source-run averaging induce (denominators

\{1,3,6,12,18\}

; only 14 of the 18 lie on thirds). Crosses are the four exact zeros, which the test drops; the point with whiskers below is the estimate and its 95% BCa interval. (b) Power against a location shift, by nonparametric bootstrap of the observed centered cluster distribution (6,000 replicates per point; Monte Carlo standard error at most

0.007

). Shading marks shifts the test has under

80\%

power against: solid to

0.40

, fringed to

0.42

, the two simulated shifts that bracket the crossing, which the lattice makes a step rather than a smooth one. The vertical rule is the interval’s upper limit,

+0.353

. The hollow point at zero shift is off the curve: centering on the mean violates the signed-rank symmetry null there, so

0.080

is power against a nonzero pseudomedian and not the test’s size,

0.049

(App. E ).