Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

Parth Kulshreshtha; Shivali Dalmia; Abhishek Mukherji

cs.CLOct 2, 2026

Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

Authors: Parth Kulshreshtha, Shivali Dalmia, Abhishek Mukherji

Organizations: Centific

Abstract

A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the second. The retained annotation record of a six-language benchmark for personally identifiable information contains two independent annotator labellings, the aggregate shipped as gold, and a reviewer gold from independent expert re-annotation of a sample. Using it, we hold the scored output fixed and exchange the reference. The score moves by 4.95 F1 points for one output and 2.00 for the other (95% CIs [3.23, 6.27] and [0.26, 3.37]), and by 7.55 in the worst language. The comparison between two outputs moves as well: the paired interaction between reference and system is +2.95 points (CI [+1.97, +3.87]), survives correction for multiple testing, and changes one language's margin outright. The cause is an undocumented aggregation default that usually kept one annotator when adjudication did not fire, making the shipped reference a partial copy of an output being scored. We report the resulting reference-sensitivity band, show how to compute one from any retained annotation record, and argue that the quantity belongs beside the score.

Figures & tables

Language	Prompts	Str.	Rel.	Identical	Type	3rd pass	Silent
		Agreement $F_{1}$
Hindi	4,082	73.0	76.4	50.0	95.6	61.3	9.0
Portuguese	4,173	85.5	88.1	63.5	97.1	28.3	11.3
Polish	4,172	84.9	86.9	68.8	97.6	25.1	8.2
Swedish	4,099	72.8	75.9	48.4	95.9	38.7	14.5
Chinese	4,213	87.8	90.8	71.8	96.7	26.4	4.2

Table 1: The annotation record, per language. Agreement is micro-

F_{1}

between the two annotator labellings over the whole corpus. Identical is the share of prompts on which their span sets match exactly. Type is the share of boundary-matched spans that also agree on type. 3rd pass is how often the selective third pass (the pipeline’s adjudication step) fired. Silent is the share of prompts where the annotators differed, no third pass fired, and the delivered gold reproduced Annotator A.

Figure 1: The annotation pipeline as designed: two independent passes, a third on disagreement, and a reviewer pool that re-annotates a sample. In practice the third pass skipped some disagreements and covered some agreed prompts (Section 2 ). Both references are outputs of this one process.

Reference	Ann.	Deliv.	Rev.	$\Delta$ [95% CI]
Pooled	A	91.62	86.67	+4.95 [+3.23, +6.27]
	B	88.79	86.79	+2.00 [+0.26, +3.37]
Hindi (largest)	A	90.29	82.74	+7.55 [+4.90, +10.91]
Swedish (smallest)	A	89.54	88.76	+0.78 [-7.42, +6.58]

Table 2: Effect of exchanging the reference, Strict matching. Each annotator’s output is held fixed and scored against the delivered gold, then the reviewer gold.

\Delta

is the movement attributable to the reference alone. Bold marks intervals excluding zero. Table 4 gives all six languages and Relaxed .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Delivered gold			Reviewer gold
Language	A	B	Skew	A	B	Skew
Hindi	74.6	68.6	+6.0	61.7	64.0	-2.3
Portuguese	84.6	76.5	+8.1	74.1	74.8	-0.6
Polish	82.9	80.2	+2.7	72.6	73.5	-0.9
Swedish	84.2	71.3	+12.9	71.3	66.0	+5.3
Chinese	90.7	83.9	+6.7	75.8	75.6	+0.2

Appendix

Table 3: Neutrality of each reference. Percentage of audited prompts on which a reference reproduces an annotator’s span set exactly. The delivered gold favours Annotator A in every language; the reviewer gold does not.

		Strict			Relaxed
Language	Ann.	Deliv.	Rev.	$\Delta$ [95% CI]	Deliv.	Rev.	$\Delta$ [95% CI]
Hindi	A	90.29	82.74	+7.55 [+4.90, +10.91]	93.17	87.31	+5.85 [+3.49, +8.98]
	B	86.32	84.04	+2.28 [-0.33, +5.50]	88.92	87.85	+1.06 [-1.22, +3.99]
Portuguese	A	94.78	90.11	+4.67 [+3.31, +6.34]	97.12	93.26	+3.86 [+2.58, +5.43]
	B	90.19	89.93	+0.26 [-1.23, +1.82]	92.80	93.31	-0.52 [-1.91, +0.94]
Polish	A	89.43	85.82	+3.61 [+0.78, +5.92]	90.99	87.68	+3.31 [+0.54, +5.60]

Appendix

Table 4: Effect of exchanging the reference. Each annotator’s output is held fixed and scored against the delivered gold, then the reviewer gold.

\Delta

is the movement attributable to the reference alone. Bold marks intervals excluding zero.

Language	Ann.	vs. other ann.	vs. deliv.	vs. rev.	Range
Hindi	A	79.69	90.29	82.74	10.60
	B	79.69	86.32	84.04	6.63
Portuguese	A	86.06	94.78	90.11	8.73
	B	86.06	90.19	89.93	4.13
Polish	A	86.45	89.43	85.82	3.61
	B	86.45	90.89	88.64	4.44

Appendix

Table 5: Score range available from the choice of reference alone, Strict matching. Each row holds one annotator’s output fixed and varies which other labelling is designated gold. The maximum is obtained against the delivered gold in 13 of the 14 rows; the exception is Swedish B.

Language	Rule	Delivered gold	Reviewer gold	Swing [95% CI]	Holm $p$
		Margin [95% CI]
Hindi	strict	+3.97 [+0.65, +7.14]	-1.30 [-4.39, +1.55]	+5.27 [+2.64, +8.05]	$<$ 0.01
Portuguese	strict	+4.59 [+2.41, +6.71]	+0.18 [-1.75, +2.12]	+4.42 [+2.53, +6.45]	$<$ 0.01
Polish	strict	-1.46 [-4.59, +2.34]	-2.82 [-5.28, -0.29]	+1.37 [-0.67, +3.63]	0.73
Swedish	strict	+4.47 [+1.55, +7.68]	+0.94 [-1.66, +3.75]	+3.53 [+0.80, +6.58]	0.08
Chinese	strict	+1.75 [-0.54, +3.97]	+0.73 [-1.04, +2.50]	+1.02 [-0.87, +2.80]	0.73

Appendix

Table 6: Does the ranking of the two annotators survive a change of reference? Margin is

F_{1}(\text{A})-F_{1}(\text{B})

under each reference. Swing is the paired interaction, (margin under delivered gold)

-

(margin under reviewer gold), recomputed inside every bootstrap resample; it tests directly whether the reference changes the comparison. Bold marks intervals excluding zero.

p

is a two-sided bootstrap

p

-value for the swing, Holm-adjusted over all 14 tests. Strict matching; Table 7 adds the relaxed rows, which are included in the Holm family above.

		Margin [95% CI]
Language	Rule	Delivered gold	Reviewer gold	Swing [95% CI]	Holm $p$
Hindi	strict	+3.97 [+0.65, +7.14]	-1.30 [-4.39, +1.55]	+5.27 [+2.64, +8.05]	$<$ 0.01
	relaxed	+4.25 [+1.28, +7.04]	-0.54 [-3.23, +1.90]	+4.79 [+2.25, +7.37]	$<$ 0.01
Portuguese	strict	+4.59 [+2.41, +6.71]	+0.18 [-1.75, +2.12]	+4.42 [+2.53, +6.45]	$<$ 0.01
	relaxed	+4.33 [+2.57, +6.19]	-0.05 [-1.62, +1.58]	+4.37 [+2.57, +6.41]	$<$ 0.01
Polish	strict	-1.46 [-4.59, +2.34]	-2.82 [-5.28, -0.29]	+1.37 [-0.67, +3.63]	0.73

Appendix

Table 7: Does the ranking of the two annotators survive a change of reference? Margin is

F_{1}(\text{A})-F_{1}(\text{B})

under each reference. Swing is the paired interaction, (margin under delivered gold)

-

(margin under reviewer gold), recomputed inside every bootstrap resample; it tests directly whether the reference changes the comparison. Bold marks intervals excluding zero.

p

is a two-sided bootstrap

p

-value for the swing, Holm-adjusted over all 14 tests. Both matching rules.

Domain, task	Invented prompt and disputed span
Finance, Rewrite	Make this note more formal: “Dr. Anna Berg phoned about the card ending 0000, please call her back.” A: Dr. Anna Berg B: Anna Berg (NAME)
Travel, Extract	List the contact details in this booking: guest Omar Haddad, [email protected], +46 70 000 00 00. A: +46 70 000 00 00 B: 70 000 00 00 (PHONE)
Insurance, Summarize	Summarise this claim: the policyholder reports water damage at Flat 4B, 12 Harbour Road, after a pipe burst on 3 March. A: 12 Harbour Road B: Flat 4B, 12 Harbour Road (ADDRESS)

Appendix

Table 8: Invented prompts illustrating boundary disagreement. Each row shows only the disputed span; the other spans in the prompt are identical under both labellings. The A/B assignment is illustrative and makes no claim about which annotator did what. All names, numbers, and addresses are fictitious.

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 175 papers

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards
Aug 17, 2026Anik JhaAnnotator Disagreement LLM Evaluation
Independent Researcher
Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM
Aug 17, 2026Atul AnandLLM-as-a-Judge Language Model Generation Evaluation
Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator
Aug 23, 2026Alhasan Mahmood, Samir Abdaljalil, Hasan KurbanMultilingual Language Model Evaluation LLM-as-a-Judge
College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar · Department of Computer and Electrical Engineering, Texas A&M University, College Station, TX, USA

Aug 17, 2026J/K move · Enter open · S save

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

Anik Jha

Independent Researcher

Annotator Disagreement LLM Evaluation Language Model Generation Evaluation

Open paper AskPDF arXiv

Preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail. We measure what that detail buys. On the 2,885 MultiPref items where both pools are internally unanimous, so no tie-breaking convention is consulted at all, expert and crowd annotators assign a different majority label to 23.6% and name the opposite winner on 9.2%; on the 246 comparably unanimous MT-Bench cells, benchmark authors and recruited experts differ on 30.5% and reverse on 8.5%. Yet on both corpora the resulting model leaderboards are bit-identical: Kendall tau = 1.00 with zero of six models displaced. That invariance is far weaker evidence than it looks, and we quantify how weak. Switching pools moves a model's win rate by 1.9pp (SD), one adjacent pair in our own leaderboard sits 0.8pp apart and had a 38% chance of swapping, and an item-level bootstrap displaces at least one model in 28% of resamples. The observed zero is the common outcome, not a property of aggregation: on the same measured perturbation, a ten-model leaderboard is displaced with probability 0.86 and a twenty-model leaderboard with probability 0.9997. Reporting a six-model leaderboard is safe; the safety does not generalise, and everything that consumes labels per item is not safe at any size. We make the distinction precise, show that a widely used dataset's stated assumption of no intra-group annotator variability is false, and show that an LLM judge tracks the crowd pool over the expert pool on all three models we test, including one from a different vendor. All code, per-call outputs, and pre-registered decision rules will be released upon acceptance.

Abstract

Benchmark Validity

Explore similar work

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator