Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

Jewon Yeom; Jaewon Sok; Seonghyeon Park; Jeongjae Park; Hwiyeong Lee; Taesup Kim

cs.CLOct 23, 2026

Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

Authors: Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim

Organizations: Graduate School of Data Science, Seoul National University

Abstract

When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, and that of the verifier instruction asking for the judgment. Multilingual confidence evaluations seldom separate the three. We vary them independently in a six-language factorial design with ten open-weight models (3B-14B) on LSQA, MMMLU and TriviaQA, holding each answer fixed across verifier languages and measuring confidence with a first-token P(True) probe. The three languages move different parts of the evaluation: which of them governs accuracy depends on the benchmark, matching the response language to the question or verifier language sharpens discrimination but not calibration, and the verifier language alone shifts calibration error by up to 0.04 without changing accuracy. The scorer matters as well: an unvalidated judge would have shrunk the MMMLU English-Hindi accuracy gap from 0.198 to 0.038. Multilingual confidence results, model rankings included, are therefore comparable only when all three languages are reported and the scorer is validated.

Figures & tables

Figure 1: Model ranks as the question (

Q

), requested response (

R

) and verifier (

V

) languages switch one by one from English to the same

\ell

(mean over five

\ell

and three benchmarks). Six model pairs swap accuracy order from first to last setting (95% bootstrap; Appendix E ).

Scorer	Subset	$n$	Accuracy	$\Delta$
Judge	All	935	.803 [.777,.829]	$+.047$
Judge	LSQA	316	.816 [.771,.860]	$-.057$
Judge	TriviaQA	313	.898 [.865,.930]	$-.077$
Judge	MMMLU	306	.693 [.638,.747]	$+.281$
EM	MMMLU	306	.856 [.814,.896]	$+.137$
EM	MMMLU a	310	.981 [.961,.997]	$+.000$

Table 1: Agreement of automated labels with single primary human ratings (Qwen3-8B excluded). EM: exact option match.

\Delta

: the scorer’s correct rate minus the human correct rate. a Against the reviewer’s option-level label rather than overall correctness.

Figure 2: Range (maximum minus minimum) of each axis’s six language-marginal means, with 95% item-bootstrap intervals.

V

cannot change accuracy. Hatched: MMMLU scored by the judge instead of exact match. Ranges are non-negative; the intervals show sampling variability, not tests against zero.

Dataset	Acc	AUROC	ECE 15	Brier
LSQA	.511	.799	.270	.274
MMMLU	.523	.691	.369	.377
TriviaQA	.316	.822	.287	.287

Table 2: Mean over the 2,160 configuration cells (10 models

\times

216

Q/R/V

cells) per dataset. MMMLU uses exact match; LSQA and TriviaQA use the judge.

Figure 3: (a) Mean share of explained accuracy variation attributed to question and response language on LSQA and TriviaQA in the provisional configuration-level decomposition. (b) TriviaQA accuracy and AUROC by the English position of question and response. (c) LSQA accuracy and AUROC by alignment with the content-subset language; “unaligned” pools the 25 conditions matching neither.

LSQA regime	Acc	AUROC	ECE 15	Brier
$Q{=}R{=}s$ (native)	.608	.797	.264	.260
$R{=}s$ only	.554	.797	.257	.260
$Q{=}s$ only	.549	.784	.261	.265
$Q{=}R{\neq}s$	.514	.823	.285	.287
$Q{\neq}R$ , both ${\neq}s$	.485	.788	.270	.277

Table 3: LSQA metrics by alignment with the item’s content-subset language

s

: means over the 360 model–subset–

V

units of each regime, each pooling the regime’s

Q/R

conditions.

	Term	LSQA	MMMLU	TriviaQA
AUROC	$Q{=}R$	$+.024^{*}$ (1.00)	$+.006^{*}$ (.940)	$+.023^{*}$ (1.00)
	$R{=}V$	$+.028^{*}$ (1.00)	$+.005^{*}$ (1.00)	$+.018^{*}$ (1.00)
	$Q{=}V$	$+.001$ (.675)	$+.002$ (.945)	$+.001$ (.720)
	$Q{=}R{=}V$	$-.027^{*}$ (1.00)	$-.001$ (.755)	$-.016^{*}$ (1.00)
ECE 15	$Q{=}R$	$+.014^{*}$ (.960)	$-.005$ (1.00)	$+.016^{*}$ (.990)
	$R{=}V$	$-.002$ (.625)	$-.004$ (1.00)	$+.015^{*}$ (.985)

Table 4: Mixed-model matching coefficients. ∗ Wald 95% interval excludes zero. Parentheses: fraction of 200 model-cluster resamples with the same sign. Positive ECE coefficients mean worse calibration.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: LSQA AUROC by question language (rows) and requested response language (columns) for four models fixed before the analysis (the smallest and largest Qwen2.5 models, Llama-3.1-8B and Aya Expanse 8B), with English verifier instructions, pooled across content subsets. The supplementary data give the cells of all ten models.

Subject group	$Q,R$ =EN	$Q$ =EN	$R$ =EN	Neither
Humanities	.581	.562	.463	.442
STEM	.581	.569	.505	.482
Social sciences	.708	.684	.561	.538
Other	.668	.653	.560	.539

Appendix

Table 5: MMMLU exact-match accuracy by subject group and English position.

Q

=EN: English question with a non-English requested response;

R

=EN: the converse.

Model	Exact match	Judge
Qwen2.5-3B	.484	.551
Qwen2.5-7B	.565	.729
Qwen2.5-14B	.653	.838
Qwen3-8B	.562	.741
Llama-3.1-8B	.523	.640
Llama-3.2-3B	.449	.540

Appendix

Table 6: MMMLU accuracy over all 34,560 responses per model. The judge column is the sensitivity analysis, not the primary label.

	Compl.	Non.	Mixed	Indet.
Qwen2.5-3B	.620	.157	.074	.130
Qwen2.5-7B	.704	.093	.037	.157
Qwen2.5-14B	.639	.139	.065	.139
Llama-3.1-8B	.491	.306	.074	.120
Llama-3.2-3B	.435	.370	.056	.111
Gemma-2-9B	.648	.037	.120	.194

Appendix

Table 7: Human-rated realized response language on the 972-response sample without Qwen3-8B (108 per model, 162 per requested language). Rows omit the empty/refusal share (1.0% overall).

Figure 5: Model ranks by benchmark along the path of Figure 1 , averaged over the five target languages.

		Reversed pairs
Metric	$\tau$	$Q$	$R$	$V$	Ends	$V$ only
Accuracy	.51 [.42, .60]	8	5	0	6	0
AUROC	.73 [.56, .87]	1	0	0	0	0
ECE 15	.87 [.64, .91]	5	0	1	2	6
Brier	.73 [.51, .78]	4	0	3	3	0

Appendix

Table 8: Rank changes along the path of Figure 1 (means over the three benchmarks).

\tau

: Kendall’s

\tau

between the all-English and fully localized rankings, with 95% bootstrap interval. The other columns count reversed model pairs (of 45) at the

Q

R

and

V

steps, between the two ends of the path, and when only

V

is switched.

Phenomenon	Question and response (excerpts)	Labels and $P(\mathrm{True})$ by verifier language
Judge over-credits an MMMLU answer (Aya Expanse 8B; $Q$ =FR, $R$ =EN)	Q: Lequel des énoncés suivants sur le clivage embryonnaire n’est PAS vrai ? (gold: C) R: La réponse incorrecte est A) La présence de jaune inhibe le clivage. […]	Judge score 5 (correct). Reviewer: option A, incorrect; answered in French. Exact match: incorrect. $P(\mathrm{True})$ : FR 0.002, ZH 0.68, other $V$ $\geq$ 0.99.
Correct but not in the requested language (Gemma-2-9B, TriviaQA; $Q$ =EN, $R$ =KO)	Q: Cobweb is a fairy in which Shakespeare play? R: A Midsummer Night’s Dream	Judge score 4 (correct). Reviewer: correct; answered in English. $P(\mathrm{True})\geq$ 0.999 for all six $V$ .
Verifier-language spread (Qwen2.5-7B, LSQA; $Q$ =EN, $R$ =FR)	Q: When did the Great Fire of London occur? R: Le Grand Incendie de Londres s’est produit du 2 au 6 septembre 1666.	Judge score 5 (correct). Reviewer: correct; compliant. $P(\mathrm{True})$ : EN 1.00, FR 1.00, KO 0.18, JA 0.03, ZH 0.0001, HI $3\times 10^{-8}$ .

Appendix

Table 9: Human-validated examples. For each of the first two phenomena we took the first qualifying unit, in random review-ID order, among reviewed responses (Qwen3-8B excluded) whose question and response are in Latin script; the last row is the unit with the widest

P(\mathrm{True})

range across verifier languages among the 109 such units whose reviewer and final labels agree. Examples illustrate phenomena and do not estimate their prevalence; Markdown is omitted.

Language Model Self-Assessment

Language model self-assessment studies how models judge their own confidence, capabilities, errors, and behavior.

See all 93 papers

LLM Evaluators are Biased across Languages
Jul 16, 2026Ej Zhou, Lucas Resck, Zheng Hui +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Language Technology Lab, University of Cambridge
Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models
Oct 19, 2026Debojeet Ghosh, Shreyansh Kumar, Ayushman GargConfidence Estimation in Language Models Language Model Calibration
When Should a Language Model Trust Itself? Same-Model Self-Verification as a Conditional Confidence Signal
Apr 8, 2026Aditya Ajay PhalodLanguage Model Self-Verification Language Model Self-Assessment
Independent Researcher

Jul 16, 2026J/K move · Enter open · S save

LLM Evaluators are Biased across Languages

Ej Zhou, Lucas Resck, Zheng Hui, Anna Korhonen

Language Technology Lab, University of Cambridge

Multilingual Language Model Evaluation Language Model Safety Evaluation Language Model Bias Evaluation Linguistic Bias in Language Models Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

LLM evaluators (trained reward models and prompted LLM-as-a-Judge) are routinely validated via pairwise accuracy. In a multilingual setting, this operates under the premise that high pairwise accuracy implies reliable, language-neutral scoring. We show that this assumption does not hold. We conduct experiments with semantically identical instruction-response pairs across 23 languages, and find that multilingual evaluators assign significantly different scores to different evaluation languages. The bias is statistically significant and consistent across eight open-weight evaluators of different architectures and training paradigms, persists in frontier judges, and is strongly correlated with language resource level: lower-resource languages are scored more generously. Meanwhile, these biases are invisible to pairwise accuracy: evaluators achieve above 90% pairwise accuracy, yet have up to 43% difference in acceptance rate across languages under a global decision threshold, meaning, for instance, that harmful content in lower-resource languages is more likely to pass safety filters. Per-language thresholds would require language identification, which can be defeated by code-switched prompts. We then investigate why lower-resource languages receive higher rather than lower scores, and we find that model uncertainty is linked with the effect: models tend to give higher scores when less confident, both under negative log-likelihood and under token-free uncertainty measures; however, language identity remains a significant predictor after controlling for uncertainty, and the bias cannot be explained away by content difficulty alone, but is a structural, language-level misalignment.

Abstract

Language Model Self-Assessment

Explore similar work

LLM Evaluators are Biased across Languages

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

When Should a Language Model Trust Itself? Same-Model Self-Verification as a Conditional Confidence Signal