Fair Fact-Checking: Closing the Cross-Lingual Gap in LLM Factual Judgement with RoSh

Muhammad Ahmad; Fatemeh Seyedin; Adrian Weller; Dongwon Lee; Mahmoudreza Babaei

cs.CLSep 28, 2026

Fair Fact-Checking: Closing the Cross-Lingual Gap in LLM Factual Judgement with RoSh

Authors: Muhammad Ahmad, Fatemeh Seyedin, Adrian Weller, Dongwon Lee, Mahmoudreza Babaei

Organizations: Department of Computer Science, Shifa Tameer-e-Millat University, Islamabad, Pakistan · BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany · Max Planck Institute for Security and Privacy, Bochum, Germany · University of Cambridge, United Kingdom · The Pennsylvania State University, University Park, PA, USA · GISMA University of Applied Sciences, Potsdam, Germany

Abstract

Misinformation on social media remains a critical problem, and more and more people settle it by asking a language model instead of a fact checker. Whether models judge such claims reliably is debated; whether they judge them equally well in every language people ask in has gone almost unasked. We test eight models from five families, 3B to 70B, on 1,500 encyclopedic factual claims that exist in identical form in eight languages. English is judged better than every other language on every model, and the gap is widest on the smallest ones, where Llama-3B on Arabic is no better than guessing. Existing remedies retrain on more multilingual data or fit an unconstrained map between language representations, and neither asks whether the model already holds the answer and simply fails to say it. It largely does: a linear probe recovers the truth from the very activations the model fails to express. We propose RoSh, a per-language shift and rotation of the residual stream, computed in closed form at three layers, with no training and no weight modified. It improves every model and closes 75% of the gap on average, helping most where the model was worst: Arabic on Llama-3B goes from chance to nearly the English level, and a fifth fewer of the claims answered correctly in English are lost in translation. What remains is no longer a read-out failure: afterwards the head recovers as much of what is encoded outside English as it does in English. An unconstrained map fitted on the same pairs falls below the untouched baseline, so the orthogonality constraint is doing the work, and every model clears a scrambled-correspondence control and ten further controls. On the two benchmarks of the closest inference-time method, latent-space intervention, run with its own data and metric code, RoSh's gains are five to thirteen times larger.

Figures & tables

Model		EN	PT	PL	DE	AR	ZH	RU	TR
Llama-3B	probe	0.922	0.908	0.906	0.908	0.871	0.875	0.888	0.884
	read-out	0.857	0.787	0.794	0.851	0.523	0.772	0.787	0.669
Gemma-9B	probe	0.935	0.928	0.936	0.937	0.912	0.899	0.927	0.920
	read-out	0.878	0.859	0.862	0.862	0.844	0.762	0.857	0.820

Table 1: The truth is encoded even where the verdict fails: linear probe against the model’s own read-out, held out, at the first selected layer (for English, the layer most often selected first). Read-out values are those of Table 3 .

Figure 1: Language-conditioned correction at inference. The language of the incoming claim selects a recipe: the layers to correct and the rotation $\mathbf{R}_{\ell}$ and shift to apply at each. Shown is Llama-3B with the Portuguese recipe. Only residual-stream activations at the listed layers are transformed; the model’s weights are shared across all languages and never modified. The inset is a schematic of what the correction does at one layer, not a measurement.

Figure 2: What the read-out head sees. Each dot is one claim projected onto the read-out direction, true above the centre line and false below. English (top) separates cleanly; non-English before correction (middle) overlaps almost completely; after RoSh (bottom) the separation matches English, without retraining. Same synthetic geometry as the inset of Figure 1 .

Model	EN	before	after	before	after	closed	accuracy	consistency
		non-EN		gap		gap		directional
Llama-3B	0.857	0.740	0.829	$+0.116$	$+0.028$	76%	$0.671\to 0.736$	$0.750\to 0.841$
Mistral-7B	0.834	0.704	0.809	$+0.130$	$+0.025$	80%	$0.649\to 0.726$	$0.715\to 0.806$
Gemma-9B	0.878	0.838	0.865	$+0.040$	$+0.012$	69%	$0.740\to 0.759$	$0.879\to 0.905$
Mistral-24B	0.888	0.866	0.878	$+0.022$	$+0.011$	53%	$0.756\to 0.767$	$0.883\to 0.898$
Gemma-27B	0.860	0.833	0.844	$+0.027$	$+0.016$	42%	$0.729\to 0.746$	$0.839\to 0.851$

Table 2: Non-English mean, held out. Gap is English minus non-English, and gap closed is

(\mathrm{gap}_{\text{before}}-\mathrm{gap}_{\text{after}})/\mathrm{gap}_{\text{before}}

(Appendix A ). Directional consistency is the share of claims the model gets right in English that it also gets right in the other language. The Mean row averages the eight model rows; its gap closed is the mean of the per-model values, with values above 100% counted as 100%.

Model	PT	PL	DE	AR	ZH	RU	TR
Llama-3B	$0.787\to 0.840$	$0.794\to 0.829$	$0.851\to 0.856$	$0.523\to 0.841$	$0.772\to 0.798$	$0.787\to 0.838$	$0.669\to 0.798$
Mistral-7B	$0.646\to 0.896$	$0.802\to 0.878$	$0.807\to 0.847$	$0.585\to 0.674$	$0.659\to 0.702$	$0.791\to 0.869$	$0.639\to 0.795$
Gemma-9B	$0.859\to 0.872$	$0.862\to 0.884$	$0.862\to 0.871$	$0.844\to 0.853$	$0.762\to 0.845$	$0.857\to 0.868$	$0.820\to 0.865$
Mistral-24B	$0.874\to 0.874$	$0.875\to 0.890$	$0.893\to 0.897$	$0.859\to 0.892$	$0.824\to 0.828$	$0.889\to 0.902$	$0.846\to 0.860$
Gemma-27B	$0.856\to 0.855$	$0.840\to 0.860$	$0.847\to 0.863$	$0.826\to 0.841$	$0.790\to 0.787$	$0.830\to 0.863$	$0.842\to 0.841$
Qwen-32B	$0.879\to 0.882$	$0.888\to 0.898$	$0.891\to 0.900$	$0.877\to 0.893$	$0.862\to 0.875$	$0.892\to 0.908$	$0.866\to 0.868$

Table 3: Baseline and corrected AUROC per language, held out. Shading shows the change: green for a gain, darker the larger it is; red for a drop; grey for no change.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Checkpoint	depth $L$	width $d$
Llama-3B	unsloth/Llama-3.2-3B-Instruct	28	$3{,}072$
Mistral-7B	unsloth/mistral-7b-instruct-v0.3	32	$4{,}096$
Gemma-9B	unsloth/gemma-2-9b-it	42	$3{,}584$
Mistral-24B	unsloth/Mistral-Small-24B-Instruct-2501	40	$5{,}120$
Gemma-27B	unsloth/gemma-2-27b-it	46	$4{,}608$
Qwen-32B	Qwen/Qwen2.5-32B-Instruct	64	$5{,}120$

Appendix

Table 4: Models evaluated. Depth is the number of transformer blocks and

d

the residual width; both determine the size of the layer sweep and of the fitted transform.

Language	Message
English	Is the following statement true or false? Answer with only the word TRUE or FALSE. Statement: {claim}
Portuguese	A seguinte afirmação é verdadeira ou falsa? Responda apenas com a palavra TRUE ou FALSE. Afirmação: {claim}
Polish	Czy poniższe stwierdzenie jest prawdziwe czy fałszywe? Odpowiedz tylko słowem TRUE albo FALSE. Stwierdzenie: {claim}
German	Ist die folgende Aussage wahr oder falsch? Antworte nur mit dem Wort TRUE oder FALSE. Aussage: {claim}
Turkish	Aşağıdaki ifade doğru mu yanlış mı? Sadece TRUE veya FALSE kelimesiyle cevap ver. İfade: {claim}

Appendix

Table 5: The instruction in each Latin-script language, verbatim. The Arabic, Chinese and Russian versions are equivalent translations and are given verbatim in the code release ( INSTR in run_lookup.py ). In every language the answer words stay true and false .

Language	true	false
English	Colorado shares border with Wyoming.	Croatia shares border with Bulgaria.
Portuguese	Colorado compartilha fronteira com Wyoming.	Croácia compartilha fronteira com Bulgária.
Polish	Kolorado graniczy z Wyoming.	Chorwacja graniczy z Bułgaria.
German	Colorado teilt die Grenze mit Wyoming.	Kroatien teilt die Grenze mit Bulgarien.
Turkish	Colorado, Wyoming ile sınır paylaşıyor.	Hırvatistan, Bulgaristan ile sınır paylaşıyor.

Appendix

Table 6: Two claims of the dataset in the five Latin-script languages.

Template	true	false	total
$S$ and $O$ are twin cities.	290	283	573
$S$ shares border with $O$ .	163	145	308
The official language of $S$ is $O$ .	100	117	217
$S$ is the capital of $O$ .	55	62	117
$S$ is located in $O$ .	51	50	101
The capital of $S$ is $O$ .	43	33	76

Appendix

Table 7: Relation templates in the claim set, with the number of claims (

S

subject,

O

object).

Model	$r$	$\det\mathbf{R}=-1$	$\lVert\mathbf{R}-\mathbf{I}\rVert_{F}$	$\sqrt{2d}$	in-span share	non-EN AUROC
Llama-3B	737	12/21 / 0/21	76.5 / 38.7	78.4	99.1%	0.8274 / 0.8277
Gemma-9B	745	5/21 / 0/21	82.1 / 36.6	84.7	98.8%	0.8657 / 0.8649

Appendix

Table 8: Diagnostics for the fitted transform over the 21 selected layers (7 languages

\times

3 layers).

r

is the mean numerical rank of

\mathbf{M}

(singular values above

10^{-6}\sigma_{\max}

);

\det\mathbf{R}=-1

counts reflections; where two values are given they are for the implementation-default / canonical solution.

\sqrt{2d}

is the value

\lVert\mathbf{R}-\mathbf{I}\rVert_{F}

takes for a Haar-random rotation, shown for reference.

Llama-3B ( $d=3{,}072$ , $\sqrt{2d}=78.4$ )
Language	Layer	rank $(\mathbf{M})$	$\sigma_{\max}$	$\det\mathbf{R}$	$\lVert\mathbf{R}-\mathbf{I}\rVert_{F}$	$\lVert\bm{\mu}^{\mathrm{en}}-\bm{\mu}^{\ell}\rVert$	in-span
PT	14	738	$1.4{\times}10^{3}$	$-1$ / $+1$	77.2 / 41.1	6.02	99.3%
PT	20	717	$3.6{\times}10^{3}$	$-1$ / $+1$	75.3 / 34.0	4.15	99.4%
PT	24	721	$1.4{\times}10^{4}$	$-1$ / $+1$	73.9 / 30.4	6.14	98.9%
PL	14	741	$1.7{\times}10^{3}$	$-1$ / $+1$	77.5 / 41.7	5.99	99.4%
PL	22	736	$5.5{\times}10^{3}$	$+1$ / $+1$	75.2 / 35.8	7.12	99.1%

Appendix

Table 9: Per-layer diagnostics for the two refitted models.

\det\mathbf{R}

and

\lVert\mathbf{R}-\mathbf{I}\rVert_{F}

are given for the implementation-default / canonical solution; in-span is the share of a centred held-out activation’s squared norm inside the span of the fitting activations, averaged over held-out claims.

Model	read-out	probe	gap	baseline	RoSh	probe	before	after	closed
	English			non-English			non-EN gap
Llama-3B	0.857	0.923	$+0.066$	0.740	0.829	0.893	$+0.153$	$+0.064$	58%
Mistral-7B	0.834	0.927	$+0.093$	0.704	0.809	0.857	$+0.153$	$+0.048$	69%
Gemma-9B	0.878	0.941	$+0.063$	0.838	0.865	0.928	$+0.090$	$+0.063$	30%
Mistral-24B	0.888	0.939	$+0.051$	0.866	0.878	0.931	$+0.065$	$+0.053$	18%
Gemma-27B	0.860	0.928	$+0.068$	0.833	0.844	0.911	$+0.078$	$+0.067$	14%

Appendix

Table 10: Read-out against a linear probe, held out, for all eight models. Read-out and RoSh are from Table 2 . The probe is an L2-regularised logistic regression on the fit half, scored on the held-out half, at the best candidate layer; the non-English probe averages the seven languages. Gap is probe minus read-out. Closed is

(\textsc{RoSh}-\text{baseline})/\text{gap before}

; the Mean row’s share is computed from the mean gap.

Benchmark	Model	base	LSI	$\Delta$	base	RoSh	$\Delta$	base	LSI	$\Delta$	base	RoSh	$\Delta$
		accuracy						agreement with English
		LSI (reported)			RoSh (ours)			LSI (reported)			RoSh (ours)
KLAR	Qwen3-8B	87.51	88.25	$+0.74$	85.53	90.17	$+4.64$	86.64	87.38	$+0.74$	86.78	91.57	$+4.79$
KLAR	Llama-3.1-8B	78.21	79.05	$+0.84$	74.06	80.21	$+6.15$	76.31	78.29	$+1.98$	75.40	81.47	$+6.07$
KLAR	Aya-8B	85.59	85.32	$-0.27$	83.55	89.64	$+6.09$	85.09	85.43	$+0.34$	84.79	90.36	$+5.57$
KLAR	mean			$+0.44$			$+5.63$			$+1.02$			$+5.48$

Appendix

Table 11: External benchmarks, in %. The LSI columns are exactly as reported by Ghorbanpour et al. (2026) (their Tables 1, 4 and 5): their baseline and their variant with the highest agreement with English, ties broken by accuracy (Qwen3-8B mean-shift, Llama-3.1-8B AE + mean-shift, Aya-8B AE on KLAR and AE + PCA on mParaRel); on mParaRel they report Aya only, so we do the same. The RoSh columns are our runs, with accuracy the non-English mean over Arabic, Dutch, Russian and Chinese. The two methods are evaluated on different splits, so the gains

\Delta

are the comparable quantity.

MT system	$N$	non-EN baseline	non-EN RoSh	$\Delta$ [95% CI]
Google	90	94.7	95.0	$+0.28$ [ $-0.56$ , $+1.11$ ]
Bing	279	86.0	88.5	$+2.51$ [ $+1.16$ , $+3.85$ ]
OPUS-MT	180	92.1	91.9	$-0.14$ [ $-1.11$ , $+0.83$ ]
mBART50 + m2m100	59	72.5	74.6	$+2.12$ [ $+0.00$ , $+4.24$ ]
All	608	87.8	89.1	$+1.36$ [ $+0.66$ , $+2.10$ ]

Appendix

Table 12: Correction by translation system on mParaRel (Qwen3-8B, held out), restricted to claims answered correctly in English. Non-English accuracy in %;

\Delta

in points with a

95\%

bootstrap interval.

Figure 3: Building the lookup table. From 1,500 parallel facts ( $\times\,8$ languages), we reserve half for fitting and half for held-out evaluation (scored once). On the fit split, for each language we: (1) capture residual-stream activations at every second or third layer between $15\%$ and $97\%$ of depth, reducing the candidate space while retaining coverage across network depth, (2) rank candidate layers by AUROC on an inner fold of the fit split, and (3) greedily select up to $D{=}3$ layers, fitting the orthogonal Procrustes rotation $\mathbf{R}$ at each step. This shallow forward search avoids the combinatorial cost of exhaustively evaluating layer combinations while allowing multiple interventions when they provide complementary gains. Eleven controls (Section 3.5 ), among them scrambled pairing, random rotation, random shift and random shift + rotation, must each score below the full method, confirming the result is not an artefact. The held-out half is scored exactly once to report final accuracy and agreement.

Model	depth	PT	PL	DE	AR	ZH	RU	TR
Llama-3B	28	14, 20, 24	14, 22, 26	20, 6, 22	16, 20, 14	20, 22, 12	14, 20, 12	14, 12, 6
Mistral-7B	32	18, 16, 12	18, 30, 28	18, 30, 28	10, 8, 6	16, 12, 14	18, 28, 30	18, 10, 26
Gemma-9B	42	28, 22, 26	26, 24, 22	24, 22, 28	22, 26, 24	38, 22, 36	24, 22, 26	28, 26, 24
Mistral-24B	40	24, 26, 28	36, 18, 14	20, 24, 34	24, 20, 6	18, 6, 8	26, 24, 14	26, 14, 12
Gemma-27B	46	42, 24, 30	22, 14, 32	22, 30, 18	24, 20, 34	38, 14, 36	24, 22, 30	16, 8, 32
Qwen-32B	64	48, 54, 12	51, 54, 57	54, 48, 45	51, 48, 15	42, 36, 57	51, 54, 45	48, 57, 54

Appendix

Table 13: The layer stack chosen for each language, and the model’s layer count.

Model	fitted	scrambled pairing	beats	$z$
Llama-3B	0.821	0.583	100%	$+1.83$
Mistral-7B	0.779	0.520	100%	$+8.13$
Gemma-9B	0.857	0.730	100%	$+2.05$
Mistral-24B	0.877	0.640	100%	$+3.21$
Gemma-27B	0.839	0.723	100%	$+1.87$
Qwen-32B	0.886	0.644	100%	$+3.88$

Appendix

Table 14: Scrambled-correspondence control at depth 1, where the control mirrors the fitting procedure exactly. Beats is the share of the three scrambled permutations that the fitted map outperforms;

z

is computed over those three.

Arm	What it tests	AUROC	vs method	success
baseline	nothing is applied	0.820	$-0.033$	yes
shift only	whether moving the cloud is enough, without turning it	0.834	$-0.019$	yes
rotation only	whether turning it is enough, without moving it	0.844	$-0.008$	yes
unconstrained linear map	whether the orthogonality constraint earns its place	0.786	$-0.067$	yes
scrambled pairing	whether the fact-to-fact correspondence matters, or only the two clouds	0.621	$-0.232$	yes
random rotation	whether any rotation would do as well as the fitted one	0.658	$-0.194$	yes

Appendix

Table 15: Every arm, averaged over 8 models and seven languages, held out.

Explore similar work

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection
Sep 8, 2026Sanghyeok Park, Minji Kang, Hosung Kwak +1Cross-Lingual Consistency Distortion
Soongsil University, Seoul, Republic of Korea · KAIST, Daejeon, Republic of Korea
Multilingual Fact-Checking at Scale: Fine-Tuned Compact Models vs LLMs
Jun 7, 2026Pratuat Amatya, Vinay SettyBert-Based Models
Factiverse AS and University of Stavanger Stavanger, Norway
Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies
May 28, 2026Yuxuan Ye, Raul Santos-Rodriguez, Edwin SimpsonLLM Reasoning Strategies Facts
Intelligent Systems Laboratory University of Bristol United Kingdom

Sep 8, 2026, cs.CLJ/K move · Enter open · S save

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

Sanghyeok Park, Minji Kang, Hosung Kwak, Jinhyuk Yun

Soongsil University, Seoul, Republic of Korea · KAIST, Daejeon, Republic of Korea

Cross-Lingual Consistency Distortion

Open paper AskPDF arXiv

Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating genuine factual understanding. We introduce Systematic Wikidata-based Object-Relation Distortion (SWORD), a benchmark that evaluates whether models consistently reject factual errors across languages. SWORD generates syntactically well-formed but factually incorrect statements in eight widely spoken languages through controlled perturbations of Wikidata triples, ranging from random entity substitutions to semantically plausible property-based selections. Our distortion-based evaluation surfaces two critical insights that remain entirely obscured by conventional benchmarks. First, models counterintuitively achieve higher accuracy on semantically plausible distortions than on nonsensical random substitutions, suggesting reliance on distributional familiarity rather than genuine factual verification. Second, models exhibiting comparable baseline accuracy across languages show substantial performance degradation specifically on (East) Asian languages when presented with distorted statements, with cross-lingual performance gaps reaching up to 28 percentage points (49% relative reduction) in some models. These findings demonstrate that multilingual factual reasoning involves asymmetric capabilities that aggregate accuracy metrics systematically obscure.

Abstract

Explore similar work

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

Multilingual Fact-Checking at Scale: Fine-Tuned Compact Models vs LLMs

Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies