Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models

Athira Gopal; Patil Prathamesh Pravin; Ankush Kadu; Arpit Jain; Ashwanth Krishnan

cs.CLOct 22, 2026

Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models

Authors: Athira Gopal, Patil Prathamesh Pravin, Ankush Kadu, Arpit Jain, Ashwanth Krishnan

Organizations: QpiAI India

Abstract

Multilingual language models can give different identity judgments when the same people's names are written in different scripts. We study this effect using 601 audited identity pairs with attested Devanagari and Latin forms, holding the question and relation fixed. Qwen2.5-7B's same-person accuracy falls from 35.1% in Latin to 8.5% in Devanagari under Marathi prompts, while different-person accuracy remains above 98%. Evaluation across five models reveals distinct response biases, and reduced tokenization fragmentation does not reliably close the gap. Replicating the design on 374 audited Malayalam pairs built from the same entities reproduces the effect in a second script and language family: same-person accuracy falls to 5.3%, about half the matched Devanagari rate, while the margin still ranks matching above non-matching names at 0.925 AUROC. Latin-trained probes recover label information from Devanagari states, exceeding a string-feature baseline under similarity-matched negatives. Scalar calibration recovers 87% of the raw model-probe balanced-accuracy gap, identifying decision criterion placement as a major contributor. Activation patching localises causal sensitivity to the second entity in Qwen's middle layers; one shared sensitivity direction transfers about half the full-state margin correction. A fact-independent intervention at this site raises held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy. Together, these results distinguish script-dependent discrimination from criterion placement and identify a causal site for improving identity judgments.

Figures & tables

Figure 1: Controlled identity verification. Both names and the English question are supplied; only the attested script-associated name forms change. Qwen and Llama reject this illustrative Dev–Dev pair and accept its Lat–Lat counterpart.

		Dev–Dev		Dev–Lat		Lat–Dev		Lat–Lat
Model	Prompt	pos.	neg.	pos.	neg.	pos.	neg.	pos.	neg.
Qwen2.5-7B	Marathi	8.5	99.8	29.4	99.1	35.0	99.8	35.1	99.7
	Hindi	16.4	99.9	45.2	98.6	52.2	99.5	41.1	99.6
	English	28.2	100.0	61.3	98.9	66.0	98.7	56.3	99.6
Gemma-2-2b	Marathi	91.8	25.2	96.7	27.6	92.2	29.0	96.6	34.1
	Hindi	94.1	44.6	97.5	38.4	91.0	52.2	97.2	49.9

Table 1: Accuracy (%) over 601 items, three paraphrases and both answer keys. Pos. and neg. denote same-person and different-person pairs. English rows use the Marathi corpus; the 24B models were evaluated on that corpus only. Textual gaps use unrounded values.

	Malayalam ( $n=374$ )			Devanagari ( $n=374$ )
Name scripts	pos.	neg.	AUROC	pos.	neg.	AUROC
Both native	5.3	100.0	0.925	10.4	100.0	0.935
Native, Latin	33.5	98.6	0.941	33.2	98.8	0.960
Latin, native	38.1	99.6	0.964	37.8	99.8	0.970
Both Latin	48.8	99.9	0.990	41.0	99.6	0.981
Gap, points	$+43.5\ [40.0,47.1]$			$+30.6\ [26.6,34.5]$

Table 2: Qwen2.5-7B on 374 Malayalam items and the same 374 entities in Devanagari, each corpus queried in its own language. Script configurations are Mlym–Mlym, Mlym–Lat, Lat–Mlym and Lat–Lat for Malayalam, and Dev–Dev, Dev–Lat, Lat–Dev and Lat–Lat for Devanagari. Pos. and neg. are same-person and different-person accuracy; AUROC is entity-averaged over six prompt margins. The gap rows are Lat–Lat minus the both-native condition, paired by entity, with 95% item-clustered intervals over 2,000 resamples. The two accuracy gaps differ by 12.9 points, 5.1 of which come from the both-native cell and 7.8 from the differing Lat–Lat baselines. The Devanagari column is a new run on this 374-item subset with its own distractor map; it is not the 601-item corpus of Table 1 and the two are not interchangeable. English-prompt results and the held-out split appear in Table 66 .

Prompt	Negative map	Probe	String baseline	Difference
Marathi	Original	0.925	0.933	$-0.008$ [ $-0.033$ , $+0.017$ ]
	Matched	0.908	0.796	$+0.112$ [ $+0.067$ , $+0.163$ ]
English	Original	0.950	0.933	$+0.017$ [ $-0.004$ , $+0.042$ ]
	Matched	0.879	0.796	$+0.083$ [ $+0.025$ , $+0.142$ ]

Table 3: Question-mark probe and string baseline on the same 120 held-out entities under the original and partially similarity-matched negative maps. Values are balanced accuracy; paired differences include 95% item-clustered intervals. The matched map permits tight matching on 25 test items. Full controls appear in Appendix T .

	Marathi			English
Rule	BA	TPR	FPR	BA	TPR	FPR
Raw model	0.525	0.058	0.008	0.633	0.267	0.000
Margin + constant	0.879	0.808	0.050	0.904	0.875	0.067
E2 probe	0.933	0.883	0.017	0.954	0.942	0.033
String features	0.933	0.883	0.017	0.933	0.883	0.017

Table 4: Entity-level decisions on 120 held-out Dev–Dev items, averaging six prompt margins before classification. BA is balanced accuracy; TPR/FPR are true-/false-positive rates. All settings are validation-selected. The original negative map is used here; similarity-matched results appear in Table 52 .

Site / contrast	Marathi ( $n=12$ )	Hindi ( $n=15$ )	English ( $n=24$ )
First entity (E1)	0.12 [ $-0.13$ , 0.38]	$-0.03$ [ $-0.25$ , 0.16]	0.27 [0.02, 0.53]
Second entity (E2)	2.66 [1.67, 3.71]	3.16 [2.47, 3.90]	2.85 [2.28, 3.45]
Question mark	0.61 [0.33, 0.97]	0.33 [0.16, 0.52]	0.79 [0.59, 1.00]
Readout	1.26 [0.96, 1.58]	1.08 [0.87, 1.30]	1.33 [1.12, 1.57]
E2: correct $-$ wrong entity	2.09 [0.41, 3.84]	$-0.51$ [ $-2.06$ , 1.08]	0.70 [ $-0.21$ , 1.54]

Table 5: Held-out Qwen patching: mean margin change with 95% item-clustered intervals. Site effects average layers 10–18 and three Latin-bearing donors. The last row compares correct and wrong-entity Lat–Lat donors on matched layers and items. Full profiles appear in Appendix I .

	Marathi ( $n=12$ )			English ( $n=24$ )
Direction	repair	signed	$\|\Delta\|$	repair	signed	$\|\Delta\|$
Whole patch	2.54	0.48	1.17	4.90	1.03	1.98
Local sensitivity	1.30	0.23	0.47	2.65	0.79	1.01
Mean difference	3.50	1.81	1.82	4.33	2.67	2.67
Donor PCA	0.09	$-0.05$	0.16	0.43	0.19	0.22
Probe	$-0.01$	0.00	0.13	$-0.01$	0.02	0.08

Table 6: Rank-one donor-projected patching at E2, layer 16, held-out fold t01e2. Repair is the gated margin change; signed and

|\Delta|

measure changes on never-correct same-person pairs outside the gate. Negative accuracy is 1.00 throughout. The whole-state patch is the reference.

	Entity-averaged AUROC		TPR (FPR)
Arm	Dev–Dev	change	vector	offset	threshold only
Marathi	0.952	$+0.003$ $[-0.008,+0.015]$	0.335 (0.015)	0.312 (0.012)	0.336 (0.015)
English (mr corpus)	0.961	$-0.003$ $[-0.016,+0.012]$	0.590 (0.024)	0.714 (0.037)	0.644 (0.024)
Hindi	0.935	$-0.016$ $[-0.038,+0.005]$	0.546 (0.131)	0.656 (0.178)	0.594 (0.131)

Table 7: Held-out intervention and calibration. AUROC averages six margins per entity; change includes a 95% item-clustered interval. TPR (FPR) pools prompts. The offset is validation-selected under a negative-accuracy floor; threshold only is an oracle test-curve diagnostic. Appendix Y gives selection rules.

Appendix figures & tables65 assets

Supplementary material from the paper’s appendix.

Appendix

Tier	Items
Alias observed only	404
Explicit relation plus attested alias, birth name	171
Explicit plus alias, pseudonym	13
Explicit plus alias, nickname	7
Explicit relation observed, birth name	5
Explicit relation observed, nickname	1

Appendix

Table 8: Relation tiers, 601 retained items.

Language	#	Template
Marathi	0	[a] ani [b] he ekac vyakti ahet ka? Are [a] and [b] one and the same person?
	1	[a] ani [b] hi nave ekac vyaktila darshavatat ka? Do the names [a] and [b] denote the same person?
	2	[a] he [b] yance dusre nav ahe ka? Is [a] another name of [b]?
Hindi	0	kya [a] aur [b] ek hi vyakti hain? Are [a] and [b] the same person?
	1	kya [a] aur [b] ek hi vyakti ke nam hain? Are [a] and [b] names of the same person?
	2	kya [a] [b] ka dusra nam hai? Is [a] another name for [b]?

Appendix

Table 9: The three question paraphrases per prompt language. Both Marathi and Hindi templates have been verified by native speakers. The paraphrases are matched in function across languages: the same person, names for one person, and whether one is another name for the other.

Model	Prompt	Gate	Run	Test
Qwen2.5-7B	Marathi	53	53	12
	Hindi	50	50	15
	English (Marathi run)	116	116	24
	English (Hindi run)	107	107	24
Gemma-2-2b	Marathi	33	33	10
	Hindi	65	60	12

Appendix

Table 10: Causal sample sizes. “Gate” counts all qualifying items, “run” counts executed items after the per-language cap, and “test” counts unique held-out items used for test estimates. Qwen runs its complete gate in every cell; the Gemma and Llama rows retain the cap of 60 they were produced with. Qwen and Llama are gated on positive pairs; Gemma is gated on negative pairs.

Dev–Dev rate	0.00	0.17	0.33	0.50	0.67	0.83	1.00
Qwen	497	25	21	8	41	3	6
Gemma	0	0	0	1	81	131	388

Appendix

Table 11: Distribution of Dev–Dev positive rate among candidate items. The causal gate requires exactly 0.0.

Figure 2: Qwen accuracy across the four ways of writing the two names. The large change is concentrated on SAME pairs; DIFFERENT accuracy remains near ceiling. Values pool all 601 items, three paraphrases and both answer-key orders.

Prompt	Paraphrase	Dev–Dev	Dev–Lat	Lat–Dev	Lat–Lat
Marathi	P0	10.6	37.7	51.0	43.9
	P1	11.6	46.9	51.2	51.7
	P2	3.2	3.5	2.7	9.8
Hindi	P0	14.8	61.8	73.4	50.0
	P1	28.2	71.6	79.7	59.2
	P2	6.2	2.2	3.5	14.1

Appendix

Table 12: Qwen positive-pair accuracy (%) by paraphrase. Negatives, omitted, remain between 97.2 and 100.0 in every cell.

Prompt	Tier	Dev–Dev	Dev–Lat	Lat–Dev	Lat–Lat	$n$
Marathi	Alias observed	10.4	27.7	36.3	32.5	404
	Explicit plus alias, birth name	4.3	34.8	33.0	43.0	171
	Explicit plus alias, pseudonym	0.0	9.0	16.7	17.9	13
	Explicit plus alias, nickname	14.3	33.3	42.9	45.2	7
	Explicit observed, birth name	13.3	33.3	33.3	16.7	5
	Explicit observed, nickname	0.0	0.0	0.0	0.0	1

Appendix

Table 13: Qwen positive accuracy (%) by relation tier: Marathi prompts on the Marathi corpus in the upper block, Hindi prompts on the Hindi corpus in the lower.

Model	Prompt	Dev–Dev	Lat–Lat	Gap (CI)
Qwen	Marathi	1.494	2.376	$+0.88\ [0.14,1.55]$
	Hindi	2.122	2.425	$+0.30\ [-0.22,0.76]$
	English	2.765	2.833	$+0.07\ [-0.45,0.74]$
Gemma	Marathi	0.724	1.410	$+0.69\ [0.57,0.80]$
	Hindi	1.428	1.910	$+0.48\ [0.33,0.65]$
	English	1.136	1.866	$+0.73\ [0.56,0.91]$

Appendix

Table 14: Lat–Lat minus Dev–Dev

d^{\prime}

gap, 95% bootstrap CI over the 601 items of each corpus. Marathi and English rows are from the Marathi run, Hindi from the Hindi run. Llama’s Marathi and Hindi arms (

\dagger

) are excluded from the analysis as instruction-following failures (Appendix Q ); their intervals exclude zero but they are not among the cells interpreted in § 4.1 .

Corpus	Token count	Names	Mean tokens	Retrieved at rank 1
Marathi	$\leq 10$	217	8.4	41.0%
	11–15	396	13.0	22.0%
	16–20	303	17.8	17.5%
	$\geq 21$	286	26.2	16.8%
Hindi	$\leq 10$	220	8.4	42.7%
	11–15	400	13.0	24.0%

Appendix

Table 15: Isolated-name retrieval by Devanagari token count, all 1,202 names, Qwen at layer 8.

Relationship	$\rho$	$p$	$n$
Marathi
Token ratio vs Latin-script benefit, pooled	$+0.146$	$3.3\times 10^{-4}$	601
foreign-origin names	$+0.168$	$1.9\times 10^{-4}$	489
native-origin names	$+0.091$	$0.34$	112
Token ratio vs layer of first recognition	$+0.220$	$4.6\times 10^{-8}$	604 of 1202
Hindi

Appendix

Table 16: Fragmentation correlations, Spearman

\rho

. The last row of each block uses the first layer at which the correct counterpart reaches rank 1, and is necessarily restricted to names that ever do so. The origin split is a property of the entity, taken from the Marathi corpus and applied to the Hindi items by fact_id ; the two corpora cover the same 601 entities.

Corpus	Quartile	Mean token ratio	Mean benefit
Marathi	Q1, least fragmented	2.12	3.78
	Q2	2.60	3.82
	Q3	3.16	5.74
	Q4, most fragmented	4.49	6.34
Hindi	Q1, least fragmented	2.12	3.29
	Q2	2.60	3.43

Appendix

Table 17: Mean Latin-script benefit in margin points by quartile of fragmentation, Qwen, 601 items. Each corpus is read under its own prompt language.

Feature set	Qwen (mr)	Qwen (en)	Gemma (mr)	Qwen (hi)	Gemma (hi)
Surface features only	0.602	0.564	0.588	0.572	0.561
+ early-layer retrieval	0.590	0.568	0.557	0.574	0.626
+ retrieval across layers	0.694	0.728	0.672	0.722	0.653
+ relation states	0.954	0.962	0.864	0.937	0.915

Appendix

Table 18: Out-of-fold AUC for predicting which items are script-sensitive. The two Hindi columns, right of the rule, are the same measurement on Hindi renderings of the same entities. The columns rest on unequal evidence: the number of script-sensitive items is 201, 180, 63, 199 and 33 respectively, so Gemma’s two columns are the least well supported.

Model	Prompt	Dev–Dev	replaced	anchored	Lat–Lat
Qwen2.5-7B	Marathi	0.537	0.523–0.548	0.535–0.554	0.706
	Hindi	0.546	0.508–0.525	0.550–0.562	0.748
	English	0.667	0.550–0.579	0.671–0.733	0.781
Gemma-2-2b	Marathi	0.754	0.621–0.683	0.675–0.731	0.871
	Hindi	0.829	0.646–0.702	0.681–0.721	0.827
	English	0.529	0.510–0.548	0.533–0.615	0.646

Appendix

Table 19: Romanization on held-out test items, against the Dev–Dev baseline and the Lat–Lat reference. Ranges span the five automatic schemes. Replacing the native name with any of them does not reliably exceed the baseline; anchoring beside it helps but does not reach the reference. English rows are from the Marathi run; the Hindi corpus is a Hindi rendering of the same 601 entities.

Version 1	Version 2	Canonical
imdira gamdhi	imdira gamdhi	Indira Gandhi
jaoNrja iliyata	jorja iliyata	George Eliot
meri eNna ivhansa	meri aena ivhansa	Mary Anne Evans
naremdra modi	naremdra modi	Narendra Modi

Appendix

Table 20: Transliteration outputs against canonical Latin spellings.

Fit on	Evaluate on	Qwen	Gemma	Llama
English	English	0.950 (L18)	0.921 (L11)	0.942 (L11)
English	Marathi	0.929 (L15)	0.883 (L10)	0.954 (L9)
Marathi	English	0.942 (L21)	0.838 (L20)	0.925 (L21)
Marathi	Marathi	0.925 (L16)	0.925 (L14)	0.950 (L19)

Appendix

Table 21: Held-out balanced accuracy, 120 test items. The layer is chosen on the validation split alone and the accuracy reported on the test split at that layer, so no test item participates in the choice. The selected layer is given in parentheses.

		Qwen (28 layers)			Gemma (26 layers)
Prompt	Layer	Bal.	S	D	Bal.	S	D
Marathi	0	0.525	0.411	0.639	0.491	0.691	0.291
	4	0.832	0.915	0.749	0.665	0.927	0.403
	8	0.902	0.820	0.983	0.913	0.857	0.970
	12	0.871	0.742	1.000	0.910	0.834	0.987
	16	0.913	0.832	0.993	0.901	0.807	0.995

Appendix

Table 22: Cross-script probe by layer, Marathi prompts, question-mark position. Bal. is balanced accuracy, S and D are recall on same-person and different-person items.

Figure 3: Qwen contextual alignment at E2 under Marathi prompts. Robust and script-sensitive items are close at layer 3 and separate by layer 8. Because E2 contains the preceding name, this is a fact-context alignment measure rather than an isolated-name identity measure.

Failure recall by behavioural group, Marathi, question-mark position
	Marathi		Hindi
	Question mark	Readout	Question mark	Readout
Balanced accuracy, held-out test	0.925	0.892	0.942	0.925
Selected layer	16	6	8	17
Same recall on failures	0.789	0.570	0.829	0.927
Robust across scripts	0.944

Appendix

Table 23: Probe by position and by behavioural group, Qwen, Marathi and Hindi prompts. Position figures are held-out test items at the layer chosen on validation alone; ties in validation are broken toward the lower layer, and two of the four selections are exact ties, Marathi’s question mark with layer 17 and Hindi’s with layer 15. Failure figures are same recall on exact failures, averaged over layers 10–18, and do not depend on a single-layer selection. Group figures are out-of-fold over the same layers and are Marathi only. All are from this run.

		Qwen		Gemma		Llama
Prompt	Switched span	at E2	at Q	at E2	at Q	at E2	at Q
Marathi	first	0.967	0.918	0.859	0.839	0.937	0.928
	second	0.491	0.899	0.394	0.615	0.547	0.842
Hindi	first	0.938	0.708	0.788	0.793	0.932	0.923
	second	0.627	0.899	0.404	0.683	0.616	0.913
English	first	0.906	0.760	0.797	0.687	0.936	0.903

Appendix

Table 24: Top-1 contextual cross-script alignment when the first or second entity span is switched, layer 8. The first-entity position cannot encode a later change to the second entity under causal masking, so that cell is degenerate and is omitted. Because the unchanged name can identify the fact at later positions, these are fact-context rather than isolated-name identity scores. Llama’s Marathi row is reported despite its behavioural arm being excluded. English rows are from the Marathi run, as in Table 1 .

Prompt	E1	E2	Q	R	wrong ent.	wrong pos.	reverse	12–17
	Site				Control			Window
Marathi ( $n_{\rm test}{=}12$ )	0.12	2.66	0.61	1.26	0.70	0.57	$-2.79$	12.95
Hindi ( $n_{\rm test}{=}15$ )	$-0.03$	3.16	0.33	1.08	2.08	0.40	$-1.44$	12.35
English ( $n_{\rm test}{=}24$ )	0.27	2.85	0.79	1.33	0.86	0.37	$-0.72$	11.65

Appendix

Table 25: Mean margin change from patching, held-out Qwen test items; Hindi from the Hindi run. E1 and E2 are the entities, Q the question mark, R the readout, and the window column patches E1, E2 and Q jointly over layers 12–17. Site columns average layers 10–18 and the three Latin-bearing donors, control and reverse columns all of 8–20, so the two are not comparable; Appendix I gives the matched contrast and the gate sizes.

Held-out test items ( $n=12$ , $15$ , $24$ )
Quantity	Marathi	Hindi	English
First entity (E1)	0.12 [ $-$ 0.13, 0.38]	$-$ 0.03 [ $-$ 0.25, 0.16]	0.27 [0.02, 0.53]
Second entity (E2)	2.66 [1.67, 3.71]	3.16 [2.47, 3.90]	2.85 [2.28, 3.45]
Question mark (Q)	0.61 [0.33, 0.97]	0.33 [0.16, 0.52]	0.79 [0.59, 1.00]
Readout (R)	1.26 [0.96, 1.58]	1.08 [0.87, 1.30]	1.33 [1.12, 1.57]
E2 minus wrong-entity donor	2.09 [0.41, 3.84]	$-$ 0.51 [ $-$ 2.06, 1.08]	0.70 [ $-$ 0.21, 1.54]

Appendix

Table 26: Item-clustered intervals for the patching effects of Table 25 , 2000 resamples of entities. The upper block is the held-out split and is the estimate we rely on; the lower adds the train and validation items patched under the same gate, which is better powered but not independent of the layer choice. Appendix I gives the gate sizes, the entity-specificity contrast and why both blocks are reported.

		Native-script prompts				English prompts
Corpus	Layer	E1	E2	Q	R	E1	E2	Q	R
Marathi	8	0.28	0.31	0.42	0.02	0.95	$-0.31$	0.01	0.11
	9	0.14	0.53	0.47	0.05	0.93	$-0.02$	$-0.08$	0.10
	10	0.14	0.59	0.35	0.12	0.78	$-0.25$	0.01	0.20
	11	0.12	0.79	0.38	0.09	0.60	0.52	0.08	0.19
	12	0.17	1.80	0.23	0.08	0.45	1.94	0.16	0.25

Appendix

Table 27: Mean margin change by layer and site, Lat–Lat source, pooled over splits. E1 and E2 are the first and second entity, Q the question mark, R the readout position. Gate sizes are

n{=}53

Marathi and

n{=}116

English for the Marathi corpus,

n{=}50

Hindi and

n{=}107

English for the Hindi corpus.

Patch	Marathi	English
Layers 9–11	0.77 (0.00)	0.04 (0.07)
Layers 12–14	4.04 (0.04)	3.10 (0.22)
Layers 15–17	9.08 (0.30)	6.91 (0.50)
Layers 12–17	11.70 (0.43)	9.05 (0.64)
Layers 18–20	0.56 (0.00)	0.28 (0.00)
E1+E2, 12–17	12.23 (0.51)	9.99 (0.69)

Appendix

Table 28: Multi-layer and joint-site patching at the second entity, Lat–Lat source, pooled over splits. Values are mean margin change with flip rate in parentheses.

	Variant	Marathi	English	Hindi
Donor, same language	Lat–Lat (used throughout)	5.97	4.45	7.22
	Lat–Dev	5.88	6.88	4.61
	Dev–Lat	4.03	5.15	5.66
Donor, other language	Lat–Lat	6.26	4.21	4.30
	Dev–Dev	0.11	0.53	1.09
Negative pairs	margin change, joint 12–17	$-0.76$	$-1.10$	$-2.73$

Appendix

Table 29: Patching variants at E2 in Qwen, held-out test items, peak over layers 10–18. Marathi and English columns are from the Marathi run, Hindi from the Hindi run. Cross-language rows take the donor state from the other prompt language. The final block applies the 12–17 window at E1 and E2 jointly to the matched negative items. The spread across the three same-language donors is 1.94 margin points in Marathi, 2.43 in English and 2.61 in Hindi.

Figure 4: Two stages of the dissociation in Qwen. Left: a probe fitted on Lat–Lat states reads Dev–Dev states far above the model’s own uncalibrated balanced accuracy, a gap Table 4 shows to be 87 percent a calibration difference. Right: patching at E2 becomes effective before the corrective effect moves to the question position. The patching profile is pooled over splits for localisation; held-out effect sizes and sample sizes are reported in Table 25 .

Patched	Marathi	English	Hindi
Full residual stream, single layer	2.16	1.31	1.58
Attention output only, single layer	0.52	0.29	0.41
MLP output only, single layer	0.07	0.48	0.26
Attention output, cumulative	5.41	4.51	4.16
MLP output, cumulative	0.51	1.10	0.56
Best single head	1.70	1.44	1.66

Appendix

Table 30: Component and head patching at E2 in Qwen. Marathi and English columns are from the Marathi run, Hindi from the Hindi run. Single-layer components are averaged over layers 8–20; cumulative components patch the same component at every layer up to and including the current one. Heads are screened on training items over layers 8–20 and confirmed on validation and test items. “Mean of 12” averages the twelve selected heads patched individually.

	E2 patch		E1 patch		Dev–Dev acc.		Lat–Lat acc.
Prompt	canon.	swap.	canon.	swap.	canon.	swap.	canon.	swap.
Marathi	4.79	3.78	-0.11	0.03	0.551	0.572	0.719	0.751
Hindi	5.78	4.45	-0.08	0.41	0.572	0.632	0.746	0.814
English	3.72	3.36	-0.42	0.09	0.675	0.759	0.799	0.856

Appendix

Table 31: Name-order swap in Qwen. Left: causal effect of patching each entity site, canonical against swapped order. Right: behavioural accuracy, canonical against swapped.

Figure 5: Local sensitivity at E2 in Qwen under Marathi prompts. Dev–Dev and Lat–Lat gradient directions overlap but do not coincide (left), and the Dev–Dev output is less sensitive at the same position (right). Values are means over 60 pairs per layer.

Marathi prompts
Layer	deficit	$\cos(g_{\mathrm{dev}},g_{\mathrm{lat}})$	$\cos(g_{\mathrm{dev}},r)$	$\\|g_{\mathrm{dev}}\\|$	$\\|g_{\mathrm{lat}}\\|$	Latin frame	own frame
12	0.33	0.37	+0.001	3.19	4.85	$-$ 0.01	$-$ 0.40
13	0.53	0.39	+0.002	2.87	4.57	$-$ 0.09	$-$ 0.22
14	0.63	0.44	+0.000	2.73	4.47	0.07	0.19
15	0.89	0.49	$-$ 0.002	2.37	4.01	0.34	$-$ 0.18
16	1.32	0.53	+0.001	1.76	3.01	0.58	$-$ 0.36

Appendix

Table 32: Local sensitivity at E2 in Qwen, layers 12 to 17. The deficit is positive and grows with depth when both states are read through the Devanagari run’s direction, and neither grows nor stays positive once each state is read in its own frame.

Figure 6: Held-out Qwen recovery controls, all of them single-layer. Added at one layer a universal additive vector yields a small gain, while supplying familiar Latin names helps substantially (left). Automatic romanizations shorten the strings but remain below the familiar human Latin form (right). The small gain here is a property of the single-layer setting: Appendix N gives the same vector across layers 12–17, where it moves accuracy substantially, and § 7 says what that movement is.

		Native-script prompts				English prompts
Corpus	Layer	$\alpha{=}0.5$	$2.0$	$4.0$	$8.0$	$\alpha{=}0.5$	$2.0$	$4.0$	$8.0$
Marathi	10	0.058	0.058	0.058	0.075	0.333	0.333	0.333	0.350
	12	0.058	0.058	0.058	0.067	0.333	0.333	0.342	0.367
	14	0.058	0.058	0.067	0.075	0.333	0.333	0.333	0.342
	16	0.058	0.058	0.075	0.083	0.333	0.333	0.333	0.333
	18	0.058	0.058	0.058	0.058	0.333	0.333	0.333	0.333

Appendix

Table 33: Validation grid, Qwen, positive-pair accuracy. Negative-pair accuracy is 1.000 in every cell. No floor was imposed during selection (Appendix Y ); the point is that no negative accuracy was given up anywhere in this grid, not that a constraint was satisfied.

Model	Prompt	Layer	$\alpha$	Positives	Negatives
Qwen2.5-7B	Marathi	16	6.0	0.083	0.993
	English	12	8.0	0.350	1.000
Gemma-2-2b	Marathi	10	0.5	0.925	0.246
	English	12	3.0	0.979	0.231
Llama-3.1-8B	Marathi	11	8.0	0.354	0.661
	English	21	4.0	0.478	0.849

Appendix

Table 34: What the validation split selected, per model and prompt language, and the held-out result of the additive vector at that setting. Four cells select the largest scale in the grid, which is what left the single-layer result unresolved rather than negative; Appendix N resolves it by extending the grid, and § 7 says what the resulting movement is. Gemma is a different case: it accepts almost every positive pair before any intervention, so its rows measure the absence of headroom rather than the effect of the vector.

	Marathi		English		Hindi
Condition	positives	negatives	positives	negatives	positives	negatives
No intervention	0.065	0.994	0.283	1.000	0.122	1.000
Prompt anchor, first	0.201	1.000	0.547	0.997	0.260	0.997
Prompt anchor, second	0.189	0.996	0.588	0.992	0.353	0.999
Prompt anchor, both	0.219	0.990	0.613	1.000	0.362	0.997
Automatic romanization anchor	0.108	0.997	0.317	0.989	0.157	0.994

Appendix

Table 35: Test-set recovery, Qwen, 120 held-out items. Prompt anchoring supplies a Latin form of one or both names in the prompt itself.

	Marathi prompts		English prompts
Scale	positives	negatives	positives	negatives
$\alpha{=}0.5$	0.067	1.000	0.333	1.000
$\alpha{=}1$	0.075	1.000	0.333	1.000
$\alpha{=}1.5$	0.083	1.000	0.333	1.000
$\alpha{=}2$	0.083	1.000	0.350	1.000
$\alpha{=}3$	0.117	1.000	0.358	1.000

Appendix

Table 36: Validation sweep for the vector applied across layers 12–17, Qwen, 60 tuning items. The single-layer sweep applied the vector at one layer and stopped at

\alpha{=}8

; both changes are needed. At the single-layer scale

\alpha{=}6

the window alone lifts Marathi from 0.083 to 0.233, and the extended grid lifts it again to 0.392. Positive-pair accuracy peaks and turns over, which it cannot do within the single-layer grid’s range.

	Marathi		English
Condition	positives	negatives	positives	negatives
No intervention	0.065	0.994	0.283	1.000
Vector, full	0.113	0.993	0.292	0.996
Vector, first entity	0.276	0.992	0.694	0.971
Vector, second entity	0.219	0.992	0.481	0.986
Vector, additive	0.335	0.985	0.590	0.976

Appendix

Table 37: Held-out test recovery with the vector applied across layers 12–17, Qwen, 120 items, at the scale selected on validation (

\alpha{=}12

Marathi,

\alpha{=}24

English). The additive row is the variant selection optimised; the others are reported for completeness and must not be read as the result. Marathi comes within 0.005 of the Lat–Lat reference and English exceeds it, but § 7 applies the same vector to Lat–Lat inputs and raises accuracy there as well, by

+0.289

under Marathi prompts and

+0.227

under English, so the reference is not a ceiling and these figures should not be read as closure toward one.

Prompt language	Layer	Median coordinate	Beyond grid
Marathi	12	15.3	0.00
	13	15.3	0.00
	14	16.3	0.00
	15	17.7	0.00
	16	17.8	0.00
	17	17.8	0.01

Appendix

Table 38: How far each held-out item must move along the rank-one direction to reach its Lat–Lat counterpart, in the same units as the scale

\alpha

. The single-layer grid stopped at

\alpha{=}8

, roughly half the required distance at every layer, which accounts for the null without any appeal to the direction being wrong. The scales selected on validation, 12 for Marathi and 24 for English, bracket these medians.

Configuration	Prompt	$n$	whole $\Delta$	rank-one $\Delta$	share	rank-one flips
Layer 16 only	Marathi	10	2.44	1.38	0.56	0.00
Layers 12–17	Marathi	10	13.66	8.68	0.63	0.40
Layers 12–17, held out	English	20	7.39	3.88	0.52	0.15
Layers 12–17, held out	Marathi	10	7.80	3.52	0.45	0.00

Appendix

Table 39: The reference arm: the rank-one direction applied across layers 12–17 with each item’s own donor coordinate, which is the most informed additive intervention short of replacing the state. Rows one and two are validation-split and validate the window implementation against the validation single-layer figures of § 6.2 , not against the held-out numbers of Table 6 ; the body text below says why the two should not be compared. The upper bound moves margins but flips few gated items, which is the tension discussed in § 7 .

Prompt	Template	Dev–Dev	with vector	Lat–Lat
Marathi	0	7.5	42.5	42.1
	1	10.0	39.2	51.2
	2	2.1	18.8	8.8
English	0	33.3	70.4	56.2
	1	35.0	75.8	60.8
	2	16.7	30.8	41.7

Appendix

Table 40: The intervention across question templates, 120 held-out items. Layer, window and scale were selected on template 0 alone, and the mean-difference vector was estimated on 361 training entities none of which appear here, so templates 1 and 2 are held out twice over. Positive-pair accuracy rises in every cell. Template 2 is the hardest for every condition, not only for the repair: its Lat–Lat reference is 8.8 in Marathi, which the treated model exceeds.

	Balanced accuracy			Positives / negatives, repaired
Arm	baseline	with vector	Lat–Lat	positives	negatives	reference
Marathi	53.0	66.0	66.9	33.5	98.5	34.0
English, Marathi corpus	64.2	78.3	76.5	59.0	97.6	52.9
Hindi	56.1	70.8	70.1	54.6	86.9	40.1
English, Hindi corpus	63.0	77.9	76.0	58.5	97.4	52.1

Appendix

Table 41: The window intervention across all four arms, held-out test split, 120 items each, additive vector applied across layers 12–17 at the scale validation selected. Balanced accuracy is given because positive-pair accuracy alone flatters the Hindi arm: it reaches 0.546 against a 0.401 reference, but pays 13 points of negative-pair accuracy for it, against under 3 in every other arm. Priced that way, Marathi comes within 0.9 of its Lat–Lat reference and the other three arms exceed theirs; Marathi approaches the reference rather than reaching it. What repeats across the four arms is the accuracy movement. § 7 shows that movement to be largely a shift of the decision criterion, with a smaller ranking component on two of three templates, so this table reports a replicated effect and not a replicated repair. The scales are

\alpha{=}12

Marathi, 24 English on the Marathi corpus, 16 Hindi and 32 English on the Hindi corpus.

	Mean difference		Local sensitivity
Scale	positives	negatives	positives	negatives
$\alpha{=}0.5$	6.7	100.0	25.8	100.0
$\alpha{=}1$	7.5	100.0	68.3	98.3
$\alpha{=}1.5$	8.3	100.0	85.8	80.0
$\alpha{=}2$	8.3	100.0	96.7	39.2
$\alpha{=}3$	11.7	100.0	100.0	0.8

Appendix

Table 42: The two correction directions across the same scale grid, Marathi prompts, 60 validation tuning items, both applied across layers 12–17 and both rescaled to unit length. At their selected scales the sensitivity direction is the better of the two on both classes, 68.3 positives at 98.3 negatives against 38.3 at 97.5, so neither is exclusively effective under a negative-pair constraint. What separates them is the width of the usable range. The mean-difference vector holds negative-pair accuracy at 100.0 through

\alpha{=}8

and 93.3 at the largest scale searched, so its operating region is broad. The sensitivity direction runs 100.0, 98.3, 80.0, 39.2 and 0.8 across

\alpha\in\{0.5,1,1.5,2,3\}

, so this grid steps over its usable range entirely; the finer grid of Table 65 resolves a Marathi plateau from

\alpha{=}1.0

to 1.5. Both reach a higher balanced accuracy than the single-layer sweep of Appendix M , and § 7 shows that for the mean-difference vector the gain is largely a shift of the decision criterion, with a smaller within-template ranking component.

Amount	fold	Marathi	Hindi	English
Item’s own	t01e2	0.16	0.15	0.20
	t02e1	0.08	0.06	0.01
	t12e0	0.06	0.02	$-0.15$
Another item’s	t01e2	0.23	0.14	0.18
	t02e1	0.10	0.11	0.00
	t12e0	0.13	0.05	$-0.06$

Appendix

Table 43: Signed margin change on never-correct items divided by signed margin change on gated items, for the sensitivity direction at layer 16, rank 1, on validation. Lower means the intervention acts more narrowly on the gate it was selected for. An amount derived from an item, whether its own or another item’s, moves off-gate items by between

-0.15

and 0.23 times what it repairs; one constant amount for every item moves them almost as much as it repairs, in every fold and every language. Negative-pair accuracy is 1.00 throughout.

Marathi prompts
Layer	Whole	Jac 1	Jac 8	Mean diff	PCA	Probe	Random
12	0.97	0.61	0.81	1.52	0.13	0.00	$-0.03$
13	2.05	1.25	1.32	2.01	0.03	0.02	0.02
14	2.50	2.04	2.41	2.38	$-0.01$	0.04	0.02
15	5.01	3.40	4.01	3.32	$-0.15$	0.01	0.02
16	5.52	4.00	4.54	3.62	0.40	0.03	0.06

Appendix

Table 44: Rank-one and rank-eight subspace patching at the second entity, by layer, on validation items. Whole is the full-state patch at the same layer, the reference the others are read against. Jac is the local sensitivity direction, at rank one and rank eight; the remaining columns are rank one. Mean diff, PCA, probe and random are the alternative directions of § 6.2 . Higher is more repair; the columns are not closures and are not normalised by the whole patch.

		Native-script prompts				English prompts
Corpus	Layer	E1	E2	Q	R	E1	E2	Q	R
Marathi	8	1.05	2.21	0.16	0.03	0.38	1.18	$-0.13$	$-0.12$
	9	0.88	1.64	1.22	0.17	0.36	0.98	$-0.08$	$-0.05$
	10	1.03	1.66	1.51	0.20	0.38	1.44	$-0.24$	$-0.09$
	11	0.19	1.71	3.43	0.48	0.26	2.07	1.37	$-0.22$
	12	$-0.01$	1.68	4.21	0.53	0.03	1.82	1.66	$-0.19$

Appendix

Table 45: Gemma, mean correctness-oriented margin change by layer and site, gated on negative pairs, Lat–Lat source, pooled over splits. Positive means the intervention moved the model toward the correct answer. Gate sizes are

n{=}33

Marathi and

n{=}127

English for the Marathi corpus,

n{=}65

Hindi and

n{=}125

English for the Hindi corpus.

Prompt	Paraphrase	Dev–Dev	Dev–Lat	Lat–Dev	Lat–Lat
Marathi	P0	1.2	5.2	3.5	5.1
	P1	51.4	52.0	51.1	51.3
	P2	50.0	50.0	50.0	50.0
Hindi	P0	1.4	14.2	21.0	13.6
	P1	6.6	25.3	30.0	20.5
	P2	50.0	49.9	49.9	50.0

Appendix

Table 46: Llama positive-pair accuracy (%) by paraphrase, Marathi prompts.

	Qwen2.5-7B	Gemma-2-2b	Llama-3.1-8B
Layers	28	26	32
Devanagari tokens per character, Marathi	0.98	0.56	0.61
Devanagari tokens per character, Hindi	0.99	0.55	0.60
$d^{\prime}$ gap, English prompts	$+0.07$	$+0.73$	$+1.00$
$d^{\prime}$ gap, Marathi prompts	$+0.88$	$+0.69$	$+0.08$ (excluded)
$d^{\prime}$ gap, Hindi prompts	$+0.30$	$+0.48$	$+0.40$ (excluded)

Appendix

Table 47: The three models compared. Probe figures are held-out test items; the causal rows report items passing the gate. Llama’s Marathi arm is excluded throughout. Gemma’s peak effect is negative because its gate reads negative pairs, where the correct direction is away from “same”; its magnitude, not its sign, is comparable with the others.

Layer	Marathi	English
11	0.60	0.33
12	0.60	0.44
13	0.56	0.59
14	0.36	0.69
15	0.30	0.81
16	0.21	0.94

Appendix

Table 48: Mistral-Small second-entity margin change by layer, depth-matched sweep. The band is layers 11–29 of 40, which is Qwen’s 8–20 of 28 rescaled, so it covers layer 23, the depth-matched equivalent of Qwen’s layer 16. Nothing is there: every layer from 21 to 29 is within 0.02 of zero in both prompt languages. The effect peaks at layer 11 under Marathi prompts and 18 under English, both shallower in relative depth than Qwen’s 12–17 of 28. 60 gated items per prompt language, Lat–Lat donor.

Swept band, 11–29
Prompt	E1	E2	Q	R	E2 minus wrong entity
Marathi	$-$ 0.004	0.154	0.093	1.471	+0.023
English	0.003	0.336	0.135	2.301	+0.104
Depth-matched, 14–26
Marathi	$-$ 0.002	0.090	0.134	1.491	+0.023
English	0.005	0.389	0.164	2.429	+0.122

Appendix

Table 49: Mistral-Small site effects over two bands. The upper block is the whole swept band, layers 11–29 of 40. The lower block is the sub-band depth-matched to the layers Qwen’s Table 26 averages, 10–18 of 28 rescaled to 14–26 of 40. The first entity is at zero in both. The second entity clears the question mark over the swept band, but not in Marathi on the depth-matched sub-band, where it has already decayed below it: Mistral locates this effect shallower than Qwen does, so matching Qwen’s depth window misses the Marathi peak rather than centring on it. Magnitudes are roughly tenfold smaller than Qwen’s 2.42 on either band, which the depth-matched sweep was run to establish and does. The final column is the paired entity-specificity contrast at the second entity, correct donor minus wrong-entity donor over the items present in both arms.

Fact	Devanagari field reads	Latin field reads	Kind
q485697	Kepler Laveran Lima Ferreira	Pepe	different name
q855252	Rama IX, Thailand	Bhumibol Adulyadej	different name
q213812	George Herman Ruth	Babe Ruth	different name
q35811	Zarathustra	Zoroaster	different tradition
q61047	Ferdinand Eisenstein	Gotthold Eisenstein	different given name
q11576	Raúl González	Raúl	full vs short

Appendix

Table 50: Items whose two script fields carry different name forms. Devanagari readings are given in transliteration. In every case the Latin field holds the more familiar form, which biases the measured gap in the reported direction.

	Marathi	Hindi
Surface baseline, balanced accuracy	0.933	0.946
Surface baseline, AUC	0.970	0.967
Probe, balanced accuracy	0.925	0.942
Surface recovery on the model’s failures	0.869	0.896
Probe recovery on the model’s failures	0.789	—
Probe where the surface baseline is correct	0.973	0.996

Appendix

Table 51: A classifier reading only the two Devanagari strings, against the probe on the same held-out items. The probe layer is the one validation selected. Recovery is the share of the model’s own Dev–Dev failures on which the method returns the correct label.

	Negatives	Probe	Baseline	Probe $-$ baseline
Marathi	shipped	0.925	0.933	$-$ 0.008 [ $-$ 0.033, $+$ 0.017]
Marathi	matched	0.908	0.796	$+$ 0.112 [ $+$ 0.067, $+$ 0.163]
English	shipped	0.950	0.933	$+$ 0.017 [ $-$ 0.004, $+$ 0.042]
English	matched	0.879	0.796	$+$ 0.083 [ $+$ 0.025, $+$ 0.142]
Probe where the surface baseline fails, $n{=}16$ shipped and 49 matched
Marathi	shipped	0.250 [0.059, 0.476]

Appendix

Table 52: The probe against the surface baseline under the two negative maps, paired on the same 120 held-out entities and 240 pairs, with item-clustered bootstrap intervals from 2000 draws. The shipped map is the one Table 51 reports; the matched map is the similarity-matched one of Table 55 . Under the shipped map the probe does not beat the baseline in either language. Under the matched map it does, and it stays correct on about two thirds of the pairs the strings get wrong, where under the shipped map it was correct on a quarter. The two where the baseline fails intervals do not overlap in either language.

	At layer 16		Best layer
Position	Marathi	English	Marathi	English
Second entity	0.942	0.954	0.954 (17)	0.958 (14)
Question mark	0.925	0.933	0.942 (6)	0.950 (18)
Readout	0.838	0.879	0.925 (7)	0.950 (20)

Appendix

Table 53: Held-out probe balanced accuracy by position, on the 120 test entities. The middle columns are at layer 16, the layer the causal comparison of § 6 uses; the right-hand columns give the maximum of each position’s test-layer profile. Those are descriptive: unlike the layers of Table 21 , which validation selected, they are peaks located on the evaluation split, so their position is not independent evidence for localisation. The second entity is the best position at layer 16 in both languages, and it is the only one whose optimum falls inside the 12–17 causal window: the question mark and the readout peak at layers 6 and 7 under Marathi prompts, well before it.

Sequence ratio	Items	Surface	Probe
0.3 and below	3	0.000	0.333
0.3 to 0.5	15	0.267	0.400
0.5 to 0.7	34	1.000	0.912
above 0.7	68	1.000	0.971

Appendix

Table 54: Held-out Marathi positives by how alike the two Devanagari strings are.

	String	Dev–Dev		Lat–Lat
Negatives	classifier	positives	AUC	positives	AUC
Occupation and country	0.800	7.9	0.755	36.0	0.893
Also string similarity	0.680	8.0	0.753	36.0	0.901

Appendix

Table 55: Similarity-matched negatives, Qwen, on the 144 items where the matching succeeded to within 0.05 (positive similarity 0.409, negative 0.406). The classifier is the one of Table 51 , reading only the Devanagari strings; on this subset it falls from 0.800 to 0.680, losing two fifths of its margin over chance, and on the full corpus from 0.933 to 0.796. The model does not move with it: positive-pair accuracy, the script gap and the threshold-free separation all hold to within 0.01, and Dev–Dev negatives stay at 100.0 under both maps. Whatever the model is using, it is not what the classifier lost.

		Real corpus		Nonce corpus
Prompt	Condition	positives	negatives	positives	negatives
Marathi	Dev–Dev	0.085	0.998	0.046	1.000
	Dev–Lat	0.294	0.991	0.024	1.000
	Lat–Dev	0.350	0.998	0.169	1.000
	Lat–Lat	0.351	0.997	0.036	1.000
English	Dev–Dev	0.282	1.000	0.206	1.000

Appendix

Table 56: Nonce-name control, Qwen. 600 invented people built from recombined attested name parts, sharing no fact identifier with the real corpus. Read the negatives column first: on nonce names the model answers different to essentially everything, so positive-pair accuracy falls in every condition, not only Dev–Dev. The script gap shrinks as a consequence of that shift rather than independently of it, and Marathi is uninformative at 0.036 even in the Lat–Lat condition.

	Marathi prompts			English prompts
Latin overlap	Dev–Dev	Lat–Lat	gap	Dev–Dev	Lat–Lat	gap
$<$ 0.4	7.2	38.5	31.3	14.9	49.4	34.5
0.4–0.6	6.9	36.5	29.6	26.8	61.0	34.2
0.6–0.8	8.0	35.6	27.6	28.8	55.8	27.1
$>$ 0.8	10.8	32.1	21.3	32.9	56.9	24.0
items	58 / 94 / 295 / 154			same items

Appendix

Table 57: The script gap on positive pairs, split by how alike the two Latin forms are (difflib sequence ratio on the corpus of the Marathi run). If the model were matching on surface similarity and Devanagari merely made that harder, the gap should shrink as the names become less alike. It does the opposite: the gap is largest in the least similar band and smallest in the most similar one, in both prompt languages.

Prompt	template 0	template 1	template 2
Marathi	$-0.008$ $[-0.041,+0.025]$	$-0.036$ $[-0.063,-0.015]$	$-0.048$ $[-0.080,-0.016]$
English	$-0.009$ $[-0.027,+0.004]$	$-0.005$ $[-0.024,+0.014]$	$-0.003$ $[-0.017,+0.013]$
Hindi	$-0.007$ $[-0.030,+0.014]$	$-0.016$ $[-0.044,+0.011]$	$-0.131$ $[-0.182,-0.079]$

Appendix

Table 58: Lat–Dev minus Lat–Lat AUROC of the yes-minus-no margin, per question template, held-out entities, item-clustered bootstrap over 2000 draws. Reported per template rather than pooled because pooling averages each entity’s margin across templates with different distributions and can hide within-template ranking changes, as § 7 finds for the correction vector. A negative value means Lat–Dev ranks positives against negatives less well than Lat–Lat does, while Table 1 shows it is the more accurate condition under English and Hindi prompts.

Arm	Condition	Positives	Negatives
Marathi	Dev–Dev, unmodified	0.065	0.994
	Dev–Dev, $+$ vector	0.335	0.985
	Lat–Lat, unmodified	0.340	0.997
	Lat–Lat, $+$ vector	0.629	0.976
English, mr corpus	Dev–Dev, unmodified	0.283	1.000
	Dev–Dev, $+$ vector	0.590	0.976

Appendix

Table 59: The same intervention applied to the failing condition and to the healthy one, on the 120 held-out entities of each arm, at the scale that arm’s validation selected. Positives are SAME pairs. The vector raises Lat–Lat acceptance in every arm, by 0.222 to 0.289, which is why we do not describe it as closing the script gap: a recovery stated as a fraction of the gap to Lat–Lat presumes that reference is fixed, and it is not. The two gains are not equal, though. Marathi is the only arm where the healthy condition gains more than the failing one; in the other three the Dev–Dev gain is larger, so we claim that the intervention is not specific to the failing condition and not that the two conditions gain equally.

Marathi corpus
	Native prompts		English prompts
Site	positives	negatives	positives	negatives
unmodified	0.065	0.994	0.283	1.000
second entity	0.335	0.985	0.590	0.976
first entity	0.071	0.993	0.332	1.000
question mark	0.250	0.953	0.433	0.990

Appendix

Table 60: The same vector, window and scale added at four token positions, Dev–Dev inputs, 120 held-out entities per arm. The second entity is the position the vector was estimated at and the one § 6 localises; the other three are controls. One result holds in every arm: the first entity moves accuracy by at most 0.054, so the gain is not available anywhere in the prompt. The readout is weaker and less consistent: it lowers accuracy in three arms but matches the second entity under Hindi prompts, 0.550 against 0.546 at comparable negatives, which we report without an account for it. We therefore read the intervention as depending on position rather than as acting only at the second entity.

	Positive-pair gap			Balanced gap
Arm	untreated	both treated	Change	untreated
Marathi	27.5 [+21.1, +34.4]	29.4	+1.9	+13.9 [+10.6, +17.4]
English, mr corpus	24.6 [+16.2, +33.8]	16.6	-8.0	+12.3 [+8.1, +16.9]
Hindi	27.9 [+21.8, +34.7]	14.4	-13.5	+14.0 [+10.9, +17.4]
English, hi corpus	26.1 [+18.3, +34.4]	15.8	-10.3	+13.1 [+9.2, +17.2]

Appendix

Table 61: The script disparity in positive-pair accuracy, in points, before any intervention and with the same frozen vector applied to both conditions. The untreated columns carry paired item-clustered intervals over 2000 draws, since Dev–Dev and Lat–Lat are two conditions of one run; the treated column is a point estimate, because treated Dev–Dev and treated Lat–Lat come from separate evaluations and the entities cannot be paired across them from the files we retain. Treating both conditions equally, the disparity narrows in three of the four arms and widens slightly in Marathi. A percentage closure does not make this comparison: that figure sets a treated Dev–Dev against an untreated Lat–Lat, which measures movement rather than parity. The intervention improves both conditions and does not eliminate the disparity between them.

	Entity AUROC			TPR at the vector’s FPR
Template	baseline	$+$ vector	change	vector	threshold	offset
0	0.851	0.885	+0.034 $[+0.006,+0.062]$	0.425 (0.021)	0.308	0.267 (0.004)
1	0.938	0.941	+0.003 $[-0.009,+0.015]$	0.392 (0.012)	0.371	0.637 (0.037)
2	0.924	0.944	+0.020 $[+0.008,+0.035]$	0.188 (0.012)	0.242	0.421 (0.021)

Appendix

Table 62: The window intervention template by template, Marathi prompts, Dev–Dev, 120 held-out entities. The AUROC columns average an entity’s two answer keys within a template and then rank, with paired item-clustered intervals over 2000 draws; the operating-point columns are prompt-level. The threshold column is the oracle diagnostic of Table 7 , placed per template on the test curve; the offset column is chosen on the validation tuning entities under that arm’s own validation negative-pair accuracy and reads nothing from test. Ranking improves on templates 0 and 2, which a constant offset cannot do. The vector beats both calibration columns on template 0 alone, the one its scale was tuned on. Template 2 carries an apparent tension: its ranking gain excludes zero while its true-positive rate at the operating point falls, which is consistent because the ranking gain lies away from the low-false-positive region the operating point sits in.

Mean-difference direction, gated margin change
Prompts	Fold	one constant for every item	the item’s own	another item’s
en	t01e2	5.51	3.97	4.18
en	t02e1	3.30	3.18	3.26
en	t12e0	4.13	3.85	4.05
hi	t01e2	4.35	4.47	4.25
hi	t02e1	7.00	6.62	6.20

Appendix

Table 63: How much each item moves, holding the direction fixed. Mean signed margin change on gated items, by fold and prompt language, at layer 16 and rank 1 on validation. The upper block varies the coefficient for the mean-difference direction: across the six Marathi and Hindi cells a single constant for every item spans 3.55 to 7.00 and an item’s own coefficient spans 2.52 to 6.62, so a constant amount is not what limits recovery. The lower block gives the paired difference between an item’s own coefficient and another item’s for the sensitivity direction: it spans zero in 9 of 9 cells, the largest magnitude being

-1.14

[-4.86,+1.73]

in mr fold t12e0. With 10 to 23 entities per fold these intervals bound an advantage loosely rather than excluding one.

Intervention	Fitted on	Selected on	Templates	Reported on
Single-layer vector	361 train entities	60 of 120 validation	0 for selection, 0–2 for test	120 test
Window vector	361 train entities	60 of 120 validation	0 for selection, 0–2 for test	120 test
Sensitivity sweep	31 Marathi, 60 English gated train items	60 of 120 validation	0 only	validation only
Scalar offset	nothing fitted; one constant selected	60 of 120 validation	0 only	120 test
Rank-one subspace	31 Marathi, 60 English gated train items	fold t01e2	train 0–1, evaluate 2	gated test items
Condition control	frozen, nothing fitted	nothing selected	0–2	120 test

Appendix

Table 64: What each intervention was fitted on, selected on and reported on. The selection metric is balanced accuracy in every case. No intervention is selected under a floor on negative-pair accuracy: negative accuracy enters only as a tiebreak between settings with equal balanced accuracy, which is why the selected settings are free to lower it, and do, on validation, to 0.975 in the Marathi window arm and 0.925 in the Hindi one. The scalar offset of Table 7 is the one exception: it was held to a floor, which handicaps the offset and so favours the vector in that comparison.

	Local sensitivity				Mean difference
$\alpha$	positives	negatives	balanced	$\alpha$	positives	negatives	balanced
Marathi prompts
0.5	0.258	1.000	0.629	0.5	0.067	1.000	0.533
0.6	0.308	1.000	0.654	1	0.075	1.000	0.537
0.7	0.342	1.000	0.671	1.5	0.083	1.000	0.542
0.8	0.483	1.000	0.742	2	0.083	1.000	0.542

Appendix

Table 65: Both correction directions on the validation tuning items, selected by balanced accuracy exactly as in Table 36 . Scales are not comparable across the two panels: each direction is unit length, so its own scale carries all magnitude, and the grids are the ones each was searched on. Two things the coarse grid of Table 42 is too sparse to show. The sensitivity direction has a plateau: 6 consecutive Marathi scales from

\alpha{=}1.0

to 1.5 sit within 0.025 of its maximum. And under the same rule it reaches higher balanced accuracy than the mean-difference vector in both languages, 0.846 against 0.679 in Marathi and 0.892 against 0.850 in English, the Marathi gap because the mean-difference vector never lifts positive-pair accuracy above 0.392 at any scale while the sensitivity direction reaches 0.858. It is fitted on 31 gated training items against the other’s 361. No held-out test evaluation was run for the sensitivity direction, so these figures cannot be set beside the test numbers of Table 37 .

Entity-averaged AUROC
		Both native		Native, Lat		Lat, native		Both Latin
Corpus	Prompt	pos.	neg.	pos.	neg.	pos.	neg.	pos.	neg.
Malayalam	Malayalam	5.3	100.0	33.5	98.6	38.1	99.6	48.8	99.9
	English	26.5	98.9	58.2	97.1	66.4	95.6	62.9	99.6
Devanagari	Marathi	10.4	100.0	33.2	98.8	37.8	99.8	41.0	99.6
	English	30.0	99.6	64.4	98.6	66.0	97.5	62.9	99.6

Appendix

Table 66: The full Malayalam grid and its matched Devanagari comparator, 374 entities, Qwen2.5-7B. Accuracy is a percentage over three paraphrases and both answer keys; AUROC is entity-averaged over the six prompt margins. Gaps are Lat–Lat minus the both-native condition, paired by entity, with 95% item-clustered intervals over 2,000 resamples; the test column repeats the gap on the 76 held-out entities. The English Lat–Lat cells of the two corpora come from byte-identical prompts and agree to 0.0 points.

Prompt	Paraphrase	Mlym–Mlym	Mlym–Lat	Lat–Mlym	Lat–Lat
Malayalam	P1	10.5	48.7	62.5	65.8
	P2	10.5	56.6	64.5	70.4
	P3	2.6	7.9	2.0	4.6
English	P1	26.3	73.7	86.2	61.2
	P2	28.3	75.7	90.8	62.5
	P3	12.5	40.8	23.7	47.4

Appendix

Table 67: Same-person accuracy (%) on the 76 held-out Malayalam entities by paraphrase. P1 asks whether the two names are one person, P2 whether they are names of one person, P3 whether one is another name of the other. All three were checked by a Malayalam speaker. P3 sits near the floor in Malayalam and recovers in English on the same items, so its collapse is specific to the Malayalam rendering of that construction.

Cross-Lingual Consistency in Language Models

Cross-lingual consistency in language models studies whether equivalent inputs across languages produce comparable knowledge, reasoning, and safety behavior.

See all 71 papers

May 27, 2026

DEPART: DEcomposing PARiTy across Multilingual LLMs

Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain

R^2_{\text{ling}} = 79\%

of this variance on understanding tasks and

92\%

on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model

\times

benchmark

\times

language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding (

66.7\%

of variance), whereas the benchmark

\times

model interaction dominates reasoning (

46.3\%

). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.

Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3

Microsoft Research India · IIT Gandhinagar

Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models

Ask PDF

Oct 6, 2026

Are Language Models Script-Aware?

Language models frequently generate outputs in unintended languages or scripts, a phenomenon known as off-target generation. While existing research has focused on language selection, the dimension of script knowledge remains understudied: before any linguistic understanding can occur, users must recognize the graphic symbols in a model's response. We investigate whether Small and Large Language Models (SLMs and LLMs) possess script knowledge by testing them on multi-scriptic languages. Through two complementary experiments, we evaluate whether models (1) adapt their output script to match the input, and (2) follow explicit instructions to generate text in a specified script. The models we tested demonstrate substantial script knowledge: they all achieve a near-perfect Latin script fidelity (more than 98%) and follow script instructions with high frequency. Nevertheless, we notice differences between LLMs and SLMs, with higher scores for LLMs including for non-standard script combinations.

David Kletz, Sandra Mitrović, Ljiljana Dolamić +1

SUPSI, IDSIA, Switzerland · armasuisse, Science & Technology, Switzerland

Multilingual Language Models Multilingual Language Model Evaluation

Ask PDF

May 29, 2026

The Latin Substrate: How Language Models Represent and Mediate Script Choice

Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the representational level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely preserving semantic content. The vector generalizes to writing systems unseen during construction: it flips non-Latin output to Latin, but maps Latin output into varied non-Latin scripts. At the mechanistic level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the model. Overall, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.

Daniil Gurgurov, Alan Saji, Katharina Trinley +2

D German Research Center for AI (DFKI) · I Indian Institute of Technology, Madras · H University of Hamburg +1

Multilingual Language Models LLM Interpretability

Ask PDF

Abstract

Cross-Lingual Consistency in Language Models

Explore similar work

DEPART: DEcomposing PARiTy across Multilingual LLMs

Are Language Models Script-Aware?

The Latin Substrate: How Language Models Represent and Mediate Script Choice