Beyond Accuracy: What Language-Subspace Interventions Capture, Remove, and Improve

Shaanil Punglia; Dadhichi Telwadkar; Vaibhav Rathore; Siddharth Sureka; Neha Bhargava

cs.CLNov 3, 2026

Beyond Accuracy: What Language-Subspace Interventions Capture, Remove, and Improve

Authors: Shaanil Punglia, Dadhichi Telwadkar, Vaibhav Rathore, Siddharth Sureka, Neha Bhargava

Organizations: Motilal Oswal Financial Services, India · Indian Institute of Technology Bombay, India

Abstract

Language-subspace interventions aim to improve multilingual reasoning by suppressing language-related components in mid-layer activations. Using MATH and MGSM across four open-weight LLMs, we ask three questions: do estimated language subspaces capture translation-associated variation? Does projecting them out remove language information? And do they improve accuracy without compromising response language? For the first question, we build subspaces using the existing Mean-SVD construction and our DiffPCA. Both capture substantial translation-associated variation and show similar patterns across model depth. For the second, we project out each subspace and train linear probes to classify language from the projected activations. The probes still identify language almost perfectly, showing that neither subspace exhaustively captures language information. For the third, we evaluate several intervention configurations on Qwen2.5-7B, by varying subspace creation methods and intervention operators. We find that the intervention that most improves MGSM accuracy sacrifices response-language fidelity, with only one intervention improving joint accuracy--answers that are correct and in the target language--slightly, by 1.60 points. Interestingly, subtle changes in the prompt alone can reproduce the accuracy gains albeit with significant damage to language fidelity. We argue that causal interventions must be judged on more than accuracy, to fully account for language substitution trade-offs.

Figures & tables

Setting	Accuracy	Fidelity	Joint
	(%) $\uparrow$	(%) $\uparrow$	(%) $\uparrow$
No intervention	62.80	99.82	62.62
Mean-SVD , rank 10 projection, dose 0.5	65.75	80.58	53.82
Mean-SVD , rank 1 projection, dose 1	64.47	99.31	64.22

Table 1: No intervention and the Mean-SVD configurations with the highest accuracy and joint accuracy on Qwen2.5 across eleven languages.

	Causal intervention *		English switch		Chinese switch
Model	Accuracy	Fidelity	Accuracy	Fidelity	Accuracy	Fidelity
Qwen2.5	$+0.94$	$+4.69$	$+1.78$	$-64.71$	$+3.42$	$-95.45$
Qwen3 (thinking)	$+1.27$	$+0.03$	$+1.87$	$-80.44$	$-1.73$	$-77.33$
Qwen3 (no thinking)	–	–	$+1.42$	$-72.36$	$+0.36$	$-50.89$
R1-Distill	$+2.40$	$+1.20$	$+1.07$	$-22.49$	$+2.58$	$-31.09$
GLM-Z1	$+1.52$	$+0.08$	$+6.53$	$-76.76$	$-0.27$	$-72.76$

Table 2: Accuracy and fidelity changes under causal intervention and prompt-language switches. * Causal-intervention values are imported from Zhao et al. (2025) ; switch values average the nine non-English/Chinese MGSM languages. Comparable accuracy gains thus conceal radically different language behavior.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Approximate the language means at rank $r$ .
1	$m_{\ell}\leftarrow\frac{1}{n}\sum_{i}h_{i,k}^{\ell}$ ; $W\leftarrow[m_{1},\ldots,m_{L}]$ .
2	$a_{0}\leftarrow\frac{1}{L}W\mathbf{1}$ ; $U\Sigma V^{\top}\leftarrow\operatorname{SVD}(W-a_{0}\mathbf{1}^{\top})$ .
3	$\widehat{W}\leftarrow a_{0}\mathbf{1}^{\top}+U_{r}\Sigma_{r}V_{r}^{\top}$ .

Appendix

Table 3: Layerwise construction of Mean-SVD and DiffPCA subspaces. Subscript

r

selects the leading

r

singular components;

W^{+}

is the Moore–Penrose pseudoinverse and

\epsilon>0

is a numerical norm floor.

Model/data	Mean-SVD	DiffPCA	$k$ /depth	$r_{\mathrm{P}}$
Qwen2.5/ MATH	0.431	0.479	18/0.67	0.9971
Qwen2.5/ MGSM	0.297	0.352	18/0.67	0.9918
Qwen3/ MATH	0.592	0.608	21/0.60	0.9995
Qwen3/ MGSM	0.448	0.462	21/0.60	0.9972
R1/ MATH	0.459	0.494	16/0.59	0.9940
R1/ MGSM	0.240	0.296	17/0.63	0.9882

Appendix

Table 4: Rank-10 translation capture at each model–corpus pair’s Mean-SVD interior minimum.

k

is the block index; depth is

k/(K-1)

r_{\mathrm{P}}

is the Pearson correlation between the two estimators’ full depth profiles.

Model	$k$	Mean-SVD	DiffPCA	Activation-PCA	Random	Perm.	Min. retr.
Qwen2.5	18	0.357 / 1.000	0.441 / 1.000	0.364 / 1.000	1.000 / 1.000	0.086	0.9993
Qwen3	21	0.243 / 1.000	0.296 / 1.000	0.186 / 1.000	1.000 / 1.000	0.092	0.9971
R1-Distill	16	0.946 / 1.000	0.925 / 1.000	0.992 / 1.000	1.000 / 1.000	0.089	0.9985
GLM-Z1	20	0.337 / 1.000	0.494 / 1.000	0.291 / 1.000	1.000 / 1.000	0.093	0.9993

Appendix

Table 5: Language probes at the held-out MATH capture minimum, using rank ten and full projection (

\lambda=1

). Each method cell gives frozen / retrained held-out accuracy. Clean accuracy is 1.000 for every model. The permutation reference uses shuffled labels on clean activations. The final column is the lowest retrained accuracy over all tested intervention cells across the depth, rank, and strength sweeps, reported to four decimals.

English
	Prompt-language switch			Translate-test
Model	Accuracy	Fidelity	Joint	Accuracy	Fidelity	Joint
Qwen2.5	$+1.78$	$-64.71$	$-40.49$	$+27.02$	$-99.82$	$-61.33$
Qwen3 (thinking)	$+1.87$	$-80.44$	$-65.24$	$+14.09$	$-99.60$	$-82.04$
Qwen3 (non-thinking)	$+1.42$	$-72.36$	$-53.29$	$+20.04$	$-99.96$	$-74.89$
R1-Distill	$+1.07$	$-22.49$	$-6.71$	$+32.58$	$-97.85$	$-48.71$

Appendix

Table 6: Prompt-language switches and translate-test on the same nine MGSM languages. Values are percentage-point changes from the matched native-prompt baseline. Fidelity and joint accuracy are scored against the original problem language, including for translate-test.

Prompt	Seed 0	Seed 1
Native prompt	70.44	73.78
English switch	14.71	15.47
English + target instruction	72.49	72.04
Chinese switch	17.69	17.47
Chinese + target instruction	59.69	57.60

Appendix

Table 7: GLM joint accuracy (%) on the same nine languages, with and without an explicit target-language instruction.

Causal Interventions in Language Models

Causal interventions in language models test and change how internal representations and computational components cause outputs.

See all 505 papers

Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops
Aug 28, 2026Faeze Ghorbanpour, Constanza Fierro, Alexander Fraser +1Cross-Lingual Consistency in Language Models Multilingual QA
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML) · Department of Computer Science, University of Copenhagen
Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer
Aug 31, 2026Minju Song, Hyeon Hwang, Junhyun Lee +1Cross-Lingual Reasoning in Language Models Low-Resource Language Processing
Korea University · Hankuk University of Foreign Studies · Noah’s Farm +1
DEPART: DEcomposing PARiTy across Multilingual LLMs
May 27, 2026Manan Uppadhyay, Prashant Kodali, Pranjal Chitale +3Multilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
Microsoft Research India · IIT Gandhinagar

Aug 28, 2026J/K move · Enter open · S save

Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops

Faeze Ghorbanpour, Constanza Fierro, Alexander Fraser, Anders Søgaard

School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML) · Department of Computer Science, University of Copenhagen

Cross-Lingual Consistency in Language Models Multilingual QA Cross-Lingual Representation Alignment Factual Knowledge in Language Models Causal Interventions in Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) often answer the same factual question differently across languages. We study whether cross-lingual latent-space intervention can reduce this inconsistency. We train layer-specific autoencoders on parallel multilingual representations and apply inference-time corrections to factual QA prompts. We find that latent intervention improves geometric alignment between languages, and that this improvement translates into consistent gains in cross-lingual consistency with English across both open-ended and multiple-choice QA formats, without degrading factual accuracy. In open-ended QA, Spearman's rank correlation between English and non-English languages improves substantially, with gains of 0.16 for English-Arabic and 0.20 for English-Russian pairs. In multiple-choice QA, answer agreement with English improves consistently across both KLAR and mParaRel. Ablations show that AE reconstruction yields consistent gains at no accuracy cost, while PCA projection contributes marginally, and mean-shift produces substantially larger consistency gains in open-ended QA at the cost of some accuracy.

Abstract

Causal Interventions in Language Models

Explore similar work

Latent-Space Intervention for Cross-Lingual Factual Consistency: Consistency Improvements without Accuracy Drops

Enhancing Low-Resource Language Reasoning via High-Resource Language Feature Transfer

DEPART: DEcomposing PARiTy across Multilingual LLMs