Multilingual language models can solve the same mathematical problem in different languages, but it remains unclear whether they rely on shared features or on language-specific computations that only produce similar outputs. We study this question in five models from four families using the Multilingual Grade School Math (MGSM) dataset, with problems solved in English, German, French, Spanish, Russian, and Chinese, retaining problems with valid reasoning traces in all six languages and replaying those traces through the model to record representations at multiple layers. For each model, we first use Centered Kernel Alignment (CKA) to identify layers with cross-language alignment. At each selected layer, we train two sparse autoencoders (SAE): a baseline reconstruction-only model and a contrastive variant introduced in this work, the Geometry-Invariant SAE (GI-SAE). GI-SAE supplements the reconstruction loss with an Information Noise-Contrastive Estimation (InfoNCE) loss that trains the encoder to produce similar activations for traces of the same problem, regardless of language or token position. We then test whether the resulting shared features are functionally interchangeable by swapping their values between languages during the model's forward pass and measuring the resulting change in output, quantified by Kullback-Leibler (KL) divergence per feature. Although GI-SAE yields higher CKA and Jaccard similarity at nearly every layer, higher geometric similarity does not consistently imply greater functional interchangeability. We find that cross-language feature sharing is model- and architecture-dependent in this sample and appears at different depths in different models. GI-SAE primarily amplifies cross-language structure already present: the pattern is model-specific, with strengthening in Qwen, no functional benefit in Gemma, and mixed layer-dependent effects in Llama and Phi.
Figures & tables
Model
Family
Params
dmodel
Sweep
Valid
Activation vectors
problems
Train
Val
Qwen3-1.7B
Qwen
1.7B
2048
L16–L26 (11)
151
2.77M
0.65M
Qwen3-4B
Qwen
4B
2560
L21–L34 (14)
202
5.42M
1.06M
Llama-3.2-3B
Llama
3B
3072
L11–L27 (17)
60
0.79M
0.20M
Gemma-3-4B
Gemma
4B
2560
L15–L30 (16)
137
1.95M
0.39M
Phi-3-mini
Phi
3.8B
3072
L1–L20 0 (20)
63
1.18M
0.24M
Table 1: Models, layer sweeps, and dataset sizes. dsae=4×dmodel ; K=128 for all models. Sweep ranges are determined by the CKA survey. Valid problems are those solved correctly in all six languages. Activation vectors are the total number of dmodel -dimensional residual-stream vectors extracted across all tokens, languages, and swept layers.
Figure 1: Cross-language shared fraction (fraction of 128 active features shared between two languages for the same problem) vs. relative depth. Left : baseline SAE (reconstruction only). Right : GI-SAE (reconstruction + contrastive). Colored bands mark the three sharing profiles: low-sharing ( < 15%, orange), convergent (15–60%, green), saturated ( > 60%, red).
Figure 2: Each point is one (model, layer) observation. x -axis: CKA improvement from GI-SAE over baseline (geometric similarity gain). y -axis: KL/feature improvement (positive = GI-SAE has lower KL/feature). Marker size scales with baseline shared fraction. The upper-right quadrant (green) indicates both geometric and functional benefit; the lower-right (orange) indicates geometric improvement without functional benefit.
Figure 3: Phase diagram: GI-SAE KL/feature benefit vs. baseline shared fraction (test split). Points above the dashed line indicate GI-SAE wins. Three profiles emerge with distinct win rates.
Profile
Baseline shared
Obs.
Win rate
95% CI
Low-sharing
< 15%
37
49%
[33, 64]%
Convergent
15–60%
24
83%
[64, 93]%
Saturated
> 60%
17
6%
[1, 27]%
Table 2: GI-SAE win rates by baseline sharing profile (test split). 95% Wilson confidence intervals in brackets.
Figure 4: Three representative family profiles. Top : baseline shared fraction vs. layer. Middle : CKA (baseline blue, GI-SAE orange). Bottom : KL/feature (lower = greater functional interchangeability; shading marks GI-SAE advantage in green, disadvantage in orange). Left: Gemma (saturated). Center: Qwen3-4B (convergent). Right: Phi-3-mini (early-extraction).
Model
Layers
Test wins
95% CI
CKA Δ
BL shared
GI shared
Profile
Qwen3-1.7B
11
11/11
[74, 100]%
+0.19
11–62%
16–67%
Conv
Qwen3-4B
14
14/14
[78, 100]%
+0.23
13–51%
24–69%
Conv
Llama-3.2-3B
17
7/17
[22, 64]%
+0.57
4–6%
1–17%
Mixed
Gemma-3-4B
16
0/16
[0, 19]%
+0.33
87–96%
85–94%
Sat
Phi-3-mini
20
7/20
[18, 57]%
+0.24
1–17%
9–39%
Mixed
Table 3: Five-model summary (test split). 95% Wilson CIs for win rates. Full per-layer results in Appendix B .
Table 12: SAE training diagnostics (representative layers). Val loss is reconstruction MSE on the validation split. Alive% is the fraction of dsae features firing on at least one validation sample.
Profile
Variant
Raw KL
∣S∣
∣S∣/128
KL/feat
AFR
Low-sharing
BL
0.198
15.2
11.9%
0.0124
15.2%
GI
0.301
30.8
24.0%
0.0114
14.9%
Convergent
BL
0.426
53.1
41.5%
0.0080
28.2%
GI
0.249
77.0
60.1%
0.0033
17.9%
Saturated
BL
1.581
113.6
88.8%
0.0138
32.5%
GI
3.516
114.4
89.3%
0.0303
47.3%
Appendix
Table 13: KL decomposition by sharing profile (test split, all layers). Raw KL = mean cross-language KL divergence. ∣S∣ = mean shared feature count. ∣S∣/128 = shared fraction. KL/feat = primary metric. AFR = autoregressive flip rate.
Figure 5: Val vs. test KL/feature benefit for all 78 observations (descriptive Pearson r=0.894 ; layer observations are nested within models and should not be treated as independent samples).
Scope
Wins
N
Rate
95% Wilson CI
Qwen3-1.7B
11
11
100%
[74%, 100%]
Qwen3-4B
14
14
100%
[78%, 100%]
Llama-3.2-3B
7
17
41%
[22%, 64%]
Gemma-3-4B
0
16
0%
[0%, 19%]
Phi-3-mini
7
20
35%
[18%, 57%]
Low-sharing
18
37
49%
[33%, 64%]
Appendix
Table 14: GI-SAE win rates with 95% Wilson confidence intervals (test split).
Model
Layer
Feature
Langs
Category
Representative snippets
Qwen-4B
L29
F9662
5/6
Boundary
EN: “should be 29 [.] ” / DE: “$29.00 [.] ” / ZH: “29 (yuan) [.] ”
F8519
5/6
Boundary
EN: “answer should be 29 [.] ” / DE: “correct total [.] ”
Table 15: Cross-language shared feature audit. Bracketed tokens mark peak activation positions. All features fire on semantically equivalent content across 4–6 languages.
Model
Layer
BL raw
GI raw
BL norm
GI norm
Stable?
Qwen-4B
L29
0.0087
0.0044
0.0539
0.0040
Yes
Qwen-1.7B
L24
0.0081
0.0026
0.0739
0.0020
Yes
Llama
L17
0.0147
0.0500
0.0139
0.0076
No
Llama
L25
0.0093
0.0049
0.0202
0.0076
Yes
Phi
L3
0.0060
0.0009
0.0117
0.0438
No
Phi
L15
0.0034
0.0193
0.0130
0.0014
No
Appendix
Table 16: KL/feature under native-scale and normalized-space patching on representative layers. Lower is better. Gemma L21 is omitted because this diagnostic script produced no valid normalized-space interventions for that layer; we do not interpret this as evidence about Gemma’s sharing profile. The full main sweep reports Gemma under the native-scale convention.
Large language models (LLMs) exhibit substantial cross-lingual variation in mathematical reasoning performance, but it remains unclear whether these differences reflect language-specific parameters or a shared mechanism that manifests differently by language. We present a cross-lingual mechanistic analysis of mathematical reasoning in LLMs, enabling us to localize and compare model parameters that support mathematical reasoning across languages. We find that the extracted math-associated parameters exhibit partial cross-lingual overlap, with the strongest overlap concentrated in intermediate model layers. We further observe that English consistently produces the largest set of math-relevant parameters, whereas lower-resource languages reveal smaller sets of relevant parameters. These results suggest that math-related behavior in multilingual LLMs is neither fully language-invariant nor fully language-specific, but instead exhibits partial cross-lingual parameter overlap with systematic language-dependent differences.
Behzad Shomali, Luisa Victor, Tim Selbach +5
Lamarr Institute · University of Bonn · Fraunhofer IAIS
Languages encode distinct abstractions and inductive priors, yet most large language models (LLMs) overlook this diversity by reasoning in a single dominant language. In this work, we introduce x1, a family of reasoning models that can adaptively reason in an advantageous language on a per-instance basis. To isolate the effect of reasoning-language choice, x1 is constructed without expanding the model's knowledge boundaries and is trained by contrasting linguistically distinct reasoning trajectories for the same input. Our extensive experiments demonstrate the benefits of adaptive multilingual reasoning across multilingual mathematical reasoning and culturally grounded tasks. Moreover, our results challenge a simplistic view of scaling laws: while scaling reduces cross-lingual disparities in procedural domains such as math reasoning, it does not eliminate the advantages of culture-associated languages in culturally grounded tasks, as we empirically show that such reasoning enables more efficient and accurate cultural knowledge recall. Overall, our findings establish language choice as a functional component of reasoning, with implications for building more generalist and globally competent reasoning models.
Yangfan Ye, Xiaocheng Feng, Xiachong Feng +8
Harbin Institute of Technology · Peng Cheng Laboratory · The University of Hong Kong +1
Large Reasoning Models (LRMs) still exhibit large performance gaps between English and other languages, yet much current work assumes these gaps can be closed simply by making reasoning in every language resemble English reasoning. This work challenges this assumption by asking instead: what actually characterizes successful reasoning traces in multilingual settings, and to what extent do English-derived reasoning features genuinely help in other languages? We first define a suite of measurable reasoning features spanning multilingual alignment, reasoning step, and reasoning flow aspects of reasoning traces, and use logistic regression to quantify how each feature associates with final answer accuracy. We further train sparse autoencoders over multilingual traces to automatically discover latent reasoning concepts that instantiate or extend these features. Finally, we use the features to re-rank traces and measure their impact on accuracy at test time. Across two mathematical reasoning benchmarks, four LRMs, and ten languages, we find that most features are positively associated with accuracy, but the strength of association varies considerably across languages and can even reverse in some. Our findings challenge English-centric reward designs and point toward adaptive objectives that accommodate language-specific reasoning patterns, with concrete implications for multilingual benchmark and reward design.