Rethinking Pivot Programming Languages in Code Language Models

Andor Diera; Lukas Galke Poech; Matthias Tichy

cs.CLSep 19, 2026

Rethinking Pivot Programming Languages in Code Language Models

Authors: Andor Diera, Lukas Galke Poech, Matthias Tichy

Organizations: Ulm University Ulm, Germany · University of Southern Denmark Odense, Denmark

Abstract

Multilingual code language models transfer skills across programming languages (PLs), but whether any PL occupies a privileged pivot position remains contested: geometric analyses point to C-family languages and Go, while behavioral evidence highlights Python. We revisit this question under controls for representational anisotropy and length variation across PLs, two confounds that compromise prior cosine-based analyses. Across three code models on multilingual competitive-programming data, we study three views of cross-PL organization: pairwise PL geometry, PL-English alignment, and pivoted retrieval through candidate PL representation spaces. The results are relation-dependent. Code-code geometry reveals structured language regions but no universal center; code-English alignment favors high-level scripting languages; and pivoted retrieval favors different intermediate spaces for code-to-code and English-to-code transfer. These findings suggest that Python's special role is better understood as English-facing affinity than as universal geometric centrality.

Figures & tables

Figure 1: Pivoted retrieval split by transfer type. Each point is a candidate programming-language pivot, positioned by its mean within-task rank for code-to-code retrieval and English-to-code retrieval averaged across 3 code-LMs. For visualization, mean within-task pivot ranks are transformed into pivot scores, with higher values indicating better average rank.

Figure 2: Box-and-whisker plot of character lengths by language. English refers to problem descriptions, while all other languages refer to the source-code solutions. The shaded region indicates the 200–800 character matching window used in subsequent analyses.

Model	Code vs code	Code vs English
StarEncoder	0.31–0.40	0.01–0.23
CodeLlama	0.33–0.47	0.29–0.88
Qwen2.5-Coder	0.24–0.49	0.21–0.96

Table 2: Spearman correlation

|\rho|

min-max ranges between cosine-based alignment measures and code length, across post-embedding layers per model. All correlations are statistically significant (

p<0.05

) except StarEncoder’s code-vs-English.

Figure 3: Pairwise code–code geometry at a fixed normalized depth of $0.5$ across models. Each heatmap reports cosine similarity between mean-centered programming-language centroid directions, computed on the length-matched code subset. The resulting geometry is structured but model-dependent, with no single programming language consistently central across models.

Figure 4: Code–English alignment ranks at normalized depth 0.5. Model columns rank length-matched, mean-centered cosine similarity between each programming language and English problem descriptions. The Lexical column ranks raw Jaccard token overlap. Lower rank = closer to English. High-level scripting-like languages align most closely with English, a pattern not explained by lexical overlap alone.

Setting	Contrast	Estimate	95% CI
Code $\rightarrow$ Code	Top-six spread	0.46	[0.38, 1.43]
Code $\rightarrow$ Code	Tier–remainder gap	1.07	[0.99, 1.15]
English $\rightarrow$ Code	Top-two spread	0.80	[0.14, 1.28]
English $\rightarrow$ Code	Tier–remainder gap	3.29	[1.02, 4.54]

Table 4: Bootstrap contrasts for pivoted retrieval. The Code

\rightarrow

Code tier comprises PHP, C++, JavaScript, Ruby, Java, and Go; the English

\rightarrow

Code tier comprises Ruby and Python. Spread denotes the maximum minus minimum mean rank within a tier. The gap is the remainder’s average rank minus the tier’s average rank.

Setting	Qwen-base	Qwen-Instruct	Gain
Code $\rightarrow$ Code	0.263	0.297	+12.9%
English $\rightarrow$ Code	0.132	0.164	+24.7%

Table 5: Instruction-tuning robustness for pivoted retrieval. Values report mean MRR over matched source–target–pivot tasks and 10 subsampling seeds.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Layer-wise anisotropy across the three code language models. Left: mean cosine similarity between randomly paired code representations. Right: mean-norm ratio. Both measures remain high across depth, indicating strong representational anisotropy in all models.

Figure 6: Minimum per-language sample count as a function of the upper bound, for several choices of lower bound.

Lang	$N_{\text{full}}$	$N_{\text{win}}$	frac.	$\bar{\ell}_{\text{full}}$	$\bar{\ell}_{\text{win}}$	$\sigma_{\text{win}}$
Python	959	366	0.38	279	384	164
Ruby	369	166	0.45	332	386	159
C++	961	722	0.75	364	371	143
C	892	616	0.69	605	414	157
Java	922	617	0.67	825	500	149
PHP	454	293	0.65	881	447	164

Appendix

Table 6: Per-language statistics within the [200, 800] matching window.

N_{\text{full}}

: total solutions.

N_{\text{win}}

: solutions in window. frac.: fraction retained.

\bar{\ell}

: mean length (chars).

\sigma_{\text{win}}

: standard deviation of length within window.

Figure 7: Layer-wise Spearman

|\rho|

between pairwise centroid cosine and pairwise length difference (code-vs-code), across all layers per model, plotted on normalized depth (embedding layer at depth

0

; final layer at depth

1

Figure 8: Layer-wise Spearman

|\rho|

between per-language pointwise cosine and mean code length (code-vs-English), across all layers per model, plotted on normalized depth (embedding layer at depth

0

; final layer at depth

1

Figure 9: Layer-wise probing performance across models. The left panel reports semantic-tag prediction using micro-F1, and the right panel reports programming-language identification using macro-F1. The dashed vertical line marks normalized depth

0.5

Figure 10: Cross-programming-language semantic contrast by layer. Higher values indicate stronger problem-level semantic correspondence across programming languages. The dashed vertical line marks normalized depth

0.5

, the representative layer used in the main geometric analyses.

Figure 11: Per-model code–English cosine distance at fixed normalized depth 0.5. Bars show

1-\cos

, where cosine is the mean random-pair similarity between a programming language and English problem descriptions after length matching and mean-centering. Lower values indicate closer alignment to English. Error bars show standard deviation across 200 random-pairing rounds.

Figure 12: Code–English alignment ranks with a filtered token-overlap baseline. Model columns are identical to Figure 4 . The Lexical column ranks Jaccard overlap between code tokens and English-description tokens after removing common English stopwords and programming-language keywords. Lower rank indicates closer representation-level alignment or greater lexical overlap with English.

PCA dim.	Top code-to-code pivots by mean rank
16	Ruby, C++, PHP, JavaScript
32	Ruby, C++, Python, JavaScript
64	Ruby, C++, PHP, JavaScript
96	Ruby, PHP, C++, JavaScript
106	PHP, C++, JavaScript, Ruby
128	PHP, JavaScript, Java, C++

Appendix

Table 7: Top code-to-code candidate pivots across PCA dimensions, sorted by mean within-task pivot rank. The exact top pivot varies with PCA dimension, but the same broad bridge tier remains competitive.

PCA dim.	Top English-to-code pivots by mean rank
16	Ruby, Python, JavaScript
32	Ruby, Python, JavaScript
64	Ruby, Python, JavaScript
96	Ruby, Python, JavaScript
106	Ruby, Python, JavaScript
128	Ruby, Python, JavaScript

Appendix

Table 8: Top English-to-code candidate pivots across PCA dimensions, sorted by mean within-task pivot rank. Ruby and Python remain the top two pivots throughout the sweep, showing that the English-facing result is stable under PCA dimensionality variation.

Figure 13: No-pivot code $\rightarrow$ code retrieval by source and target language. Entries report mean MRR across models using the same evaluation problems as the pivoted retrieval experiment. Darker blue indicates higher retrieval performance.

Figure 14: No-pivot English

\rightarrow

code retrieval by target language. Bars report mean MRR across models using the same evaluation problems as the pivoted retrieval experiment; error bars denote standard deviation across models.

Explore similar work

What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations
Jul 23, 2026Piotr WilamCircuit Discovery Language Model Introspection
University College London
CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora
Apr 20, 2026Shangyu Li, Juyong Jiang, Meibo Ren +7Code Translation RL for Language Models
The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · Monash University +1
Large Language Models for Multilingual Code Intelligence: A Survey
Apr 27, 2026Chao Jiang, Dugang Liu, Cheng Wen +6Code Translation Code Generation
College of Computer Science and Software Engineering, Shenzhen University, China · Guangzhou Institute of Technology, Xidian University, China · Guangzhou University of Software, China +2

Jul 23, 2026J/K move · Enter open · S save

What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

Piotr Wilam

University College London

Circuit Discovery Language Model Introspection Circuits in Language Models Representation Geometry in Language Models Code Language Models

Open paper AskPDF arXiv

Do independently trained language models come to represent the same thing in the same way? We answer for code, extending a recently introduced concept-circuit extraction method to a 2x2 design -- Python and Rust crossed with Qwen2.5-Coder-7B and DeepSeek-Coder-V1-6.7B -- and measuring a complete inventory of grammatical concepts (58 Python, 57 Rust) identically in all four cells: the smallest design that separates what depends on the task, the language, and the model. The answer splits into three parts. What earns dedicated circuitry is set by the task: the models agree on which concepts receive circuits (Spearman

ρ

= 0.638 for Python, 0.673 for Rust, both p <

10^{-7}

). Where those circuits sit is set by the model: Qwen processes concepts in a late band (~L17-19), DeepSeek at L6-7, for both languages. How circuits grow across layers is also set by the model: Qwen gives its atomic concepts an early spike that DeepSeek does not. "Are circuits universal?" thus has no single answer: yes for What, no for Where and How -- universality is a property of representational content, not of computational organisation. None of this structure was fixed in advance. The agreement could have landed anywhere between independence and identity; it lands at

ρ\approx 0.65

. Rust constructs receive 2-3x more concept-specific circuitry than their Python equivalents, in both models. Both models share neurons between the languages (6/7 and 7/7 paired constructs), DeepSeek 1.94x more than Qwen -- a direction no prior result predicts. And Qwen binds nine keywords of Rust's type-and-trait machinery into one tight neuron cluster (Jaccard 0.535 vs null 0.112, p < 0.001), a semantic dimension invisible in surface syntax. Ablation and linear probes confirm the circuits are functional. All claims are scoped to this 2x2; whether the per-model profile predicts a third model is the designed next test.

Abstract

Explore similar work

What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora

Large Language Models for Multilingual Code Intelligence: A Survey