Large language models (LLMs) have reached expert-level performance on competition mathematics largely through the volume of search placed around them: candidate solutions are sampled in quantity and retained only when an external criterion accepts them. Such a procedure improves the outcome that survives it while leaving untouched what the model represents. We examine that question where no external criterion exists: translating statements between the dialects of neighbouring subfields, where fidelity turns on the level of generality at which content is asserted. The source leaves that level implicit in its vocabulary, so a faithful translation must recover it from the relation between the theories. We introduce an instrument that codes truth, content and scope in separate blind queues, with a judge-free measure of whether a rewrite states the hypothesis implicit in its source, and establish its sensitivity with a planted-positive control. Across seven models from four families, translating towards the general framing widens the domain of quantification in 60.6% of rewrites and narrows it in none; translating towards the concrete framing narrows it in 28.3% and widens it in 0.3%. The hypothesis that would prevent it is stated in 21.6% of model rewrites and 4.2% of human statements. Capability does not govern the asymmetry: it appears in every model tested, and the most capable widens least. It replicates on the half of the benchmark held out by a pre-registered rule, and on statements written by mathematicians. Instructing a model to state every hypothesis it requires raises that rate but not its sensitivity to direction. We argue that these systems have acquired an object-level correspondence between subfield vocabularies without the constraint under which a translation between theories carries hypotheses to hypotheses.
Figures & tables
Figure 1: The operation a cross-framing rewrite must perform, and why its failure escapes single-scale evaluation. (A) “Vector space” means “module over a field”: the field hypothesis is encoded in the noun, so a rewrite into module theory must promote it to an explicit statement, and the reverse rewrite must absorb it into the vocabulary. We call this hypothesis the pin . (B) The same surface rewrite, V↦M , applied to two corpus items. Above, the pin is not promoted and the result is false, since Z/2Z is a Z -module without a basis (the repair is shown in red). Below, no pin is required, because idempotents split over every ring, and the identical transformation yields a theorem. The outcome is determined by the mathematics rather than by anything the model varied.
Approach
Setting
Certified or measured
Scope
Verifier-guided search ( Cobbe et al., 2021 )
answer or proof
outcome correctness
no
BEq + ( Poiroux et al., 2025 )
formal (Lean)
derivability; sound, incomplete
no
driftbench ( Mohammad & Sheikh, 2026 )
formal (Lean)
typed drift categories
partial
MELD ( Ye et al., 2026 )
informal prose
cross-dialect retrieval
no
Our work
informal prose
truth, content, scope
yes
Table 1: Closest lines of work: the setting each evaluates, what it certifies or measures, and whether implicit scope , the level of generality at which a statement is asserted, is visible to it. driftbench is marked partial because it types hypothesis omission and conclusion generalisation, but only once a formal target exists.
Direction
=
wider ( sup )
narrower ( sub )
incomparable
abstracting (concrete → general)
36.5
60.6
0.0
2.9
specialising (general → concrete)
70.7
0.3
28.3
0.6
Half
vs → mod
prob → meas
set → cat
pooled
registered (this harness)
+100.0⋆
+46.7†
+100.0⋆
+82.2⋆
held out
+60.0(0.004)‡
+73.3(0.001)⋆
+80.0(0.0005)⋆
+71.1⋆
Table 2: The direction effect. Top: judged scope relative to the source, by direction, as a share of all rewrites, pooled over seven models on the held-out items ( n=315 , 314 codings) (Table 5 reports a separate two-model pool, not commensurable with this one). Bottom: Claude Opus 5 alone, the same effect in percentage points (widening when abstracting minus when specialising), item level, exact sign-flip permutation, 15 items per pair, with the registered half re-run in the same harness as a control. †p<0.05 , ‡p<0.01 , ⋆p<0.001 .
Model
abs.
spec.
Δ (pp) †
Claude Opus 5
46.7 ( 92.0 ⋆ )
0.0
+45.5
Claude Sonnet 5
73.3
0.0
+73.3
Claude Haiku 4.5
53.3
0.0
+53.3
GPT-5.6 Sol
68.9 ( 92.0 ⋆ )
0.0
+68.9
GPT-5.6 Luna
53.3
0.0
+53.3
DeepSeek V4 Pro
62.2 ( 96.0 ⋆ )
0.0
+62.2
Table 3: The direction effect across models and corpora. Widening when abstracting and when specialising. Upper block: the 45 held-out benchmark items, k=1 , seven models from four families. Parenthesised underlined values are the same model on the 25 human-authored ProofWiki items, where specialising widens 0.0% throughout. Lower block: the ProofWiki pair breakdown, item counts in brackets. †p<10−4 , ‡p<0.01 , ⋆p<0.001 .
dropping the pin makes it…
scope kept
false ( lb_truth )
6/12 (50.0)
ill-formed ( ill_posed )
1/4 (25.0)
true of other objects ( lb_ref )
1/8 (12.5)
unchanged ( inert , none)
0/21 (0.0)
p , truth-load vs. rest
0.0024
Table 4: Scope preservation by the cost of losing it. Claude Opus 5, abstracting direction, the 45 registered vertical items, current codebook; load from the adjudicated annotation. Recomputed for this revision.
axis
framing pair
n
ΔH↓
ΔC↓
Δ∀↓
ΔT↓
ΔR↓
ΔI↓
vertical
vector spaces ↔ modules
64
3.1
57.8
0.0
56.2
0.0
0.0
vertical
probability ↔ measure
74
1.4
18.9
4.1
58.1
0.0
0.0
vertical
set theory ↔ categories
61
8.2
36.1
1.6
55.7
0.0
1.6
lateral
geometry ↔ comm. algebra
74
8.1
25.7
4.1
14.9
29.7
1.4
lateral
graph theory ↔ linear algebra
98
6.1
4.1
4.1
9.2
0.0
9.2
functorial
topology ↔ algebra
95
16.8
25.3
2.1
61.1
0.0
0.0
Table 5: Failures categories. Every failing candidate in the common-judge pool, 466 of 1,188 ( 594 tasks × two models, conditions A and B ). Cells are percentages of that row’s failures , not of rewrites; labels are not exclusive and 35.0% of failures carry none. Failure rates over the pool: 30.7% vertical, 47.8% lateral, 52.8% functorial. ΔH hypothesis omission, ΔC conclusion generalisation, Δ∀ quantifier inversion, ΔT type coercion, ΔR referent drift, ΔI illocutionary drift.
Check
Result
What it establishes
Planted control, 28 planted / 28 real
28/28 sup , 0/28 sup
the near-empty cell is the models,
(Fisher p=2.6×10−16 )
not the code
Regex vs. judge, 315 abstracting
2.1% vs 85.9% widened
two independent instruments
(Fisher p=1.1×10−49 )
separate the same candidates
Second judge family, 180 candidates
κ=1.000
no detectable self-preference
Blind expert reading, 60 candidates
κ=0.900 (95% CI [0.77,1.00] )
the judged code matches a human
Table 6: Validation of the instrument. Each row was fixed before the analysis it supports; populations differ by row and are named. The convergence figure is computed on three populations, of which the seven-model one is quoted (Appendix I ).
promote ↑
added sentence
VM
PM
SC
all
absorb ↓
dsi ↑
b / c ( p )
vs. A
E
be concise
0/15
4/10
3/12
7/37
18/37
−29.7
4 / 15 ( 0.019 )
1.00
C
style will not be judged
0/15
0/10
0/12
0/37
19/37
−51.4
0 / 19 ( 3.8×10−6 )
5.1×10−3
A
bare prompt
0/15
4/10
4/12
8/37
19/37
−29.7
5 / 16 ( 0.027 )
D
style will be judged
0/15
6/10
2/12
8/37
17/37
−24.3
6 / 15 ( 0.078 )
1.00
B
state every hypothesis
15/15
6/10
12/12
33/37
25/37
+21.6
12 / 4 ( 0.077 )
4.6×10−9
Table 7: The instruction ladder. Claude Opus 5, 45 vertical items, both directions, k=1 , judge-free. Each of the four conditions B – E adds one sentence to the bare prompt A ; only B carries mathematical content. Promotion per pair (VM, PM, SC) and pooled over 37 scorable items. b / c : items carrying the hypothesis only when abstracting or only when specialising, exact McNemar. Protocol in Appendix D .
chars
markers
source statements
129
0.01
A, control axis
226
0.30
B, control axis
927 ( +311% )
1.83 ( +511% )
A, vertical axis
224
0.39
B, vertical axis
778 ( +247% )
3.26 ( +737% )
Table 8: Instruction consequences. Mean output length and hypothesis-introducing constructions per output, by condition and axis. The control axis has no hypothesis to promote.
second-family judge
orig.
rep.
95 blinded candidates
= / sup / sub / neq
68 / 7 / 13 / 7
flag =1 on non- =
27/27
—
scope_ok =1
95/95
—
11 anchors
agreement, scope ↑
9/11
9/11
Table 9: The judged scope code under two codebooks. A second-family judge reading the original codebook and the repaired one. Upper block: 95 blinded candidates, with the within-family figure the same quantity across three judges of the rewriter’s own family. Lower block: the eleven published anchors, five of them known silent widenings and one an announced widening. Appendix C .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
second-family judge
original
repaired
flag =1 on non- = codes
27/27
n/a
anchor agreement, scope_ok ↑
5/11
11/11
anchor agreement, flag ↑
5/11
9/11
anchor agreement, scope ↑
9/11
9/11
false accepts (5 silent) ↓
5/5
0/5
reliability
α↑
n
Appendix
Table 10: The judged scope variable across judge families. Top: a second-family judge under the original and repaired flag codebooks, against eleven published m1 anchors (five silent widenings). Bottom: reliability. Full tables in Appendix G .
Axis
Model
n
truth
content equiv.
widened
scope kept
vertical, abstracting
Claude Opus 5
135
94.8
92.6
64.4
46.7
vertical, abstracting
GPT-5.6 Sol
135
98.5
97.8
41.5
63.7
vertical, specialising
Claude Opus 5
135
100.0
95.6
0.0
77.0
vertical, specialising
GPT-5.6 Sol
135
100.0
98.5
0.7
81.5
lateral, A → B
Claude Opus 5
90
53.3
82.2
3.3
65.6
lateral, A → B
GPT-5.6 Sol
90
90.0
75.6
0.0
84.4
Appendix
Table 11: Failure by axis and direction (earlier two-model run, original codebook, k=3 ). A → B on the vertical axis is abstracting. Percentages of codings.
m1 (Claude Opus 5)
m2 (GPT-5.6 Sol)
axis
direction
n
truth ↑
equiv. ↑
scope ↑
truth ↑
equiv. ↑
scope ↑
vertical
abstracting (promote)
135 / 135
98.5
99.3
59.3
98.5
98.5
63.7
vertical
specialising (absorb)
135 / 135
100
97.8
95.6
100
98.5
81.5
lateral
A → B
90 / 30
94.4
95.6
95.6
90.0
66.7
90.0
lateral
B → A
90 / 30
95.6
95.6
100
93.3
76.7
93.3
functorial
A → B
45 / 15
100
95.6
88.9
66.7
86.7
80.0
Appendix
Table 12: Truth, content and scope under the bare prompt : percentage of rewrites coded true, content-equivalent on the shared domain, and scope-preserving ( scope_ok ). m1 is coded by the adjudicated majority of three judges and m2 by one judge under the repaired codebook (§ C.1 ); vertical cells pool k=3 samples, and m2 ’s control cells use one. n : m1 / m2 .
canonical coding
common judge
reference
framing pair, abstracting
m1 ↑
m2 ↑
p
m1 ↑
m2 ↑
MELD gold ↑
vector spaces → modules
73.3
33.3
3×10−4
33.3
33.3
96.7
probability → measure
80.0
64.4
0.16
66.7
66.7
43.3
set theory → categories
24.4
93.3
1.2×10−11
40.0
91.1
83.3
pooled
59.3
63.7
0.53
74.4
Appendix
Table 13: Abstracting scope preservation by framing pair (%). Canonical : each model’s registered coding, with a Fisher exact test. Common judge : both models’ rewrites of byte-identical tasks re-coded blind to generator by one judge. MELD gold : reference translations ( n=30 per pair) under m2 ’s judge.
dropping the pin makes it…
m1 ↑
m2 ↑
false ( truth )
21/24 (87.5)
26/27 (96.3)
ill-formed
n/a
31/36 (86.1)
true of other objects ( ref. )
12/30 (40.0)
0 2/6 0 (33.3)
unchanged ( inert , none)
47/81 (58.0)
27/66 (40.9)
p , truth vs. rest
0.0024
2.9×10−5
Appendix
Table 14: Abstracting scope preservation by pin load. Each model is scored under its own, independently produced load annotation ( m1 ’s has no ill-formed category); p : Fisher exact test, truth against the rest.
m1 Claude Opus 5
m2 GPT-5.6 Sol
native prose
framing pair
promote ↑
absorb ↓
dsi ↑
promote ↑
absorb ↓
dsi ↑
general
concrete
vector spaces ↔ modules
4.4%
82.2%
−77.8
17.8%
28.9%
−11.1
30/30
0/30
probability ↔ measure
20.0%
40.0%
−20.0
30.0%
26.7%
+3.3
10/30
1/30
set theory ↔ categories
33.3%
11.1%
+22.2
80.6%
13.9%
+66.7
25/30
0/30
pooled
18.0%
47.7%
−29.7
41.4%
23.4%
+18.0
Appendix
Table 15: Pin presence in both directions (judge-free, k=3 ): percentage of rewrites containing the registered pin. Promotion requires the pin ( 100% is correct), absorption its removal ( 0% is correct; a retained pin is redundant, not false); dsi in points. The item whose abstracting source contains the pin is excluded. Right: pin frequency in native prose over all 270 MELD pairs.
We present a new approach for benchmarking Large Language Model (LLM) capabilities on research-level mathematics. Existing benchmarks largely rely on static, hand-curated sets of contest or textbook-style problems as proxies for mathematical research. Instead, we establish an updatable benchmark evaluating models directly on the latest research results in mathematics. This consists of an automatic pipeline that extracts lemmas from arXiv and rewrites them into self-contained statements by making all assumptions and required definitions explicit. It results in a benchmark that can be updated regularly with new problems taken directly from human mathematical research, while previous instances can be used for training without compromising future evaluations. We benchmark current state-of-the-art LLMs, which obtain around 10-15% accuracy in theorem proving (pass@1) depending on the model, showing that there is currently a large margin of progression for LLMs to reach human-level proving capabilities in a research context.
Antoine Peyronnet, Fabian Gloeckle, Amaury Hayat
1Ecole Normale Superieure de Rennes · 2Ecole des Ponts Paris · 3Korean Institute for Advanced Study
Recent advances in Reinforcement Post-Training (RPT) have substantially improved the mathematical reasoning capabilities of large language models (LLMs), but it remains unclear how well such gains transfer across languages. In this work, we study cross-lingual mathematical reasoning transfer in English-centric LLMs under controlled multilingual evaluation. We first evaluate a range of English-centric reasoning models on multilingual mathematical reasoning benchmarks and find that transfer is highly variable across initial model choice, target language, and training paradigm. Through controlled comparative experiments, we further find that stronger English-centric initialization does not necessarily lead to stronger relative transfer efficiency, even when it yields higher multilingual reasoning accuracy. Finally, we show that the largest improvement comes from moving beyond English-only supervision: adding a single parallel language yields a substantial gain, while further gains from additional languages are smaller but consistent. Overall, our results highlight the need to evaluate multilingual mathematical reasoning directly, using both absolute performance and relative transfer-efficiency metrics.
Wen Yang, Junhong Wu, Chong Li +2
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research
When selecting mathematical training data for LLMs, a natural organizing principle is topic: probability examples for probability targets. An alternative is reasoning approach: worked solutions that share a solution method with the target, even when the mathematical domain differs. We ask which relation produces greater transfer after fine-tuning. We evaluate two counterbalanced 2×2 designs: probability and combinatorics crossed with invariant reasoning and double counting (2,000 problems), and number theory and geometry crossed with complement and pigeonhole reasoning (800 problems). In each design, every cell serves as the held-out target in turn: same-approach (SA) sources share the target's method but change the topic, while same-topic (ST) sources share the topic but change the method. Every source appears once in each role, so additive source-quality effects cancel from the equally weighted aggregate contrast. Across five base models and three training seeds per design, SA outperforms ST in all 40 seed-pooled model--target comparisons. Model-level advantages range from 8.2 to 16.2 percentage points in the primary design (mean: 10.8) and from 12.0 to 16.0 in the second design (mean: 14.3); all ten model-level 95% confidence intervals exclude zero. In both designs, ST sources are more similar to targets under embedding and lexical measures, so the SA advantage runs opposite to the measured ordering of statement-level resemblance. These findings identify reasoning approach as a more effective matching criterion than topic for mathematical transfer across the evaluated topic--approach combinations.