Recipe-Matching, Not Equivalence

Ali Habibullah; Mohammad Alshiekh; Yazan Alshoibi; Salman Khan; Naeemullah Khan

cs.IRSep 25, 2026

Recipe-Matching, Not Equivalence

Authors: Ali Habibullah, Mohammad Alshiekh, Yazan Alshoibi, Salman Khan, Naeemullah Khan

Organizations: KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · KAUST Academy & CEMSE, KAUST Lady Margaret Hall, University of Oxford

Abstract

MathNet-Retrieve asks a retriever to find, for a math problem, a document stating the same problem. An LLM under one fixed prompt writes each gold document and its near-miss distractors; LLM judges filter them. We call this procedure the "recipe", training on pairs built the same way "recipe-matching", and ask how much score it buys beyond the ability the benchmark claims to test. Two models from one base, matched in rows and settings, differ only in the training file: pairs written under the benchmark's published prompt by another vendor's LLM and judge, or computer-algebra-verified pairs with no LLM anywhere. The first leads by 45 R@1 points on the easy tier. By a non-LLM paraphrase control, half to two thirds of that gap comes from the pairs being LLM-written at all: LLM rewrites under two unrelated prompts, with the verified model's negatives, recover 30 and 22 of the 45 points; back-translations with the same negatives recover almost none. The remaining 15 to 25 points appear only under the benchmark's own prompt and vanish on real duplicates no generator wrote, the same problem in two languages. The hard tier rewards the recipe's pair structure, a deep rewrite against a minimal-edit near-miss: LLM rewrites alone score zero on it, attaching negatives unlocks it, and every negative that does so costs cross-language points; the sets scoring highest on it separate near-misses no LLM wrote worse than LLM rewrites with verified negatives. MELD also moves when a model trains on pairs built its way, without losing retention; on SABER-Math the registered attack fails, and the one gain, from its LLM-written summaries, is small but holds at a matched budget. Only on MathNet-Retrieve could we pin an inversion, benchmark score up and real retention down, to one edit of a training file. We release the generator-free duplicate evaluations, the near-miss test and three trained models.

Figures & tables

Benchmark tiers
Evaluation	Base	Verified	Reference	Recipe	Recipe $-$ verified	Recipe $-$ reference
Easy (R@1)	8.32	17.05 $\pm$ 0.79	46.91 $\pm$ 1.25	62.38 $\pm$ 0.85	45.33 $[44.25,46.41]$	15.47 $[14.11,16.73]$
Medium (R@1)	1.78	3.14 $\pm$ 0.46	4.22 $\pm$ 0.25	8.84 $\pm$ 0.36	5.70 $[5.18,6.24]$	4.62 $[4.13,5.15]$
Hard (R@1)	0.00	0.16 $\pm$ 0.06	10.30 $\pm$ 1.15	9.42 $\pm$ 0.61	9.26 $[8.71,9.85]$	$-0.88$ $[-1.85,0.10]$
Hard (R@5)	4.08	8.35 $\pm$ 1.13	60.05 $\pm$ 1.22	57.20 $\pm$ 0.82	48.84 $[47.95,49.74]$	$-2.86$ $[-4.01,-1.67]$
Real duplicates, no LLM involved

Table 1: The main experiment. Columns: the untrained base, the two matched-budget arms and the recipe-free reference (D4 restatements with the verified arm’s negatives, no benchmark prompt anywhere), each over eight seeds (mean

\pm

std), then the recipe arm’s lead over the verified arm and over the reference with

95\%

nested-bootstrap intervals (Section 3 ). Rows: the benchmark’s own tiers first, then real duplicates that no LLM wrote (cross-language reprints, scored strictly : the query’s own corpus entry is masked and only a reprint in another language counts as a hit; and the

125

-query same-language slice of Appendix A ), then the difference-in-differences (DiD), the benchmark lead minus the real-duplicate lead. Bold marks the largest value in a row; on the duplicate rows it is the untrained base. Seed-

42

breakdown: Table 17 .

Panel of Figure 1 — Figure 1: Benchmark score against real retrieval kept after training, one point per trained model. Horizontal axis: strict R@1 on cross-language duplicates; the dashed line and $\times$ mark the untrained base, so points left of it lost real retrieval by being trained. Colour gives the positives (legend); hollow means no negatives. (a) Easy tier: every model trained on LLM rewrites scores above the verified arm. (b) Hard tier: only deep LLM rewrites with negatives leave the floor. Arrows run from D4 alone to D4 with negatives attached; each label names the negatives (verified ones give the reference; “recipe” and “unrelated” are LLM near-misses from the recipe’s prompt and from an unrelated one, at the reference’s count or at the recipe arm’s full volume) and the cross-language points lost. All models: Figure 2 .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Base model	Qwen3-Embedding-0.6B, bf16, all weights trained
Loss	the in-batch contrastive loss of Eq. 1 , computed with gradient caching [ 33 ]
Batch size / mini-batch size	$256$ / $16$ (the cache makes the gradient identical to a plain batch of $256$ )
Learning rate	$2\times 10^{-5}$
Epochs	$3$
Seeds	$42$ – $49$ (eight per arm)
Training rows per arm	$6{,}145$ , with all rows of one source problem kept together

Appendix

Table 2: Training settings shared by both arms. Every row is identical for the verified arm and the recipe arm; only the training file differs. Two other experiments reuse these settings with stated deviations, applied identically to both arms: the SABER-Math attack of Section 6 caps training sequences at

1{,}024

tokens where the controlled arms use the backbone’s default, and the

4

B replication uses rank-

16

LoRA adapters instead of full fine-tuning, a

1\times 10^{-4}

learning rate instead of

2\times 10^{-5}

, and gradient checkpointing.

Control	Explanation it removes	Easy-tier gap (R@1)
Matched budget, eight seeds (Table 1 )	seed noise	$45.33\pm 0.93$
Queries the original gate kept out of both training files ( $8{,}663$ )	leaked test items	$46.37\pm 0.85$
No copy of the query in either arm’s rows ( $12{,}760$ , strictest)	leaked test items	$44.73\pm 0.86$
Retrain under the corrected filter, $33\%$ fewer rows	near-verbatim copies of queries	$43.11\pm 0.90$
Retrain under the corrected filter, removed third regenerated from fresh sources	near-verbatim copies of queries	$44.45\pm 0.87$
Retrain under the corrected filter, removed third regenerated from the same sources	near-verbatim copies of queries	$44.38\pm 1.46$

Appendix

Table 3: Does the easy-tier gap survive when a possible explanation for it is removed? Each row removes one and reports the recipe arm’s lead over the verified arm on the easy tier (first row: the headline comparison of Table 1 ). Rows marked “intact” change the verified arm only, so they report its score rather than a gap. No

0.6

B control brings the gap below

41

points; the other-backbone and

4

B rows shrink it but keep its sign (Appendix D ). The last two rows are the two recipe-arm variants of Appendix C , each against the eight-seed verified arm.

Query slice	$n$	Base	Verified $-$ base	Recipe $-$ base	D4 $-$ base
All queries	393	84.73	$-28.21$ $[-33.21,-23.15]$	$-18.48$ $[-23.47,-13.68]$	$+9.08$ $[5.85,12.55]$
Pair found through a shared answer	25	88.00	$-28.00$ $[-45.00,-12.50]$	$-35.50$ $[-55.50,-16.00]$	$0.00$ $[-12.00,12.00]$
Pair found through matching text	15	86.67	$-18.33$ $[-37.50,-3.33]$	$-8.33$ $[-23.33,0.00]$	$0.00$ $[0.00,0.00]$
Pair found through formulas only	353	84.42	$-28.65$ $[-33.50,-23.16]$	$-17.71$ $[-22.59,-12.96]$	$+10.10$ $[6.42,13.69]$
Least shared formula content (bottom quartile)	99	76.77	$-29.42$ $[-38.89,-20.20]$	$-20.45$ $[-31.19,-9.47]$	$+15.15$ $[6.06,23.23]$
Most shared formula content (top quartile)	99	93.94	$-23.36$ $[-31.69,-15.40]$	$-12.12$ $[-19.44,-5.18]$	$+3.03$ $[-1.01,7.41]$

Appendix

Table 4: Cross-language retention by query slice. Each row is a subset of the

393

cross-language queries; Base is the untrained model’s strict R@1 on that subset, and the three gap columns give how far the verified arm, the recipe arm and D4 sit from it, with

95\%

nested bootstrap intervals over queries and training seeds (

B=2{,}000

; the arms eight seeds, D4 three). Rows two to four split the queries by how the pair was found, the two quartile rows keep the queries with the least and the most shared rare-formula content, and the last two split by how formula-heavy the query text is. On no slice does either arm reach the base; the recipe arm loses more where formulas are scarce (bottom quartile, formula-light) than where they are plentiful, the reverse of a formula-matching artefact; and D4 falls below the base on no slice.

Model	Primary ( $125$ )	English ( $115$ )	Leaked ( $82$ )	Exact reprints ( $291$ )	Primary gap to base
Base $0.6$ B (untrained)	62.40	59.13	50.00	100.00
Base $+$ V2 instruction	62.40	59.13	52.44	100.00	$0.00$ $[-3.23,3.17]$
BM25 (lexical baseline)	54.40	51.30	32.93	100.00	$-8.00$ $[-17.07,0.79]$
Verified arm ( $8$ seeds)	51.30	47.07	34.91	99.40	$-11.10$ $[-18.90,-3.89]$
Recipe arm ( $8$ seeds)	46.30	41.63	35.82	85.61	$-16.10$ $[-26.37,-6.10]$
D4 (recipe-free, no negatives)	61.07	57.68	44.72	100.00	$-1.33$ $[-7.20,3.73]$

Appendix

Table 5: Same-language reprints: R@1 for retrieving a reprint of the query in the same language, for every model we trained. The four score columns are the four slices: the primary slice (

125

queries from reworded clusters that no training file touches), its English queries, the leaked slice (queries from reworded clusters that some training file does touch), and the exact reprints (near-identical copies that any model should retrieve, a ceiling check). The last column is each model’s gap to the untrained base on the primary slice, with a

95\%

nested bootstrap interval over clusters and seeds. Every model trained with negatives of any kind sits well below the base; only the models trained without negatives, the back-translated model, V2 and the MELD arm stay within noise of it. Seeds: the two arms and the reference eight; the ladder rungs, CAS positives without negatives, the other factorial cells, V2 and the MELD arm three; the rest single runs.

Model	Benchmark hard-tier R@1	Probe R@1, full corpus	Probe R@1, near-misses only
Base $0.6$ B (untrained)	0.00	10.84	10.93
BM25 (lexical baseline)	0.01	72.27	73.8
Verified arm ( $8$ seeds) †	0.16	97.42 $\pm$ 0.32	97.53 $\pm$ 0.33
Recipe arm ( $8$ seeds)	9.42	60.49 $\pm$ 2.50	68.38 $\pm$ 2.40
D2 arm (paraphrase of the benchmark’s prompt)	6.76	62.54 $\pm$ 1.23	69.35 $\pm$ 1.99
D3 arm (different-style prompt)	4.67	68.50 $\pm$ 1.38	72.30 $\pm$ 2.21

Appendix

Table 6: The generator-free near-miss probe:

1{,}107

held-out benchmark queries, each with a computer-algebra-verified positive and counterexampled minimal-edit negatives. Each model’s benchmark hard-tier R@1 is set beside its probe R@1 scored two ways: against the full public corpus plus the probe documents (the hard tier’s shape without a generator) and against the probe documents alone (a pure near-miss test). Means

\pm

sample std over seeds where more than one: the arms and the reference eight, the other cells three, the base one. Rows marked

\dagger

are in-family, since the probe uses their own transformation families (the verified arm, V2 and the three CAS-positive cells); their scores are shown but not read as evidence. Among the other rows, the two with the highest hard-tier scores have the lowest probe scores of D4’s cells, while back-translated positives, near the hard-tier floor, score high on the probe.

Query slice	Queries	Recipe $-$ verified, easy R@1
All queries	$15{,}000$	45.33 $\pm$ 0.93
Queries the original gate did catch (clean, up to $98$ missed copies)	$8{,}761$	46.25 $\pm$ 0.86
No copy of the query in either arm’s training rows (strictest clean)	$12{,}760$	44.73 $\pm$ 0.86
Query text trained on by the recipe arm (leaked)	$1{,}960$	49.80 $\pm$ 1.51
A copy exists in the corpus, but neither arm trained on it	$4{,}018$	41.16 $\pm$ 0.94

Appendix

Table 7: Does the easy-tier gap depend on leaked queries? The

15{,}000

easy-tier queries are split by whether a copy of the query reached training, and each row gives the recipe arm’s lead over the verified arm on that slice (R@1, mean

\pm

sample std over the eight training seeds; slice sizes at seed

42

). Leakage is judged against every row of each arm’s training file; the recipe arm’s rows are the same at every seed, the verified arm’s are redrawn per seed. The gap is present on every slice, including the strictest clean one; the last two rows isolate the raw leakage effect, the gap on queries the recipe arm trained on against the gap on comparable queries neither arm trained on.

Recipe $-$ verified	Headline (original gate, $6{,}145$ rows)	Filtered (corrected gate, $4{,}101$ rows)	Regenerated, fresh sources (corrected gate, $6{,}145$ rows)	Regenerated, same sources (corrected gate, $6{,}145$ rows)
Easy R@1	45.33 $\pm$ 0.93	43.11 $\pm$ 0.90	44.45 $\pm$ 0.87	44.38 $\pm$ 1.46
Hard R@1	9.26 $\pm$ 0.60	9.34 $\pm$ 0.31	9.21 $\pm$ 0.66	9.05 $\pm$ 0.77
Real duplicates (strict R@1)	9.74	11.04	17.40	18.73
Difference-in-differences	35.59	32.07	27.05	25.65

Appendix

Table 8: The arm gap under the original and the corrected contamination gate. Each column is one version of the two-arm comparison, all at the same eight seeds: the headline arms (original gate,

6{,}145

rows); the arms filtered under the corrected gate to the

4{,}101

matched rows it leaves; the arms restored to

6{,}145

rows with the recipe arm’s missing third regenerated from fresh sources; and the arms restored to

6{,}145

rows with that third regenerated from the same sources, a second rewrite per source, against the same verified arm as the fresh-source column. Each row is the recipe arm’s lead over the verified arm on one evaluation. The easy- and hard-tier gaps barely move. In the fresh-source column the two arms no longer share one source list, and only the shared list still contains members of the real-duplicate clusters, so that column’s real-duplicate row, and the difference-in-differences built on it, are not like-for-like with the others; the same-source column restores the shared list.

Training set	Easy R@1	Hard R@1	Cross-language dup. R@1
Base 0.6B (untrained)	8.32	0.00	84.73
Verified arm	17.05 $\pm$ 0.79	0.16 $\pm$ 0.06	56.52 $\pm$ 4.05
D4: restatement prompt unrelated to the recipe	30.03 $\pm$ 1.45	0.04 $\pm$ 0.02	93.81 $\pm$ 0.39
D3: rewrite prompt in a different style	67.88 $\pm$ 0.76	4.67 $\pm$ 0.40	65.39 $\pm$ 1.83
D2: paraphrase of the benchmark’s prompt	67.54 $\pm$ 1.95	6.76 $\pm$ 1.01	68.02 $\pm$ 3.51
D1: the benchmark’s prompt ( $=$ recipe arm)	61.86 $\pm$ 0.52	9.90 $\pm$ 0.74	66.41 $\pm$ 1.78

Appendix

Table 9: The rewriter-prompt ladder. Rows run from the prompt farthest from the benchmark’s (D4) to the benchmark’s own (D1), with the untrained base and the verified arm above for reference; columns are R@1 on the easy tier, the hard tier and cross-language duplicates, mean

\pm

sd over the same three seeds (

42

–

44

). The hard-tier column falls step by step from D1 to D4, the registered prediction that passed; the easy-tier column does not, since D2 and D3 outscore D1 and only D4 drops; on duplicates D4 tops every model including the base. Three notes: D1 is the recipe arm read at three seeds, so its numbers differ slightly from the eight-seed ones of Table 1 (

62.38\pm 0.85

9.42\pm 0.61

); D3 trained on the

5{,}591

positives its prompt yielded rather than the matched

6{,}145

, which a control shows flatters it on the hard tier; D4 attaches no negatives by design, and Table 12 separates that from its prompt. On real duplicates only the contrasts between D4 and the rest replicate across seeds. The points are plotted in Figure 1 .

recipe	how a benchmark builds itself: the prompts and filters under which an LLM writes part of it
template	the exact wording of that prompt
genre	the style of LLM-written rewrites, whichever prompt produced them
pair structure	a deep LLM rewrite as the positive against a minimal-edit near-miss as the negative
surface	the part of a benchmark its LLM writes, the part an attacker can imitate: documents, summaries or evaluation pairs
reference	the recipe-free comparison model: LLM restatements written under a prompt unrelated to the recipe (D4), paired with the verified arm’s computer-algebra negatives; no benchmark prompt anywhere in its training file

Appendix

Table 10: The five terms the paper relies on, and the reference model that Table 1 compares against.

	Negatives
Positives	none	computer-algebra counterexamples	LLM near-misses, unrelated prompt	LLM near-misses, recipe prompt
Computer-algebra near-copies	0.00 / 52.08	0.16 / 56.52 verified	0.23 / 13.48	0.11 / 12.47
Back-translated paraphrases	0.05 / 94.57	1.09 / 84.57	1.88 / 35.20	4.18 / 38.67
LLM restatements, unrelated prompt (D4)	0.04 / 93.81 D4	10.30 / 78.02 reference	16.96 / 85.50; 20.93 / 84.91	20.71 / 83.89; 26.97 / 78.88
LLM rewrites, recipe prompt (D1)	0.00 / 83.21	2.09 / 70.82	3.45 / 47.41	9.42 / 66.26 recipe

Appendix

Table 11: Every training set in the paper, arranged by what its positives are (rows) and what its negatives are (columns). Each cell is one trained model and gives two numbers: R@1 on the benchmark’s hard tier, then strict R@1 on real cross-language duplicates. Two cells show two pairs because that set was trained twice, first with as many negatives as the reference has and then with as many as the recipe arm has; every other LLM-near-miss cell uses the recipe arm’s volume, and every computer-algebra-negative cell the reference’s negatives. The two arms and the reference have eight seeds, every other cell three (Appendix C ). The four models the paper names are in italic.

The $2\times 2$ : CAS or D4 positives, with or without CAS counterexamples
Positives	Negatives	Easy R@1	Hard R@1	Cross-language dup. R@1
CAS (verified)	CAS counterexamples ( $=$ verified arm)	17.05 $\pm$ 0.79	0.16 $\pm$ 0.06	56.52 $\pm$ 4.05
CAS (verified)	none	7.42 $\pm$ 0.24	0.00 $\pm$ 0.00	52.08 $\pm$ 1.06
D4 (recipe-free)	none ( $=$ D4)	30.03 $\pm$ 1.45	0.04 $\pm$ 0.02	93.81 $\pm$ 0.39
D4 (recipe-free)	CAS counterexamples ( $=$ reference)	46.91 $\pm$ 1.25	10.30 $\pm$ 1.15	78.02 $\pm$ 0.91
D4 positives with LLM-written near-misses (matched: the reference’s count; full: the recipe arm’s volume)

Appendix

Table 12: Every training set of the factorial, a kind of positive crossed with a kind of negative: R@1 on the easy and hard tiers and strict R@1 on cross-language duplicates, mean

\pm

sd over training seeds (the verified arm and the reference eight, every other cell three; the D4 row is the three-seed mean of Table 9 ). The group labels say what each block varies. Three readings: neither half of a pair scores on the hard tier on its own, and near-copy or back-translated positives barely do even with negatives; D4’s restatements with any negatives leave the floor, matching the recipe arm (

9.42\pm 0.61

over eight seeds, Table 1 ) with no recipe wording in the file and passing it with LLM-written near-misses; and every kind of negative costs D4’s restatements cross-language points from their

93.81

, except an unrelated prompt’s near-misses, which leave them at the untrained base (

84.73

). The last block fills the grid: positives of any kind trained alone never register on the hard tier, and paraphrase positives alone, LLM-written or back-translated, sit at the top on cross-language duplicates (

94.57

for back-translations, within noise of D4’s

93.81

); near-copy or back-translated positives with LLM near-misses of either prompt stay near the floor and lose most of their cross-language retention; and D1’s own positives with the reference’s negatives reach a fifth of the reference’s hard-tier score.

Panel of Figure 2 — Figure 2: The full plot behind Figure 1 : every $0.6$ B model of Tables 1 , 9 , 12 and 19 , with the corrected-gate retrains and V2’s no-instruction reading omitted. Same axes and conventions as Figure 1 : horizontal, strict R@1 on cross-language duplicates, the dashed line and $\times$ the untrained base; colour, what the positives are; hollow, no negatives; the labels on the D4 cells name the negatives attached (“recipe” and “unrelated” are LLM near-misses from the recipe’s prompt and from an unrelated one, “full” at the recipe arm’s negative volume). Two groups appear here only: the verified squares also include computer-algebra positives with no negatives (CAS alone) and the uncapped verified arm, and the crossed circles are the mixed-data, soup and V2 models of Appendix E , which sit near the hard-tier floor. The MELD-recipe arm is seed $42$ ( $20.23$ easy, $0.11$ hard, $90.84$ real; Section 6 ). (a) Easy tier: every model trained on LLM rewrites scores above the verified ones. (b) Hard tier: only deep LLM rewrites paired with negatives reach it; the arrow is the edit from D4 alone to the reference, attaching verified negatives, which lifts hard R@1 off the floor and costs $15.79$ retention points, and LLM-written near-misses lift the same restatements further (Table 12 ). Computer-algebra positives with the recipe’s near-misses lie off the axis at $12.47$ real.

Rewrite prompt	Held-out positives	AUC vs. organic	$95\%$ CI
D1: the benchmark’s Appendix-F template	$1{,}282$	0.8802	$[0.8679,0.8917]$
D2: paraphrase of that template	$1{,}249$	0.8756	$[0.8624,0.8873]$
D3: rewrite prompt in a different style	$1{,}096$	0.8654	$[0.8520,0.8788]$
D4: restatement prompt unrelated to the recipe	$1{,}265$	0.6758	$[0.6569,0.6948]$

Appendix

Table 13: What the style detector detects. Each row takes one prompt’s rewrites (held-out sources only) and gives the AUC with which the detector separates them from the same

1{,}282

held-out organic problems, with

95\%

CIs from

2{,}000

bootstrap replicates over held-out source ids. The three recipe-family prompts D1–D3 are detected at about the same level and cannot be told apart within their intervals; the recipe-unrelated prompt D4 is detected far less strongly but still well above chance. The detector therefore mostly picks up the genre of LLM rewrites, which D4 shares, plus a recipe increment of

0.204

[0.186,0.222]

that only the recipe-family prompts add.

Model	$\rho$ (similarity, style)	Top- $10$ style percentile
Base 0.6B (untrained)	$-0.061$	44.8
Verified arm	$-0.019$	45.5
D4: restatement prompt unrelated to the recipe	$-0.000$	48.1
Reference (D4 $+$ CAS negatives)	$+0.025$	52.0
D4 $+$ recipe-prompt near-misses, count-matched	$+0.046$	53.1
D4 $+$ recipe-prompt near-misses, full	$+0.050$	53.5

Appendix

Table 14: Style tracking: does a model rate LLM-styled documents as more similar? For each of

2{,}000

sampled queries against the full

117{,}088

-document corpus (single runs at seed

42

), the first column is the Spearman correlation between the model’s similarity to each document and the artefact-stripped detector’s style score for it, averaged over queries; positive means similarity rises with LLM style. The second column is where the model’s top-

10

documents sit in the corpus’s style distribution, as a percentile, where

50

means no preference (we report percentiles because the detector’s raw score is an unbounded log-odds whose ratios have no scale meaning). The base and the verified arm show no style preference, D4’s restatements alone none, and D4 with negatives of either kind only a faint one; the recipe-family prompts track clearly, in ladder order, with the exact template strongest.

Model	$\rho$ (all documents)	Top- $10$ style percentile
Public models, never trained by us
Qwen3-Embedding-8B	$-0.0705$	45.21
Qwen3-Embedding-0.6B (the untrained base)	$-0.0610$	44.83
Qwen3-Embedding-4B	$-0.0548$	45.61
MathLeap-Qwen-8B	$-0.0214$	47.04
MathLeap-Octen-8B	$-0.0091$	47.94

Appendix

Table 15: Do public models show the recipe-style signature? The same two measures as Table 14 :

\rho

is the mean per-query Spearman correlation between a model’s query–document similarity and the document’s LLM-style score, and the last column is where the model’s top-

10

documents sit in the corpus’s style distribution, as a percentile, with

50

meaning no preference. Every public model sits in the same narrow band as the untrained base and our verified arm, between

-0.07

and

+0.004

with top-

10

percentiles of

45

48

; our recipe-family models sit far away, at

+0.10

+0.17

and

57

62

. No public model approaches the recipe arm.

Model	Hard-tier R@1	Cross-language duplicates, strict R@1 / R@5 / R@10
Qwen3-Embedding-0.6B (base, untrained)	0.00	84.73 / 95.17 / 96.95
BM25 (lexical baseline)	0.01	10.69 / 15.78 / 19.59
V2 (ours, Appendix E )	0.01	74.30 / 89.31 / 93.13
Recipe arm	9.42 $\pm$ 0.61	66.26 $\pm$ 2.31 / 83.68 $\pm$ 1.61 / 86.93 $\pm$ 1.10
Verified arm	0.16 $\pm$ 0.06	56.52 $\pm$ 4.05 / 76.21 $\pm$ 3.63 / 82.92 $\pm$ 3.14
Reference (D4 $+$ CAS negatives)	10.30 $\pm$ 1.15	78.02 $\pm$ 0.91 / 83.94 $\pm$ 1.51 / 85.62 $\pm$ 1.19

Appendix

Table 16: Real retrieval kept after training, beside benchmark score: each model’s strict R@1 / R@5 / R@10 on cross-language duplicates against its hard-tier R@1. The two arms and the reference are means

\pm

sample std over eight seeds, the factorial’s other cells over three; the rest are single runs. Read the columns against each other. The recipe arm keeps a

{\sim}10

-point edge over the verified arm on duplicates, positive in every seed and not equivalent at the pre-specified

11.33

-point margin, but both sit well below the untrained base, and the

{\sim}60\times

hard-tier separation between them on the benchmark is nowhere in sight. Three training sets keep retention at the base (

85.50

84.91

and

84.57

against

84.73

): D4 with an unrelated prompt’s near-misses at either volume, which also buy

16.96

and

20.93

hard R@1, and back-translated positives with verified negatives. Near-copy CAS positives under the recipe arm’s near-misses fall to

12.47

and under an unrelated prompt’s to

13.48

, the two lowest rows; back-translated positives trained alone keep the most,

94.57

, then D4’s and D5’s restatements alone (

93.81

93.72

). The last seven rows are the cells that complete the grid of Table 11 .

Model	Easy R@1 / R@5 / R@10	Medium R@1 / R@5 / R@10	Hard R@1 / R@5 / R@10
Qwen3-Embedding-0.6B (base, untrained)	8.32 / 78.61 / 87.78	1.78 / 60.91 / 75.49	0.00 / 4.08 / 17.60
Verified arm	17.31 / 85.10 / 92.27	2.47 / 41.91 / 57.74	0.13 / 9.59 / 26.49
Recipe arm	61.88 / 89.94 / 93.63	9.06 / 55.18 / 66.79	9.37 / 57.06 / 70.13
Reference (D4 $+$ CAS negatives)	46.51 / 88.06 / 93.75	4.34 / 42.20 / 60.62	10.81 / 60.27 / 75.58
D4 $+$ recipe-prompt near-misses, matched	45.85 / 82.89 / 89.03	2.76 / 27.85 / 40.01	20.31 / 64.69 / 73.99
D4 $+$ recipe-prompt near-misses, full	38.58 / 70.74 / 77.01	2.67 / 20.83 / 28.73	26.47 / 63.82 / 71.40

Appendix

Table 17: The matched-budget experiment at seed

42

, with the full recall curves: R@1 / R@5 / R@10 on each tier over the full

117{,}088

-document corpus and

15{,}000

queries per tier. Both arms train on

6{,}145

rows with identical hyperparameters and differ only in the training file; the reference and the two cells that pair D4’s restatements with the recipe arm’s own near-misses (Table 12 ) are added at the same seed. Eight-seed means and intervals are in Table 1 ; over seeds the paired gap is

45.33\pm 0.93

easy,

5.70\pm 0.44

medium and

9.26\pm 0.60

hard R@1, and

48.84\pm 0.70

hard R@5. Two things the R@1 columns hide: on the easy tier the recipe arm’s lead over the verified arm nearly vanishes by R@5 and R@10, so its edge is mostly about rank

1

; and the two cells with the highest hard-tier R@1 fall far below every other model on the medium tier at R@5 and R@10. The recipe arm’s numbers are the recipe-matching exhibit, not capability.

Public models
Model	Easy R@1 / R@5 / R@10	Medium R@1 / R@5 / R@10	Hard R@1 / R@5 / R@10
RaDeR-gte-Qwen2-7B	18.94 / 95.57 / 99.05	1.12 / 63.88 / 83.24	0.01 / 8.63 / 34.63
RaDeR-Qwen2.5-7B	17.41 / 96.91 / 99.51	0.92 / 60.05 / 80.14	0.01 / 9.53 / 37.86
MathLeap-Octen-8B	15.87 / 84.17 / 94.40	3.51 / 75.90 / 91.97	0.01 / 17.53 / 62.57
MathLeap-Qwen-8B	13.79 / 81.32 / 93.16	3.37 / 75.56 / 91.15	0.01 / 17.01 / 60.86
Qwen3-Embedding-4B	11.15 / 85.65 / 93.34	1.93 / 70.19 / 85.15	0.01 / 6.04 / 28.58

Appendix

Table 18: Public models on MathNet-Retrieve (R@1 / R@5 / R@10 per tier, sorted by easy R@1), the comparison behind the statement that no public model exceeds

0.01

hard-tier R@1: RaDeR [ 10 ] , MathLeap [ 4 ] , Qwen3-Embedding [ 5 ] and ReasonIR [ 9 ] . It is a comparison set, not everything we ran: Qwen3-Embedding-8B was also evaluated on all three tiers (

10.05

2.23

0.00

) and all-mpnet-base-v2, the harness calibration model, on the easy tier only. Bold marks the best value in each column among the public models (hard-tier R@1 ties at

0.01

). Baselines use their own encoding conventions; the Qwen3-Embedding-4B row is the unprompted

2{,}048

-token run (its prompted

1{,}024

-token run scores

11.95

easy R@1, and neither reproduces the

14.76

the benchmark publishes; Appendix A ). Our own models sit below the rule and are excluded from the ranking because their training material is of the benchmark’s own kind: the recipe arm trains on the benchmark’s recipe, and the verified-supervision models’ rename and reformulation positives, though symbolically verified, are in the same paraphrase genre the easy tier tests; “—” marks the one configuration we did not run.

Model	Easy R@1 / R@5 / R@10	Medium R@1 / R@5 / R@10	Hard R@1 / R@5 / R@10	Cross-language dup. R@1
Base 0.6B (untrained)	8.32 / 78.61 / 87.78	1.78 / 60.91 / 75.49	0.00 / 4.08 / 17.60	84.73
Uncapped verified arm	15.92 / 78.18 / 87.21	3.86 / 40.80 / 55.92	0.16 / 6.59 / 19.47	47.58
Soup $\alpha{=}0.3$ (weights: $0.3$ arm, $0.7$ base)	13.55 / 93.57 / 97.93	1.59 / 59.93 / 76.35	0.01 / 4.80 / 23.13	80.15
Soup $\alpha{=}0.5$	15.77 / 93.42 / 97.68	1.77 / 57.30 / 74.17	0.01 / 6.28 / 27.51	73.54
Soup $\alpha{=}0.7$	16.41 / 90.44 / 96.04	2.33 / 52.23 / 69.83	0.01 / 7.54 / 28.14	62.34
V1 (verified pairs $+$ replay)	20.83 / 91.96 / 96.71	3.67 / 63.57 / 78.80	0.03 / 7.67 / 30.01	59.80

Appendix

Table 19: The two remedies of Appendix E : mixing in weight space against mixing in data space (R@1 / R@5 / R@10 per tier; strict R@1 on cross-language duplicates). The soups blend the uncapped verified arm’s weights with the untrained base’s,

\alpha

being the share of the trained arm, so

\alpha{=}0.3

sits nearest the base and

\alpha{=}0.7

nearest the arm; V1 and V2 instead train on verified pairs mixed with problem-to-solution replay, V2 under an instruction prompt. The soups recover retention as

\alpha

falls but lose the hard-tier R@1 the arm had bought at every blend; V2 keeps most of both score and retention. The two control rows show how much the instruction alone does: applied to the untrained base it gives the best retention in the table, and V2 without it loses about six points. Base, uncapped-arm and soup rows are deterministic single evaluations; V1 and V2 show the seed-

42

run, with three-seed means in Appendix E . The soups of V1 test whether the two remedies compose (Appendix E ).

Model	MELD R@1	BRIGHT-TT nDCG@10	BRIGHT-AoPS nDCG@10	ImpliRet nDCG@10
Qwen3-Embedding-0.6B (base, untrained)	10.19	21.24	7.44	16.03
Verified arm	9.44	1.83	0.00	13.14
Recipe arm	18.89	10.19	0.75	9.23
V1 (verified pairs $+$ replay)	9.63	16.90	1.59	16.04
V2 (the same mix $+$ instruction)	10.19	15.26	2.55	16.20
Qwen3-Embedding-4B (untrained)	18.89	31.94	8.69	14.99

Appendix

Table 20: Three benchmarks we had no hand in building, single runs, every cell measured: MELD R@1, BRIGHT nDCG@10 on its two math theorem splits (TT

=

theoremqa-theorems, AoPS

=

the AoPS split), and ImpliRet [ 31 ] nDCG@10, macro-averaged over its six subsets (category

\times

style) of

1{,}500

queries each. On MELD, whose pairs are LLM-written, the recipe arm doubles the base’s score while the verified arm stays flat; on BRIGHT both arms fall below the base, the verified arm to almost nothing, the recipe arm less; on ImpliRet’s

9{,}000

queries the recipe arm is the worst model we measure while the two mixed-data models hold the base’s level. Our pre-registered prediction that the two arms would score within noise of each other is therefore partially refuted, on MELD and BRIGHT, in the recipe arm’s favour, on BRIGHT by intervals that exclude zero. The untrained

4

B model is shown for scale.

Model	Easy	Medium	Hard	Same-lang. primary	Same-lang. all
ColBERTv2 (untrained)	3.97 / 52.79 / 60.86	1.51 / 32.80 / 42.37	0.01 / 1.33 / 3.43	46.40	74.30
Verified arm	6.57 / 59.83 / 69.66	2.17 / 30.19 / 41.43	0.01 / 1.24 / 3.53	46.40	73.90
Recipe arm	39.05 / 75.69 / 81.44	4.28 / 30.02 / 40.09	0.08 / 2.35 / 5.64	35.20	66.06

Appendix

Table 21: The two-arm experiment on a late-interaction retriever (fact source: results/colbert/ , results/colbert_summary.json ). ColBERTv2 fine-tuned through PyLate on the recipe arm’s and the verified arm’s training files, the same

6{,}145

rows, gates and dev split as the dense arms; seed

42

, one run each. Benchmark columns are R@1 / R@5 / R@10 over the full

117{,}088

-document corpus with the query’s own entry masked, from a PLAID index’s top-

1{,}000

candidates re-scored by exact MaxSim, a gold outside the candidates counting as a miss. The retention columns are same-language R@1 on the reprint probe’s primary slice (

125

queries) and on all

498

of its queries (Table 5 ); the cross-language probe is unusable for this English-only backbone (untrained

1.78

strict R@1, the arms

1.78

and

6.36

Task	$n$	Base	Verified arm	Recipe arm	Recipe $-$ verified	Recipe $-$ base
MSE duplicate questions	25116	48.94	32.64	28.05	$-4.59$ $[-5.07,-4.11]$	$-20.90$ $[-21.41,-20.38]$
MathOverflow duplicate questions	225	72.67	52.37	48.22	$-4.15$ $[-9.81,1.45]$	$-24.45$ $[-30.37,-18.72]$
MSE question to answer (ARQMath-3 Task 1)	78	40.04	29.40	41.66	$+12.26$ $[7.72,16.91]$	$+1.62$ $[-2.91,6.41]$
ProofWiki question to answer	1099	75.41	59.35	56.63	$-2.72$ $[-5.43,0.02]$	$-18.78$ $[-21.26,-16.30]$
Stacks question to answer	776	44.58	34.59	29.19	$-5.40$ $[-8.07,-2.84]$	$-15.39$ $[-18.00,-12.79]$
ProofWiki premises	1086	20.12	15.89	11.29	$-4.60$ $[-5.87,-3.39]$	$-8.83$ $[-10.17,-7.53]$

Appendix

Table 22: The eleven natural-language tasks of MIRB [ 32 ] , a human-curated mathematical retrieval benchmark, scored with its protocol (nDCG@10

\times 100

over the top

1{,}000

, each query’s excluded ids dropped, documents as title plus text; the model’s own query prompt, as for every external benchmark here). Untrained base against the two seed-

42

arms; the last two columns are paired query bootstraps (

5{,}000

resamples,

95\%

intervals), bold where the interval excludes zero. The MSE duplicate-question task ranks

1{,}350{,}505

documents per query; the three formal-library premise tasks are outside the paper’s domain.

	Documents written by an LLM			Computer algebra	Human authors
Trained on	Easy tier (Gemini, the benchmark’s prompt)	Golds rebuilt by Qwen, the benchmark’s prompt	Golds rebuilt by Qwen, a different-style prompt	Near-miss probe	Cross-language duplicates
Untrained base	8.32	10.69	3.28	10.84	84.73
Verified arm: computer-algebra pairs	17.05	15.02	15.80	97.42 †	56.52
Recipe arm: benchmark’s prompt, Qwen	62.38	50.75	44.13	60.49	66.26
D3 arm: different-style prompt, Qwen	67.88	50.56	55.41	68.50	65.39
Reference: unrelated prompt, verified negatives	46.91	46.03	44.00	53.51	78.02

Appendix

Table 23: Who built the test decides who wins. Rows are training files; columns are test sets, grouped by who wrote the documents being retrieved; each cell is R@1. The verified arm wins only on the test that computer algebra built, as its own pairs were (

\dagger

: in-family, disclosed rather than read). The two arms trained on the recipe family’s pairs take the first two places on every test an LLM built, and the D3 arm leads the recipe arm by most on the golds its own prompt wrote; the reference barely moves when the golds are rebuilt, so on the different-style golds it is level with the recipe arm (

44.00

against

44.13

; Table 24 ). On the near-miss probe the D3 arm outscores the recipe arm. The untrained base wins on duplicates no generator wrote. A better training file would win everywhere. Means over eight seeds for the two arms and the reference, three for the D3 arm, single seed-

42

checkpoints in the two rebuilt-golds columns (

1{,}525

queries). Sources: Tables 1 , 9 , 24 and 6 .

Gold documents	Base	Verified arm	D1 (recipe)	D2	D3	D4	Reference	Gap
Original benchmark golds	7.34	15.54	61.18	64.92	67.41	30.43	46.89	45.64
Regenerated with the benchmark’s template	10.69	15.02	50.75	55.67	50.56	28.59	46.03	35.73
Regenerated with a paraphrase of it (D2’s prompt)	8.39	16.72	54.36	60.07	56.92	28.79	45.97	37.64
Regenerated in a different style (D3’s prompt)	3.28	15.80	44.13	52.92	55.41	21.70	44.00	28.33
Each gold from one of the three, at random	7.02	16.26	47.74	54.95	54.23	25.84	43.93	31.48

Appendix

Table 24: The regeneration defence of Appendix E (fact source: results/defense_eval/ ). Each row is the same

1{,}525

easy-tier queries with their gold documents regenerated a different way; the columns give each model’s easy-tier R@1 on that version over the full

117{,}088

-document corpus (seed-

42

checkpoints), and Gap is the recipe arm (D1) minus the verified arm. Down the Gap column: regenerating the golds with the benchmark’s own template trims the lead only from

45.64

35.73

, a different-style prompt to

28.33

, and drawing each gold from one of three prompts to

31.48

. Across rows: D2 beats D1 on every version and D3 on all but the exact-template one, so the easy-tier plateau of Section 5 recurs, and D3’s largest lead over D1 is on the golds written with D3’s own prompt. The reference column was scored after the registered readings were taken ( eval_<version>_reference.json ): the reference drops only from

46.89

44.00

across the versions where the recipe arm drops from

61.18

44.13

, so on the different-style golds the two are level, and the recipe arm’s lead over the reference on these items shrinks from

14.29

4.72

under the benchmark’s own template and to

0.13

under the different style.

Gold documents	Base	Verified arm	D1 (recipe)	D2	D3	D4	Gap
Original benchmark golds	7.10	16.33	60.43	63.27	65.04	28.48	44.10
Regenerated with the benchmark’s template	0.00	0.53	20.14	17.48	10.56	0.53	19.61
Regenerated with a paraphrase of it (D2’s prompt)	0.00	0.71	22.54	19.61	11.80	0.44	21.83
Regenerated in a different style (D3’s prompt)	0.00	0.62	17.21	18.10	14.02	0.53	16.59
Each gold from one of the three, at random	0.00	0.80	18.54	18.19	11.71	0.53	17.74

Appendix

Table 25: The regeneration defence rerun with the benchmark’s own vendor (fact source: results/defense_eval_gemini/ ). The same

1{,}700

sampled easy-tier queries, the same three prompts and the same single judge as Table 24 , but the golds are rewritten by gemini-3-flash-preview through the vendor’s API instead of by Qwen3-32B-AWQ; the

1{,}127

queries the judge accepted under all three prompts are scored over the full corpus with the same seed-

42

checkpoints. Easy-tier R@1; Gap is D1 minus the verified arm. The original-gold row differs from Table 24 ’s because the slice does. The Gap column keeps its shape (paraphrase at or above exact, style lowest, mixed between), but every regenerated row collapses for every model, the base to

0.00

at rank

1

, because the vendor’s fresh rewrites are a deeper rewrite than the golds the benchmark shipped (text).

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 175 papers

Feb 27, 2026

LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics

We present a new approach for benchmarking Large Language Model (LLM) capabilities on research-level mathematics. Existing benchmarks largely rely on static, hand-curated sets of contest or textbook-style problems as proxies for mathematical research. Instead, we establish an updatable benchmark evaluating models directly on the latest research results in mathematics. This consists of an automatic pipeline that extracts lemmas from arXiv and rewrites them into self-contained statements by making all assumptions and required definitions explicit. It results in a benchmark that can be updated regularly with new problems taken directly from human mathematical research, while previous instances can be used for training without compromising future evaluations. We benchmark current state-of-the-art LLMs, which obtain around 10-15

\%

accuracy in theorem proving (pass@1) depending on the model, showing that there is currently a large margin of progression for LLMs to reach human-level proving capabilities in a research context.

Antoine Peyronnet, Fabian Gloeckle, Amaury Hayat

1Ecole Normale Superieure de Rennes · 2Ecole des Ponts Paris · 3Korean Institute for Advanced Study

Mathematical Reasoning Benchmarks Automated Theorem Proving LLM Mathematical Reasoning Large Language Model-Guided Theorem Proving

Ask PDF

May 9, 2026

Re $^2$ Math: Benchmarking Theorem Retrieval in Research-Level Mathematics

Large language models are increasingly capable at closed-world mathematical reasoning, but research assistance also requires source-grounded use of the literature. When a proof reaches a non-trivial step, a useful assistant should determine whether the needed tool (e.g., a lemma) already exists, identify a suitable scholarly source, and verify that its assumptions align with the current proof context. To rigorously evaluate such capabilities, we introduce Re

^2

Math, a benchmark for tool-grounded retrieval from partial mathematical proofs. Each instance is built from a candidate instrumental citation in the proof of a main theorem, with hierarchical context and an optional leakage-controlled anchor hint. We also make the task source-grounded yet citation-agnostic in that any admissible theorem sufficient for the proof transition is accepted. Evaluation uses a release-frozen retrieval artifact, ensuring reproducibility, while the benchmark itself supports automatic, continual expansion with newly constructed instances. On the current benchmark test set, the best fixed-judge ToolAcc reaches 7.0%, despite substantially higher rates of source grounding, indicating that current systems often retrieve valid statements but fail to establish their applicability to the local proof step. By decoupling citation recall, grounding, and proof-gap sufficiency, Re

^2

Math transforms literature-grounded mathematical tool use into a controlled diagnostic task.

Zicheng Lyu, Wenjie Yang, Shengzhong Zhang +1

Fudan University Shanghai, China · Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China · Fudan University Shanghai Innovation Institute Shanghai, China

Mathematical Reasoning Benchmarks LLM Evaluation Mathematical Reasoning Information Retrieval Evidence Retrieval

Ask PDF

Aug 30, 2026

RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving

Data contamination undermines the reliable evaluation of large language models (LLMs) on mathematical problem solving. While rewriting-based evaluation mitigates memorization, existing methods lack guarantees of problem validity and answer correctness. We propose Proof-Verified Benchmark Rewriting (RePro), the first framework to integrate Lean-oriented neural automated theorem provers (ATPs) into benchmark rewriting, which rewrites problems and regenerates answers with correctness ensured by Lean-verified proofs. Experiments on GSM8K and MATH show that RePro's retained rewritten instances achieve 100% well-definedness, feasibility, and answer correctness, while existing methods still produce invalid or incorrect instances. Moreover, several models exhibit accuracy drops on proof-verified rewritten benchmarks, suggesting that their performance is sensitive to surface-level and structural variations and may partly reflect memorization effects. Our source code and data are available at https://github.com/AI4Engi/RePro.

Xiyuan Zhou, Zhuoqi Li, Xinlei Wang +6

Nanyang Technological University · The Chinese University of Hong Kong, Shenzhen · INSAIT, Sofia University “St. Kliment Ohridski” +2

Mathematical Reasoning Benchmarks LLM Mathematical Reasoning Data Contamination in Language Models

Ask PDF

Abstract

Benchmark Validity

Explore similar work

LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics

Re2^22Math: Benchmarking Theorem Retrieval in Research-Level Mathematics

RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving

Re $^2$ Math: Benchmarking Theorem Retrieval in Research-Level Mathematics