cs.IRSep 25, 2026

Recipe-Matching, Not Equivalence

Authors: Ali Habibullah, Mohammad Alshiekh, Yazan Alshoibi, Salman Khan, Naeemullah Khan

Organizations: KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · KAUST Academy & CEMSE, KAUST Lady Margaret Hall, University of Oxford

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

MathNet-Retrieve asks a retriever to find, for a math problem, a document stating the same problem. An LLM under one fixed prompt writes each gold document and its near-miss distractors; LLM judges filter them. We call this procedure the "recipe", training on pairs built the same way "recipe-matching", and ask how much score it buys beyond the ability the benchmark claims to test. Two models from one base, matched in rows and settings, differ only in the training file: pairs written under the benchmark's published prompt by another vendor's LLM and judge, or computer-algebra-verified pairs with no LLM anywhere. The first leads by 45 R@1 points on the easy tier. By a non-LLM paraphrase control, half to two thirds of that gap comes from the pairs being LLM-written at all: LLM rewrites under two unrelated prompts, with the verified model's negatives, recover 30 and 22 of the 45 points; back-translations with the same negatives recover almost none. The remaining 15 to 25 points appear only under the benchmark's own prompt and vanish on real duplicates no generator wrote, the same problem in two languages. The hard tier rewards the recipe's pair structure, a deep rewrite against a minimal-edit near-miss: LLM rewrites alone score zero on it, attaching negatives unlocks it, and every negative that does so costs cross-language points; the sets scoring highest on it separate near-misses no LLM wrote worse than LLM rewrites with verified negatives. MELD also moves when a model trains on pairs built its way, without losing retention; on SABER-Math the registered attack fails, and the one gain, from its LLM-written summaries, is small but holds at a matched budget. Only on MathNet-Retrieve could we pin an inversion, benchmark score up and real retention down, to one edit of a training file. We release the generator-free duplicate evaluations, the near-miss test and three trained models.

Figures & tables

EvaluationBaseVerifiedReferenceRecipeRecipe −- verifiedRecipe −- reference
Benchmark tiers
Easy (R@1)8.3217.05 ±\pm 0.7946.91 ±\pm 1.2562.38 ±\pm 0.8545.33 [44.25,46.41][44.25,46.41]15.47 [14.11,16.73][14.11,16.73]
Medium (R@1)1.783.14 ±\pm 0.464.22 ±\pm 0.258.84 ±\pm 0.365.70 [5.18,6.24][5.18,6.24]4.62 [4.13,5.15][4.13,5.15]
Hard (R@1)0.000.16 ±\pm 0.0610.30 ±\pm 1.159.42 ±\pm 0.619.26 [8.71,9.85][8.71,9.85]−0.88-0.88 [−1.85,0.10][-1.85,0.10]
Hard (R@5)4.088.35 ±\pm 1.1360.05 ±\pm 1.2257.20 ±\pm 0.8248.84 [47.95,49.74][47.95,49.74]−2.86-2.86 [−4.01,−1.67][-4.01,-1.67]
Real duplicates, no LLM involved
Table 1: The main experiment. Columns: the untrained base, the two matched-budget arms and the recipe-free reference (D4 restatements with the verified arm’s negatives, no benchmark prompt anywhere), each over eight seeds (mean ±\pm std), then the recipe arm’s lead over the verified arm and over the reference with 95%95\% nested-bootstrap intervals (Section 3 ). Rows: the benchmark’s own tiers first, then real duplicates that no LLM wrote (cross-language reprints, scored strictly : the query’s own corpus entry is masked and only a reprint in another language counts as a hit; and the 125125 -query same-language slice of Appendix A ), then the difference-in-differences (DiD), the benchmark lead minus the real-duplicate lead. Bold marks the largest value in a row; on the duplicate rows it is the untrained base. Seed- 4242 breakdown: Table 17 .
Figure 1: Benchmark score against real retrieval kept after training, one point per trained model. Horizontal axis: strict R@1 on cross-language duplicates; the dashed line and ×\times mark the untrained base, so points left of it lost real retrieval by being trained. Colour gives the positives (legend); hollow means no negatives. (a) Easy tier: every model trained on LLM rewrites scores above the verified arm. (b) Hard tier: only deep LLM rewrites with negatives leave the floor. Arrows run from D4 alone to D4 with negatives attached; each label names the negatives (verified ones give the reference; “recipe” and “unrelated” are LLM near-misses from the recipe’s prompt and from an unrelated one, at the reference’s count or at the recipe arm’s full volume) and the cross-language points lost. All models: Figure 2 .
Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Base modelQwen3-Embedding-0.6B, bf16, all weights trained
Lossthe in-batch contrastive loss of Eq. 1 , computed with gradient caching [ 33 ]
Batch size / mini-batch size256256 / 1616 (the cache makes the gradient identical to a plain batch of 256256 )
Learning rate2×10−52\times 10^{-5}
Epochs33
Seeds4242 – 4949 (eight per arm)
Training rows per arm6,1456{,}145 , with all rows of one source problem kept together
Appendix
Table 2: Training settings shared by both arms. Every row is identical for the verified arm and the recipe arm; only the training file differs. Two other experiments reuse these settings with stated deviations, applied identically to both arms: the SABER-Math attack of Section 6 caps training sequences at 1,0241{,}024 tokens where the controlled arms use the backbone’s default, and the 44 B replication uses rank- 1616 LoRA adapters instead of full fine-tuning, a 1×10−41\times 10^{-4} learning rate instead of 2×10−52\times 10^{-5} , and gradient checkpointing.
ControlExplanation it removesEasy-tier gap (R@1)
Matched budget, eight seeds (Table 1 )seed noise45.33±0.9345.33\pm 0.93
Queries the original gate kept out of both training files ( 8,6638{,}663 )leaked test items46.37±0.8546.37\pm 0.85
No copy of the query in either arm’s rows ( 12,76012{,}760 , strictest)leaked test items44.73±0.8644.73\pm 0.86
Retrain under the corrected filter, 33%33\% fewer rowsnear-verbatim copies of queries43.11±0.9043.11\pm 0.90
Retrain under the corrected filter, removed third regenerated from fresh sourcesnear-verbatim copies of queries44.45±0.8744.45\pm 0.87
Retrain under the corrected filter, removed third regenerated from the same sourcesnear-verbatim copies of queries44.38±1.4644.38\pm 1.46
Appendix
Table 3: Does the easy-tier gap survive when a possible explanation for it is removed? Each row removes one and reports the recipe arm’s lead over the verified arm on the easy tier (first row: the headline comparison of Table 1 ). Rows marked “intact” change the verified arm only, so they report its score rather than a gap. No 0.60.6 B control brings the gap below 4141 points; the other-backbone and 44 B rows shrink it but keep its sign (Appendix D ). The last two rows are the two recipe-arm variants of Appendix C , each against the eight-seed verified arm.
Query slicennBaseVerified −- baseRecipe −- baseD4 −- base
All queries39384.73−28.21-28.21 [−33.21,−23.15][-33.21,-23.15]−18.48-18.48 [−23.47,−13.68][-23.47,-13.68]+9.08+9.08 [5.85,12.55][5.85,12.55]
Pair found through a shared answer2588.00−28.00-28.00 [−45.00,−12.50][-45.00,-12.50]−35.50-35.50 [−55.50,−16.00][-55.50,-16.00]0.000.00 [−12.00,12.00][-12.00,12.00]
Pair found through matching text1586.67−18.33-18.33 [−37.50,−3.33][-37.50,-3.33]−8.33-8.33 [−23.33,0.00][-23.33,0.00]0.000.00 [0.00,0.00][0.00,0.00]
Pair found through formulas only35384.42−28.65-28.65 [−33.50,−23.16][-33.50,-23.16]−17.71-17.71 [−22.59,−12.96][-22.59,-12.96]+10.10+10.10 [6.42,13.69][6.42,13.69]
Least shared formula content (bottom quartile)9976.77−29.42-29.42 [−38.89,−20.20][-38.89,-20.20]−20.45-20.45 [−31.19,−9.47][-31.19,-9.47]+15.15+15.15 [6.06,23.23][6.06,23.23]
Most shared formula content (top quartile)9993.94−23.36-23.36 [−31.69,−15.40][-31.69,-15.40]−12.12-12.12 [−19.44,−5.18][-19.44,-5.18]+3.03+3.03 [−1.01,7.41][-1.01,7.41]
Appendix
Table 4: Cross-language retention by query slice. Each row is a subset of the 393393 cross-language queries; Base is the untrained model’s strict R@1 on that subset, and the three gap columns give how far the verified arm, the recipe arm and D4 sit from it, with 95%95\% nested bootstrap intervals over queries and training seeds ( B=2,000B=2{,}000 ; the arms eight seeds, D4 three). Rows two to four split the queries by how the pair was found, the two quartile rows keep the queries with the least and the most shared rare-formula content, and the last two split by how formula-heavy the query text is. On no slice does either arm reach the base; the recipe arm loses more where formulas are scarce (bottom quartile, formula-light) than where they are plentiful, the reverse of a formula-matching artefact; and D4 falls below the base on no slice.
ModelPrimary ( 125125 )English ( 115115 )Leaked ( 8282 )Exact reprints ( 291291 )Primary gap to base
Base 0.60.6 B (untrained)62.4059.1350.00100.00 
Base ++ V2 instruction62.4059.1352.44100.000.000.00 [−3.23,3.17][-3.23,3.17]
BM25 (lexical baseline)54.4051.3032.93100.00−8.00-8.00 [−17.07,0.79][-17.07,0.79]
Verified arm ( 88 seeds)51.3047.0734.9199.40−11.10-11.10 [−18.90,−3.89][-18.90,-3.89]
Recipe arm ( 88 seeds)46.3041.6335.8285.61−16.10-16.10 [−26.37,−6.10][-26.37,-6.10]
D4 (recipe-free, no negatives)61.0757.6844.72100.00−1.33-1.33 [−7.20,3.73][-7.20,3.73]
Appendix
Table 5: Same-language reprints: R@1 for retrieving a reprint of the query in the same language, for every model we trained. The four score columns are the four slices: the primary slice ( 125125 queries from reworded clusters that no training file touches), its English queries, the leaked slice (queries from reworded clusters that some training file does touch), and the exact reprints (near-identical copies that any model should retrieve, a ceiling check). The last column is each model’s gap to the untrained base on the primary slice, with a 95%95\% nested bootstrap interval over clusters and seeds. Every model trained with negatives of any kind sits well below the base; only the models trained without negatives, the back-translated model, V2 and the MELD arm stay within noise of it. Seeds: the two arms and the reference eight; the ladder rungs, CAS positives without negatives, the other factorial cells, V2 and the MELD arm three; the rest single runs.
ModelBenchmark hard-tier R@1Probe R@1, full corpusProbe R@1, near-misses only
Base 0.60.6 B (untrained)0.0010.8410.93
BM25 (lexical baseline)0.0172.2773.8
Verified arm ( 88 seeds) †0.1697.42 ±\pm 0.3297.53 ±\pm 0.33
Recipe arm ( 88 seeds)9.4260.49 ±\pm 2.5068.38 ±\pm 2.40
D2 arm (paraphrase of the benchmark’s prompt)6.7662.54 ±\pm 1.2369.35 ±\pm 1.99
D3 arm (different-style prompt)4.6768.50 ±\pm 1.3872.30 ±\pm 2.21
Appendix
Table 6: The generator-free near-miss probe: 1,1071{,}107 held-out benchmark queries, each with a computer-algebra-verified positive and counterexampled minimal-edit negatives. Each model’s benchmark hard-tier R@1 is set beside its probe R@1 scored two ways: against the full public corpus plus the probe documents (the hard tier’s shape without a generator) and against the probe documents alone (a pure near-miss test). Means ±\pm sample std over seeds where more than one: the arms and the reference eight, the other cells three, the base one. Rows marked †\dagger are in-family, since the probe uses their own transformation families (the verified arm, V2 and the three CAS-positive cells); their scores are shown but not read as evidence. Among the other rows, the two with the highest hard-tier scores have the lowest probe scores of D4’s cells, while back-translated positives, near the hard-tier floor, score high on the probe.
Query sliceQueriesRecipe −- verified, easy R@1
All queries15,00015{,}00045.33 ±\pm 0.93
Queries the original gate did catch (clean, up to 9898 missed copies)8,7618{,}76146.25 ±\pm 0.86
No copy of the query in either arm’s training rows (strictest clean)12,76012{,}76044.73 ±\pm 0.86
Query text trained on by the recipe arm (leaked)1,9601{,}96049.80 ±\pm 1.51
A copy exists in the corpus, but neither arm trained on it4,0184{,}01841.16 ±\pm 0.94
Appendix
Table 7: Does the easy-tier gap depend on leaked queries? The 15,00015{,}000 easy-tier queries are split by whether a copy of the query reached training, and each row gives the recipe arm’s lead over the verified arm on that slice (R@1, mean ±\pm sample std over the eight training seeds; slice sizes at seed 4242 ). Leakage is judged against every row of each arm’s training file; the recipe arm’s rows are the same at every seed, the verified arm’s are redrawn per seed. The gap is present on every slice, including the strictest clean one; the last two rows isolate the raw leakage effect, the gap on queries the recipe arm trained on against the gap on comparable queries neither arm trained on.
Recipe −- verifiedHeadline (original gate, 6,1456{,}145 rows)Filtered (corrected gate, 4,1014{,}101 rows)Regenerated, fresh sources (corrected gate, 6,1456{,}145 rows)Regenerated, same sources (corrected gate, 6,1456{,}145 rows)
Easy R@145.33 ±\pm 0.9343.11 ±\pm 0.9044.45 ±\pm 0.8744.38 ±\pm 1.46
Hard R@19.26 ±\pm 0.609.34 ±\pm 0.319.21 ±\pm 0.669.05 ±\pm 0.77
Real duplicates (strict R@1)9.7411.0417.4018.73
Difference-in-differences35.5932.0727.0525.65
Appendix
Table 8: The arm gap under the original and the corrected contamination gate. Each column is one version of the two-arm comparison, all at the same eight seeds: the headline arms (original gate, 6,1456{,}145 rows); the arms filtered under the corrected gate to the 4,1014{,}101 matched rows it leaves; the arms restored to 6,1456{,}145 rows with the recipe arm’s missing third regenerated from fresh sources; and the arms restored to 6,1456{,}145 rows with that third regenerated from the same sources, a second rewrite per source, against the same verified arm as the fresh-source column. Each row is the recipe arm’s lead over the verified arm on one evaluation. The easy- and hard-tier gaps barely move. In the fresh-source column the two arms no longer share one source list, and only the shared list still contains members of the real-duplicate clusters, so that column’s real-duplicate row, and the difference-in-differences built on it, are not like-for-like with the others; the same-source column restores the shared list.
Training setEasy R@1Hard R@1Cross-language dup. R@1
Base 0.6B (untrained)8.320.0084.73
Verified arm17.05 ±\pm 0.790.16 ±\pm 0.0656.52 ±\pm 4.05
D4: restatement prompt unrelated to the recipe30.03 ±\pm 1.450.04 ±\pm 0.0293.81 ±\pm 0.39
D3: rewrite prompt in a different style67.88 ±\pm 0.764.67 ±\pm 0.4065.39 ±\pm 1.83
D2: paraphrase of the benchmark’s prompt67.54 ±\pm 1.956.76 ±\pm 1.0168.02 ±\pm 3.51
D1: the benchmark’s prompt ( == recipe arm)61.86 ±\pm 0.529.90 ±\pm 0.7466.41 ±\pm 1.78
Appendix
Table 9: The rewriter-prompt ladder. Rows run from the prompt farthest from the benchmark’s (D4) to the benchmark’s own (D1), with the untrained base and the verified arm above for reference; columns are R@1 on the easy tier, the hard tier and cross-language duplicates, mean ±\pm sd over the same three seeds ( 4242 – 4444 ). The hard-tier column falls step by step from D1 to D4, the registered prediction that passed; the easy-tier column does not, since D2 and D3 outscore D1 and only D4 drops; on duplicates D4 tops every model including the base. Three notes: D1 is the recipe arm read at three seeds, so its numbers differ slightly from the eight-seed ones of Table 1 ( 62.38±0.8562.38\pm 0.85 / 9.42±0.619.42\pm 0.61 ); D3 trained on the 5,5915{,}591 positives its prompt yielded rather than the matched 6,1456{,}145 , which a control shows flatters it on the hard tier; D4 attaches no negatives by design, and Table 12 separates that from its prompt. On real duplicates only the contrasts between D4 and the rest replicate across seeds. The points are plotted in Figure 1 .
recipehow a benchmark builds itself: the prompts and filters under which an LLM writes part of it
templatethe exact wording of that prompt
genrethe style of LLM-written rewrites, whichever prompt produced them
pair structurea deep LLM rewrite as the positive against a minimal-edit near-miss as the negative
surfacethe part of a benchmark its LLM writes, the part an attacker can imitate: documents, summaries or evaluation pairs
referencethe recipe-free comparison model: LLM restatements written under a prompt unrelated to the recipe (D4), paired with the verified arm’s computer-algebra negatives; no benchmark prompt anywhere in its training file
Appendix
Table 10: The five terms the paper relies on, and the reference model that Table 1 compares against.
 Negatives
Positivesnonecomputer-algebra counterexamplesLLM near-misses, unrelated promptLLM near-misses, recipe prompt
Computer-algebra near-copies0.00 / 52.080.16 / 56.52 verified0.23 / 13.480.11 / 12.47
Back-translated paraphrases0.05 / 94.571.09 / 84.571.88 / 35.204.18 / 38.67
LLM restatements, unrelated prompt (D4)0.04 / 93.81 D410.30 / 78.02 reference16.96 / 85.50; 20.93 / 84.9120.71 / 83.89; 26.97 / 78.88
LLM rewrites, recipe prompt (D1)0.00 / 83.212.09 / 70.823.45 / 47.419.42 / 66.26 recipe
Appendix
Table 11: Every training set in the paper, arranged by what its positives are (rows) and what its negatives are (columns). Each cell is one trained model and gives two numbers: R@1 on the benchmark’s hard tier, then strict R@1 on real cross-language duplicates. Two cells show two pairs because that set was trained twice, first with as many negatives as the reference has and then with as many as the recipe arm has; every other LLM-near-miss cell uses the recipe arm’s volume, and every computer-algebra-negative cell the reference’s negatives. The two arms and the reference have eight seeds, every other cell three (Appendix C ). The four models the paper names are in italic.
PositivesNegativesEasy R@1Hard R@1Cross-language dup. R@1
The 2×22\times 2 : CAS or D4 positives, with or without CAS counterexamples
CAS (verified)CAS counterexamples ( == verified arm)17.05 ±\pm 0.790.16 ±\pm 0.0656.52 ±\pm 4.05
CAS (verified)none7.42 ±\pm 0.240.00 ±\pm 0.0052.08 ±\pm 1.06
D4 (recipe-free)none ( == D4)30.03 ±\pm 1.450.04 ±\pm 0.0293.81 ±\pm 0.39
D4 (recipe-free)CAS counterexamples ( == reference)46.91 ±\pm 1.2510.30 ±\pm 1.1578.02 ±\pm 0.91
D4 positives with LLM-written near-misses (matched: the reference’s count; full: the recipe arm’s volume)
Appendix
Table 12: Every training set of the factorial, a kind of positive crossed with a kind of negative: R@1 on the easy and hard tiers and strict R@1 on cross-language duplicates, mean ±\pm sd over training seeds (the verified arm and the reference eight, every other cell three; the D4 row is the three-seed mean of Table 9 ). The group labels say what each block varies. Three readings: neither half of a pair scores on the hard tier on its own, and near-copy or back-translated positives barely do even with negatives; D4’s restatements with any negatives leave the floor, matching the recipe arm ( 9.42±0.619.42\pm 0.61 over eight seeds, Table 1 ) with no recipe wording in the file and passing it with LLM-written near-misses; and every kind of negative costs D4’s restatements cross-language points from their 93.8193.81 , except an unrelated prompt’s near-misses, which leave them at the untrained base ( 84.7384.73 ). The last block fills the grid: positives of any kind trained alone never register on the hard tier, and paraphrase positives alone, LLM-written or back-translated, sit at the top on cross-language duplicates ( 94.5794.57 for back-translations, within noise of D4’s 93.8193.81 ); near-copy or back-translated positives with LLM near-misses of either prompt stay near the floor and lose most of their cross-language retention; and D1’s own positives with the reference’s negatives reach a fifth of the reference’s hard-tier score.
Figure 2: The full plot behind Figure 1 : every 0.60.6 B model of Tables 1 , 9 , 12 and 19 , with the corrected-gate retrains and V2’s no-instruction reading omitted. Same axes and conventions as Figure 1 : horizontal, strict R@1 on cross-language duplicates, the dashed line and ×\times the untrained base; colour, what the positives are; hollow, no negatives; the labels on the D4 cells name the negatives attached (“recipe” and “unrelated” are LLM near-misses from the recipe’s prompt and from an unrelated one, “full” at the recipe arm’s negative volume). Two groups appear here only: the verified squares also include computer-algebra positives with no negatives (CAS alone) and the uncapped verified arm, and the crossed circles are the mixed-data, soup and V2 models of Appendix E , which sit near the hard-tier floor. The MELD-recipe arm is seed 4242 ( 20.2320.23 easy, 0.110.11 hard, 90.8490.84 real; Section 6 ). (a) Easy tier: every model trained on LLM rewrites scores above the verified ones. (b) Hard tier: only deep LLM rewrites paired with negatives reach it; the arrow is the edit from D4 alone to the reference, attaching verified negatives, which lifts hard R@1 off the floor and costs 15.7915.79 retention points, and LLM-written near-misses lift the same restatements further (Table 12 ). Computer-algebra positives with the recipe’s near-misses lie off the axis at 12.4712.47 real.
Rewrite promptHeld-out positivesAUC vs. organic95%95\% CI
D1: the benchmark’s Appendix-F template1,2821{,}2820.8802[0.8679,0.8917][0.8679,0.8917]
D2: paraphrase of that template1,2491{,}2490.8756[0.8624,0.8873][0.8624,0.8873]
D3: rewrite prompt in a different style1,0961{,}0960.8654[0.8520,0.8788][0.8520,0.8788]
D4: restatement prompt unrelated to the recipe1,2651{,}2650.6758[0.6569,0.6948][0.6569,0.6948]
Appendix
Table 13: What the style detector detects. Each row takes one prompt’s rewrites (held-out sources only) and gives the AUC with which the detector separates them from the same 1,2821{,}282 held-out organic problems, with 95%95\% CIs from 2,0002{,}000 bootstrap replicates over held-out source ids. The three recipe-family prompts D1–D3 are detected at about the same level and cannot be told apart within their intervals; the recipe-unrelated prompt D4 is detected far less strongly but still well above chance. The detector therefore mostly picks up the genre of LLM rewrites, which D4 shares, plus a recipe increment of 0.2040.204 [0.186,0.222][0.186,0.222] that only the recipe-family prompts add.
Modelρ\rho (similarity, style)Top- 1010 style percentile
Base 0.6B (untrained)−0.061-0.06144.8
Verified arm−0.019-0.01945.5
D4: restatement prompt unrelated to the recipe−0.000-0.00048.1
Reference (D4 ++ CAS negatives)+0.025+0.02552.0
D4 ++ recipe-prompt near-misses, count-matched+0.046+0.04653.1
D4 ++ recipe-prompt near-misses, full+0.050+0.05053.5
Appendix
Table 14: Style tracking: does a model rate LLM-styled documents as more similar? For each of 2,0002{,}000 sampled queries against the full 117,088117{,}088 -document corpus (single runs at seed 4242 ), the first column is the Spearman correlation between the model’s similarity to each document and the artefact-stripped detector’s style score for it, averaged over queries; positive means similarity rises with LLM style. The second column is where the model’s top- 1010 documents sit in the corpus’s style distribution, as a percentile, where 5050 means no preference (we report percentiles because the detector’s raw score is an unbounded log-odds whose ratios have no scale meaning). The base and the verified arm show no style preference, D4’s restatements alone none, and D4 with negatives of either kind only a faint one; the recipe-family prompts track clearly, in ladder order, with the exact template strongest.
Modelρ\rho (all documents)Top- 1010 style percentile
Public models, never trained by us
Qwen3-Embedding-8B−0.0705-0.070545.21
Qwen3-Embedding-0.6B (the untrained base)−0.0610-0.061044.83
Qwen3-Embedding-4B−0.0548-0.054845.61
MathLeap-Qwen-8B−0.0214-0.021447.04
MathLeap-Octen-8B−0.0091-0.009147.94
Appendix
Table 15: Do public models show the recipe-style signature? The same two measures as Table 14 : ρ\rho is the mean per-query Spearman correlation between a model’s query–document similarity and the document’s LLM-style score, and the last column is where the model’s top- 1010 documents sit in the corpus’s style distribution, as a percentile, with 5050 meaning no preference. Every public model sits in the same narrow band as the untrained base and our verified arm, between −0.07-0.07 and +0.004+0.004 with top- 1010 percentiles of 4545 to 4848 ; our recipe-family models sit far away, at +0.10+0.10 to +0.17+0.17 and 5757 to 6262 . No public model approaches the recipe arm.
ModelHard-tier R@1Cross-language duplicates, strict R@1 / R@5 / R@10
Qwen3-Embedding-0.6B (base, untrained)0.0084.73 / 95.17 / 96.95
BM25 (lexical baseline)0.0110.69 / 15.78 / 19.59
V2 (ours, Appendix E )0.0174.30 / 89.31 / 93.13
Recipe arm9.42 ±\pm 0.6166.26 ±\pm 2.31 / 83.68 ±\pm 1.61 / 86.93 ±\pm 1.10
Verified arm0.16 ±\pm 0.0656.52 ±\pm 4.05 / 76.21 ±\pm 3.63 / 82.92 ±\pm 3.14
Reference (D4 ++ CAS negatives)10.30 ±\pm 1.1578.02 ±\pm 0.91 / 83.94 ±\pm 1.51 / 85.62 ±\pm 1.19
Appendix
Table 16: Real retrieval kept after training, beside benchmark score: each model’s strict R@1 / R@5 / R@10 on cross-language duplicates against its hard-tier R@1. The two arms and the reference are means ±\pm sample std over eight seeds, the factorial’s other cells over three; the rest are single runs. Read the columns against each other. The recipe arm keeps a ∼10{\sim}10 -point edge over the verified arm on duplicates, positive in every seed and not equivalent at the pre-specified 11.3311.33 -point margin, but both sit well below the untrained base, and the ∼60×{\sim}60\times hard-tier separation between them on the benchmark is nowhere in sight. Three training sets keep retention at the base ( 85.5085.50 , 84.9184.91 and 84.5784.57 against 84.7384.73 ): D4 with an unrelated prompt’s near-misses at either volume, which also buy 16.9616.96 and 20.9320.93 hard R@1, and back-translated positives with verified negatives. Near-copy CAS positives under the recipe arm’s near-misses fall to 12.4712.47 and under an unrelated prompt’s to 13.4813.48 , the two lowest rows; back-translated positives trained alone keep the most, 94.5794.57 , then D4’s and D5’s restatements alone ( 93.8193.81 , 93.7293.72 ). The last seven rows are the cells that complete the grid of Table 11 .
ModelEasy R@1 / R@5 / R@10Medium R@1 / R@5 / R@10Hard R@1 / R@5 / R@10
Qwen3-Embedding-0.6B (base, untrained)8.32 / 78.61 / 87.781.78 / 60.91 / 75.490.00 / 4.08 / 17.60
Verified arm17.31 / 85.10 / 92.272.47 / 41.91 / 57.740.13 / 9.59 / 26.49
Recipe arm61.88 / 89.94 / 93.639.06 / 55.18 / 66.799.37 / 57.06 / 70.13
Reference (D4 ++ CAS negatives)46.51 / 88.06 / 93.754.34 / 42.20 / 60.6210.81 / 60.27 / 75.58
D4 ++ recipe-prompt near-misses, matched45.85 / 82.89 / 89.032.76 / 27.85 / 40.0120.31 / 64.69 / 73.99
D4 ++ recipe-prompt near-misses, full38.58 / 70.74 / 77.012.67 / 20.83 / 28.7326.47 / 63.82 / 71.40
Appendix
Table 17: The matched-budget experiment at seed 4242 , with the full recall curves: R@1 / R@5 / R@10 on each tier over the full 117,088117{,}088 -document corpus and 15,00015{,}000 queries per tier. Both arms train on 6,1456{,}145 rows with identical hyperparameters and differ only in the training file; the reference and the two cells that pair D4’s restatements with the recipe arm’s own near-misses (Table 12 ) are added at the same seed. Eight-seed means and intervals are in Table 1 ; over seeds the paired gap is 45.33±0.9345.33\pm 0.93 easy, 5.70±0.445.70\pm 0.44 medium and 9.26±0.609.26\pm 0.60 hard R@1, and 48.84±0.7048.84\pm 0.70 hard R@5. Two things the R@1 columns hide: on the easy tier the recipe arm’s lead over the verified arm nearly vanishes by R@5 and R@10, so its edge is mostly about rank 11 ; and the two cells with the highest hard-tier R@1 fall far below every other model on the medium tier at R@5 and R@10. The recipe arm’s numbers are the recipe-matching exhibit, not capability.
ModelEasy R@1 / R@5 / R@10Medium R@1 / R@5 / R@10Hard R@1 / R@5 / R@10
Public models
RaDeR-gte-Qwen2-7B18.94 / 95.57 / 99.051.12 / 63.88 / 83.240.01 / 8.63 / 34.63
RaDeR-Qwen2.5-7B17.41 / 96.91 / 99.510.92 / 60.05 / 80.140.01 / 9.53 / 37.86
MathLeap-Octen-8B15.87 / 84.17 / 94.403.51 / 75.90 / 91.970.01 / 17.53 / 62.57
MathLeap-Qwen-8B13.79 / 81.32 / 93.163.37 / 75.56 / 91.150.01 / 17.01 / 60.86
Qwen3-Embedding-4B11.15 / 85.65 / 93.341.93 / 70.19 / 85.150.01 / 6.04 / 28.58
Appendix
Table 18: Public models on MathNet-Retrieve (R@1 / R@5 / R@10 per tier, sorted by easy R@1), the comparison behind the statement that no public model exceeds 0.010.01 hard-tier R@1: RaDeR [ 10 ] , MathLeap [ 4 ] , Qwen3-Embedding [ 5 ] and ReasonIR [ 9 ] . It is a comparison set, not everything we ran: Qwen3-Embedding-8B was also evaluated on all three tiers ( 10.0510.05 / 2.232.23 / 0.000.00 ) and all-mpnet-base-v2, the harness calibration model, on the easy tier only. Bold marks the best value in each column among the public models (hard-tier R@1 ties at 0.010.01 ). Baselines use their own encoding conventions; the Qwen3-Embedding-4B row is the unprompted 2,0482{,}048 -token run (its prompted 1,0241{,}024 -token run scores 11.9511.95 easy R@1, and neither reproduces the 14.7614.76 the benchmark publishes; Appendix A ). Our own models sit below the rule and are excluded from the ranking because their training material is of the benchmark’s own kind: the recipe arm trains on the benchmark’s recipe, and the verified-supervision models’ rename and reformulation positives, though symbolically verified, are in the same paraphrase genre the easy tier tests; “—” marks the one configuration we did not run.
ModelEasy R@1 / R@5 / R@10Medium R@1 / R@5 / R@10Hard R@1 / R@5 / R@10Cross-language dup. R@1
Base 0.6B (untrained)8.32 / 78.61 / 87.781.78 / 60.91 / 75.490.00 / 4.08 / 17.6084.73
Uncapped verified arm15.92 / 78.18 / 87.213.86 / 40.80 / 55.920.16 / 6.59 / 19.4747.58
Soup α=0.3\alpha{=}0.3 (weights: 0.30.3 arm, 0.70.7 base)13.55 / 93.57 / 97.931.59 / 59.93 / 76.350.01 / 4.80 / 23.1380.15
Soup α=0.5\alpha{=}0.515.77 / 93.42 / 97.681.77 / 57.30 / 74.170.01 / 6.28 / 27.5173.54
Soup α=0.7\alpha{=}0.716.41 / 90.44 / 96.042.33 / 52.23 / 69.830.01 / 7.54 / 28.1462.34
V1 (verified pairs ++ replay)20.83 / 91.96 / 96.713.67 / 63.57 / 78.800.03 / 7.67 / 30.0159.80
Appendix
Table 19: The two remedies of Appendix E : mixing in weight space against mixing in data space (R@1 / R@5 / R@10 per tier; strict R@1 on cross-language duplicates). The soups blend the uncapped verified arm’s weights with the untrained base’s, α\alpha being the share of the trained arm, so α=0.3\alpha{=}0.3 sits nearest the base and α=0.7\alpha{=}0.7 nearest the arm; V1 and V2 instead train on verified pairs mixed with problem-to-solution replay, V2 under an instruction prompt. The soups recover retention as α\alpha falls but lose the hard-tier R@1 the arm had bought at every blend; V2 keeps most of both score and retention. The two control rows show how much the instruction alone does: applied to the untrained base it gives the best retention in the table, and V2 without it loses about six points. Base, uncapped-arm and soup rows are deterministic single evaluations; V1 and V2 show the seed- 4242 run, with three-seed means in Appendix E . The soups of V1 test whether the two remedies compose (Appendix E ).
ModelMELD R@1BRIGHT-TT nDCG@10BRIGHT-AoPS nDCG@10ImpliRet nDCG@10
Qwen3-Embedding-0.6B (base, untrained)10.1921.247.4416.03
Verified arm9.441.830.0013.14
Recipe arm18.8910.190.759.23
V1 (verified pairs ++ replay)9.6316.901.5916.04
V2 (the same mix ++ instruction)10.1915.262.5516.20
Qwen3-Embedding-4B (untrained)18.8931.948.6914.99
Appendix
Table 20: Three benchmarks we had no hand in building, single runs, every cell measured: MELD R@1, BRIGHT nDCG@10 on its two math theorem splits (TT == theoremqa-theorems, AoPS == the AoPS split), and ImpliRet [ 31 ] nDCG@10, macro-averaged over its six subsets (category ×\times style) of 1,5001{,}500 queries each. On MELD, whose pairs are LLM-written, the recipe arm doubles the base’s score while the verified arm stays flat; on BRIGHT both arms fall below the base, the verified arm to almost nothing, the recipe arm less; on ImpliRet’s 9,0009{,}000 queries the recipe arm is the worst model we measure while the two mixed-data models hold the base’s level. Our pre-registered prediction that the two arms would score within noise of each other is therefore partially refuted, on MELD and BRIGHT, in the recipe arm’s favour, on BRIGHT by intervals that exclude zero. The untrained 44 B model is shown for scale.
ModelEasyMediumHardSame-lang. primarySame-lang. all
ColBERTv2 (untrained)3.97 / 52.79 / 60.861.51 / 32.80 / 42.370.01 / 1.33 / 3.4346.4074.30
Verified arm6.57 / 59.83 / 69.662.17 / 30.19 / 41.430.01 / 1.24 / 3.5346.4073.90
Recipe arm39.05 / 75.69 / 81.444.28 / 30.02 / 40.090.08 / 2.35 / 5.6435.2066.06
Appendix
Table 21: The two-arm experiment on a late-interaction retriever (fact source: results/colbert/ , results/colbert_summary.json ). ColBERTv2 fine-tuned through PyLate on the recipe arm’s and the verified arm’s training files, the same 6,1456{,}145 rows, gates and dev split as the dense arms; seed 4242 , one run each. Benchmark columns are R@1 / R@5 / R@10 over the full 117,088117{,}088 -document corpus with the query’s own entry masked, from a PLAID index’s top- 1,0001{,}000 candidates re-scored by exact MaxSim, a gold outside the candidates counting as a miss. The retention columns are same-language R@1 on the reprint probe’s primary slice ( 125125 queries) and on all 498498 of its queries (Table 5 ); the cross-language probe is unusable for this English-only backbone (untrained 1.781.78 strict R@1, the arms 1.781.78 and 6.366.36 ).
TasknnBaseVerified armRecipe armRecipe −- verifiedRecipe −- base
MSE duplicate questions2511648.9432.6428.05−4.59-4.59 [−5.07,−4.11][-5.07,-4.11]−20.90-20.90 [−21.41,−20.38][-21.41,-20.38]
MathOverflow duplicate questions22572.6752.3748.22−4.15-4.15 [−9.81,1.45][-9.81,1.45]−24.45-24.45 [−30.37,−18.72][-30.37,-18.72]
MSE question to answer (ARQMath-3 Task 1)7840.0429.4041.66+12.26+12.26 [7.72,16.91][7.72,16.91]+1.62+1.62 [−2.91,6.41][-2.91,6.41]
ProofWiki question to answer109975.4159.3556.63−2.72-2.72 [−5.43,0.02][-5.43,0.02]−18.78-18.78 [−21.26,−16.30][-21.26,-16.30]
Stacks question to answer77644.5834.5929.19−5.40-5.40 [−8.07,−2.84][-8.07,-2.84]−15.39-15.39 [−18.00,−12.79][-18.00,-12.79]
ProofWiki premises108620.1215.8911.29−4.60-4.60 [−5.87,−3.39][-5.87,-3.39]−8.83-8.83 [−10.17,−7.53][-10.17,-7.53]
Appendix
Table 22: The eleven natural-language tasks of MIRB [ 32 ] , a human-curated mathematical retrieval benchmark, scored with its protocol (nDCG@10 ×100\times 100 over the top 1,0001{,}000 , each query’s excluded ids dropped, documents as title plus text; the model’s own query prompt, as for every external benchmark here). Untrained base against the two seed- 4242 arms; the last two columns are paired query bootstraps ( 5,0005{,}000 resamples, 95%95\% intervals), bold where the interval excludes zero. The MSE duplicate-question task ranks 1,350,5051{,}350{,}505 documents per query; the three formal-library premise tasks are outside the paper’s domain.
 Documents written by an LLMComputer algebraHuman authors
Trained onEasy tier (Gemini, the benchmark’s prompt)Golds rebuilt by Qwen, the benchmark’s promptGolds rebuilt by Qwen, a different-style promptNear-miss probeCross-language duplicates
Untrained base8.3210.693.2810.8484.73
Verified arm: computer-algebra pairs17.0515.0215.8097.42 †56.52
Recipe arm: benchmark’s prompt, Qwen62.3850.7544.1360.4966.26
D3 arm: different-style prompt, Qwen67.8850.5655.4168.5065.39
Reference: unrelated prompt, verified negatives46.9146.0344.0053.5178.02
Appendix
Table 23: Who built the test decides who wins. Rows are training files; columns are test sets, grouped by who wrote the documents being retrieved; each cell is R@1. The verified arm wins only on the test that computer algebra built, as its own pairs were ( †\dagger : in-family, disclosed rather than read). The two arms trained on the recipe family’s pairs take the first two places on every test an LLM built, and the D3 arm leads the recipe arm by most on the golds its own prompt wrote; the reference barely moves when the golds are rebuilt, so on the different-style golds it is level with the recipe arm ( 44.0044.00 against 44.1344.13 ; Table 24 ). On the near-miss probe the D3 arm outscores the recipe arm. The untrained base wins on duplicates no generator wrote. A better training file would win everywhere. Means over eight seeds for the two arms and the reference, three for the D3 arm, single seed- 4242 checkpoints in the two rebuilt-golds columns ( 1,5251{,}525 queries). Sources: Tables 1 , 9 , 24 and 6 .
Gold documentsBaseVerified armD1 (recipe)D2D3D4ReferenceGap
Original benchmark golds7.3415.5461.1864.9267.4130.4346.8945.64
Regenerated with the benchmark’s template10.6915.0250.7555.6750.5628.5946.0335.73
Regenerated with a paraphrase of it (D2’s prompt)8.3916.7254.3660.0756.9228.7945.9737.64
Regenerated in a different style (D3’s prompt)3.2815.8044.1352.9255.4121.7044.0028.33
Each gold from one of the three, at random7.0216.2647.7454.9554.2325.8443.9331.48
Appendix
Table 24: The regeneration defence of Appendix E (fact source: results/defense_eval/ ). Each row is the same 1,5251{,}525 easy-tier queries with their gold documents regenerated a different way; the columns give each model’s easy-tier R@1 on that version over the full 117,088117{,}088 -document corpus (seed- 4242 checkpoints), and Gap is the recipe arm (D1) minus the verified arm. Down the Gap column: regenerating the golds with the benchmark’s own template trims the lead only from 45.6445.64 to 35.7335.73 , a different-style prompt to 28.3328.33 , and drawing each gold from one of three prompts to 31.4831.48 . Across rows: D2 beats D1 on every version and D3 on all but the exact-template one, so the easy-tier plateau of Section 5 recurs, and D3’s largest lead over D1 is on the golds written with D3’s own prompt. The reference column was scored after the registered readings were taken ( eval_<version>_reference.json ): the reference drops only from 46.8946.89 to 44.0044.00 across the versions where the recipe arm drops from 61.1861.18 to 44.1344.13 , so on the different-style golds the two are level, and the recipe arm’s lead over the reference on these items shrinks from 14.2914.29 to 4.724.72 under the benchmark’s own template and to 0.130.13 under the different style.
Gold documentsBaseVerified armD1 (recipe)D2D3D4Gap
Original benchmark golds7.1016.3360.4363.2765.0428.4844.10
Regenerated with the benchmark’s template0.000.5320.1417.4810.560.5319.61
Regenerated with a paraphrase of it (D2’s prompt)0.000.7122.5419.6111.800.4421.83
Regenerated in a different style (D3’s prompt)0.000.6217.2118.1014.020.5316.59
Each gold from one of the three, at random0.000.8018.5418.1911.710.5317.74
Appendix
Table 25: The regeneration defence rerun with the benchmark’s own vendor (fact source: results/defense_eval_gemini/ ). The same 1,7001{,}700 sampled easy-tier queries, the same three prompts and the same single judge as Table 24 , but the golds are rewritten by gemini-3-flash-preview through the vendor’s API instead of by Qwen3-32B-AWQ; the 1,1271{,}127 queries the judge accepted under all three prompts are scored over the full corpus with the same seed- 4242 checkpoints. Easy-tier R@1; Gap is D1 minus the verified arm. The original-gold row differs from Table 24 ’s because the slice does. The Gap column keeps its shape (paraphrase at or above exact, style lowest, mixed between), but every regenerated row collapses for every model, the base to 0.000.00 at rank 11 , because the vendor’s fresh rewrites are a deeper rewrite than the golds the benchmark shipped (text).
More in this topic

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 175 papers

Explore similar work

CardsList

  1. LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics

    Feb 27, 2026Antoine Peyronnet, Fabian Gloeckle, Amaury HayatMathematical Reasoning BenchmarksAutomated Theorem Proving

    1Ecole Normale Superieure de Rennes · 2Ecole des Ponts Paris · 3Korean Institute for Advanced Study

  2. Re2^2Math: Benchmarking Theorem Retrieval in Research-Level Mathematics

    May 9, 2026Zicheng Lyu, Wenjie Yang, Shengzhong Zhang +1Mathematical Reasoning BenchmarksLLM Evaluation

    Fudan University Shanghai, China · Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China · Fudan University Shanghai Innovation Institute Shanghai, China

  3. RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving

    Aug 30, 2026Xiyuan Zhou, Zhuoqi Li, Xinlei Wang +6Mathematical Reasoning BenchmarksLLM Mathematical Reasoning

    Nanyang Technological University · The Chinese University of Hong Kong, Shenzhen · INSAIT, Sofia University “St. Kliment Ohridski” +2