How Much Is Left Matters: False Positives and False Negatives in Partial-Prompt Contamination Tests
Organizations: Meta
Abstract
Partial-prompt completion, where a model receives the first part of a benchmark problem and is checked for whether it reproduces the rest, is a widely used test for benchmark contamination. We audit this test on seven open-weight models using MATH-500 and recent AIME 2025/2026 problems, which serve as true negatives. Memorization flags depend strongly on how much text remains: for every model, problems with at most 8 leftover words are flagged several times more often than problems with at least 25 (31.5% vs. 5.5% for Qwen2.5-7B-Instruct), and half to three-quarters of all ROUGE-based flags come from short leftovers. On true negatives, a ROUGE-L >= 0.75 threshold flags up to 6.7% of problems at a 60% prefix, and at an 80% prefix even exact match flags 8.6% (pooled over models; 6.2% on AIME 2026, which postdates every model), against 0% at 40-60%. The test also misses contamination upstream: models distilled from Qwen2.5-Math, whose base is heavily flagged in prior work, show almost no model-specific recall. Calibrating each model against models from other families removes most flags while isolating a consistent Qwen2.5-specific signal at every prefix length. We recommend length-stratified reporting, prefixes of at most 60%, style-matched post-cutoff controls, and cross-family calibration.
Figures & tables
| MATH-500 (%) | AIME 25/26 (%) | ||||||
|---|---|---|---|---|---|---|---|
| Model | Raw | Specific | EM | 8 w | 25 w | ROUGE FP | EM FP |
| Qwen2.5-1.5B-Inst. | 14.6 | 5.8 | 4.4 | 20.2 | 3.6 | 3.3 | 0.0 |
| Qwen2.5-7B-Inst. | 22.6 | 11.4 | 8.8 | 31.5 | 5.5 | 6.7 | 0.0 |
| R1-Distill-Qwen-1.5B | 5.8 | 0.8 | 1.2 | 9.6 | 0.0 | 1.7 | 0.0 |
| R1-Distill-Qwen-7B | 9.8 | 1.6 | 2.6 | 15.7 | 0.0 | 0.0 | 0.0 |
| Qwen3-1.7B | 6.6 | 0.6 | 1.2 | 13.5 | 0.0 | 0.0 | 0.0 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Prefix 40% | Prefix 60% | Prefix 80% | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Raw | Spec. | EM | EM FP | Raw | Spec. | EM | EM FP | Raw | Spec. | EM | EM FP |
| Qwen2.5-1.5B-Inst. | 7.4 | 3.6 | 1.4 | 0.0 | 14.6 | 5.8 | 4.4 | 0.0 | 24.6 | 6.4 | 16.4 | 11.7 |
| Qwen2.5-7B-Inst. | 13.0 | 9.8 | 3.6 | 0.0 | 22.6 | 11.4 | 8.8 | 0.0 | 35.8 | 12.0 | 25.4 | 13.3 |
| R1-Distill-Qwen-1.5B | 1.6 | 0.2 | 0.2 | 0.0 | 5.8 | 0.8 | 1.2 | 0.0 | 14.2 | 1.8 | 8.2 | 5.0 |
| R1-Distill-Qwen-7B | 2.8 | 0.2 | 0.6 | 0.0 | 9.8 | 1.6 | 2.6 | 0.0 | 19.2 | 3.4 | 10.4 | 6.7 |
| Qwen3-1.7B | 1.8 | 0.4 | 0.2 | 0.0 | 6.6 | 0.6 | 1.2 | 0.0 | 13.2 | 1.0 | 5.0 | 1.7 |
| Words left | 8 ( =178) | 9–14 ( =162) | 15–24 ( =105) | 25+ ( =55) |
|---|---|---|---|---|
| Qwen2.5-1.5B-Inst. | 20.2 | 16.0 | 8.6 | 3.6 |
| Qwen2.5-7B-Inst. | 31.5 | 22.8 | 16.2 | 5.5 |
| R1-Distill-Qwen-1.5B | 9.6 | 6.2 | 1.9 | 0.0 |
| R1-Distill-Qwen-7B | 15.7 | 9.9 | 4.8 | 0.0 |
| Qwen3-1.7B | 13.5 | 4.9 | 1.0 | 0.0 |
| Qwen3-4B-Inst.-2507 | 17.4 | 8.0 | 2.9 | 0.0 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,527 papers