Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of k independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@k. However, prior work reports a trade-off: pass@k improves while pass@1 degrades under such methods. This trade-off is practically important because pass@1 often remains a hard operational constraint due to latency and cost budgets, imperfect verifier coverage, and the need for a reliable single-shot fallback. We study the origin of this trade-off and provide a theoretical characterization of when pass@k policy optimization can reduce pass@1 through gradient conflict induced by prompt interference. This interference arises because shared model parameters couple prompts, so improving performance on some prompts can adversely affect others. We show that pass@k policy gradients can conflict with pass@1 gradients because pass@k optimization implicitly reweights prompts toward low-success prompts; when these prompts are negatively interfering, their upweighting can rotate the pass@k update direction away from the pass@1 direction. We illustrate our findings with large language model experiments on verifiable mathematical reasoning and coding tasks.
Figures & tables
Figure 1: (a) Empirical trade-off. Under pass@ k policy optimization, pass@ k increases while pass@ 1 may decrease. We explain this empirically observed trade-off in (b) and (c) , which schematically illustrate the pass@ 1 and pass@ k ( k>1 ) gradients for three prompts and their expectations in policy-parameter space. (b) Pass@1 gradients with negatively interfering prompt. This panel shows a setting in which prompt 3 is negatively interfering with prompts 1 and 2 , i.e., the per-prompt pass@1 gradient for prompt 3 has negative inner product with the per-prompt pass@ 1 gradients for prompts 1 and 2 (see Section 3 ). Here, ∇ pass@ 1 denotes the population pass@ 1 gradient, given by the average (expectation over prompts; here under a uniform distribution) of the per-prompt pass@ 1 gradients. (c) Pass@ k vs pass@1 gradient conflict. Per-prompt pass@k gradients are scaled versions of the corresponding per-prompt pass@1 gradients (Eq. ( 2 )). This reweighting amplifies the magnitude of the pass@ k gradient for the negatively interfering prompt 3 , causing the resulting population pass@ k gradient to conflict with the population pass@ 1 gradient (their inner product becomes negative, corresponding to an obtuse angle, shown here as approximately 120∘ ). Hence, a policy update in the pass@ k gradient direction can increase pass@ k while decreasing pass@1.
Figure 2: The prompt distribution has prompts with negative and positive agreement scores aθ(x) ( ) on the toy example (Appendix ). Prompts with negative scores are responsible for gradient conflict and hence pass@1 decrease. After pass@ k optimization, negative scores get even more negative.
Figure 4: Pass@ k -induced reweighting produces population-gradient conflict in filtered MATH subsets. Both models demonstrate how pass@ k reweighting causes gradient conflict with pass@1, despite different threshold settings. Agreement scores are computed relative to the pass@1 gradient of filtered prompts. Left column (A): Agreement scores show separation between hard negatives (red) and easy positives (green), with unweighted means shown as blue dashed lines. Middle column (B): Pass@ k weights grouped by pass@1 probability, showing extreme weight disparity ( ∼ 10 28 :1) between hard and easy prompts. Right column (C): Weighted contributions (estimates of pass@ k vs pass@1 inner products) demonstrate how pass@ k reweighting induces negative pass@ k /pass@1 gradient alignment. Purple arrows highlight the critical downward shift from the unweighted mean (blue dotted, same as plot A) to the weighted mean (red dashed). (a) DeepSeek-Llama: The downward shift from +2.80×10−3 to −1.12×10−3 ( Δ=−3.92×10−3 ) flips the gradient from positive to negative, resulting in inner product −0.613 . (b) DeepSeek-Qwen: The dramatic downward shift from +2.97×10−2 to −2.74×10−1 ( Δ=−3.04×10−1 ) demonstrates even stronger gradient conflict. Despite only 29 hard prompts vs 627 easy (21.6:1 ratio), the extreme weight disparity causes hard negatives to completely dominate, resulting in inner product −181 .
Figure 5: Cosine kernel heatmap: cos(∇pθ(x),∇pθ(x′)) for subsamples of prompts: 120 easy and 80 hard among a total of 6000 samples. Blue regions show negative prompt interference.
Figure 6: Contour plots of pass@1 and pass@ k objectives in the policy parameter space. Gradients of pass@ k and pass@1 with respect to policy parameters are conflicting in the gray area.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Gradient conflict across all threshold configurations (DeepSeek-R1-Distill-Llama-8B). Each panel shows agreement scores, weights, and weighted contributions for a different (δ1,δ2) configuration. All exhibit negative inner products, confirming robust misalignment across thresholds. The consistent downward shift from unweighted to weighted means shows pass@ k optimization systematically opposes pass@1 improvement.
Figure 8: Pass@ k amplification pattern across models and thresholds. Each point represents one prompt, with the corresponding agreement score (x-axis), pass@ k weight (y-axis), and pass@1 probability shown by color (red = low, green = high). Consistent pattern: Prompts with negative alignment (left), low pass@1 (red), and high weights (4-12) cluster upper-left, while prompts with positive alignment (right), high pass@1 (green), and zero weights ( ∼ 10 -28 ) appear lower-right. Top row (a-c): DeepSeek-R1-Distill-Llama-8B across three δ2 values (0.05, 0.10, 0.15). Bottom row (d-f): DeepSeek-R1-Distill-Qwen-7B with same thresholds.
Figure 9: Pass@ k amplification and negative prompt interference in HumanEval dataset using a Qwen 1.5B model for different values of k . Each point represents one prompt, with the corresponding agreement score (x-axis), pass@ k weight (y-axis), and pass@1 probability shown by color (red = low, green = high). Consistent pattern: Prompts with negative alignment (left), low pass@1 (red), and high weights (4-12) cluster upper-left, while prompts with positive alignment (right), high pass@1 (green), and small weights appear lower-right.
Post-training is central to mathematical reasoning in modern large language models (LLMs), but endpoint pass@1 alone underidentifies what has changed. Gains may reflect newly reachable solutions, cheaper sampling of latent solutions, surface robustness, or memorisation. We compare three post-training paths under a common diagnostic readout: our sufficiently trained off-policy distillation trajectories, released Qwen3 off-policy-plus-on-policy distillation endpoints, and a released DeepSeek-Math endpoint trained with Group Relative Policy Optimisation (GRPO). Our probe uses cross-surface pass@K over verbatim prompts, paraphrases, numerical isomorphisms, and translations, plus consistency, distribution-shape, and verified supervised-fine-tuning (SFT) membership analyses. We find two regimes. On easier AMC problems, large-K ceilings are near saturation, so post-training mainly compresses sample cost. On harder AIME problems, post-training expands the large-K ceiling over the base model: sufficient off-policy distillation already raises this ceiling, Qwen3 released endpoints raise it further, and DeepSeek-Math GRPO does not dominate sufficient off-policy distillation at large K. English-dominant distillation improves non-English reasoning but preserves language-tier gaps. A controlled-overfit audit finds limited sensitivity in current SFT-membership probes. Compression is one regime of post-training, not a universal explanation.
Hongyang Li, Yiming Zhu, Xiao Li +3
University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay
Pass@k measures whether a model reaches a correct answer under repeated sampling, but never how: a lucky guess counts the same as sound reasoning. CoT-Pass@k was proposed to close that gap, adding an LLM-as-judge that must assess a solution's reasoning chain before it counts. Its value rests entirely on one assumption: that the judge catches flawed reasoning. That assumption has never been tested inside the metric that depends on it, and never outside English, though the metric's claims concern models used in many languages. We report the first audit of that verification step, run under the metric's own protocol on a multilingual suite of five mathematical benchmarks in English, Turkish and Portuguese, two of them natively written. We corrupt correct solutions with deterministic edits that damage the chain and the final answer separately. We observe that all three judges accept corrupted chains almost as often as clean ones. V4-Flash and Qwen3.6 reject a solution sharply only when its final answer is wrong and accept a wrong answer more readily when the chain agrees with it; the metric's own judge accepts most wrong answers as well. Our study shows that chain-answer agreement dominates the two larger judges' verdicts and that all three fail to reliably detect the tested reasoning errors. Consequently the difference Pass@k - CoT-Pass@k averages 19.7 points on an earlier solver generation but only 4.1 on the current one. What little remains depends on the token budgets on both sides and on the generation mode; raising the generation budget moves Pass@64 by more than fifty points while the difference stays at zero. We close with two checks any judged reasoning metric should pass before its numbers are read as evidence about reasoning.
Tarık Tuna Taşaltı, Burcu Hüdaverdi, David Semedo
Dokuz Eylül University, ˙Izmir, Türkiye · NOVA School of Science and Technology, Universidade NOVA de Lisboa, Caparica, Portugal
Majority voting over multiple LLM attempts improves mathematical reasoning, but correlated errors limit the effective sample size. A natural fix is to assign different reasoning strategies to different voters. The approach, Diverse Prompt Mixer, is tested on the AIMO 3 competition: 3 models, 23+ experiments, 50 IMO-level problems, one H100 80 GB, 5-hour limit. Every prompt-level intervention fails. High-temperature sampling already decorrelates errors; weaker strategies reduce accuracy more than they reduce correlation. Across an 8-point capability gap at equal N=8 and every optimization tested, model capability dominates. The gap between the best majority-vote score (42/50) and pass@20 (~45.5) is selection loss, not prompt loss. A verifier-based selector could close it. Prompt engineering cannot.