cs.CLSep 26, 2026
SaveDoes CoT-Pass@k Really Check the CoT? A Multilingual Mathematical Audit
Organizations: Dokuz Eylül University, ˙Izmir, Türkiye · NOVA School of Science and Technology, Universidade NOVA de Lisboa, Caparica, Portugal
Abstract
Pass@k measures whether a model reaches a correct answer under repeated sampling, but never how: a lucky guess counts the same as sound reasoning. CoT-Pass@k was proposed to close that gap, adding an LLM-as-judge that must assess a solution's reasoning chain before it counts. Its value rests entirely on one assumption: that the judge catches flawed reasoning. That assumption has never been tested inside the metric that depends on it, and never outside English, though the metric's claims concern models used in many languages. We report the first audit of that verification step, run under the metric's own protocol on a multilingual suite of five mathematical benchmarks in English, Turkish and Portuguese, two of them natively written. We corrupt correct solutions with deterministic edits that damage the chain and the final answer separately. We observe that all three judges accept corrupted chains almost as often as clean ones. V4-Flash and Qwen3.6 reject a solution sharply only when its final answer is wrong and accept a wrong answer more readily when the chain agrees with it; the metric's own judge accepts most wrong answers as well. Our study shows that chain-answer agreement dominates the two larger judges' verdicts and that all three fail to reliably detect the tested reasoning errors. Consequently the difference Pass@k - CoT-Pass@k averages 19.7 points on an earlier solver generation but only 4.1 on the current one. What little remains depends on the token budgets on both sides and on the generation mode; raising the generation budget moves Pass@64 by more than fifty points while the difference stays at zero. We close with two checks any judged reasoning metric should pass before its numbers are read as evidence about reasoning.
Figures & tables
Figure 1: CoT acceptance under injected errors per benchmark and judge (rows), majority-correct strategy of Section 4.1 , 95% Wilson intervals; the dark grey bar is the rate pooled over the five benchmarks. Every judge assesses the same 720 original solutions, 144 per benchmark and condition. R1-distill is the judge the metric specifies.
Figure 2: CoT-Pass@64 (open) and Pass@64 (filled) per solver and 64-sample benchmark under the anchor judge Qwen3.6 at 16k; row-end numbers give the difference. Q2.5/Q3.5 = Qwen2.5/Qwen3.5, G4 = Gemma-4-it.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Variant | any | majority | all |
|---|---|---|---|
| V4-Flash | |||
| Clean | 96.2 | 92.9 | 89.2 |
| Intermediate numeric error | 96.1 | 93.1 | 90.1 |
| Truncation error | 89.3 | 85.1 | 78.8 |
| Final-answer error | 47.2 | 18.1 | 6.4 |
| Consistent final-answer error | 49.0 | 24.6 | 12.4 |
Table 1: Acceptance rates (%) under the three verification strategies of the original CoT-Pass@ judge: a variant is accepted when at least one of its three judge generations approves it (any-correct), when more approve than reject (majority-correct), or when all three do (all-correct). The majority-correct column is the one plotted in Figure 1 . 720 original solutions per error type for each judge; R1-distill is the judge the metric specifies.
Figure 4: Acceptance (%, majority-correct) by the kind and the size of the injected edit, with 95% Wilson intervals, for the three conditions that change a number (columns). Top, how the wrong number was formed; bottom, its relative change . Each row of panels partitions the same 720 solutions of every condition, and the number of solutions in a category is given under its label.
Figure 5: Language of the error-injection study, by benchmark. Left, the 720 original solutions of Section 4.2 (Qwen3.5-4B and 9B, thinking mode), classed by the share of their letters, mathematics removed, in the benchmark’s language (at least 80%, benchmark language ), in English (at least 80%, English ) or in between ( mixed ); the edits do not change a chain’s language, so the five variants share the classification. Right, the judges’ output on the clean controls, thinking and visible answer together, classed the same way. On English AIME every chain and every judgment is English, so that benchmark is left out. Under the wrong-answer conditions Qwen3.6’s shares stay within 1 point of these, R1-distill’s all-English share on the Turkish benchmarks falls from 36 to 37% to 19 to 26%, and V4-Flash’s mixed share on the Portuguese exams falls from 10% to under 3%. Appendix C.5 gives the method.
| Solver | Qwen3.6 | Gemma4 | V4-Flash | R1-distill |
|---|---|---|---|---|
| Q2.5-7B | 15.6 | 13.9 | 18.0 | 15.6 |
| Q2.5-32B | 20.5 | 22.1 | 23.0 | 21.3 |
| Q2.5-7B-I | 22.1 | 23.0 | 23.8 | 22.1 |
| Q2.5-32B-I | 20.5 | 20.5 | 21.3 | 21.3 |
| Average | 19.7 | 19.9 | 21.5 | 20.1 |
Table 2: Pass@64 CoT-Pass@64 per earlier-generation solver (rows), averaged over the four 64-sample benchmarks (weighted by how many questions each contributes, as in Figures 6 and 7 ) under all four judges (columns). Judge generation budgets: 16,384 for Qwen3.6, Gemma4 and R1-distill, 20,480 for V4-Flash; every judge runs in thinking mode with three judgments per solution. The last row averages the solvers. Q2.5 = Qwen2.5 (-I Instruct).
| Solver | Qwen3.6 | Gemma4 |
|---|---|---|
| Q3.5-4B-B | 0.8 | 1.6 |
| Q3.5-9B-B | 0.0 | 0.8 |
| Q3.5-4B | 4.9 | 6.6 |
| Q3.5-9B | 3.3 | 4.1 |
| G4-E2B | 8.2 | 8.2 |
| G4-E4B | 7.4 | 4.9 |
Table 3: As Table 2 , for the current-generation solvers under the two judges that cover them. Q3.5 = Qwen3.5 (-B base), G4 = Gemma-4-it.
Figure 6: The difference Pass@ CoT-Pass@ against the number of sampled generations, under the anchor judge Qwen3.6 at 16k. Each solver is a weighted mean over its four benchmarks, weighted by how many questions each contributes; each curve is then the unweighted mean over the solvers of that generation. Endpoints are labelled; Figure 7 gives the per-solver spread at .
Figure 7: Question-level bootstrap 95% intervals for Pass@ CoT-Pass@ under the anchor judge Qwen3.6 at 16k, in points, with the numeric interval at the right of each row. Each solver is a weighted mean over its four benchmarks, weighted by how many questions each contributes; mean is the unweighted mean over the solvers of a generation. 10,000 resamples of questions, percentile intervals; cells are resampled independently. Q2.5/Q3.5 = Qwen2.5/Qwen3.5, G4 = Gemma-4-it.
Figure 8: Acceptance rate of answer-correct generations per solver and benchmark, the share of a solver’s correct generations whose reasoning chain the judge approved, under Qwen3.6 (top) and Gemma4 (bottom), for the solvers of Section 5.2 at the 16k budget. The number above each bar is that rate in percent; the number at its base is the count of answer-correct generations in the cell, given in parentheses after the rate where the bar is too short. The rejected share is 100 minus the rate. The dark grey bar of each solver gives its rate pooled over the four benchmarks, with the pooled count at its base, and the panel headers give the rate pooled over each generation. The vertical axis starts at 30. Q2.5/Q3.5 = Qwen2.5/Qwen3.5 (-Inst instruct, -Base base), G4 = Gemma-4-it.
Figure 9: CoT-Pass@ after conditioning on , the number of a question’s 64 generations whose final answer is correct, counted before judging, for the ten solvers of Section 5.2 at the 16k budget, pooled over the four benchmarks, under both judges (rows), with 95% Wilson intervals. Left, the share of question and solver pairs in the bin in which the judge approved at least one correct generation, which is CoT-Pass@ for that bin since Pass@ there is 100 by construction; right, the share of answer-correct generations in the bin that the judge approved, the acceptance rate of Figure 8 at fixed , so that the two generations can be compared at the same . The tick labels give the number of question and solver pairs in each bin, earlier | current. Per benchmark, the Turkish olympiad has the lowest value in both panels in every bin of the earlier generation, and Figure 8 gives the acceptance rates per benchmark.
Figure 10: Language of the reasoning chains (left) and acceptance by language (right) for the answer-correct generations of the ten solvers of Section 5.2 at the 16k budget, on the three non-English benchmarks pooled (top) and on the Portuguese exams (bottom), where thinking runs stand in for the non-thinking runs they lack. A chain is in the benchmark language when at least 80% of its letters, mathematics removed, are in that language, English when at least 80% are English, mixed otherwise. Acceptance is the majority verdict of Qwen3.6 (filled) and Gemma4 (open); a class with fewer than 20 chains of a solver is not plotted, and is the number of chains. The dotted line separates the earlier generation from the current one. Pooling mixes benchmarks with different acceptance rates, so within-solver comparisons are read per benchmark in Appendix C.5 . Q2.5/Q3.5 = Qwen2.5/Qwen3.5 (-Inst instruct, -Base base, -Think thinking mode), G4 = Gemma-4-it.
Figure 11: Language of the answer-correct chains and of the judgments in the judge matrix of Section 5.2 , per benchmark. Top, the ten solvers at the 16k budget, earlier generation left of the dotted line and current generation right of it (-Inst instruct, -Base base); bottom, a sample of 300 judgments per judge and benchmark. Classes as in Figure 5 . English AIME is left out, every chain and judgment there being English. Figure 10 gives the same shares pooled over the three benchmarks together with acceptance by class.
| solver | Gemma4 | Qwen3.6 | ||||
|---|---|---|---|---|---|---|
| Benchmark | Solver | stops | stops | diff. | stops | diff. |
| AIME (EN) | Q3.5-4B | 89.6 | 35.0 | 3.33 | 15.7 | 0.00 |
| Q3.5-9B | 83.9 | 33.0 | 3.33 | 17.5 | 0.00 | |
| G4-E2B | 0.2 | 2.1 | 6.67 | 6.5 | 3.33 | |
| G4-E4B | 1.4 | 2.7 | 10.00 | 6.0 | 3.33 | |
| AIME (PT) | Q3.5-4B | 47.7 | 53.5 | 13.33 | 5.2 | 0.00 |
Table 4: Stops at the max-token limit on both sides, in thinking mode at the 16k generation budget: the share of solver generations that stop at the max-token limit (stops, %), and per judge the share of its judgments that do, with the difference Pass@ CoT-Pass@ (diff.) under that judge. is 64, or 16 for the Portuguese exams; stop rates come from each generation’s recorded finish state. Q3.5 = Qwen3.5, G4 = Gemma-4-it.
Figure 12: The difference Pass@ CoT-Pass@ per judge and solver mode, split into the part carried by the judge’s verdicts (solid) and the part that disappears when judgments stopped at the max-token limit are counted as approvals (hatched). Cell means over the four 64-sample benchmarks and the four current-generation solvers, 16k budget on both sides.
| stops at max | Pass@1 | Pass@64 | |||||
|---|---|---|---|---|---|---|---|
| tokens (%) | (%) | (%) | |||||
| Benchmark | Solver | low | high | low | high | low | high |
| AIME (EN) | 4B | 89.6 | 18.4 | 10.5 | 75.4 | 40.0 | 96.7 |
| 9B | 83.9 | 8.9 | 16.2 | 84.5 | 40.0 | 96.7 | |
| AIME (PT) | 4B | 47.7 | 7.6 | 51.0 | 71.0 | 83.3 | 93.3 |
| 9B | 44.0 | 3.8 | 55.3 | 78.2 | 80.0 | 96.7 | |
Table 5: The budget ladder: the same solver on the same benchmark at its low and high generation budget. Low is 16,384 tokens everywhere; high is 32,768, except 65,536 on English AIME (the text explains the jump). Share of solver generations stopping at the max-token limit, Pass@ and Pass@ ; thinking mode throughout. 4B/9B = Qwen3.5.
| Mode | Lang. | stops at max | Pass@1 | Pass@64 |
|---|---|---|---|---|
| tokens (%) | (%) | (%) | ||
| non-thinking | EN | 27.9 | 55.8 | 91.7 |
| PT | 2.4 | 54.7 | 95.0 | |
| TR | 16.6 | 35.7 | 88.3 | |
| thinking | EN | 86.7 | 13.4 | 40.0 |
| PT | 45.8 | 53.2 | 81.7 |
Table 6: The same thirty AIME problems in English and in the Portuguese and Turkish translations: Qwen3.5-{4B,9B} at the 16k generation budget, both modes. Columns give the share of generations that stop at the max-token limit, Pass@1 and Pass@64, averaged over questions and over the two models.
Figure 13: Language of the answer-correct chains under the generation mode and the token budget of Section 5.3 , per benchmark. Top, the four current-generation solvers at the 16k budget in non-thinking and in thinking mode; bottom, the two Qwen3.5 thinking-mode solvers at the low and the raised generation budget of the ladder (Table 5 ). Classes as in Figure 5 ; English AIME left out. Turning thinking on moves every chain out of the benchmark’s language, and raising the budget moves the mixed chains further toward English.
More in this topic
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 555 papers