Verifiers provide important reward signals for reinforcement learning of large language models (LLMs). However, it is challenging to develop or create reliable verifiers, especially for code generation tasks. A well-disguised wrong solution program may only be detected by carefully human-written edge cases that are difficult to synthesize automatically. To address this issue, we propose HARDTESTGEN, an approach to synthesize high-quality test cases for algorithmic coding problems. We curate a comprehensive algorithmic programming dataset HARDTESTS with 26.6k problems and high-quality synthetic tests. Compared with existing tests, HARDTESTGEN tests demonstrate significantly higher accuracy in verifying LLM-generated code (+11.22 percentage points in precision, the percentage of actually correct code within the predicted correct ones). We also show that downstream post-training -- including rejection sampling and reinforcement learning (RL) -- using HARDTESTS verifier results in improved performance of LLM code generation. We open-source our dataset and synthesis pipeline at https://leililab.github.io/HardTests/.
Figures & tables
Figure 1: HardTests test cases are significantly better than the baselines. The large improvement in precision indicates that our tests greatly reduce false positives and are indeed harder .
Figure 2: The procedure of generating test inputs and test outputs in HardTestGen .
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4+
Average
prec.
recall
prec.
recall
prec.
recall
prec.
recall
prec.
recall
Qwen2.5-Coder-7B-Instruct
TACO
96.41
79.5
75.96
80.92
53.81
65.47
17.83
90
61
78.97
CodeContests
92.6
92.67
63.86
85.69
39.3
65.57
10.81
100
51.64
85.98
HT–L
88.28
98.66
44.42
99.29
29.02
76.18
7.97
95
42.42
92.28
HT–L+R/S
94.97
98.31
53.18
99.29
62.8
75.43
47.73
95
64.67
92.01
Table 1: Precision and recall of the test cases of TACO, CodeContests, HardTests , and ablative baseline on the combined dataset of problems from AtCoder and Codeforces. HT–L refers to the results using only the test cases of LLMGen from HardTests . while HT–L+RS refers to the results using only the test cases of LLMGen, in addition to RPGen, or SPGen from HardTests .
Table 4
Figure 3: RL Validation Rewards Over Time. Reward from HardTests makes the training better.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The result status distribution of our test case generation pipeline HardTestGen .
Figure 7
platform
URL
Problem Specification Sources
Oracle Program Sources
Codeforces
https://codeforces.com/
Codeforces
TACO, CodeContests, Luogu
AtCoder
https://atcoder.jp/contests/
AtCoder
AtCoder, TACO, Luogu
Luogu
https://www.luogu.com.cn/
Luogu
Luogu
UVa
https://onlinejudge.org/
Luogu
Luogu
SPOJ
https://www.spoj.com/
Luogu
Luogu
Aizu
https://onlinejudge.u-aizu.ac.jp/
TACO
TACO
Appendix
Table 4: Problem specification sources and oracle solution sources of each online judge platform.
Oracle Program Source
Reliability
Languages
Notes
User-submitted and accepted programs from AtCoder
High
Python, C++
Some code (either Python or C++) may use AtCoder’s custom library .
Code solutions from CodeContests
High
Python 2/3, C++, Java
—
Community-authored editorials from Luogu
Medium
C++
Some editorials may lack complete, directly executable code. But if the code has no compilation or runtime errors, it is very likely to be completely correct.
Verified programs from TACO , i.e., programs that can pass all TACO’s own test cases
Medium
Python
There’s some false positives in TACO’s test cases.
Other programs from TACO
Low
Python
Reliability is not zero due to some false negatives in TACO’s test cases.
Appendix
Table 5: Oracle program sources with reliability, languages, and notes
False Positive Rate (FPR)
False Negative Rate (FNR)
AceCoder
32.49
2.59
HardTestGen
17.67
2.19
Appendix
Table 6: Performance comparison of oracle-free test generation algorithms based on error rates.
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
HT+GPT-4o
99.53
99.18
100.0
97.43
96.04
98.45
84.18
98.03
HT+Claude-4-Sonnet
99.48
99.86
100.0
95.70
98.28
99.35
93.21
96.86
HT+Kimi-K2*
99.41
99.87
98.30
97.01
98.06
99.13
87.11
98.04
HT+Qwen3-Coder*
99.47
99.14
99.62
98.88
95.20
99.13
76.83
98.82
Appendix
Table 7: Precision and recall of HardTestGen when using different LLMs as test case generators. * denotes open-weight models, and HT denotes HardTestGen .
Diffculty 1
Diffculty 2
Diffculty 3
Diffculty 4
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
SymPrompt+GPT-4
98.74
98.95
92.64
90.91
81.72
90.99
28.13
93.18
TACO+GPT-4
100.0
73.06
99.75
67.29
92.74
74.08
62.07
71.05
HT+Qwen3-Coder*
99.47
99.14
99.62
98.88
95.20
99.13
76.83
98.82
Appendix
Table 8: Comparison between HardTestGen with Qwen3-Coder and TACO/SymPrompt with GPT-4. HT denotes HardTestGen .
Figure 7: Percentage of retained test cases vs precision.
Precision
Recall
AFL++
70.00
30.43
HardTestGen
100.00
86.96
Appendix
Table 9: Comparison between HardTestGen and AFL++.
Precision
Recall
TrickCatcher
75.76
49.50
HardTestGen
96.43
77.88
Appendix
Table 10: Comparison between HardTestGen and TrickCatcher (both using the same model, GPT-4o).
Precision
Recall
F1
Line Coverage
Branch Coverage
SymPrompt
62.28
94.67
75.13
81.59
82.84
HardTestGen
95.77
90.67
93.15
92.83
93.36
Appendix
Table 11: Comparison between HardTestGen and SymPrompt (both using the same model, GPT-4o).
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4+
Average
prec.
recall
prec.
recall
prec.
recall
prec.
recall
prec.
recall
Qwen2.5-Coder-7B-Instruct
TACO
99.48
77.09
89.66
62.9
69.07
81.71
26
86.67
71.05
77.09
CodeContests*
95.24
93.93
64.12
67.74
52.5
76.83
13.64
100
56.38
84.63
HT–L
94.63
99.84
74.7
100
42.2
89.02
9.79
93.33
55.33
95.55
HT–L+R+S
97.85
99.35
97.64
100
74.23
87.8
56
93.33
81.43
95.12
Appendix
Table 12: Precision and recall of the test cases of TACO, CodeContests*, HardTests , and ablative baseline on AtCoder. HT–L refers to the results using only the test cases of LLMGen from HardTests . while HT–L+R+S refers to the results using only the test cases of LLMGen, RPGen, and SPGen from HardTests . The asterisk next to CodeContests indicates that this dataset does not provide test cases for AtCoder problems. We implemented the method described in their paper to generate the test cases.
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4+
Average
prec.
recall
prec.
recall
prec.
recall
prec.
recall
prec.
recall
Qwen2.5-Coder-7B-Instruct
TACO
89.64
86.13
71.07
92.91
31.06
39.47
9.82
100
50.4
79.63
CodeContests
85.74
89.24
63.73
97.64
23.8
47.54
6.67
100
44.99
83.61
HT–L
74.03
95.45
34.9
98.82
16.12
55.61
5.24
100
32.57
87.47
HT–L+R+S
87.61
95.45
40.7
98.82
45.2
55.61
33.75
100
51.82
87.47
Appendix
Table 13: Precision and recall of the test cases of TACO, CodeContests, HardTests , and ablative baseline on Codeforces. HT–L refers to the results using only the test cases of LLMGen from HardTests . while HT–L+R+S refers to the results using only the test cases of LLMGen, RPGen, and SPGen from HardTests .
Large language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic knowledge and reasoning about how to apply it. However, existing LLM pipelines are opaque, with no explicit separation between these two components. We argue that for well-known algorithms whose canonical implementations are widely accessible in pretraining corpora, code generation is better measured as parametric code retrieval: reproducing a named algorithm from internalised knowledge rather than synthesizing a novel one. We introduce AlgoREval, a benchmark of 599 problems spanning classical 77 algorithms across 14 domains, 7 programming languages, and 4 graph-input representations to evaluate this capability in isolation, and assess 15 models (7B--34B parameters) in a zero-shot setting. We find substantial variation in retrieval accuracy across languages and input representations, even for widely documented algorithms and show that prompt augmentation with retrieved code snippets or structured algorithmic hints improve accuracy on complex algorithms, while SFT achieves broader language gains and GRPO achieves larger per-language gains on specific languages. Together, our results establish parametric code retrieval as a distinct, measurable capability and caution against deploying AI-generated algorithmic code without systematic validation.\footnote{Code and dataset are available at https://github.com/Nickil21/AlgoREval
Nickil Maveli, Antonio Vergari, Shay B. Cohen
School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB
Reinforcement Learning from Verifiable Rewards (RLVR) is pivotal for enhancing LLM code generation, yet its efficacy is often hindered by insufficient test case coverage, leading to reward hacking and policy degradation. To address this, we propose RobustTests, a framework featuring a faulty-code-driven test case synthesis strategy. By leveraging "near-correct" faulty codes, RobustTests captures latent logical discrepancies and employs validator agents with behavioral feature clustering to filter invalid or redundant test cases. Additionally, a stepwise dense reward function based on pass rates is introduced to mitigate false negatives and enhance training robustness. Using this pipeline, we construct an augmented version of the CodeContests+ dataset with superior diagnostic utility. Experimental results show that RL fine-tuning of Qwen3-32B via RobustTests achieves a 3% absolute gain on LiveCodeBench, demonstrating its effectiveness in advancing LLM code generation proficiency. Codes and data are available at https://huggingface.co/datasets/sid6/RobustTests.
Test case synthesis is crucial for evaluating and ranking programs generated by large language models (LLMs). However, constructing high-quality test cases remains challenging because reliable expected outputs are often difficult to obtain. We propose Confidence-Gated Transductive Test Generation (CoTT), which first uses an efficient inductive procedure and invokes transductive generation only when inductive confidence is low. This adaptive design improves output reliability while allocating extra computation only when needed. On code reranking benchmarks, CoTT outperforms prior baselines across the reported metrics while reducing cost relative to applying transductive generation to every input. These results show that confidence-based allocation of test-time computation provides a favorable efficiency-effectiveness trade-off with a single efficient LLM.
Sungjae Lee, Youngsik Yoon, Seockbean Song +3
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, China