Verifiers provide important reward signals for reinforcement learning of large language models (LLMs). However, it is challenging to develop or create reliable verifiers, especially for code generation tasks. A well-disguised wrong solution program may only be detected by carefully human-written edge cases that are difficult to synthesize automatically. To address this issue, we propose HARDTESTGEN, an approach to synthesize high-quality test cases for algorithmic coding problems. We curate a comprehensive algorithmic programming dataset HARDTESTS with 26.6k problems and high-quality synthetic tests. Compared with existing tests, HARDTESTGEN tests demonstrate significantly higher accuracy in verifying LLM-generated code (+11.22 percentage points in precision, the percentage of actually correct code within the predicted correct ones). We also show that downstream post-training -- including rejection sampling and reinforcement learning (RL) -- using HARDTESTS verifier results in improved performance of LLM code generation. We open-source our dataset and synthesis pipeline at https://leililab.github.io/HardTests/.
Figures & tables
Figure 1: HardTests test cases are significantly better than the baselines. The large improvement in precision indicates that our tests greatly reduce false positives and are indeed harder .
Figure 2: The procedure of generating test inputs and test outputs in HardTestGen .
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4+
Average
prec.
recall
prec.
recall
prec.
recall
prec.
recall
prec.
recall
Qwen2.5-Coder-7B-Instruct
TACO
96.41
79.5
75.96
80.92
53.81
65.47
17.83
90
61
78.97
CodeContests
92.6
92.67
63.86
85.69
39.3
65.57
10.81
100
51.64
85.98
HT–L
88.28
98.66
44.42
99.29
29.02
76.18
7.97
95
42.42
92.28
HT–L+R/S
94.97
98.31
53.18
99.29
62.8
75.43
47.73
95
64.67
92.01
Table 1: Precision and recall of the test cases of TACO, CodeContests, HardTests , and ablative baseline on the combined dataset of problems from AtCoder and Codeforces. HT–L refers to the results using only the test cases of LLMGen from HardTests . while HT–L+RS refers to the results using only the test cases of LLMGen, in addition to RPGen, or SPGen from HardTests .
Table 4
Figure 3: RL Validation Rewards Over Time. Reward from HardTests makes the training better.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The result status distribution of our test case generation pipeline HardTestGen .
Figure 7
platform
URL
Problem Specification Sources
Oracle Program Sources
Codeforces
https://codeforces.com/
Codeforces
TACO, CodeContests, Luogu
AtCoder
https://atcoder.jp/contests/
AtCoder
AtCoder, TACO, Luogu
Luogu
https://www.luogu.com.cn/
Luogu
Luogu
UVa
https://onlinejudge.org/
Luogu
Luogu
SPOJ
https://www.spoj.com/
Luogu
Luogu
Aizu
https://onlinejudge.u-aizu.ac.jp/
TACO
TACO
Appendix
Table 4: Problem specification sources and oracle solution sources of each online judge platform.
Oracle Program Source
Reliability
Languages
Notes
User-submitted and accepted programs from AtCoder
High
Python, C++
Some code (either Python or C++) may use AtCoder’s custom library .
Code solutions from CodeContests
High
Python 2/3, C++, Java
—
Community-authored editorials from Luogu
Medium
C++
Some editorials may lack complete, directly executable code. But if the code has no compilation or runtime errors, it is very likely to be completely correct.
Verified programs from TACO , i.e., programs that can pass all TACO’s own test cases
Medium
Python
There’s some false positives in TACO’s test cases.
Other programs from TACO
Low
Python
Reliability is not zero due to some false negatives in TACO’s test cases.
Appendix
Table 5: Oracle program sources with reliability, languages, and notes
False Positive Rate (FPR)
False Negative Rate (FNR)
AceCoder
32.49
2.59
HardTestGen
17.67
2.19
Appendix
Table 6: Performance comparison of oracle-free test generation algorithms based on error rates.
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
HT+GPT-4o
99.53
99.18
100.0
97.43
96.04
98.45
84.18
98.03
HT+Claude-4-Sonnet
99.48
99.86
100.0
95.70
98.28
99.35
93.21
96.86
HT+Kimi-K2*
99.41
99.87
98.30
97.01
98.06
99.13
87.11
98.04
HT+Qwen3-Coder*
99.47
99.14
99.62
98.88
95.20
99.13
76.83
98.82
Appendix
Table 7: Precision and recall of HardTestGen when using different LLMs as test case generators. * denotes open-weight models, and HT denotes HardTestGen .
Diffculty 1
Diffculty 2
Diffculty 3
Diffculty 4
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
SymPrompt+GPT-4
98.74
98.95
92.64
90.91
81.72
90.99
28.13
93.18
TACO+GPT-4
100.0
73.06
99.75
67.29
92.74
74.08
62.07
71.05
HT+Qwen3-Coder*
99.47
99.14
99.62
98.88
95.20
99.13
76.83
98.82
Appendix
Table 8: Comparison between HardTestGen with Qwen3-Coder and TACO/SymPrompt with GPT-4. HT denotes HardTestGen .
Figure 7: Percentage of retained test cases vs precision.
Precision
Recall
AFL++
70.00
30.43
HardTestGen
100.00
86.96
Appendix
Table 9: Comparison between HardTestGen and AFL++.
Precision
Recall
TrickCatcher
75.76
49.50
HardTestGen
96.43
77.88
Appendix
Table 10: Comparison between HardTestGen and TrickCatcher (both using the same model, GPT-4o).
Precision
Recall
F1
Line Coverage
Branch Coverage
SymPrompt
62.28
94.67
75.13
81.59
82.84
HardTestGen
95.77
90.67
93.15
92.83
93.36
Appendix
Table 11: Comparison between HardTestGen and SymPrompt (both using the same model, GPT-4o).
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4+
Average
prec.
recall
prec.
recall
prec.
recall
prec.
recall
prec.
recall
Qwen2.5-Coder-7B-Instruct
TACO
99.48
77.09
89.66
62.9
69.07
81.71
26
86.67
71.05
77.09
CodeContests*
95.24
93.93
64.12
67.74
52.5
76.83
13.64
100
56.38
84.63
HT–L
94.63
99.84
74.7
100
42.2
89.02
9.79
93.33
55.33
95.55
HT–L+R+S
97.85
99.35
97.64
100
74.23
87.8
56
93.33
81.43
95.12
Appendix
Table 12: Precision and recall of the test cases of TACO, CodeContests*, HardTests , and ablative baseline on AtCoder. HT–L refers to the results using only the test cases of LLMGen from HardTests . while HT–L+R+S refers to the results using only the test cases of LLMGen, RPGen, and SPGen from HardTests . The asterisk next to CodeContests indicates that this dataset does not provide test cases for AtCoder problems. We implemented the method described in their paper to generate the test cases.
Difficulty 1
Difficulty 2
Difficulty 3
Difficulty 4+
Average
prec.
recall
prec.
recall
prec.
recall
prec.
recall
prec.
recall
Qwen2.5-Coder-7B-Instruct
TACO
89.64
86.13
71.07
92.91
31.06
39.47
9.82
100
50.4
79.63
CodeContests
85.74
89.24
63.73
97.64
23.8
47.54
6.67
100
44.99
83.61
HT–L
74.03
95.45
34.9
98.82
16.12
55.61
5.24
100
32.57
87.47
HT–L+R+S
87.61
95.45
40.7
98.82
45.2
55.61
33.75
100
51.82
87.47
Appendix
Table 13: Precision and recall of the test cases of TACO, CodeContests, HardTests , and ablative baseline on Codeforces. HT–L refers to the results using only the test cases of LLMGen from HardTests . while HT–L+R+S refers to the results using only the test cases of LLMGen, RPGen, and SPGen from HardTests .
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, China