The evaluation of Large Language Models (LLMs) for code generation relies heavily on the quality and robustness of test cases. However, existing benchmarks often lack coverage for subtle corner cases, allowing incorrect solutions to pass. To bridge this gap, we propose CodeHacker, an automated agent framework dedicated to generating targeted adversarial test cases that expose latent vulnerabilities in program submissions. Mimicking the hack mechanism in competitive programming, CodeHacker employs a multi-strategy approach, including stress testing, anti-hash attacks, and logic-specific targeting to break specific code submissions. To ensure the validity and reliability of these attacks, we introduce a Calibration Phase, where the agent iteratively refines its own Validator and Checker via self-generated adversarial probes before evaluating contestant code.Experiments demonstrate that CodeHacker significantly improves the True Negative Rate (TNR) of existing datasets, effectively filtering out incorrect solutions that were previously accepted. Furthermore, generated adversarial cases prove to be superior training data, boosting the performance of RL-trained models on benchmarks like LiveCodeBench.
Figures & tables
Figure 1: The overall architecture of the CodeHacker framework. Phase I (Evaluation Tool Calibration): The agent iteratively refines the judging infrastructure to ensure reliability. This process begins by refining the Validator to strictly enforce input constraints, followed by refining the Checker to eliminate false verdicts. Phase II (Adversarial Case Generation): Utilizing the calibrated tools, the Code Analyst guides three distinct generation strategies (Stress, LLM-based, and Anti-hash) to synthesize adversarial test cases that expose specific vulnerabilities in the contestant’s submission.
Dataset
Overall
Traditional Judge
Special Judge
VPR ( %↑ )
TPR ( %↓ )
TNR ( %↑ )
TPR ( %↓ )
TNR ( %↑ )
CodeContests Li et al. (2022b)
71.41
98.96
76.33
84.73
77.69
HardTests He et al. (2025)
97.32
98.33
79.25
86.56
75.37
TACO Li et al. (2023a)
81.84
96.46
83.70
82.67
85.48
CodeContest + Wang et al. (2025c)
99.66
96.02
85.72
96.62
84.04
CodeContest ++ (Ours)
100.00
95.86
96.31
96.38
96.05
Table 1: Comparison of Overall Validation Pass Rate (VPR) of original datasets, and correctness metrics (TPR/TNR) across 2000 problems with among different datasets.
Backbone Model
HSR (%)
Avg. #T
Reasoning Models
DeepSeek V3.2
64.83
1.24
GPT-5-Mini
51.40
1.35
Qwen3-Next-80B-A3B
38.37
1.58
Gemini-3.0-Flash-Preview
35.65
1.62
Qwen3-32B
23.64
1.89
Table 2: Hack performance across different models. HSR : Hack Success Rate. Avg. #T : The average number of refinement turns required to successfully hack a submission by directly LLM generated (lower is better, max 5). The symbol * indicates non-thinking mode for hybrid-reasoning models.
Model
Pass@1
Adjusted Pass@1
Δ
DeepSeek V3.2
83.3%
81.6%
-1.7%
GPT-5-Mini
78.4%
76.7%
-1.7%
Gemini-3.0-Flash
78.6%
76.5%
-2.1%
DeepSeek V3.2*
61.9%
59.7%
-2.2%
Qwen3-32B*
51.0%
47.5%
-3.5%
Table 3: Pass@1 Performance Correction on CodeHackerBench. The drop quantifies the proportion of "lucky" submissions that contained latent bugs but passed the original weak tests.The symbol * indicates non-thinking mode for hybrid-reasoning models.
Figure 2: Pass@5 performance comparison on LiveCodeBench. Models trained with adversarial data show consistent improvements. The model trained on our augmented subset achieves the highest accuracy across all difficulty levels.
Variant
Hack Success Rate (% ↑ )
CodeHacker (Full)
51.40
w/o Code Analyst
49.86
w/o Stress Test
50.12
w/o Anti-Hash Generator
51.00
w/o Refinement Loop
46.60
Table 4: Ablation study of CodeHacker agent components.
Configuration
TNR (% ↑ )
TPR (%)
Baseline (CodeContest + )
82.18
95.34
+ Refined Validator
82.08
95.50
+ Refined Checker
84.04
96.62
+ Hack Cases (Ours)
96.05
96.38
Table 5: Stepwise analysis of the dataset enhancement pipeline on Special Judge problems. Adding adversarial Hack Cases provides the decisive boost to TNR.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Term
Description
Submission
In programming competitions, a submission is a program submitted by a contestant. This submission is evaluated by a judging system, resulting in a verdict such as Accepted (AC) , Wrong Answer (WA) , Time Limit Exceeded (TLE) , Runtime Error (RE) , or Compile Error (CE) , among others.
Test case
A test case is used to check whether the participant’s submission is correct. It usually consists of input data and the corresponding correct output (reference answer).
Test input
The input data of a test case will be fed into the contestant’s program. The program’s output will then be compared with the reference output to determine correctness.
Test output
The output data of a test case is the correct answer corresponding to the input data. For problems with multiple correct solutions, the reference output typically represents one of the possible correct answers.
Validator
A validator is a program used to check whether the input data satisfies the problem constraints and format.
Generator
A generator is a program used to produce test input automatically, especially when the input data is large, random, or complex and cannot be easily handcrafted.
Table 7: C++ compilation environments reproduced from the official Codeforces judge configuration.
Hyperparameter
Value
Algorithm
DAPO
Base Model
Qwen3-4B
Train Batch Size
32
Samples per Prompt ( N )
8
Actor Learning Rate
5×10−7
Max Generation Length
24,000
Appendix
Table 8: Hyperparameters for Reinforcement Learning (DAPO).
Training Stage
Easy pass@1
Easy pass@5
Medium pass@1
Medium pass@5
Hard pass@1
Hard pass@5
Base Model (Qwen3-4B)
91.04
97.01
63.24
73.53
11.84
25.00
RL w/ CodeContests +
94.03
97.01
58.82
75.00
15.13
26.97
RL w/ CodeContests + +
94.03
98.51
66.18
80.88
17.11
27.63
Appendix
Table 9: Detailed Pass@ k performance on LiveCodeBench across different difficulty levels. Comparing the Base Model against versions trained via RL using standard vs. adversarial datasets.