cs.AIApr 1, 2026

RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning

Authors: Shaopeng Fu, Xingxing Zhang, Li Dong, Furu Wei

Organizations: Work done during an internship at Microsoft Research · KAUST · Microsoft Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While large language models (LLMs) have demonstrated strong performance on complex reasoning tasks such as competitive programming (CP), existing methods predominantly focus on single-attempt settings, overlooking their capacity for iterative refinement. In this paper, we present RefineRL, a novel approach designed to unleash the self-refinement capabilities of LLMs for CP problem solving. RefineRL introduces two key innovations: (1) Skeptical-Agent, an iterative self-refinement agent equipped with local execution tools to validate generated solutions against public test cases of CP problems. This agent always remains skeptical of its own outputs and thereby enforces rigorous self-refinement even when validation suggests correctness. (2) A reinforcement learning (RL) solution to incentivize LLMs to self-refine with only standard RLVR data (i.e., problems paired with their verifiable answers). Extensive experiments on Qwen3-4B and Qwen3-4B-2507 demonstrate that our method yields substantial gains: after our RL training, these compact 4B models integrated with the Skeptical-Agent not only outperform much larger 32B models but also approach the single-attempt performance of 235B models. Notably, although our models are trained solely on CP coding data, RefineRL also substantially improves their performance on mathematical reasoning benchmarks, suggesting that the learned self-refinement behavior can transfer beyond coding-specific tasks. These findings highlight self-refinement as a promising direction for scaling LLMs' general reasoning ability.

Figures & tables

Figure 1: Overview of our RefineRL approach. (a) Skeptical-Agent employs local tools to evaluate generated solutions against visible public test cases and enforces rigorous self-refinement even when evaluation results suggest correctness. (b) Self-Refinement RL utilizes self-generated refinement trajectory data from the Skeptical-Agent on real-world CP problems data, along with a novel Squared-incentive reward function, to advance the self-refinement capabilities of LLMs.
 CP BenchmarksMath Benchmarks
ModelLCB-v5LCB-v6USACOAIME24AIME25AIME26
Qwen3-235B-A22B ( Yang et al., 2025 )67.2559.9360.7187.2977.9282.71
Qwen3-235B-A22B-2507 ( Yang et al., 2025 )79.0872.5075.5193.3390.6390.00
Qwen3-8B ( Yang et al., 2025 )55.3949.3246.1777.9264.5864.58
Dpsk-R1-0528-Qwen3-8B ( Guo et al., 2025 )59.8451.8443.1679.3766.4669.38
X-Coder-RL-Qwen3-8B ( Wu et al., 2026 )55.8048.3943.4033.7526.6733.13
Table 1: Performance on various CP and math benchmarks. We compare our RefineRL models (4B and 4B-2507 versions) against baselines ranging from 4B to 235B parameters. Bold indicates the best performance, and underline indicates the second-best performance. Our best RefineRL model equipped with Skeptical-Agent (RefineRL-4B-2507 + Skeptical@16) outperforms most 32B baselines and achieves performance comparable to the 235B model.
Figure 3
Model & StrategyAvg. Refine RoundsAvg. Response Len.
Qwen3-4B10.3810,297.97
RefineRL-4B (Ours)9.7811,371.62
Qwen3-4B-25079.2610,330.52
RefineRL-4B-2507 (Ours)8.6617,558.65
Table 3: Impact of Self-Refinement RL on reasoning depth. We compare the average number of self-refinement rounds and the response length per round for models before and after RL training.
Model (w/ Skeptical@16)(%) / Round(%) / Problem
RefineRL-4B1.131.86
RefineRL-4B-25070.532.29
Table 4: Degradation rates of Skeptical-Agent (Skeptical@16) on LCB-v6.
Figure 6
Table 7
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Data TypeQwen3-4BQwen3-4B-2507
Initial CP Problems from DCPD_{\text{CP}}4,5004,500
Error-driven Trajectory2,8112,721
[True Negative] Wrong Answer (WA)2,1272,077
[True Negative] Runtime Error (RE)377364
[True Negative] Time Limit Exceeded (TLE)307280
[True Negative] Compile Error (CE)00
Appendix
Table 8: Statistics of the training dataset for Self-Refinement RL. This table details the sample counts for each interaction category. “WA/RE/TLE/CE/MLE” represent trajectories triggered by error-driven feedback. “AC →\rightarrow WA/RE/TLE/CE/MLE” denote trajectories involving skepticism-driven feedback, where the solution passed public test cases but failed on private ones due to the indicated errors.
MethodData SizeTrajectory TypeReward Function
Vanilla RL∼\sim 4.5KNoneSquared-incentive
Refinement (Binary)∼\sim 8.7K (4.5K Real Data + 4.2K Syn. Data)RefinementPass-or-fail
RefineRL (Ours)∼\sim 8.7K (4.5K Real Data + 4.2K Syn. Data)RefinementSquared-incentive
Appendix
Table 9: Comparison of RL Training Strategies. We contrast three RL setups: Vanilla RL (direct generation), Self-Refinement RL with pass-or-fail rewards, and our Self-Refinement RL (with squared-incentive rewards). Vanilla RL uses only the initial generation subset ( ∼\sim 4.5k), while refinement methods use the full trajectory dataset ( ∼\sim 8.7k). Our method employs a squared reward to incentivize robust self-verification.
Figure 4: Training dynamics of Qwen3-4B. The left plot shows the steady increase in reward, while the right plot shows the corresponding growth in response length, reflecting the model’s internalization of the skeptical reasoning process.
Figure 5: Training dynamics of Qwen3-4B-2507. Similar to the base model, the 2507 variant demonstrates consistent learning progress in terms of reward maximization and increased reasoning length during the RL process.
Figure 6: Validation reward curves across three held-out validation settings of Qwen3-4B-2507 throughout RL training. In each subplot, the solid line shows the smoothed trend, while the lighter transparent line shows the raw validation reward. These results indicate that the model continues to acquire self-refinement ability during later training stages, rather than merely overfitting to the training trajectories.
Figure 7: Validation reward curves across three held-out validation settings of Qwen3-4B throughout RL training. In each subplot, the solid line shows the smoothed trend, while the lighter transparent line shows the raw validation reward. These results indicate that the model continues to acquire self-refinement ability during later training stages, rather than merely overfitting to the training trajectories.
Figure 14
Model & StrategyIOI 2024 Score@50
Qwen3-235B-A22B-2507148.5
OCR-Nemotron-1.1-32B132.0
Qwen3-4B-Thinking-250790.0
+ Skeptical@32 (Ours)109.0
RefineRL-4B-2507 (Ours)109.0
+ Skeptical@32 (Ours)117.0
Appendix
Table 11: Performance on IOI 2024 under the official subtask-level scoring protocol with up to 50 submissions per problem.
Figure 9: Standard system prompt used for initial solution generation and subsequent refinement steps.
Figure 10: Prompt for error-driven feedback construction.
Figure 11: Detailed templates for constructing the “Error Reason” field based on specific execution errors (WA, TLE, RE, etc.).
Figure 12: The skeptical-driven feedback prompt ( PSkep\mathcal{P}_{\text{Skep}} ) triggered when a solution passes local tests (on all public test cases), instructing the agent to self-critique against a checklist.
Figure 13: Standard system prompt used for initial solution generation and subsequent refinement steps in solving AIME problems.
Figure 14: The skeptical-driven feedback prompt ( PSkep\mathcal{P}_{\text{Skep}} ) triggered every time when the agent tries to perform refinement.

Explore similar work

CardsList

  1. A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback

    May 18, 2026Anika Tabassum, Md Sifat Hossain, Md. Fahim Arefin +2Code GenerationFeedback

    Dept. of Computer Science and Engineering, University of Dhaka, Bangladesh · Dept. of Information Systems, University of Maryland, Baltimore County, USA

  2. X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Tests

    Jan 11, 2026Jie Wu, Haoling Li, Xin Zhang +8Raw Judge OutputsSynthetic Task

    Tsinghua University · Microsoft

  3. Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution

    May 14, 2026Han Li, Jinyu Tian, Rili Feng +10Self-Evolving AgentsTransferable Implicit Solvent Machine Learning Potential

    Nanjing University · Independent Researcher · Tsinghua University