cs.AIApr 1, 2026

RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning

Authors: Shaopeng Fu, Xingxing Zhang, Li Dong, Furu Wei

Organizations: Work done during an internship at Microsoft Research · KAUST · Microsoft Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While large language models (LLMs) have demonstrated strong performance on complex reasoning tasks such as competitive programming (CP), existing methods predominantly focus on single-attempt settings, overlooking their capacity for iterative refinement. In this paper, we present RefineRL, a novel approach designed to unleash the self-refinement capabilities of LLMs for CP problem solving. RefineRL introduces two key innovations: (1) Skeptical-Agent, an iterative self-refinement agent equipped with local execution tools to validate generated solutions against public test cases of CP problems. This agent always remains skeptical of its own outputs and thereby enforces rigorous self-refinement even when validation suggests correctness. (2) A reinforcement learning (RL) solution to incentivize LLMs to self-refine with only standard RLVR data (i.e., problems paired with their verifiable answers). Extensive experiments on Qwen3-4B and Qwen3-4B-2507 demonstrate that our method yields substantial gains: after our RL training, these compact 4B models integrated with the Skeptical-Agent not only outperform much larger 32B models but also approach the single-attempt performance of 235B models. Notably, although our models are trained solely on CP coding data, RefineRL also substantially improves their performance on mathematical reasoning benchmarks, suggesting that the learned self-refinement behavior can transfer beyond coding-specific tasks. These findings highlight self-refinement as a promising direction for scaling LLMs' general reasoning ability.

Figures & tables

Figure 1: Overview of our RefineRL approach. (a) Skeptical-Agent employs local tools to evaluate generated solutions against visible public test cases and enforces rigorous self-refinement even when evaluation results suggest correctness. (b) Self-Refinement RL utilizes self-generated refinement trajectory data from the Skeptical-Agent on real-world CP problems data, along with a novel Squared-incentive reward function, to advance the self-refinement capabilities of LLMs.
 CP BenchmarksMath Benchmarks
ModelLCB-v5LCB-v6USACOAIME24AIME25AIME26
Qwen3-235B-A22B ( Yang et al., 2025 )67.2559.9360.7187.2977.9282.71
Qwen3-235B-A22B-2507 ( Yang et al., 2025 )79.0872.5075.5193.3390.6390.00
Qwen3-8B ( Yang et al., 2025 )55.3949.3246.1777.9264.5864.58
Dpsk-R1-0528-Qwen3-8B ( Guo et al., 2025 )59.8451.8443.1679.3766.4669.38
X-Coder-RL-Qwen3-8B ( Wu et al., 2026 )55.8048.3943.4033.7526.6733.13
Table 1: Performance on various CP and math benchmarks. We compare our RefineRL models (4B and 4B-2507 versions) against baselines ranging from 4B to 235B parameters. Bold indicates the best performance, and underline indicates the second-best performance. Our best RefineRL model equipped with Skeptical-Agent (RefineRL-4B-2507 + Skeptical@16) outperforms most 32B baselines and achieves performance comparable to the 235B model.
Figure 3
Model & StrategyAvg. Refine RoundsAvg. Response Len.
Qwen3-4B10.3810,297.97
RefineRL-4B (Ours)9.7811,371.62
Qwen3-4B-25079.2610,330.52
RefineRL-4B-2507 (Ours)8.6617,558.65
Table 3: Impact of Self-Refinement RL on reasoning depth. We compare the average number of self-refinement rounds and the response length per round for models before and after RL training.
Model (w/ Skeptical@16)(%) / Round(%) / Problem
RefineRL-4B1.131.86
RefineRL-4B-25070.532.29
Table 4: Degradation rates of Skeptical-Agent (Skeptical@16) on LCB-v6.
Figure 6
Table 7
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Data TypeQwen3-4BQwen3-4B-2507
Initial CP Problems from DCPD_{\text{CP}}4,5004,500
Error-driven Trajectory2,8112,721
[True Negative] Wrong Answer (WA)2,1272,077
[True Negative] Runtime Error (RE)377364
[True Negative] Time Limit Exceeded (TLE)307280
[True Negative] Compile Error (CE)00
Appendix
Table 8: Statistics of the training dataset for Self-Refinement RL. This table details the sample counts for each interaction category. “WA/RE/TLE/CE/MLE” represent trajectories triggered by error-driven feedback. “AC →\rightarrow WA/RE/TLE/CE/MLE” denote trajectories involving skepticism-driven feedback, where the solution passed public test cases but failed on private ones due to the indicated errors.
MethodData SizeTrajectory TypeReward Function
Vanilla RL∼\sim 4.5KNoneSquared-incentive
Refinement (Binary)∼\sim 8.7K (4.5K Real Data + 4.2K Syn. Data)RefinementPass-or-fail
RefineRL (Ours)∼\sim 8.7K (4.5K Real Data + 4.2K Syn. Data)RefinementSquared-incentive
Appendix
Table 9: Comparison of RL Training Strategies. We contrast three RL setups: Vanilla RL (direct generation), Self-Refinement RL with pass-or-fail rewards, and our Self-Refinement RL (with squared-incentive rewards). Vanilla RL uses only the initial generation subset ( ∼\sim 4.5k), while refinement methods use the full trajectory dataset ( ∼\sim 8.7k). Our method employs a squared reward to incentivize robust self-verification.
Figure 4: Training dynamics of Qwen3-4B. The left plot shows the steady increase in reward, while the right plot shows the corresponding growth in response length, reflecting the model’s internalization of the skeptical reasoning process.
Figure 5: Training dynamics of Qwen3-4B-2507. Similar to the base model, the 2507 variant demonstrates consistent learning progress in terms of reward maximization and increased reasoning length during the RL process.
Figure 6: Validation reward curves across three held-out validation settings of Qwen3-4B-2507 throughout RL training. In each subplot, the solid line shows the smoothed trend, while the lighter transparent line shows the raw validation reward. These results indicate that the model continues to acquire self-refinement ability during later training stages, rather than merely overfitting to the training trajectories.
Figure 7: Validation reward curves across three held-out validation settings of Qwen3-4B throughout RL training. In each subplot, the solid line shows the smoothed trend, while the lighter transparent line shows the raw validation reward. These results indicate that the model continues to acquire self-refinement ability during later training stages, rather than merely overfitting to the training trajectories.
Figure 14
Model & StrategyIOI 2024 Score@50
Qwen3-235B-A22B-2507148.5
OCR-Nemotron-1.1-32B132.0
Qwen3-4B-Thinking-250790.0
+ Skeptical@32 (Ours)109.0
RefineRL-4B-2507 (Ours)109.0
+ Skeptical@32 (Ours)117.0
Appendix
Table 11: Performance on IOI 2024 under the official subtask-level scoring protocol with up to 50 submissions per problem.
Figure 9: Standard system prompt used for initial solution generation and subsequent refinement steps.
Figure 10: Prompt for error-driven feedback construction.
Figure 11: Detailed templates for constructing the “Error Reason” field based on specific execution errors (WA, TLE, RE, etc.).
Figure 12: The skeptical-driven feedback prompt ( PSkep\mathcal{P}_{\text{Skep}} ) triggered when a solution passes local tests (on all public test cases), instructing the agent to self-critique against a checklist.
Figure 13: Standard system prompt used for initial solution generation and subsequent refinement steps in solving AIME problems.
Figure 14: The skeptical-driven feedback prompt ( PSkep\mathcal{P}_{\text{Skep}} ) triggered every time when the agent tries to perform refinement.

Explore similar work

CardsList
May 18, 2026cs.SE

A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback

Large Language Models (LLMs) demonstrate strong potential for automated code generation, yet their ability to iteratively refine solutions using execution feedback remains underexplored. Competitive programming offers an ideal testbed for this investigation, as it demands end-to-end algorithmic reasoning, precise implementation under strict computational constraints, and complete functional correctness with rigorous evaluation. In this paper, we present A-ProS, an autonomous AI agent that solves competitive programming problems through a hybrid multi-model feedback framework separating solution generation from specialized debugging. A-ProS combines ChatGPT-based generators (GPT-4 and GPT-5) with three debugging critics: Codestral-2508, Llama-3.3-70B, and DeepSeek-R1, under a 2 x 3 factorial design. We evaluate six workflows on 367 problems from ICPC World Finals (2011-2024) and Codeforces (rated 1200-1800). The results show that GPT-5 workflows improve from 39 initial accepted solutions to 85-90 after three refinement rounds, while GPT-4 improves from 15 to 31-38. A controlled ablation on 47 problems shows that stateful refinement outperforms stateless approaches by 8.5-10.6 percentage points and reduces repeated failures by up to 3.5x. Compared to baseline agent loops, A-ProS achieves over 2x greater gains, highlighting the importance of persistent context and multi-model feedback for reliable autonomous program synthesis.

Anika Tabassum, Md Sifat Hossain, Md. Fahim Arefin +2

Dept. of Computer Science and Engineering, University of Dhaka, Bangladesh · Dept. of Information Systems, University of Maryland, Baltimore County, USA

Code GenerationFeedback
AskPDF
Jan 11, 2026cs.CL

X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Tests

Competitive programming remains challenging for code LLMs. Despite recent progress, many training pipelines still depend on scarce real-world data, raising concerns about scalability and near-duplicate benchmark contamination. In this paper, we examine whether synthetic training artifacts can support the complete SFT-to-RL cycle for competitive programming: no real-world tasks, solutions, or test cases are directly used for post-training. To this end, we synthesize tasks, verified solutions, and reliable test cases that serve as reward signals for reinforcement learning. To improve reward reliability, we introduce a dual-verification strategy that reduces noise in both selected solutions and test outputs. Using this high-quality data, we train the X-Coder model series. X-Coder-14B achieves 67.5% avg@8 on LiveCodeBench v5 and 63.4% on v6, outperforming its base model by over 40 points. Further analysis provides practical insights into synthetic post-training, highlighting the value of diverse tasks, verified long-CoT supervision, and code-centric reinforcement. Our data and models are released at https://github.com/JieWu02/X-Coder

Jie Wu, Haoling Li, Xin Zhang +8

Tsinghua University · Microsoft

Raw Judge OutputsSynthetic TaskSynthetic Training DataCode Optimization
AskPDF
May 14, 2026cs.AI

Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution

Large language models (LLMs) still struggle with the rigorous reasoning demands of hard competitive programming. While recent multi-agent frameworks attempt to bridge this reliability gap, they remain fundamentally stateless: they rely on static retrieval and discard the valuable problem-solving and debugging experience gained from previous tasks. To address this, we present Solvita, an agentic evolution framework that enables continuous learning without requiring weight updates to the underlying LLM. Solvita reorganizes problem-solving into a closed-loop system of strategy selection, program synthesis, certified supervision, and targeted hacking, executed by four specialized agents: Planner, Solver, Oracle, and Hacker. Crucially, each agent is paired with a trainable, graph-structured knowledge network. As the system operates, outcome signals, such as pass/fail verdicts, test certification quality, and adversarial vulnerabilities discovered by the Hacker, are recast as reinforcement learning updates to these network weights. This allows the agents to dynamically route future queries based on past successes and failures, effectively accumulating transferable reasoning experience over time. Evaluated across CodeContests, APPS, AetherCode, and live Codeforces rounds, Solvita establishes a new state-of-the-art among code-generation agents, outperforming existing multi-agent pipelines and nearly doubling the accuracy of single-pass baselines.

Han Li, Jinyu Tian, Rili Feng +10

Nanjing University · Independent Researcher · Tsinghua University

Self-Evolving AgentsTransferable Implicit Solvent Machine Learning Potential
AskPDF