$\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

Pinzheng Wang; Shuli Xu; Juntao Li; Yu Luo; Dong Li; Jianye Hao; Min Zhang

cs.AIMar 7, 2026

$\textbf{Re}^{2}$ : Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

Authors: Pinzheng Wang, Shuli Xu, Juntao Li, Yu Luo, Dong Li, Jianye Hao, Min Zhang

Organizations: School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · College of Intelligence and Computing, Tianjin University

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs) by increasing test-time compute. However, even after extensive RLVR training, such models still tend to generate unnecessary and low-quality steps in their chain-of-thought (CoT), leading to inefficient overthinking and lower answer quality. We show that when the initial direction or quality of the CoT is suboptimal, the model often fails to reach the correct answer, even after generating several times more tokens than when the initial CoT is well-initialized. To this end, we introduce Reinforcement Learning with Re-solving (Re $^2$ ), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary, rather than always committing to a final answer. Re $^2$ applies pure reinforcement learning without any preliminary supervised fine-tuning, successfully amplifying the rare redo behavior in vanilla models from only 0.5% to over 30%. This leads to substantial performance gains over standard RLVR under the same training compute budget, and also demonstrates notable improvements in test-time performance as the number of samples increases.

Figures & tables

Figure 1: (a) Accuracy improvements of DAPO and Re 2 on Qwen2.5-7B-Instruct at each training step, with comparable numbers of generated and trained tokens per step. (b) Test-time scaling of DAPO and Re 2 under the same training budget, as the number of samples increases.

Figure 2: Illustration of reasoning with re-solving. (a) In standard reasoning, when the initial steps are suboptimal, the LLM struggles to reach the correct answer even after generating more reasoning steps and tokens. (b) Reasoning with re-solving, allowing the model to abandon an unpromising path and restart at any point, leads to a higher overall accuracy. (c) Example trace of DAPO and Re 2 : Both models initially attempt an incorrect approach using the AM–GM inequality; however, Re 2 detects the failure, restarts, and arrives at the correct answer.

Figure 3: Correlation between CoT length and reasoning performance. The bars represent the proportion of cases within different CoT length intervals, while the line indicates the average accuracy for responses in each interval. The large plots in the top row show the correlation across the entire dataset. The small plots in the second row illustrate the correlation at the level of individual problems.

Figure 4: Number of cases where the relative drop exceeds the threshold when continuing from different proportions of incorrect response prefixes, compared to reasoning from scratch. “All Drops” counts all such cases at each prefix proportion, while “First Drops” records the earliest prefix for each response where the drop exceeds the threshold.

Figure 5: The framework of Re 2 . For each query, Re 2 samples multiple prefixes, then generates multiple continuations for each prefix. The advantage is calculated within each group, while the out-of-group accuracy is used as the reward for the redo action.

Algorithm 1 Re 2 : Reinforcement Learning with Resolving

Input initial policy model

\pi_{\theta}

; task prompts

\mathcal{D}

; maximum training steps

s

; number of prefixes

n

; number of continuations

m

; max resolve rounds

R

; clipping thresholds

\varepsilon_{\mathtt{low}},\varepsilon_{\mathtt{high}}

; update steps per batch

\mu

for step = 1,…,

s

1: Sample a mini-batch

\mathcal{D}_{b}

from

\mathcal{D}

2: For each question

q\in\mathcal{D}_{b}

, sample

n

responses from

\pi_{\theta_{\text{old}}}(\cdot\mid q)

and truncate them at random ratios to form prefixes

\{\text{Pre}_{i}\}_{i=1}^{n}

3: For each prefix

\text{Pre}_{i}

, sample

m

continuations

\{O_{i,j}\}_{j=1}^{m}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q,\mathrm{Prefix}_{i})

4: Compute rewards

\{r_{i,j}\}

according to Eq. 1

Table 1:

Models	Methods	AIME24	AIME25	AMC23	GSM8K	GPQA	Avg
Base Model
Qwen2.5-7B Base	+ DAPO	11.9	10.3	64.7	91.8	29.7	41.7
Qwen2.5-7B Base	+ Re 2	17.1	19.0	70.8	93.6	36.8	47.5 (+5.8)
Qwen2.5-14B Base	+ DAPO	18.2	15.7	64.0	94.3	44.8	47.4
Qwen2.5-14B Base	+ Re 2	28.5	23.4	68.5	94.6	49.6	52.9 (+5.5)
Instruct Model

Table 1: Experimental results on five reasoning benchmarks. Re 2 consistently improves the overall reasoning performance of each model over DAPO (

p

-value

<0.05

). Red numbers in parentheses indicate performance gains relative to DAPO.

Figure 6: Test-time scaling of DAPO and Re 2 under the same training budget as the number of samples increases.

Figure 7: The training progress of Re 2 on Qwen2.5-14B-Base.

Figure 8: Analysis of accuracy and behavior across problem difficulty levels. (a) Accuracy improvements of DAPO and Re 2 , along with the resolving rate of Re 2 , when questions are grouped by the difficulty estimated by the base model. (b) Accuracy improvements and resolving rate of Re 2 when questions are grouped by the difficulty estimated by DAPO.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: The template of Re 2 .

Models	Methods	AIME24	AIME25	AMC23	GSM8K	GPQA
Base Model
Qwen2.5-7B Base	+ DAPO	11.9 (±1.0)	10.3 (±1.0)	64.7 (±1.3)	91.8 (±0.1)	29.7 (±0.5)
Qwen2.5-7B Base	+ Re 2	17.1 (±1.4)	19.0 (±1.2)	70.8 (±0.3)	93.6 (±0.0)	36.8 (±0.3)
Qwen2.5-14B Base	+ DAPO	18.2 (±1.2)	15.7 (±1.2)	64.0 (±1.3)	94.3 (±0.1)	44.8 (±0.6)
Qwen2.5-14B Base	+ Re 2	28.5 (±1.1)	23.4 (±1.3)	68.5 (±0.3)	94.6 (±0.0)	49.6 (±0.3)
Instruct Model

Appendix

Table 2: Accuracy with 95% confidence intervals on five reasoning benchmarks, confidence intervals are given in parentheses.

Figure 10: Test-time scaling of Re 2 compared with additional baselines.

Figure 11: Bias and variance of the estimated resolving accuracy under different values of

n

and

m

Figure 12: Degeneration group rate during training for DAPO and Re 2 .

Figure 13: Reasoning lengths during training for DAPO and Re 2 .

Figure 14: Accuracy improvements of DAPO and Re 2 under the same training time.

Panel of Figure 15 — Figure 15: Case Analysis of CoTs with Different Lengths in AIME25

Panel of Figure 16 — Figure 16: Case Analysis of CoTs with Different Lengths in AIME25

Panel of Figure 17 — Figure 17: Case Analysis of CoTs with Different Lengths in AIME25

Panel of Figure 18 — Figure 18: Case Analysis of CoTs with Different Lengths in AMC

Panel of Figure 19 — Figure 19: Re 2 Examples 1

Panel of Figure 20 — Figure 20: Re 2 Examples 2

Panel of Figure 21 — Figure 21: Re 2 Examples 3

Panel of Figure 22 — Figure 22: Re 2 Examples 4

Methods	AIME25	AMC23	MATH500	GSM8K
OpenPangu-Embedded-1B	4.7	22.2	49.3	64.4
+ Re 2	8.6	35.3	65.8	81.0

Appendix

Table 3: The accuracy of OpenPangu-Embedded-1B and Re 2 on four benchmarks.

Explore similar work

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Jun 22, 2026Stanley Wei, Juno KimRL for Language Model Reasoning Efficient Language Model Reasoning
1Princeton University · University of California, Berkeley.
ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning
Aug 28, 2026Xin Jiang, Minhao Wang, Wen Wu +4Reward Shaping RL for Language Model Reasoning
School of Computer Science and Technology, East China Normal University, Shanghai 200241, China · State Key Laboratory of Estuarine and Coastal Research, School of Computer Science and Technology, East China Normal University, Shanghai 200241, China
Enhancing LLM Metacognition via Cognitive Pairwise Training
May 30, 2026Weitao Li, Hao Zhou, Xuanyu Lei +11Pairwise Preference Learning RL for Language Model Reasoning
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · WeChat AI, Tencent Inc., China

Jun 22, 2026J/K move · Enter open · S save

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

Stanley Wei, Juno Kim

1Princeton University · University of California, Berkeley.

RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Fine-Tuning LLM Reasoning

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.

Abstract

Explore similar work

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning

Enhancing LLM Metacognition via Cognitive Pairwise Training