ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

Difan Jiao; Qianfeng Wen; Blair Yang; Zhenwei Tang; Ashton Anderson

cs.AIApr 2, 2026

ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

Authors: Difan Jiao, Qianfeng Wen, Blair Yang, Zhenwei Tang, Ashton Anderson

Organizations: Department of Computer Science, University of Toronto · Coolwei AI Lab

Abstract

We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.

Figures & tables

Figure 1: (A) Prompt-only reflection can reduce top frontier LLM’s performance on AIME24, indicating brittleness. (B) ThinkTwice compared with existing method families. (C) ThinkTwice addresses these gaps by sequentially training a shared model backbone—first solving, then reflecting—yielding significant gains (+5 points reasoning, +11 points refinement) on AIME with Qwen3-4B.

Method	AIME	AMC	MATH500	Minerva	OlympiadBench	Average
Qwen3-4B
Base Model	29.18	64.87	88.47	39.61	57.90	56.01
GRPO	39.06	75.36	91.86	41.03	63.80	62.22
DrGRPO	35.46	77.65	91.83	42.75	66.51	62.84
DAPO	42.54	80.68	93.55	38.38	67.50	64.53
ThinkTwice	44.11	79.59	93.60	42.94	67.60	65.57

Table 1: Reasoning performance (pass@4,

\uparrow

) across five mathematical reasoning benchmarks. Bold and underline denote the best and second-best results per dataset within each model.

Method	AIME	AMC	MATH500	Minerva	OlympiadBench	Average
Qwen3-4B
Base Model	45.25	78.10	92.82	40.81	63.52	64.10
Reflexion †	38.47	73.17	91.48	40.87	60.89	60.98
Self-Refine †	50.37	82.40	93.86	41.19	66.33	66.83
GRPO	48.91	81.86	93.78	42.90	69.67	67.42
DrGRPO	46.98	82.66	94.46	44.84	71.75	68.14

Table 2: Self-refinement performance (pass@4,

\uparrow

) across five mathematical reasoning benchmarks. Bold and underline denote the best and second-best results per dataset within each model.

\dagger

denotes training-free methods.

Figure 3: Cross-model refinement evaluation (average pass@4, $\uparrow$ ). Rows denote the backbone reasoning model; columns denote the refinement model.

Panel of Figure 4 — Figure 4: Training dynamics of refinement across checkpoints. The vertical dashed lines mark the best checkpoints. Left (a): transition metrics on the training set across checkpoints. Right (b): formatting and length metrics across checkpoints during training. Top: boxed-answer and final-answer marker rates; bottom: average response length for self-refinement on correct-only base solutions.

Figure 5: Training-time cost and dynamics of ThinkTwice compared with GRPO. * denoted best checkpoint step of models. (a) Mean reward. (b) Response length. (c) Wall-clock time per update. (d) Accumulated training time. (e) Within-checkpoint macro average benchmark accuracy. Solid orange denotes ThinkTwice base updates, solid blue denotes GRPO, and dashed orange denotes ThinkTwice refinement updates when applicable.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Representative methods	RL?	Extra supervision?	Extra verifier / teacher / critique	Core characteristic
One-pass RLVR	GRPO, DeepSeek-R1, DAPO, Dr. GRPO	Yes	No	No	Trains first-pass reasoning only; no learned revision policy.
Training-free refinement	Self-Refine, Reflexion, self-verification prompting, CRITIC, self-consistency	No	No during training	No, or optional tools	Uses extra test-time feedback or sampling, but refinement is not learned.
Training-based refinement	Cobbe verifier, Let’s Verify, Math-Shepherd, Free Process Rewards, PRIME, AutoMathCritique, CFT, Recursive Introspection, Self-rewarding Correction, SPOC	Mixed	Often yes	Often yes or partial	Usually relies on process labels, critique data, synthetic correction traces, or stronger verifiers.
RL-based refinement	SCoRe, S 2 R, RISE, PAG, Self-Verify, Critique-GRPO	Yes	Usually yes	Usually explicit	Closest prior line, but typically adds explicit verification or critique objectives.
ThinkTwice	Two-phase RLVR with a shared policy	Yes	No	No	Same correctness reward in solve and revise phases; revision conditions only on the model’s prior answer plus a generic review instruction.

Appendix

Table 3: A compact comparison of ThinkTwice with related work. We compare whether methods use reinforcement learning, require supervision beyond final-answer correctness, and rely on an additional verifier, teacher, or explicit critique channel. Entries summarize the dominant tendency of each family.

Category	Hyperparameter
GRPO Training
Learning rate	$1\times 10^{-6}$
PPO clip ratio	$0.2$
Max response length	$3000$
Train batch size	$32$
PPO mini batch size	$8$

Appendix

Table 4: Key hyperparameters for ThinkTwice training.

Figure 6: Reasoning pass@

k

curves across five mathematical reasoning benchmarks for Qwen3-4B (top) and OLMo3-7B (bottom).

Figure 7: Self-refinement pass@

k

curves across five mathematical reasoning benchmarks for Qwen3-4B (top) and OLMo3-7B (bottom).

Case	Sample	Checkpoints	Why it matters
A	481	100, 240	Same problem shows both phases of the story: early refinement discovers the missing telescoping structure; later refinement gives a much shorter version of the now-correct base proof.
B	497	110, 250	refinement completes a stalled geometric derivation early, then later preserves the method while converting a long decimal-heavy trace into a shorter exact solution.
C	194	30, 220	Early refinement repairs a concrete derivative mistake; later refinement turns a wandering but correct proof into a short direct argument.
D	425	100, 220	Geometry case: early refinement recomputes faulty vector algebra; later refinement keeps the invariant but removes most of the symbolic overhead.

Appendix

Table 5: Case-study map for the detailed qualitative appendix.

Checkpoint	Run	Base output	refinement output
Step 100	ThinkTwice	wrong ; 1222 words; last boxed value $\boxed{85}$	correct ; 686 words; $\boxed{8n^{2}+4n+1}$
Step 100	Pure GRPO	wrong ; 1308 words; no final boxed general form	correct ; 892 words; $\boxed{8n^{2}+4n+1}$
Step 240	ThinkTwice	correct ; 1222 words; $\boxed{8n^{2}+4n+1}$	correct ; 358 words; $\boxed{8n^{2}+4n+1}$
Step 240	Pure GRPO	wrong ; 1243 words; no final boxed general form	correct ; 1067 words; $\boxed{8n^{2}+4n+1}$

Appendix

Table 6: Case A summary. The same problem exhibits both early correction and late-stage compression.

Checkpoint	Run	Base output	refinement output
Step 110	ThinkTwice	wrong ; 1280 words; no final boxed answer	correct ; 633 words; $\boxed{\frac{7}{2}}$
Step 110	Pure GRPO	wrong ; 1284 words; no final boxed answer	wrong ; 970 words; no final boxed answer
Step 250	ThinkTwice	correct ; 1303 words; $\boxed{3.5}$	correct ; 591 words; $\boxed{\frac{7}{2}}$
Step 250	Pure GRPO	correct ; 1209 words; $\boxed{\frac{7}{2}}$	correct ; 884 words; $\boxed{\frac{7}{2}}$

Appendix

Table 7: Case B summary. Early refinement completes the coordinate-shoelace derivation; late refinement keeps the method but presents it more cleanly.

Checkpoint	Run	Base output	refinement output
Step 30	ThinkTwice	wrong ; 1204 words; no final boxed answer	correct ; 719 words; $\boxed{\frac{1}{4}}$
Step 30	Pure GRPO	correct ; 1236 words; $\boxed{\frac{1}{4}}$	correct ; 1181 words; $\boxed{\frac{1}{4}}$
Step 220	ThinkTwice	correct ; 1373 words; $\boxed{\frac{1}{4}}$	correct ; 374 words; $\boxed{\frac{1}{4}}$
Step 220	Pure GRPO	correct ; 906 words; $\boxed{\frac{1}{4}}$	correct ; 755 words; $\boxed{\frac{1}{4}}$

Appendix

Table 8: Case C summary. Early refinement fixes a concrete derivative mistake; late refinement turns a wandering proof into a short direct argument.

Checkpoint	Run	Base output	refinement output
Step 100	ThinkTwice	wrong ; 1138 words; no final boxed answer	correct ; 795 words; $\boxed{3R^{2}}$
Step 100	Pure GRPO	correct ; 711 words; $\boxed{3R^{2}}$	correct ; 1046 words; $\boxed{3R^{2}}$
Step 220	ThinkTwice	correct ; 746 words; $\boxed{3R^{2}}$	correct ; 410 words; $\boxed{3R^{2}}$
Step 220	Pure GRPO	correct ; 1259 words; $\boxed{3R^{2}}$	correct ; 993 words; $\boxed{3R^{2}}$

Appendix

Table 9: Case D summary. A backup geometry example showing early recomputation and late compression.

Explore similar work

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
Oct 21, 2025Mengqi Li, Lei Zhao, Anthony Man-Cho So +2Self-Training for Language Models Mathematical Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · The Chinese University of Hong Kong
$\textbf{Re}^{2}$ : Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
Mar 7, 2026Pinzheng Wang, Shuli Xu, Juntao Li +4RL for Language Model Reasoning Reinforcement Learning
School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · College of Intelligence and Computing, Tianjin University
Learning to Refine Hidden States for Reliable LLM Reasoning
Jun 16, 2026Chia-Hsuan Hsu, Jui-Ming YaoRL for Language Model Reasoning Efficient Language Model Reasoning

Oct 21, 2025J/K move · Enter open · S save

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

Mengqi Li, Lei Zhao, Anthony Man-Cho So, Ruoyu Sun+1

The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · The Chinese University of Hong Kong

Self-Training for Language Models Mathematical Reasoning LLM Post-Training Language Model Self-Improvement LLM Reasoning

Open paper AskPDF arXiv

Can a language model improve reasoning by learning from its own imperfect responses, without rewards or teacher-provided solutions? We present Self-evolving Post-Training (SePT), a simple method that alternates temperature-controlled self-generation with next-token likelihood training. Each round uses the updated model to generate new training responses, with one response per prompt by default and no correctness filtering. Across six mathematical benchmarks, SePT improves a temperature-selected no-training baseline by 11.4 and 6.7 AVG points on Qwen2.5-Math-7B and Qwen2.5-7B, respectively, where AVG averages Pass@1, Pass@8 and Pass@32 across benchmarks. We analyze how sampling temperature shapes the learning signal and investigate the value of the resulting responses. Responses from a SePT-trained model improve a student initialized from the original weights, while their reasoning prefixes help an unchanged model complete solutions, even when matched in length to prefixes from a colder initial model. Comparing next-token predictions at identical contexts also reveals changes in token rankings that decoding-temperature adjustment cannot reproduce. Further evaluations across nine starting models, general reasoning and code generation examine broader applicability. Together, these results show that reward-free self-training can improve both a model's predictions and the supervision it provides. Our code is available at https://github.com/ElementQi/SePT.

Abstract

Explore similar work

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

Re2\textbf{Re}^{2}Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

Learning to Refine Hidden States for Reliable LLM Reasoning

$\textbf{Re}^{2}$ : Unlocking LLM Reasoning via Reinforcement Learning with Re-solving