RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

Kaiyuan Li; Jing-Cheng Pang; Yang Yu

cs.CLMar 21, 2026

RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

Authors: Kaiyuan Li, Jing-Cheng Pang, Yang Yu

Organizations: National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · Polixir.ai, China · Huawei Technologies Co., Ltd., China

Abstract

Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.

Figures & tables

Panel of Figure 1 — Figure 1: Cross-generation performance heatmaps Comparing the influence of thinking vs. answering model capacity. (Left) In reasoning tasks, performance is almost entirely dictated by the quality of the thinking trace. (Right) In general tasks, the answering model capacity remains a dominant factor, and the performance provided by superior thinking is less decisive.

Setting A: Qwen3-1.7B / GRPO / ExpertQA
Configuration	Reward	$\Delta$ R	Win Rate
Base	0.1842	—	—
$T_{\text{post}}$ + $A_{\text{pre}}$	0.1849	+0.0007	52.87%
$T_{\text{pre}}$ + $A_{\text{post}}$	0.2148	+0.0306	100.0%
Post-trained	0.2182	+0.0340	100.0%
Setting B: Qwen3-1.7B / GRPO / UltraFeedback

Table 1: Stagnation of thinking evolution in general tasks. Experimental verification of the thinking-answering decoupling effect. We report Reward and Win Rate (vs. Base) across various model scales, RL algorithms, and datasets.

\Delta

denotes the reward improvement relative to the Base model. Subscripts denote the model state before ( pre ) and after ( post ) RL fine-tuning.

Setting A: Hunyuan-1.8B-Instruct / ExpertQA
Configuration	Reward	$\Delta$ R%	Win Rate
Base	0.1628	—	—
$T_{\text{post}}$ + $A_{\text{pre}}$	0.1740	+6.88%	69.11%
$T_{\text{pre}}$ + $A_{\text{post}}$	0.1942	+19.29%	93.31%
Post-trained	0.2111	+29.67%	98.41%
Setting B: DeepSeek-R1-Distill-Qwen-1.5B / ExpertQA

Table 2: Comparison of thinking evolution across different models and domains.

\Delta R\%

denotes the percentage change in reward relative to the Base configuration.

Figure 2: Overall framework of START method.

Configuration	Post-trained		$T_{\text{post}}$ + $A_{\text{pre}}$
Configuration	Reward	Win Rate	Reward	Win Rate
GRPO	0.2182	—	0.1849	—
GRPO-MA	0.2140	34.08%	0.1859	54.14%
GRPO+START	0.2201	59.24%	0.1876	68.15%

Table 3: Main results of START on ExpertQA. Win Rates are calculated head-to-head against the post-trained GRPO baseline.

Figure 3: Reward curves during reinforcement learning fine-tuning. We compare the reward curves of GRPO, GRPO-MA, and our proposed GRPO+START on the ExpertQA dataset.

Panel of Figure 4 — Figure 4: Additional reward curves across different datasets and algorithms. For baseline models, we terminate the training process if an obvious convergence trend is observed.

ExpertQA / DAPO
Configuration	Reward	Win Rate
DAPO	0.2352	—
DAPO+START	0.2385	65.29%
UltraFeedback / GRPO
GRPO	0.1658	—
GRPO+START	0.1695	61.25%

Table 4: More results across different dataset and algorithm. Win Rates are calculated head-to-head against the post-trained base algorithm.

Configuration	Reward	$\Delta$ R	Win Rate
Base	0.1842	—	—
$T_{\text{post}}$ + $A_{\text{pre}}$	0.1888	+0.0046	69.11%
$T_{\text{pre}}$ + $A_{\text{post}}$	0.1897	+0.0055	78.98%
Post-trained	0.1927	+0.0085	84.71%

Table 5: Results of Phase I. We decouple the contributions of thinking (

T

) and answering (

A

) by cross-pairing the base and post-trained (START) components on ExpertQA using Qwen-1.7B. Win rates are evaluated relative to the base model.

\Delta R

denotes the reward improvement.

Pattern	GRPO	GRPO-MA	GRPO+START
User Needs	54.78%	61.78%	90.45%
Structural Output	61.15%	75.99%	92.36%

Table 6: Quantitative comparison of thinking patterns. We report the frequency of meta-context dimensions ( User Needs and Structural Output ) identified within the thinking traces across the test set.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Qwen3	DeepSeek-R1-Distill	Hunyuan
Temperature	0.6	0.6	0.7
Top- $p$	0.95	0.95	0.8
Top- $k$	20	20	20
Min- $p$	0	0	0
Do Sample	True	True	True
Repetition Penalty	1.0	1.0	1.05

Appendix

Table 7: Sampling parameters used across different model families.

Category	Parameter	Value
Training Setup
	Save frequency	per epoch
	Test/evaluation frequency	per epoch
	Training batch size	64
	Max prompt length	256 tokens
	Max response length	3,328 tokens

Appendix

Table 8: GRPO training hyperparameters and configuration details

Category	Parameter	Value
Training Setup
	Save frequency	per epoch
	Test/evaluation frequency	per epoch
	Training batch size	64
	Generation batch size	64
	Max prompt length	256 tokens

Appendix

Table 9: DAPO training hyperparameters and configuration details

Figure 5: Distribution of thinking-to-answer Coupling Ratio on ExpertQA. The red curve (Qwen3-1.7B) exhibits a sharp peak at

Ratio\approx 0.65

Language Model Reasoning Evaluation

See all 36 papers

$\textbf{Re}^{2}$ : Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
Mar 7, 2026Pinzheng Wang, Shuli Xu, Juntao Li +4RL for Language Model Reasoning Reinforcement Learning
School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · College of Intelligence and Computing, Tianjin University
SCOPE-RL: Optimizing Reasoning Paths Before and After Success
Jul 13, 2026Xiaojian Liu, Han Xu, Jianqiang Xia +6Reward Shaping RL for Language Model Reasoning
Baidu Inc. · Shandong University
Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
Apr 23, 2026Qinan Yu, Alexa Tartaglini, Peter Hase +2CoT Faithfulness Reinforcement Learning with Verifiable Rewards
1Stanford University · 2Schmidt Sciences

Mar 7, 2026J/K move · Enter open · S save

$\textbf{Re}^{2}$ : Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

Pinzheng Wang, Shuli Xu, Juntao Li, Yu Luo+3

School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · College of Intelligence and Computing, Tianjin University

RL for Language Model Reasoning Reinforcement Learning Efficient Language Model Reasoning Overthinking in Language Models

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs) by increasing test-time compute. However, even after extensive RLVR training, such models still tend to generate unnecessary and low-quality steps in their chain-of-thought (CoT), leading to inefficient overthinking and lower answer quality. We show that when the initial direction or quality of the CoT is suboptimal, the model often fails to reach the correct answer, even after generating several times more tokens than when the initial CoT is well-initialized. To this end, we introduce Reinforcement Learning with Re-solving (Re

^2

), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary, rather than always committing to a final answer. Re

^2

applies pure reinforcement learning without any preliminary supervised fine-tuning, successfully amplifying the rare redo behavior in vanilla models from only 0.5% to over 30%. This leads to substantial performance gains over standard RLVR under the same training compute budget, and also demonstrates notable improvements in test-time performance as the number of samples increases.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Re2\textbf{Re}^{2}Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving

SCOPE-RL: Optimizing Reasoning Paths Before and After Success

Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning

$\textbf{Re}^{2}$ : Unlocking LLM Reasoning via Reinforcement Learning with Re-solving