RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
Authors: Kaiyuan Li, Jing-Cheng Pang, Yang Yu
Organizations: National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · Polixir.ai, China · Huawei Technologies Co., Ltd., China
Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.
Figures & tables
Figure 1: Cross-generation performance heatmaps Comparing the influence of thinking vs. answering model capacity. (Left) In reasoning tasks, performance is almost entirely dictated by the quality of the thinking trace. (Right) In general tasks, the answering model capacity remains a dominant factor, and the performance provided by superior thinking is less decisive.
Configuration
Reward
Δ R
Win Rate
Setting A: Qwen3-1.7B / GRPO / ExpertQA
Base
0.1842
—
—
Tpost + Apre
0.1849
+0.0007
52.87%
Tpre + Apost
0.2148
+0.0306
100.0%
Post-trained
0.2182
+0.0340
100.0%
Setting B: Qwen3-1.7B / GRPO / UltraFeedback
Table 1: Stagnation of thinking evolution in general tasks. Experimental verification of the thinking-answering decoupling effect. We report Reward and Win Rate (vs. Base) across various model scales, RL algorithms, and datasets. Δ denotes the reward improvement relative to the Base model. Subscripts denote the model state before ( pre ) and after ( post ) RL fine-tuning.
Table 2: Comparison of thinking evolution across different models and domains. ΔR% denotes the percentage change in reward relative to the Base configuration.
Figure 2: Overall framework of START method.
Configuration
Post-trained
Tpost + Apre
Reward
Win Rate
Reward
Win Rate
GRPO
0.2182
—
0.1849
—
GRPO-MA
0.2140
34.08%
0.1859
54.14%
GRPO+START
0.2201
59.24%
0.1876
68.15%
Table 3: Main results of START on ExpertQA. Win Rates are calculated head-to-head against the post-trained GRPO baseline.
Figure 3: Reward curves during reinforcement learning fine-tuning. We compare the reward curves of GRPO, GRPO-MA, and our proposed GRPO+START on the ExpertQA dataset.
Figure 4: Additional reward curves across different datasets and algorithms. For baseline models, we terminate the training process if an obvious convergence trend is observed.
Configuration
Reward
Win Rate
ExpertQA / DAPO
DAPO
0.2352
—
DAPO+START
0.2385
65.29%
UltraFeedback / GRPO
GRPO
0.1658
—
GRPO+START
0.1695
61.25%
Table 4: More results across different dataset and algorithm. Win Rates are calculated head-to-head against the post-trained base algorithm.
Configuration
Reward
Δ R
Win Rate
Base
0.1842
—
—
Tpost + Apre
0.1888
+0.0046
69.11%
Tpre + Apost
0.1897
+0.0055
78.98%
Post-trained
0.1927
+0.0085
84.71%
Table 5: Results of Phase I. We decouple the contributions of thinking ( T ) and answering ( A ) by cross-pairing the base and post-trained (START) components on ExpertQA using Qwen-1.7B. Win rates are evaluated relative to the base model. ΔR denotes the reward improvement.
Pattern
GRPO
GRPO-MA
GRPO+START
User Needs
54.78%
61.78%
90.45%
Structural Output
61.15%
75.99%
92.36%
Table 6: Quantitative comparison of thinking patterns. We report the frequency of meta-context dimensions ( User Needs and Structural Output ) identified within the thinking traces across the test set.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Qwen3
DeepSeek-R1-Distill
Hunyuan
Temperature
0.6
0.6
0.7
Top- p
0.95
0.95
0.8
Top- k
20
20
20
Min- p
0
0
0
Do Sample
True
True
True
Repetition Penalty
1.0
1.0
1.05
Appendix
Table 7: Sampling parameters used across different model families.
Category
Parameter
Value
Training Setup
Save frequency
per epoch
Test/evaluation frequency
per epoch
Training batch size
64
Max prompt length
256 tokens
Max response length
3,328 tokens
Appendix
Table 8: GRPO training hyperparameters and configuration details
Category
Parameter
Value
Training Setup
Save frequency
per epoch
Test/evaluation frequency
per epoch
Training batch size
64
Generation batch size
64
Max prompt length
256 tokens
Appendix
Table 9: DAPO training hyperparameters and configuration details
Figure 5: Distribution of thinking-to-answer Coupling Ratio on ExpertQA. The red curve (Qwen3-1.7B) exhibits a sharp peak at Ratio≈0.65 .
Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs) by increasing test-time compute. However, even after extensive RLVR training, such models still tend to generate unnecessary and low-quality steps in their chain-of-thought (CoT), leading to inefficient overthinking and lower answer quality. We show that when the initial direction or quality of the CoT is suboptimal, the model often fails to reach the correct answer, even after generating several times more tokens than when the initial CoT is well-initialized. To this end, we introduce Reinforcement Learning with Re-solving (Re2), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary, rather than always committing to a final answer. Re2 applies pure reinforcement learning without any preliminary supervised fine-tuning, successfully amplifying the rare redo behavior in vanilla models from only 0.5% to over 30%. This leads to substantial performance gains over standard RLVR under the same training compute budget, and also demonstrates notable improvements in test-time performance as the number of samples increases.
Pinzheng Wang, Shuli Xu, Juntao Li +4
School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · College of Intelligence and Computing, Tianjin University
Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.
Reinforcement Learning from Verifiable Rewards (RLVR) on chain-of-thought reasoning has become a standard part of language model post-training recipes. A common assumption is that the reasoning chains trained through RLVR reliably represent how a model gets to its answer. In this paper, we develop two metrics for critically examining this assumption: Causal Importance of Reasoning (CIR), which measures the cumulative effect of reasoning tokens on the final answer, and Sufficiency of Reasoning (SR), which measures whether a verifier can arrive at an unambiguous answer based on the reasoning alone. Through experiments with the Qwen2.5 model series and ReasoningGym tasks, we find that: (1) while RLVR does improve task accuracy, it does not reliably improve CIR or SR, calling the role of reasoning in model performance into question; (2) a small amount of SFT before RLVR can be a remedy for low CIR and SR; and (3) CIR and SR can be improved even without SFT by applying auxiliary CIR/SR rewards on top of the outcome-based reward. This joint reward matches the accuracy of RLVR while also leading to causally important and sufficient reasoning. These results show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought, but this issue can be remedied with simple modifications to the post-training procedure.