Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learning

Yinan Xia; Haotian Zhang; Huiming Wang

cs.LGMar 19, 2026

Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learning

Authors: Yinan Xia, Haotian Zhang, Huiming Wang

Organizations: Tianjin University · Kuaishou Technology

Abstract

Large Reasoning Models (LRMs) have shown exceptional reasoning capabilities, but they also suffer from the issue of overthinking, often generating excessively long and redundant answers. For problems that exceed the model's capabilities, LRMs tend to exhibit the overconfidence phenomenon, generating overly short but incorrect answers, which may contribute to suboptimal performance. To address these issues, we propose Difficulty-Differentiated Policy Optimization (DDPO), an efficient reinforcement learning algorithm that optimizes simple and complex tasks separately based on the overconfidence phenomenon. Specifically, it reduces the output length for simple tasks without compromising accuracy, while for complex tasks, it expands the exploration space to improve performance. We further derive the theoretical conditions for maximizing expected accuracy, which require the length distribution to closely approximate the optimal length and be as concentrated as possible. Based on these conditions, we propose using the difficulty-level average as a well-founded reference for length optimization. Extensive experiments on both in-domain and out-of-domain benchmarks validate the superiority and effectiveness of DDPO. Compared to GRPO, DDPO reduces the average answer length by 12% while improving accuracy by 1.85% across multiple benchmarks, achieving a better trade-off between accuracy and length. The code is available at https://github.com/Yinan-Xia/DDPO.

Figures & tables

Figure 1: The relationship between length and difficulties for different LRMs. The output length exhibits a non-monotonic relationship with problem difficulty: it first increases and then decreases as the difficulty decreases. For overly difficult problems, the LRMs generate shorter but incorrect answers, which we define as the overconfidence phenomenon.

Figure 2: (a) Output length distribution and accuracy as a function of length on easy tasks before RL training. (b) Length distribution and length-wise accuracy after GRPO ( blue ) and DDPO ( green ) training. After training, DDPO yields a left-shifted and more concentrated length distribution compared to GRPO, with samples clustered around the optimal length (corresponding to the highest accuracy), resulting in higher accuracy. (c) Output length distribution and accuracy as a function of length on hard tasks before RL training. (d) Length distribution and length-wise accuracy after GRPO ( red ) and DDPO ( brick red ) training. After training, DDPO exhibits a clear rightward shift relative to GRPO, with a more concentrated distribution centered around the optimal length, leading to improved accuracy.

Figure 3: (a) We define the rollout accuracy as an indicator of the difficulty for each query. By analyzing the relationship between difficulty and output length (shown in the lower-right panel), we observe a inverted U-shaped pattern, which we define as “ overconfidence ”. Based on this observation, we categorize queries into hard and easy ones and propose Difficulty-Differentiated Policy Optimization (DDPO), which encourages greater exploration for difficult queries while shrinking output length for easy queries to avoid redundancy. (b) When applying the length optimization strategy, we use the optimal length as a reference and guide the final length distribution to be closer to this value and more concentrated, thereby increasing the expected accuracy. (c) We estimate the optimal length as the average length of correct answers for queries of the same difficulty within the batch.

Methods/Models	OLYMPIAD		MATH		AMC		AIME2025		AIME2024		GPQA-D		AVG.
Methods/Models	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.
GRPO	$55.39$	$5885$	91.86	$3026$	$89.84$	$4080$	$42.71$	$9569$	$56.04$	$8927$	$57.11$	$5728$	$65.49$	$6324$
L1	57.67	$6262$	92.41	$4269$	90.31	$4957$	$42.92$	9023	59.38	$8090$	57.26	$5543$	66.66	$6357$
Shorter Better	$54.23$	4920	$90.75$	1920	$88.44$	3600	$42.71$	$9519$	$58.96$	7246	$56.88$	4416	$65.33$	5270
A-DLP	$54.03$	4419	$90.28$	2052	$88.13$	$3998$	$43.13$	$9248$	$58.75$	$8519$	$56.41$	4986	$65.12$	5536
GRPO-LEAD	$53.64$	$6173$	$91.00$	$3032$	$88.44$	$4823$	44.17	$9432$	$56.88$	$9010$	$55.01$	$7147$	$64.86$	$6603$

Table 1: (Upper) Main results compared with other length optimization methods.The red and blue represent the best and second-best result respectively. Our method can best balance the accuracy and the length. (Lower) Experiments on other models with different sizes. Our method can enhance the performance and shorten the length across varying models.

Baselines	AVG.
Baselines	acc.	len.
DAPO	$65.87$	5236
DAPO+ours	66.61	$5490$
DrGRPO	$65.22$	$6528$
DrGRPO+ours	65.99	5824
GRPO	$65.49$	$6324$

Table 2: The effectiveness of our method across different baselines. The gray background indicate the performance of our method (

\theta=0.2

) adding on varying baselines; Bolded values denote the better performance between baseline and our method.

Components	AVG.
Components	acc.	len.
$L_{i}/L_{\text{max}}$	$64.51$	$5881$
$\alpha*L_{i}/L_{\text{max}}$	$65.02$	$5692$
$\alpha*(L_{i}-\mu_{\text{diff}_{q}})/L_{\text{max}}$	$65.47$	$5968$
$\alpha*(L_{i}-\mu_{\text{diff}_{q}})/L_{\text{max}}+LB$	$65.95$	$6185$
$\alpha*(L_{i}-\mu_{\text{diff}_{q}})/L_{\text{max}}+LB+UB$	67.34	5560

Table 3: Ablation Study. We validated the importance of each mechanism, showing that each improves accuracy and length control, with the optimal combination yielding significant performance gains and more concise output.

AVG-TYPE	OLYMPIAD		MATH		AMC		AIME2025		AIME2024		GPQA-D		AVG.
AVG-TYPE	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.
$\mu_{q}$	$55.46$	4954	$91.80$	$2394$	$89.38$	3846	$43.33$	8104	$57.71$	$8315$	$56.79$	$5931$	$65.74$	$5591$
$\mu_{\text{batch}}$	55.60	$6273$	$91.55$	$4110$	$89.06$	$5113$	44.79	$9486$	$59.79$	$8549$	$57.39$	$6462$	$66.36$	$6665$
$\mu_{\text{diff}_{q}}$	$55.42$	$5027$	92.05	2165	89.84	$4056$	$44.17$	$9204$	61.46	7119	57.80	$5390$	66.79	5494

Table 4: Analysis of optimal length approximation (

\theta

=0.2). The results demonstrate that approximating the optimal length by difficulty-level averages can achieve optimal average performance. The bold represents the best result.

Figure 4: The change in the number of samples with an accuracy of 0 and 1 during the training process of GRPO and DDPO. Throughout DDPO training, the number of samples with an accuracy of 0 is consistently lower than that in GRPO, while the number of samples with an accuracy of 1 consistently exceeds that in GRPO.

Figure 5: A comparison of the length distributions for different difficulty levels between GRPO and DDPO. The result shows that DDPO results in a more concentrated length distribution and significantly alleviates the overconfidence phenomenon.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

AVG-TYPE	OLYMPIAD		MATH		AMC		AIME2025		AIME2024		GPQA-D		AVG.
AVG-TYPE	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.
$\mu_{\text{similar}}$	$54.81$	$5308$	$91.10$	$2590$	89.84	$4182$	$43.75$	$8957$	$60.21$	$8104$	$57.87$	$5522$	$66.26$	$5777$
$\mu_{\text{batch}}$	$54.96$	$5220$	$91.45$	$2352$	$89.22$	$4067$	$42.92$	$9375$	$59.38$	$7900$	$57.46$	$5472$	$65.90$	$5731$
$\mu_{q}$	$54.45$	$5361$	$90.76$	$2596$	$89.22$	$4065$	$43.96$	8382	$57.50$	$7797$	$57.55$	$5620$	$65.57$	$5637$
zero	55.42	5028	92.05	2165	89.84	4056	44.17	$9204$	61.46	7119	57.80	5390	66.79	5494

Appendix

Table 5: Analysis of optimal length approximation when

acc=0

(

\theta=0.2

). When

acc=0

, the exploration space should be maximized. The results show that normalizing by

z_{i}=L_{i}/L_{\text{max}}

yields the best performance, as indicated by the bold results.

Benchmarks	ACC.		LEN.
Benchmarks	grpo	ddpo	grpo	ddpo
math_qa	$86.15$	86.46	$2555$	2313
bbeh	$21.03$	21.20	$6851$	6185
mmlu_pro	17.58	$17.41$	$1817$	1548
avg.	$41.59$	41.69	$4703$	3348

Appendix

Table 6: Comparison with GRPO on out-of-domain benchmarks based on Qwen3-14b, results demonstrate that DDPO outperforms GRPO both in terms of average accuracy and output length.

Parameters	OLYMPIAD		MATH		AMC		AIME2025		AIME2024		GPQA-D		AVG.
Parameters	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.
$\theta=0.2$	$55.42$	5028	92.05	2165	$89.84$	$4057$	$44.17$	$9205$	61.46	7120	57.80	5390	$66.79$	5494
$\theta=0.4$	55.70	$5299$	$91.84$	$2321$	91.09	3622	47.08	8359	$60.63$	$8074$	$57.68$	$5683$	67.34	$5560$
$\theta=0.6$	$54.51$	$5551$	$90.83$	$2674$	$88.44$	$4193$	$44.38$	$8839$	$59.38$	$8110$	$56.44$	$6157$	$65.66$	$5921$
$\theta=0.8$	$54.93$	$5507$	$91.56$	$2570$	$90.00$	$4427$	$44.58$	$9257$	$60.21$	$7990$	$56.79$	$5919$	$66.34$	$5945$

Appendix

Table 7: Analysis of

\theta

. The performance of

\theta=0.2

and

\theta=0.4

demonstrates superior results, corresponding to the boundary where the overconfidence phenomenon is observed in Figure 1 The bold represents the best result.

Components	OLYMPIAD		MATH		AMC		AIME2025		AIME2024		GPQA-D		AVG.
Components	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.
$L_{i}/L_{\text{max}}$	$53.80$	4955	$90.65$	1979	$87.81$	$4859$	$42.29$	$9248$	$56.67$	$8078$	$55.84$	$6168$	$64.51$	$5881$
$\alpha$	$54.00$	$5450$	$90.28$	2165	$88.13$	3822	$43.33$	$9018$	$57.71$	8062	56.69	5636	$65.02$	5692
$\mu_{\text{diff}_{q}}$	$54.57$	$5649$	$90.95$	$2512$	$89.06$	$4148$	$44.17$	8880	$57.92$	$8209$	$56.12$	$6410$	$65.47$	$5968$
LB	54.81	$6068$	91.38	$2672$	89.91	$4359$	44.58	$9042$	58.67	$8268$	$56.34$	$6701$	65.95	$6185$
UB	55.70	5299	91.84	$2321$	91.09	3622	47.08	8359	60.63	8074	57.68	5683	67.34	5560

Appendix

Table 8: Ablation study. Each line validates the effectiveness of a specific component, with the final line representing the complete DDPO method. The red and blue represent the best and second-best result respectively.

Baselines	OLYMPIAD		MATH		AMC		AIME2025		AIME2024		GPQA-D		AVG.
Baselines	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.	acc.	len.
DAPO	$55.44$	$5149$	$91.61$	2190	$89.53$	3781	$43.54$	7805	61.46	6402	$57.61$	4194	$66.53$	4920
DAPO+ours	56.13	5145	92.05	$2466$	89.84	$3899$	45.83	$8811$	$59.38$	$7777$	58.11	$5280$	66.89	$5563$
DrGRPO	$54.31$	$6171$	$91.50$	$3117$	$89.22$	$5089$	$42.29$	$9546$	58.33	$8487$	$55.68$	$6760$	$65.22$	$6528$
DrGRPO+ours	55.00	5334	91.60	2518	89.38	4113	45.00	9145	$58.13$	7923	56.85	5913	65.99	5824
GRPO	$55.39$	$5885$	$91.86$	$3026$	89.84	$4080$	$42.71$	$9569$	$56.04$	$8927$	$57.11$	$5728$	$65.49$	$6324$

Appendix

Table 9: The effectiveness of our method (

\theta=0.2

) across different baselines. The gray background indicates the performance of our method, adding on varying baselines; Bolded values denote the better performance between baseline and our method.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Jul 23, 2026

QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

Recent large reasoning models often develop long chain-of-thought responses during reinforcement learning (RL), resulting in high inference latency and deployment cost. Existing methods for response length control typically rely on explicit length penalties or additional control modules, which require careful tuning and may compromise reasoning quality. We propose Quadrant-weighted Sampling for Length-aware Policy Optimization (QLPO), a simple resampling-based variant of GRPO that introduces implicit length control without modifying the reward function. QLPO first over-generates candidate responses and then resamples the training group by preserving the empirical correct/incorrect ratio while favoring short correct responses and long incorrect responses. This reshapes the training distribution and implicitly encourages shorter model outputs. Across models ranging from 1.5B to 32B parameters, including both base models and strong reasoning models, QLPO consistently improves the accuracy-length trade-off. It reduces response length by 30% to 70% while preserving reasoning performance. These results suggest that structured resampling provides an effective and robust approach to efficient reasoning.

Siwei Chen, Siqi Chen, Xupeng Miao +1

School of Computer Science, Peking University · Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems · Department of Electronic Engineering, Tsinghua University +1

Overthinking in Language Models RL for Language Model Reasoning Group Relative Policy Optimization Efficient Language Model Reasoning RL Post-Training

Ask PDF

Aug 30, 2026

To Think or Not to Think: Allocating Reasoning Where It Helps

Reinforcement learning (RL) has proven effective in enhancing the reasoning performance of large language models (LLMs), particularly in complex mathematical and programming tasks. However, this capability comes with systematic length misallocation, in which models devote excessive reasoning to simple questions while terminating prematurely on harder ones, degrading inference efficiency with negligible accuracy improvement. Many length-adaptive methods mitigate this issue by allocating token budgets according to question difficulty, under the implicit assumption that harder questions benefit monotonically from extended reasoning. In contrast, we find that the effect of reasoning length on accuracy is concentrated on partially solvable questions. Our further analysis reveals that explicit length rewards can produce unintended training dynamics. Motivated by these findings, we propose CARE---Contrastive Accuracy Reward Estimation---which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost. Experiments across multiple reasoning benchmarks demonstrate that our method improves Pass@1 by up to

4\%

while simultaneously reducing reasoning length by

37\%

, achieving higher token efficiency. Code will be available upon the acceptance of this paper.

Zhengdong He, Yunfan Zhou, Jianguo Yao +2

Shanghai Key Laboratory of Scalable Computing and Systems · School of Computer Science, Shanghai Jiao Tong University

Efficient Language Model Reasoning RL for Language Model Reasoning

Ask PDF

Mar 9, 2026

SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning

Large reasoning models (LRMs) like OpenAI o1 and DeepSeek-R1 achieve high accuracy on complex tasks by adopting long chain-of-thought (CoT) reasoning paths. However, the inherent verbosity of these processes frequently results in redundancy and overthinking. To address this issue, existing works leverage Group Relative Policy Optimization (GRPO) to reduce LRM output length, but their static length reward design cannot dynamically adapt according to the relative problem difficulty and response length distribution, causing over-compression and compromised accuracy. Therefore, we propose SmartThinker, a novel GRPO-based efficient reasoning method with progressive CoT length calibration. SmartThinker makes a two-fold contribution: First, it dynamically estimates the optimal length with peak accuracy during training and guides overlong responses toward it to reduce response length while sustaining accuracy. Second, it dynamically modulates the length reward coefficient to avoid the unwarranted penalization of correct reasoning paths. Extensive experiment results show that SmartThinker achieves up to 52.5% average length compression with improved accuracy, and achieves up to 16.6% accuracy improvement on challenging benchmarks like AIME25. The source code can be found at https://github.com/SJTU-RTEAS/SmartThinker.

Chenzhi Hu, Qinzhe Hu, Yuhang Xu +6

Shanghai Jiao Tong University · Beihang University

Efficient Language Model Reasoning Large Reasoning Models Overthinking in Language Models RL for Language Model Reasoning

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

To Think or Not to Think: Allocating Reasoning Where It Helps

SmartThinker: Progressive Chain-of-Thought Length Calibration for Efficient Large Language Model Reasoning