The Art of Efficient Reasoning: Data, Reward, and Optimization

Taiqiang Wu; Zenan Xu; Bo Zhou; Ngai Wong

cs.CLFeb 24, 2026

The Art of Efficient Reasoning: Data, Reward, and Optimization

Authors: Taiqiang Wu, Zenan Xu, Bo Zhou, Ngai Wong

Organizations: The University of Hong Kong · LLM Department, Tencent

Abstract

Large Language Models (LLMs) consistently benefit from scaled Chain-of-Thought (CoT) reasoning, but also suffer from heavy computational overhead. To address this issue, efficient reasoning aims to incentivize short yet accurate thinking trajectories, typically through reward shaping with Reinforcement Learning (RL). In this paper, we systematically investigate the mechanics of efficient reasoning for LLMs. For comprehensive evaluation, we advocate for more fine-grained metrics, including length distribution conditioned on correctness and performance across a wide spectrum of token budgets ranging from 2k to 32k. First, we reveal that the training process follows a two-stage paradigm: length adaptation and reasoning refinement. Through extensive experiments (about 0.2 million GPU hours) in a unified protocol, we deconstruct training prompts and rollouts, reward shaping, and optimization strategies. A central finding is to maintain a sufficient density of positive reward signals and avoid the short-is-correct trap. Moreover, the learned length bias generalizes across domains and difficulty levels. We distill these findings into valuable insights and practical guidelines, and validate them across the Qwen3 models ranging from 0.6B to 30B, demonstrating the robustness and generalization. Weights are available at https://wutaiqiang.github.io/project/Art

Figures & tables

Figure 1: General pipeline for efficient reasoning via RL. The key is to promote short and accurate thinking trajectories via reward design. In this paper, we provide systematic insights ( ) considering data, reward, and optimization.

Figure 2: Performance comparison on AIME’25 for Qwen3 models ranging from 0.6B to 30B.

Figure 3: Training dynamics of various reward shaping methods on DeepSeek-R1-Distill-Qwen-1.5B . All of them follow the two-stage paradigm, including constraint satisfaction and optimization within a feasible region. The behaviors are distinct when evaluated under different token budgets.

Figure 4: Performance training on all prompts and easy/hard counterparts (rollout

L_{R}=16k

, target

L_{T}=4k

Figure 5: Performance with various rollouts

N

using DeepScaleR-Easy.

Strategy	Correct		Incorrect
Strategy	Short	Long	Short	Long
Vanilla	1	0	0	0
-I	1	0	-	-
-L&C	1	-	0	0
-L&C-S&I	1	-	-	0
-L&C-L&I	1	-	0	-

Table 1: Reward for different strategies on negative rollouts.

-

denotes masking out.

Figure 6: Performance for various reward strategies on negative rollouts (rollout

L_{R}=16k

, target

L_{T}=4k

N=24

). We also visualize

L_{R}=4k

L_{T}=4k

for comparison.

Figure 7: Performance for off-policy strategy with various staleness (i.e., 2,4,8,16).

Method	Mean@8 $\uparrow$	Pass@8 $\uparrow$	Length $\downarrow$
Qwen3-0.6B
Vanilla	13.33	26.67	14.9k
Ours (step 640)	24.58	36.67	8.9k
Qwen3-1.7B
Vanilla	35.00	60.00	17.7k
Ours (step 560)	38.75	60.00	11.2k

Table 2: Performance on AIME’25 for Qwen3 models.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Qwen3-30B-A3B-Instruct-2507			Qwen3-30B-A3B-Thinking-2507
Benchmark	Mean@4 $\uparrow$	Pass@4 $\uparrow$	Length $\downarrow$	Mean@4 $\uparrow$	Pass@4 $\uparrow$	Length $\downarrow$
Domain #1	27.5 / 28.0	40.9 / 40.9	7145 / 4881	34.2 / 34.1	50.0 / 49.8	11536 / 10237
Domain #2	35.1 / 36.0	51.7 / 50.7	7258 / 5182	42.2 / 41.6	68.8 / 68.4	11744 / 10847
Domain #3	20.5 / 21.0	31.9 / 29.8	3985 / 3102	27.0 / 26.2	34.0 / 34.7	8826 / 8278
Domain #4	15.4 / 15.8	24.0 / 25.0	7695 / 3176	16.4 / 14.7	28.9 / 26.2	4726 / 4292
Domain #5	40.6 / 40.6	53.3 / 55.4	6487 / 4500	57.6 / 58.6	78.2 / 78.2	2598 / 2351

Appendix

Table 3: Performance on private out-of-distribution benchmarks covering various domains. Each cell reports Vanilla / Ours . Our proposed model thinks shorter while maintaining comparable performance.

Figure 8: Performance on Qwen3-0.6B when adaptively setting target length

L_{T}

as 90th qualities.

Figure 9: Training dynamics of various reward shaping methods on AIME’25, MATH-500, and AMC.

Figure 10: Training dynamics of various reward shaping methods on Minerva Math, Olympiad Bench, and LiveCodeBench.

Figure 11: Performance training on various training prompts (rollout

L_{R}=16k

, target

L_{T}=8k

Figure 12: Performance training on various training prompts (rollout

L_{R}=4k

, target

L_{T}=4k

Figure 13: Performance training on various rollouts

N

(

L_{R}=16k

L_{T}=4k

Figure 14: Performance training on various rollouts

N

(

L_{R}=4k

L_{T}=4k

Question	Let $\mathcal{R}$ be the region in the complex plane consisting of all complex numbers $z$ that can be written as the sum of complex numbers $z_{1}$ and $z_{2}$ , where $z_{1}$ lies on the segment with endpoints $3$ and $4i$ , and $z_{2}$ has magnitude at most $1$ . What integer is closest to the area of $\mathcal{R}$ ? Let’s think step by step and output the final answer within \boxed{}
Hard	To determine the area of the region $\mathcal{R}$ , I first identify the segments involved. The first segment runs from $3$ to $4i$ , and the second segment encompasses all points within a unit circle centered at the origin. By translating these segments into the complex plane and applying appropriate transformations, I calculate the combined area as the sum of their respective areas. To find the area of the region $\mathcal{R}$ in the complex plane: Understanding the Components: Segment from 3 to 4i: This is a line segment in the complex plane with endpoints at 3 (which is (3, 0)) and 4i (which is (0, 4)). The length of this segment is $\sqrt{(3)^{2}+(4)^{2}}=5$ . Region defined by $z_{2}$ : This is a disk (or circle) centered at the origin with radius 1. Calculating the Area: The region $\mathcal{R}$ is the set of all points $z=z_{1}+z_{2}$ , where $z_{1}$ lies on the segment from 3 to 4i and $z_{2}$ lies within the unit disk. This effectively translates the unit disk to every point along the segment $z_{1}$ and sums the areas. Since the unit disk has an area of $\pi\times(1)^{2}=\pi$ , the total area of $\mathcal{R}$ is $\pi$ . Finding the Closest Integer: The area is approximately 3.1416, so the closest integer is 3. Final Answer: 3 ( Incorrect )
Easy	Okay, so I need to find the area of the region R in the complex plane. R consists of all complex numbers z that can be written as $z=z1+z2$ , where $z1$ lies on the segment with endpoints 3 and 4i, and z2 has magnitude at most 1. Then, I have to find the integer closest to this area. First, let me visualize this. The region R is the Minkowski sum of the segment from 3 to 4i and the disk of radius 1. So, if I think about it, R should be a rectangle with semicircles on both ends. But let me break it down step by step. The segment from 3 to 4i: In the complex plane, 3 is the point (3,0) and 4i is the point (0,4). So this is a straight line connecting (3,0) to (0,4). The length of this segment can be calculated, but maybe we don’t need it right away. z1 lies on this segment. So z1 can be parameterized. Let me define z1 as a point on the line segment between 3 and 4i. Let’s parameterize z1 with a parameter t from 0 to 1. So, $z1(t)=(3-3t)+(4i-4it)=(3(1-t),4(1-t))$ . Wait, actually, in complex terms, if I let t vary from 0 to 1, when t=0, z1 = 3, and when t=1, z1 = 4i. Alternatively, z1 can be expressed as $3+(4i-3)i$ , where i is a parameter from 0 to 1. Hmm, maybe that’s more complicated. Alternatively, the equation of the line from (3,0) to (0,4). The slope is $(4-0)/(0-3)=-4/3$ . So equation is $y=(-4/3)x+b$ . When $x=3,y=0:0=(-4/3)(3)+b=>0=-4+b=>b=4$ . So equation is $y=(-4/3)x+4$ . So, for any point z1 on the segment, z1 can be written as x + yi where x and y satisfy $y=(-4/3)x+4$ , and x ranges from 0 to 3? Wait, no, the segment goes from (3,0) to (0,4). So when x decreases from 3 to 0, y increases from 0 to 4. So, for any z1 on the segment, we can write $z1=(x,y)$ where $y=(-4/3)x+4,x\in[0,3]$ . Alternatively, parametric equations: Let’s define t as a parameter from 0 to 1. $z1(t)=(3-3t,0+4t)$ because at t=0, we get (3,0), and at t=1, we get (0,4). So in complex terms, $z1=(3-3t)+(4t)i$ . Then, z2 is a complex number with $\|z2\|\leq 1$ , which is a disk of radius 1 centered at the origin. Since R is the Minkowski sum of the segment and the disk, it is like sliding the disk along every point of the segment. So geometrically, it’s a rectangle with length equal to the length of the segment and width 2 (diameter of the disk). But with semicircular ends because the disk is only of radius 1. Wait, actually, no. The Minkowski sum of a line segment and a disk is called an "offset" of the disk by the segment. So it’s a rectangle with length equal to the segment length and width equal to twice the radius (since each end contributes a semicircle). But in this case, the radius is 1, so each end is a semicircle. Wait, but the segment is from (3,0) to (0,4), so the Minkowski sum would be the set of all points within distance 1 from the segment. So it’s like a rectangle with length equal to the segment length and width 2, but with semicircular ends of radius 1. Therefore, the area would be the area of the rectangle plus the area of the two semicircles. Area of rectangle: length * width. The length is the distance between 3 and 4i, which is $\sqrt{(}3²+4²)=5$ . The width is 2 (since radius is 1, diameter is 2). So area of rectangle is $5*2=10$ . Area of two semicircles: Each semicircle has area $(1/2)\pi(1)²$ , so two of them is $\pi$ . Therefore, total area is $10+\pi\approx 10+3.1416\approx 13.1416$ . So the area is approximately 13.1416, so the closest integer is 13. Wait, but let me verify again. The Minkowski sum of a line segment and a disk is indeed a rectangle with length equal to the segment’s length and width equal to 2r, and with two semicircular ends. So that’s correct. The length of the segment is 5, as calculated before. So area is $5\times 2+\pi\times(1)^{2}=10+\pi$ . So yes, the area is $10+\pi\approx 13.1416$ . Therefore, the integer closest to the area is 13. Final Answer 13 Double check: The region $\mathcal{R}$ consists of all complex numbers (z) that can be written as $z=z_{1}+z_{2}$ , where $z_{1}$ lies on the segment with endpoints (3) and (4i), and $z_{2}$ has magnitude at most (1). The segment from (3) to (4i) in the complex plane is a straight line connecting the points ((3, 0)) and ((0, 4)). This line has a length of (5) (calculated using the distance formula). The region ( $\mathcal{R}$ ) is the Minkowski sum of this segment and the disk of radius (1). This results in a shape that is a rectangle with length (5) and width (2) (since the radius of the disk is (1)), plus two semicircular ends each with radius (1). The area of the rectangle is ( $5\times 2=10$ ). The area of the two semicircular ends is ( $\pi\times 1^{2}=\pi$ ). Therefore, the total area of ( $\mathcal{R}$ ) is $10+\pi\approx 10+3.1416\approx 13.1416$ . The integer closest to the area of ( $\mathcal{R}$ ) is 13 . ( Correct )

Appendix

Table 4: Case comparison when trained on DeepScaleR-Easy and DeepScaleR-Hard using truncation strategy.

Question	The volume of a cone is given by the formula $V=\frac{1}{3}Bh$ , where $B$ is the area of the base and $h$ is the height. The area of the base of a cone is 30 square units, and its height is 6.5 units. What is the number of cubic units in its volume? Let’s think step by step and output the final answer within \boxed{}
Hard	To find the volume of the cone, I’ll use the formula V = (1/3) * B * h. The area of the base, B, is 30 square units, and the height, h, is 6.5 units. First, I’ll calculate one-third of the base area: (1/3) * 30 = 10. Next, I’ll multiply this result by the height: 10 * 6.5 = 65. Therefore, the volume of the cone is 65 cubic units. To find the volume of the cone, we can use the formula: $V=\frac{1}{3}\times B\times h$ Where: - ( B ) is the area of the base = 30 square units - ( h ) is the height = 6.5 units Plugging in the values: $V=\frac{1}{3}\times 30\times 6.5$ First, calculate one-third of 30: $\frac{1}{3}\times 30=10$ Then multiply by 6.5: $10\times 6.5=65$ Therefore, the volume of the cone is: 65 cubic units. ( Correct )
Easy	Okay, so I need to find the volume of a cone. The formula given is V = (1/3)Bh, where B is the area of the base and h is the height. Hmm, let me recall what each part means. First, the area of the base, which is B. The problem says the area of the base is 30 square units. So B is 30. Then, the height h is 6.5 units. Got that. So plugging these values into the formula: V = (1/3) * B * h. So that would be (1/3) * 30 * 6.5. Let me compute that step by step. First, I can compute 30 multiplied by 6.5. Let me do that: 30 * 6.5. Hmm, 30 * 6 is 180, and 30 * 0.5 is 15, so adding those together gives 195. So 30 * 6.5 is 195. Then, take that result and multiply by (1/3). So (1/3) * 195. Dividing 195 by 3. Well, 3 goes into 195 sixty-five times because 360 is 180, and 35 is 15, so 60 + 5 = 65. So (1/3)195 is 65. So V = 65 cubic units. Let me just double-check that. Alternatively, maybe I can compute it differently. Since V = (1/3) B * h, with B = 30 and h = 6.5. Compute B * h first: 30 * 6.5. As above, that’s 195. Then multiply by 1/3: 195 / 3 = 65. Yep, same result. So the volume is 65 cubic units. I think that’s correct. Final Answer The volume of the cone is 65 cubic units. Double check: The volume of a cone is given by the formula $V=\frac{1}{3}Bh$ , where $B$ is the area of the base and $h$ is the height. The area of the base $B$ is 30 square units, and the height $h$ is 6.5 units. First, we compute the product of ( B ) and ( h ): $30\times 6.5=195$ Next, we multiply this result by $\frac{1}{3}$ : $\frac{1}{3}\times 195=65$ Thus, the volume of the cone is 65 cubic units. ( Correct )

Appendix

Table 5: Case comparison when trained on DeepScaleR-Easy and DeepScaleR-Hard using truncation strategy. The model suffers from the reasoning collapse issue when trained on DeepScaleR-Hard, e.g., the double check part is skipped for shorter length.

Question	Compute $\dbinom{31}{28}$ . Let’s think step by step and output the final answer within \boxed{}
Vanilla	Okay, so I need to compute the binomial coefficient ( $\dbinom{31}{28}$ ). Hmm, binomial coefficients are those numbers that come up in combinations, right? They represent the number of ways to choose a subset of items from a larger set, disregarding the order. First, I remember that the formula for a binomial coefficient is ( $\dbinom{n}{k}=\frac{n!}{k!(n-k)!}$ ). So, in this case, (n = 31) and (k = 28). Let me write down the formula with these values: $\dbinom{31}{28}=\frac{31!}{28!(31-28)!}$ Simplifying the denominator: 31 - 28 = 3, so it becomes $\frac{31!}{28!3!}$ Hmm, but calculating factorials for such large numbers seems tedious. Maybe there’s a way to simplify this without computing the entire factorials. I remember that sometimes, in binomial coefficients, you can simplify by canceling out terms in the numerator and denominator. Let me think about how to do that. Since $(31!=31\times 30\times 29\times 28!)$ , I can rewrite the numerator as: $31!=31\times 30\times 29\times 28!$ So substituting back into the equation: $\dbinom{31}{28}=\frac{31\times 30\times 29\times 28!}{28!\times 3!}$ Oh, I see! The (28!) terms cancel out in numerator and denominator: $=\frac{31\times 30\times 29}{3!}$ Now, $(3!=3\times 2\times 1=6)$ , so substituting that in: $=\frac{31\times 30\times 29}{6}$ Alright, now I need to compute this. Let me compute the numerator first: 31 multiplied by 30 is 930, and 930 multiplied by 29. Hmm, 930 times 29. Let me do that step by step: - 930 x 20 = 18,600 - 930 x 9 = 8,370 Adding them together: 18,600 + 8,370 = 26,970 So the numerator is 26,970. The denominator is 6. So now, 26,970 divided by 6. Let’s do that: 26,970 ÷ 6. Dividing step by step: - 6 goes into 26 four times (4x6=24), remainder 2. - Bring down the 9: 29. 6 goes into 29 four times (4x6=24), remainder 5. - Bring down the 7: 57. 6 goes into 57 nine times (9x6=54), remainder 3. - Bring down the 0: 30. 6 goes into 30 five times. So putting it all together: 4, 4, 9, 5. So it’s 4,495. Wait, let me double-check that division because I might have messed up the steps. Alternatively, I can write 26,970 ÷ 6 as (26,000 + 970) ÷ 6. 26,000/6 = 4,333.333… 970/6 $\approx$ 161.666… Adding them together: 4,333.333…+161.666… = 4,495. Yep, that matches. So the result is 4,495. Therefore, ( $\dbinom{31}{28}=4,495$ ). Final Answer The value of ( $\dbinom{31}{28}$ ) is \boxed4495. To compute the binomial coefficient ( $\dbinom{31}{28}$ ), we use the formula: $\dbinom{n}{k}=\frac{n!}{k!(n-k)!}$ Substituting (n = 31) and (k = 28): $\dbinom{31}{28}=\frac{31!}{28!(31-28)!}=\frac{31!}{28!3!}$ We simplify the factorials by recognizing that ( $31!=31\times 30\times 29\times 28!$ ): $\dbinom{31}{28}=\frac{31\times 30\times 29\times 28!}{28!\times 3!}=\frac{31\times 30\times 29}{3!}$ Since (3! = 6), we have: $=\frac{31\times 30\times 29}{6}$ Calculating the numerator: $31\times 30=930][930\times 29=26970$ Dividing by the denominator: $\frac{26970}{6}=4495$ Thus, the value of ( $\dbinom{31}{28}$ ) is ( $\boxed{4495}$ ).
Ours	To compute the binomial coefficient ( $\dbinom{31}{28}$ ), I’ll start by recalling that the binomial coefficient formula is given by: $\dbinom{n}{k}=\frac{n!}{k!\cdot(n-k)!}$ Here, (n = 31) and (k = 28). Alternatively, since ( $\dbinom{n}{k}=\dbinom{n}{n-k}$ ), perhaps I can simplify the expression by choosing the smaller (k) value to make calculation easier. Calculating (31 - 28 = 3), which means: $\dbinom{31}{28}=\dbinom{31}{3}$ Now compute ( $\dbinom{31}{3}$ ): Using the formula: $\dbinom{31}{3}=\frac{31!}{3!\cdot(31-3)!}=\frac{31!}{3!\cdot 28!}$ Simplifying, since ( $31!=31\times 30\times 29\times 28!$ ): $\frac{31\times 30\times 29\times 28!}{3!\times 28!}=\frac{31\times 30\times 29}{3!}$ Calculate the denominator (3! = 6): Therefore, $\frac{31\times 30\times 29}{6}$ Computing numerator: 31 × 30 = 930; 930 × 29 = ? Compute: 930 × 29: Breakdown: = 930 × (30 - 1) = 930 × 30 - 930 × 1 = 27900 - 930 = 26970 Then divide by 6: 26970 ÷ 6 = ? Calculate: 26970 ÷ 6 = 4495 Wait perhaps: 6 × 4495 = 26970, yes. Therefore, the value becomes 4495. Thus, $\boxed{4495}$ To compute ( $\dbinom{31}{28}$ ), we use the property ( $\dbinom{n}{k}=\dbinom{n}{n-k}$ ): $\dbinom{31}{28}=\dbinom{31}{3}$ Compute ( $\dbinom{31}{3}$ ): Using the formula, $\dbinom{31}{3}=\frac{31!}{3!\cdot(31-3)!}=\frac{31\times 30\times 29}{3\times 2\times 1}$ Calculating: $31\times 30\times 29=26970$ Divide by (6) : $26970/6=4495$ Thus, $\boxed{4495}$

Appendix

Table 6: Case I for vanilla and our trained LLM. Both answers are correct .

Question	The volume of a cone is given by the formula $V=\frac{1}{3}Bh$ , where $B$ is the area of the base and $h$ is the height. The area of the base of a cone is 30 square units, and its height is 6.5 units. What is the number of cubic units in its volume? Let’s think step by step and output the final answer within \boxed{}
Vanilla	Okay, so I need to find the volume of a cone. The formula given is V = (1/3)Bh, where B is the area of the base and h is the height. Hmm, let’s break this down step by step. First, I know the area of the base is 30 square units. That’s B. Then, the height is 6.5 units, so h is 6.5. The formula is straightforward, but I should make sure I plug the values correctly. So, I can write it out as V = (1/3) * B * h. Substituting the given values, that becomes V = (1/3) * 30 * 6.5. Let me compute this step by step. First, let me calculate 1/3 of 30. Hmm, 30 divided by 3 is 10. So, 1/3 * 30 = 10. Now, I need to multiply that result by 6.5. So, 10 * 6.5. Let me think, 10 * 6 is 60, and 10 * 0.5 is 5, so 60 + 5 is 65. Therefore, 10 * 6.5 is 65. So, putting it all together, V = 65 cubic units. That should be the volume of the cone. Wait, let me double-check to make sure I didn’t make any calculation errors. 1/3 of 30 is indeed 10, correct. Then, 10 multiplied by 6.5. Yes, 6.5 is the same as 13/2, so 10(13/2) is (10/1)(13/2) = (10*13)/2 = 130/2 = 65. Yep, that’s right. So, I think I did that correctly. The volume is 65 cubic units. Final Answer The volume of the cone is $\boxed{65}$ cubic units. The volume of a cone is given by the formula $V=\frac{1}{3}Bh$ , where (B) is the area of the base and (h) is the height. Given: - The area of the base (B = 30) square units. - The height (h = 6.5) units. We substitute these values into the formula: $V=\frac{1}{3}\times 30\times 6.5$ First, calculate $\frac{1}{3}\times 30$ : $\frac{1}{3}\times 30=10$ Next, multiply the result by 6.5: $10\times 6.5=65$ Thus, the volume of the cone is ( 65 ) cubic units.
Ours	To find the volume of the cone, I’ll use the formula ( $V=\frac{1}{3}Bh$ ). Given that the base area (B) is 30 square units and the height (h) is 6.5 units. First, multiply these values together: ( $30\times 6.5=195$ ) Then, take one-third of that product: ( $\frac{1}{3}\times 195=65$ ) Therefore, the volume of the cone is 65 cubic units. Given: - Area of the base ((B)) = 30 square units - Height ((h)) = 6.5 units The formula for the volume (V) of a cone is: $V=\frac{1}{3}Bh$ Plugging in the given values: $V=\frac{1}{3}\times 30\times 6.5$ Calculate: $30\times 6.5=195$ Then, $V=\frac{195}{3}=65$ Answer: ( $\boxed{65}$ ) cubic units.

Appendix

Table 7: Case II for vanilla and our trained LLM. Both answers are correct .

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 370 papers

To Think or Not to Think: Allocating Reasoning Where It Helps
Aug 30, 2026Zhengdong He, Yunfan Zhou, Jianguo Yao +2Efficient Language Model Reasoning RL for Language Model Reasoning
Shanghai Key Laboratory of Scalable Computing and Systems · School of Computer Science, Shanghai Jiao Tong University
Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs
Feb 28, 2026Jie Cao, Tianwei Lin, Juekai Lin +7Efficient Language Model Reasoning CoT Reasoning
Zhejiang University · Tencent
LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models
May 10, 2026Songtao Wei, Yi Li, Zhikai Li +7RL for Language Model Reasoning Efficient Language Model Reasoning
University of Texas at Dallas · Emory University · University of Texas at Arlington +2

Aug 30, 2026J/K move · Enter open · S save

To Think or Not to Think: Allocating Reasoning Where It Helps

Zhengdong He, Yunfan Zhou, Jianguo Yao, Haibing Guan+1

Shanghai Key Laboratory of Scalable Computing and Systems · School of Computer Science, Shanghai Jiao Tong University

Efficient Language Model Reasoning RL for Language Model Reasoning

Open paper AskPDF arXiv

Reinforcement learning (RL) has proven effective in enhancing the reasoning performance of large language models (LLMs), particularly in complex mathematical and programming tasks. However, this capability comes with systematic length misallocation, in which models devote excessive reasoning to simple questions while terminating prematurely on harder ones, degrading inference efficiency with negligible accuracy improvement. Many length-adaptive methods mitigate this issue by allocating token budgets according to question difficulty, under the implicit assumption that harder questions benefit monotonically from extended reasoning. In contrast, we find that the effect of reasoning length on accuracy is concentrated on partially solvable questions. Our further analysis reveals that explicit length rewards can produce unintended training dynamics. Motivated by these findings, we propose CARE---Contrastive Accuracy Reward Estimation---which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost. Experiments across multiple reasoning benchmarks demonstrate that our method improves Pass@1 by up to

4\%

while simultaneously reducing reasoning length by

37\%

, achieving higher token efficiency. Code will be available upon the acceptance of this paper.

Abstract

Efficient Language Model Reasoning

Explore similar work

To Think or Not to Think: Allocating Reasoning Where It Helps

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models