Large language models can exhibit emergent reasoning behaviors, often manifested as recurring lexical patterns (e.g., "wait," indicating verification). However, complex reasoning trajectories remain sparse in unconstrained sampling, and standard RL often fails to guarantee the acquisition of diverse reasoning behaviors. We propose a systematic discovery and reinforcement of diverse reasoning patterns through structured reasoning, a paradigm that requires targeted exploration of specific reasoning patterns during the RL process. To this end, we propose Ctrl-R, a framework for learning structured reasoning via tractable trajectory control that actively guides the rollout process, incentivizing the exploration of diverse reasoning patterns that are critical for complex problem-solving. The resulting behavior policy enables accurate importance-sampling estimation, supporting unbiased on-policy optimization. We further introduce a power-scaling factor on the importance-sampling weights, allowing the policy to selectively learn from exploratory, out-of-distribution trajectories while maintaining stable optimization. Experiments demonstrate that Ctrl-R enables effective exploration and internalization of previously unattainable reasoning patterns, yielding consistent improvements across language and vision-language models on mathematical reasoning tasks.
Figures & tables
Figure 1: Examples of cognitive behaviors. Although cognitive behaviors are implicit, they often manifest through recurring lexical patterns during the reasoning process. We refer to reasoning that exhibits such patterns as structured reasoning .
Figure 2: Example of sampling a guided trajectory under Ctrl-R. We first sample a constraint α , then use the HMM to compute the marginal guidance γ(α∣x<t,xt) , which is combined with the proximal policy at each decoding step to form the guided behavior policy μα . We illustrate token-by-token decoding and visualize the guidance effect w using blue, white, and red dots. Before the constraint is satisfied, sampled tokens may be dominated by either the proximal policy (white, w>1 ) or the guidance function γ (blue, w<1 ). Once the constraint is satisfied, the behavior policy collapses to the proximal policy, with w=1 (red).
(A) Language Models
Backwarding
working backwards; thinking in reverse
Backtracking
let me go back; going back; undo the last step; try another way
Induction
try a small example; test simple numbers; look for a pattern
Counterfactual
what if I; imagine if I; alternatively, I could
Overthinking Awareness
getting too long; overcomplicating; going in circles; overthink; sufficient to answer
(B) Vision–Language Models
Table 1: Reasoning structures and their corresponding keyphrases, which are combined via logical OR to form lexical constraints. For example, any generation that includes either “working backwards” or “thinking in reverse” is considered to satisfy Backwarding constraint.
Method
AIME’24
AIME’25
MATH500
AMC’23
Minerva
Olympiad
Avg.
Acc@16
Acc@16
Acc@1
Acc@16
Acc@1
Acc@1
Score
Δ
Qwen3-8B-Base
DAPO
36.04
30.41
88.40
80.62
36.76
57.12
54.89
–
NL Guidance
34.58
31.25
89.40
81.25
36.39
57.27
55.02
+0.13
Reward Shaping
34.79
31.87
89.20
81.87
34.92
59.05
55.28
+0.39
Ctrl-R ( β=0.2 )
40.00
31.25
89.20
82.18
36.39
58.60
56.27
+1.38
Table 2: Performance comparison on math reasoning benchmarks for language models. We report Acc@16 or Acc@1 for all benchmarks. Δ denotes the absolute difference in average performance relative to the DAPO baseline. β is the power scaling factor in subsection 3.2 .
Method
MathVista
MathVerse
MathVision
MMMU-Pro
EMMA
Hallubench
Avg.
Δ
GRPO
68.10
40.61
25.39
42.66
25.62
65.09
44.57
–
Trajectory-Hint
69.80
43.22
27.43
44.34
28.05
66.14
46.49
+1.92
Auxiliary Reward
71.20
42.92
28.06
44.18
28.94
60.78
46.01
+1.44
OpenVLThinker
71.20
41.42
25.79
43.06
29.68
68.45
46.60
+2.03
Ctrl-R ( β=0.2 )
70.90
42.72
27.14
44.72
29.93
67.62
47.17
+2.60
Table 3: Performance comparison on math reasoning benchmarks for vision-language models. We report Acc@1 for all benchmarks. Δ denotes the absolute difference in average performance relative to the GRPO baseline. β is the power scaling factor in subsection 3.2
Figure 3: Trends in reasoning module usage during training and evaluation. The top panel shows rollout-time behavior. Keyword Usage (Strict) measures strict regex matches of the enforced keyphrases defined in Table 1 . Keyword Usage (Loose) captures lexical variants via expanded regex patterns ( Table 5 ). Average Scores indicate the accuracy of outputs exhibiting the corresponding reasoning patterns, as identified by loose keyword matches. All figures highlight the relative percentage change of Ctrl-R compared to the DAPO setting.
Figure 7
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Early Training Efficiency Comparison: The plot compares the reward/accuracy over time for our Medium Control method (red) against Baseline and other control settings. Our method demonstrates a +14.0% higher AUC in the first 150 steps (highlighted as the ”Early Efficiency Zone”), indicating significantly faster convergence during the initial exploration phase.
Figure 6: Deviation in keyword accuracy relative to base average across training steps. Four panels show the accuracy improvement or decline associated with specific keywords during progressive training intervals (Steps 1-100 through 301-330). The data highlights how the influence of keywords across General, Reflection, and VisualGrounding categories shifts relative to the evolving base accuracy of the model.
Figure 7: Math reasoning performance and output length across AIME’24, AIME’25, MATH500, AMC’23, Minerva, and Olympiad for Qwen3-1.7B-Base.
Natural-Language Reasoning Hints for Trajectory Prompting used in DAPO
This reminds me of a problem I’ve seen before; I’ll try using the same strategy here. Maybe I can solve this the same way I solved that other similar problem. Let me recall a specific case that feels comparable and see if the same idea works again. I think I’ve tackled a similar puzzle before; I’ll apply the same method now. The result hints at a certain pattern; I’ll assume that’s what’s happening and see if it leads to a solution. Based on the clues, I have a hunch about the method involved; I’ll test that assumption as I go. Maybe the simplest explanation is the right one; I’ll go with it and check if everything adds up. Let me hypothesize a reason behind the result and work backward to see if it solves the problem. I’ll check a few specific cases to see a pattern, then guess the general rule. Let me start with small examples to spot a trend I can use. I notice a pattern in these outputs; I bet I can generalize it to solve the problem. From the examples I’ve tried, I think I see a general rule; I’ll apply it now. I know a general formula that applies here; I’ll use it to work out the answer. Given these facts, I can apply a standard rule to deduce the answer. I have some relevant theorems in mind; I’ll follow them step by step to derive the solution. I’ll approach this systematically; I’ll start with general principles and work my way down to the specific answer.
Natural-Language Reasoning Hints for Trajectory Prompting used in VLM
Try to use the following keywords: [notation, symbols, title, alignment, coordinates, layout, scale, shape, image, describe, see, identify, observe, inspect]. Try to use the following keywords: [backwards, reverse, recall, imagine, alternatively, maybe, small example, pattern]. Try to use the following keywords: [Wait, double-check, doube check, verify, Re-examine, re-examine, missed].
Appendix
Table 4: Natural Language Hint used in NL Guidance experiments.
Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@k. On Geometry3K, UEC-RL achieves a 37.9% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.
Chen Wang, Lai Wei, Yanzhi Zhang +5
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3
Reinforcement learning (RL) has emerged as a promising paradigm for training reasoning-oriented models by leveraging rule-based reward signals. However, RL training typically tends to improve single-sample success rates (i.e., Pass@1) while offering limited exploration of diverse reasoning trajectories, which is crucial for multi-sample performance (i.e., Pass@k). Our preliminary analysis reveals that this limitation stems from a fundamental squeezing effect, whereby probability mass is excessively concentrated on a narrow subset of high-reward trajectories, restricting genuine exploration and constraining attainable performance under RL training. To address this issue, in this work, we propose Steering Probability Squeezing (SPS), a training paradigm that interleaves conventional RL with inverse reinforcement learning (IRL). SPS treats on-policy rollouts as demonstrations and employs IRL to explicitly reshape the induced trajectory distribution, thereby enhancing exploration without introducing external supervision. Experiments on five commonly used reasoning benchmarks demonstrate that SPS can enable better exploration and improve Pass@k. Beyond algorithmic contributions, we provide an analysis of RL learning dynamics and identify an empirical upper bound on Pass@k, shedding light on intrinsic exploration limits in RL-based reasoning models. Our findings suggest that alternating between RL and IRL offers an effective pathway toward extending the exploration capacity of reasoning-oriented large language models.
Yifu Huo, Chenglong Wang, Ziming Zhu +9
1Northeastern University, Shenyang, China · 2CAS Key Laboratory of Behavioral Science, Beijing, China · 3Independent Researcher, Beijing, China +1
Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward. We introduce ExTra (Exploratory Trajectory Optimization), a GRPO-compatible framework that extracts exploration signals from the model's own rollouts. ExTra combines two mechanisms: (i) a novelty reward that adds embedding-based diversity bonuses after GRPO normalization, rewarding diverse correct solutions; and (ii) entropy-guided prefix regeneration, which scores partial trajectories using entropy signals and continues exploration from promising intermediate steps. Across six mathematical reasoning benchmarks, ExTra improves Qwen3-1.7B over GRPO by about +5 points on pass@1 and +7 points on pass@16, showing that trajectory-level exploration signals can improve both single-sample accuracy and inference-time coverage.