Learning Structured Reasoning via Tractable Trajectory Control

Po-Nien Kung; Zhen Yang; Jeffrey Luo; Cheng-Fu Yang; Haikang Deng; Zi-Yi Dou; Yinfei Yang; Nanyun Peng

cs.AIMar 2, 2026

Learning Structured Reasoning via Tractable Trajectory Control

Authors: Po-Nien Kung, Zhen Yang, Jeffrey Luo, Cheng-Fu Yang, Haikang Deng, Zi-Yi Dou, Yinfei Yang, Nanyun Peng,

Organizations: University of California, Los Angeles · Apple

Abstract

Large language models can exhibit emergent reasoning behaviors, often manifested as recurring lexical patterns (e.g., "wait," indicating verification). However, complex reasoning trajectories remain sparse in unconstrained sampling, and standard RL often fails to guarantee the acquisition of diverse reasoning behaviors. We propose a systematic discovery and reinforcement of diverse reasoning patterns through structured reasoning, a paradigm that requires targeted exploration of specific reasoning patterns during the RL process. To this end, we propose Ctrl-R, a framework for learning structured reasoning via tractable trajectory control that actively guides the rollout process, incentivizing the exploration of diverse reasoning patterns that are critical for complex problem-solving. The resulting behavior policy enables accurate importance-sampling estimation, supporting unbiased on-policy optimization. We further introduce a power-scaling factor on the importance-sampling weights, allowing the policy to selectively learn from exploratory, out-of-distribution trajectories while maintaining stable optimization. Experiments demonstrate that Ctrl-R enables effective exploration and internalization of previously unattainable reasoning patterns, yielding consistent improvements across language and vision-language models on mathematical reasoning tasks.

Figures & tables

Figure 1: Examples of cognitive behaviors. Although cognitive behaviors are implicit, they often manifest through recurring lexical patterns during the reasoning process. We refer to reasoning that exhibits such patterns as structured reasoning .

Figure 2: Example of sampling a guided trajectory under Ctrl-R. We first sample a constraint $\alpha$ , then use the HMM to compute the marginal guidance $\gamma(\alpha\mid x_{<t},x_{t})$ , which is combined with the proximal policy at each decoding step to form the guided behavior policy $\mu_{\alpha}$ . We illustrate token-by-token decoding and visualize the guidance effect $w$ using blue, white, and red dots. Before the constraint is satisfied, sampled tokens may be dominated by either the proximal policy (white, $w>1$ ) or the guidance function $\gamma$ (blue, $w<1$ ). Once the constraint is satisfied, the behavior policy collapses to the proximal policy, with $w=1$ (red).

(A) Language Models
Backwarding	working backwards; thinking in reverse
Backtracking	let me go back; going back; undo the last step; try another way
Induction	try a small example; test simple numbers; look for a pattern
Counterfactual	what if I; imagine if I; alternatively, I could
Overthinking Awareness	getting too long; overcomplicating; going in circles; overthink; sufficient to answer
(B) Vision–Language Models

Table 1: Reasoning structures and their corresponding keyphrases, which are combined via logical OR to form lexical constraints. For example, any generation that includes either “working backwards” or “thinking in reverse” is considered to satisfy Backwarding constraint.

Method	AIME’24	AIME’25	MATH500	AMC’23	Minerva	Olympiad	Avg.
	Acc@16	Acc@16	Acc@1	Acc@16	Acc@1	Acc@1	Score	$\Delta$
Qwen3-8B-Base
DAPO	36.04	30.41	88.40	80.62	36.76	57.12	54.89	–
NL Guidance	34.58	31.25	89.40	81.25	36.39	57.27	55.02	+0.13
Reward Shaping	34.79	31.87	89.20	81.87	34.92	59.05	55.28	+0.39
Ctrl-R ( $\beta=0.2$ )	40.00	31.25	89.20	82.18	36.39	58.60	56.27	+1.38

Table 2: Performance comparison on math reasoning benchmarks for language models. We report Acc@16 or Acc@1 for all benchmarks.

\Delta

denotes the absolute difference in average performance relative to the DAPO baseline.

\beta

is the power scaling factor in subsection 3.2 .

Method	MathVista	MathVerse	MathVision	MMMU-Pro	EMMA	Hallubench	Avg.	$\Delta$
GRPO	68.10	40.61	25.39	42.66	25.62	65.09	44.57	–
Trajectory-Hint	69.80	43.22	27.43	44.34	28.05	66.14	46.49	+1.92
Auxiliary Reward	71.20	42.92	28.06	44.18	28.94	60.78	46.01	+1.44
OpenVLThinker	71.20	41.42	25.79	43.06	29.68	68.45	46.60	+2.03
Ctrl-R ( $\beta=0.2$ )	70.90	42.72	27.14	44.72	29.93	67.62	47.17	+2.60

Table 3: Performance comparison on math reasoning benchmarks for vision-language models. We report Acc@1 for all benchmarks.

\Delta

denotes the absolute difference in average performance relative to the GRPO baseline.

\beta

is the power scaling factor in subsection 3.2

Panel of Figure 3 — Figure 3: Trends in reasoning module usage during training and evaluation. The top panel shows rollout-time behavior. Keyword Usage (Strict) measures strict regex matches of the enforced keyphrases defined in Table 1 . Keyword Usage (Loose) captures lexical variants via expanded regex patterns ( Table 5 ). Average Scores indicate the accuracy of outputs exhibiting the corresponding reasoning patterns, as identified by loose keyword matches. All figures highlight the relative percentage change of Ctrl-R compared to the DAPO setting.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Early Training Efficiency Comparison: The plot compares the reward/accuracy over time for our Medium Control method (red) against Baseline and other control settings. Our method demonstrates a +14.0% higher AUC in the first 150 steps (highlighted as the ”Early Efficiency Zone”), indicating significantly faster convergence during the initial exploration phase.

Figure 6: Deviation in keyword accuracy relative to base average across training steps. Four panels show the accuracy improvement or decline associated with specific keywords during progressive training intervals (Steps 1-100 through 301-330). The data highlights how the influence of keywords across General, Reflection, and VisualGrounding categories shifts relative to the evolving base accuracy of the model.

Natural-Language Reasoning Hints for Trajectory Prompting used in DAPO

This reminds me of a problem I’ve seen before; I’ll try using the same strategy here. Maybe I can solve this the same way I solved that other similar problem. Let me recall a specific case that feels comparable and see if the same idea works again. I think I’ve tackled a similar puzzle before; I’ll apply the same method now. The result hints at a certain pattern; I’ll assume that’s what’s happening and see if it leads to a solution. Based on the clues, I have a hunch about the method involved; I’ll test that assumption as I go. Maybe the simplest explanation is the right one; I’ll go with it and check if everything adds up. Let me hypothesize a reason behind the result and work backward to see if it solves the problem. I’ll check a few specific cases to see a pattern, then guess the general rule. Let me start with small examples to spot a trend I can use. I notice a pattern in these outputs; I bet I can generalize it to solve the problem. From the examples I’ve tried, I think I see a general rule; I’ll apply it now. I know a general formula that applies here; I’ll use it to work out the answer. Given these facts, I can apply a standard rule to deduce the answer. I have some relevant theorems in mind; I’ll follow them step by step to derive the solution. I’ll approach this systematically; I’ll start with general principles and work my way down to the specific answer.

Natural-Language Reasoning Hints for Trajectory Prompting used in VLM

Try to use the following keywords: [notation, symbols, title, alignment, coordinates, layout, scale, shape, image, describe, see, identify, observe, inspect]. Try to use the following keywords: [backwards, reverse, recall, imagine, alternatively, maybe, small example, pattern]. Try to use the following keywords: [Wait, double-check, doube check, verify, Re-examine, re-examine, missed].

Appendix

Table 4: Natural Language Hint used in NL Guidance experiments.

Reasoning Structure	Lexical Patterns (Regex)
Backwarding	\bwork(?:ing)?\s+backwards?\b \bwork\s+back\s+from\ s+(?:the\s+)?(?:answer\|goal\|target\|result\|conclusion)\b \bstart(?:ing)?\s+from\ s+(?:the\s+)?(?:answer\|goal\|target\|end\|desired\s+result)\b \bthink(?:ing)?\s+in\s+reverse\b \breverse[-\s]?engineer(?:ing)?\b \bback[-\s]?substitut(?:e\|ing\|ion)\b \bback\s+solve\b

Backtracking	\bbacktrack(?:ing\|ed)?\b \bgo(?:ing)?\s+back\b \bundo\s+(?:the\s+)?last\s+step\b \bretrace(?:ing)?\s+(?:my\s+)?steps\b \bI\s+(?:made\|have)\s+a\s+mistake\b \bthat\s+(?:was\|is)\s+wrong\b \btry\s+(?:another\|a\s+different)\s+(?:way\|method\|approach)\b

Induction	\blook\s+for\s+a\s+pattern\b \btry(?:ing)?\s+a\s+small\s+example\b \bcheck(?:ing)?\s+small\s+cases\b \bfor\s+(?:n\|k)\s=\s\d+\b \bgeneraliz(?:e\|ing)\s+from\s+(?:these\|the)\s+(?:examples\|cases)\b \bconjectur(?:e\|ing)\b \bbase\s+case\b

Appendix

Table 5: Loose lexical patterns used to identify reasoning structures during rollout analysis.

Structured Reasoning in Language Models

See all 36 papers

Targeted Exploration via Unified Entropy Control for Reinforcement Learning
Apr 16, 2026Chen Wang, Lai Wei, Yanzhi Zhang +5Reinforcement Learning RL for Language Model Reasoning
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3
SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models
Apr 18, 2026Yifu Huo, Chenglong Wang, Ziming Zhu +9Reinforcement Learning RL for Language Model Reasoning
1Northeastern University, Shenyang, China · 2CAS Key Laboratory of Behavioral Science, Beijing, China · 3Independent Researcher, Beijing, China +1
ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning
Jun 23, 2026Wenyang Hu, Junxiang Jia, Zhen Shu +3RL for Language Models RL for Language Model Reasoning
1National University of Singapore · 2SAP

Apr 16, 2026J/K move · Enter open · S save

Targeted Exploration via Unified Entropy Control for Reinforcement Learning

Chen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao+4

College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University+3

Reinforcement Learning RL for Language Model Reasoning RL Exploration Maximum Entropy RL RL for VLMs

Open paper AskPDF arXiv

Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@

k

. On Geometry3K, UEC-RL achieves a 37.9% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.

Abstract

Structured Reasoning in Language Models

Explore similar work

Targeted Exploration via Unified Entropy Control for Reinforcement Learning

SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models

ExTra: Exploratory Trajectory Optimization for Language Model Reinforcement Learning