$R^3$: Training Robots to Reason in Natural Language via Reinforcement Learning

Lehong Wu; Yuxiao Qu; Zheyuan Hu; Ivan Zhang; Limin Wei; Zackory Erickson; Aviral Kumar

cs.ROAug 26, 2026

$R^3$ : Training Robots to Reason in Natural Language via Reinforcement Learning

Authors: Lehong Wu, Yuxiao Qu, Zheyuan Hu, Ivan Zhang, Limin Wei, Zackory Erickson, Aviral Kumar

Organizations: Carnegie Mellon University

Abstract

Reasoning in language allows foundation models to spend more test-time compute on hard problems, such as those requiring decomposition, constraint tracking, and prediction of future consequences. Whether this mechanism can improve robotic manipulation remains unclear, where long-horizon tasks require tracking partial progress, reasoning about object relations, recovering from mistakes, and steering noisy low-level policies. In this paper, we study whether VLMs can be trained to reason directly in natural language to guide low-level manipulation policies. We introduce $R^3$ , a simple post-training recipe that turns off-the-shelf VLMs into robotic reasoners: it first mid-trains a VLM on expert-generated reasoning traces to initialize the desired reasoning style, then improves the reasoner with single-step rubric-based RL from offline action data. Unlike prior robotic reasoning methods that mostly use structured traces as auxiliary supervision, $R^3$ trains free-form language reasoning to produce test-time guidance for action. We instantiate $R^3$ on Language Table and simulated bimanual grocery packing, two controlled testbeds for studying robotic reasoning and long-horizon manipulation. $R^3$ improves exploration and generalization across unseen tasks and significantly outperforms instruction-only imitation learning baselines on both benchmarks. Our analyses suggest that free-form language reasoning can function as a test-time compute mechanism for steering low-level policies. Our project page is available at https://robotic-reasoner.github.io/.

Figures & tables

Figure 1: Two-stage training of $\mathcal{R}^{3}$ . $\mathcal{R}^{3}$ trains a high-level VLM to reason in natural language and steer a fixed low-level robot policy for robotic manipulation tasks. $\mathcal{R}^{3}$ proceeds in two stages: Stage I (mid-training) imbues an off-the-shelf VLM with the reasoning style and behaviors needed to produce useful instructions for the low-level policy. Stage II (single-step RL) further improves the VLM by training it to generate reasoning traces that match the expert instruction in offline data.

Figure 2: Policy architecture. A high-level VLM generates a reasoning trace and an instruction given the scene, goal, and previous response. A language-conditioned low-level actor takes the instruction as input and emits the action that controls the robot.

Figure 3: Example of expert-collected trajectory and reasoning trace on the line task of Language Table. The notation $\times n$ indicates that the expert repeats this instruction $n$ times.

Figure 4: Example grocery packing task. From left to right: left-wrist, base, and right-wrist camera views.

Task	w/o history	w/ history
line	44.9%	51.0%
V	52.3%	57.6%

Table 1: Ablation of interaction history on the expert. Incorporating history improves the expert VLM’s performance.

	Imitation			Ours					$\Delta$	Ref.
	Base (w/o reason)	IL (mid only)	IL	Base	$\mathcal{R}^{3}$ (mid only)	$\mathcal{R}^{3}$ (RL only)	$\mathcal{R}^{3}$ (1/4th mid)	$\mathcal{R}^{3}$	$\mathcal{R}^{3}$ – IL	Gemini
$\mathcal{T}_{\mathrm{M}}$ (mid-training tasks)
group	11.9 $\pm$ 1.9	54.4 $\pm$ 2.9	64.7 $\pm$ 2.8	24.1 $\pm$ 2.5	53.8 $\pm$ 2.9	38.9 $\pm$ 3.3	55.3 $\pm$ 2.9	65.8 $\pm$ 3.0	$+$ 1.1 $\pm$ 4.1	71.3
line	5.7 $\pm$ 1.4	23.6 $\pm$ 2.6	33.9 $\pm$ 2.9	8.8 $\pm$ 1.7	22.9 $\pm$ 2.5	19.4 $\pm$ 2.7	31.8 $\pm$ 1.9	32.4 $\pm$ 3.1	$-$ 1.5 $\pm$ 4.2	51.0
V	22.8 $\pm$ 2.5	25.3 $\pm$ 2.6	40.9 $\pm$ 3.0	21.1 $\pm$ 2.4	33.3 $\pm$ 2.8	37.4 $\pm$ 3.2	69.4 $\pm$ 2.6	69.2 $\pm$ 2.9	$+$ 28.3 $\pm$ 4.2	57.6
L	24.8 $\pm$ 2.6	29.6 $\pm$ 2.7	34.9 $\pm$ 2.8	25.2 $\pm$ 2.6	28.1 $\pm$ 2.7	27.6 $\pm$ 3.1	29.4 $\pm$ 2.7	35.2 $\pm$ 3.4	$+$ 0.3 $\pm$ 4.4	36.2

Table 2: Main results. We compare base models, imitation baselines, and

\mathcal{R}^{3}

variants. Values are percentages with 95% confidence intervals. The green / red cells in the

\Delta

column mark significant gains/losses (

|\Delta|>

CI). Gemini’s performance during data collection is shown for reference. Bold / underlined values mark the best/second-best non-expert model.

	Imitation			Ours
	IL	IL (Pre-train)	IL (Co-train)	$\mathcal{R}^{3}$	$\mathcal{R}^{3}$ (trunc@100)	$\mathcal{R}^{3}$ (trunc@50)	$\mathcal{R}^{3}$ (w/o reason)
$\mathcal{T}_{\mathrm{M}}$ (mid-training tasks)
group	64.7	65.9 $(+1.2)$	66.5 $(+1.8)$	65.8	60.9 $(-4.9)$	53.1 $(-12.7)$	39.8 $(-26.0)$
line	33.9	34.6 $(+0.7)$	33.9 $(0.0)$	32.4	28.9 $(-3.5)$	21.1 $(-11.3)$	17.6 $(-14.8)$
V	40.9	43.4 $(+2.5)$	36.6 $(-4.3)$	69.2	59.4 $(-9.8)$	26.6 $(-42.6)$	20.3 $(-48.9)$
L	34.9	38.3 $(+3.4)$	35.4 $(+0.5)$	35.2	30.5 $(-4.7)$	30.5 $(-4.7)$	32.0 $(-3.2)$

Table 3: IL with pre-training or co-training on reasoning, and truncation of

\mathcal{R}^{3}

reasoning at test time. Values are percentages. Parenthetical values for IL (Pre-train) and IL (Co-train) indicate absolute changes relative to IL; for truncated

\mathcal{R}^{3}

variants, they indicate absolute changes relative to full

\mathcal{R}^{3}

Figure 6: RL affects instruction distributions. RL from the base model broadly rewrites the instruction distribution, while RL after mid-training makes localized edits.

	Ours	Post-hoc	Ours	Post-hoc
			+ ECoT	+ ECoT
$\mathcal{T}_{\mathrm{M}}$ (mid-training tasks)
group	53.8	53.9	51.0	46.6
line	22.9	19.7	20.6	20.6
V	33.3	28.7	28.4	30.2
L	28.1	31.1	28.3	25.9

Table 4: Evaluation results of ECoT variants. Trained by SFT on only our mid-training data. Values are percentages. Bold values mark the best model.

	Success Rate				Progress
	Base (w/o reason)	IL (w/o reason)	Base (w/ reason)	Ours	Base (w/o reason)	IL (w/o reason)	Base (w/ reason)	Ours
Task 1	76.0 $\pm$ 12.0	42.0 $\pm$ 13.3	84.0 $\pm$ 15.7	90.0 $\pm$ 7.9	81.0 $\pm$ 10.1	69.0 $\pm$ 7.6	86.0 $\pm$ 14.1	91.0 $\pm$ 7.4
Task 2	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	96.0 $\pm$ 7.8	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	98.7 $\pm$ 2.6	100.0 $\pm$ 0.0
Task 3	0.0 $\pm$ 0.0	40.0 $\pm$ 14.1	0.0 $\pm$ 0.0	60.0 $\pm$ 14.2	59.0 $\pm$ 2.7	78.0 $\pm$ 5.7	38.7 $\pm$ 4.9	84.0 $\pm$ 6.5
Task 4	4.0 $\pm$ 5.5	12.0 $\pm$ 9.2	12.0 $\pm$ 13.6	26.0 $\pm$ 12.7	48.0 $\pm$ 6.5	49.0 $\pm$ 7.2	45.0 $\pm$ 10.2	66.5 $\pm$ 7.8
Task 5	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	16.0 $\pm$ 10.3	34.0 $\pm$ 4.8	31.6 $\pm$ 3.4	38.4 $\pm$ 6.2	66.0 $\pm$ 6.5

Table 5: Main results on grocery packing. Success rate and normalized progress on 12 held-out tasks. Values are percentages with 95% confidence intervals. Bold values mark the best model.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Stages	Goal
Task 1	2	Move the gelatin box from the medium tray to the small tray, then pack the foam brick into the small tray.
Task 2	3	Pack the gelatin box and foam brick into the medium tray alongside the sugar box.
Task 3	3	Remove the cracker box from the medium tray, then pack the sugar box and foam brick there; the gelatin box stays in the small tray.
Task 4	4	Move the gelatin box and foam brick from the medium tray to the small tray, then pack the sugar box into the medium tray.
Task 5	5	Redistribute five objects from the large tray: cracker box stays in large; sugar box and soup can to medium; gelatin box and foam brick to small.
Task 6	4	Move the soup can from small to medium, move the gelatin box and foam brick from medium to small, and pack the sugar box into medium.

Appendix

Table 6: Goal descriptions and stage counts for 12 held-out packing specifications for evaluation.

	Language Table	Bimanual packing
Embodiment	single cylindrical pusher	dual 7-DoF xArm-7 + Robotiq 2F-85
Action	2D tabletop pushing	14-D end-effector (7 per arm)
Cameras	top-down RGB	base + left/right wrist RGB
Low-level policy	pre-trained language-conditioned policy	VLA fine-tuned from $\pi_{0.5}$
Control frequency	10 Hz	60 Hz
Max episode length	400 steps (40s)	21,600 steps (300s)

Appendix

Table 7: Comparison of key environment and evaluation configs for Language Table and simulated bimanual grocery packing.

Figure 10: Successful execution of tasks. For each task, we show the task name, a long-horizon goal, and the image of the goal state. Task-related blocks are annotated with white dots.

Question Class	Question Type	Think?	Qwen3.5-4B variants				Reference
			Base	$\mathcal{R}^{3}$ (mid only)	$\mathcal{R}^{3}$ (RL only)	$\mathcal{R}^{3}$	Gemini	Random
Absolute Position	Perception	No	35.7	39.5	41.8	41.3	62.3	–
Relative Position	Perception	No	39.3	56.8	48.0	59.3	66.7	–
Distance	Perception	No	29.3	48.1	38.5	52.5	82.3	–
Instruction Execution	Instruction Comp.	Yes	67.8	65.0	61.2	64.3	77.3	50.0
Instruction Inference	Instruction Comp.	Yes	30.9	45.2	43.2	55.3	85.1	20.0

Appendix

Table 10: Accuracy by VQA question class. Gemini is reported as an external reference and is not included in the ranking. Bold values mark the best non-reference model.

Signal	Base	$\mathcal{R}^{3}$ (mid only)	$\mathcal{R}^{3}$
Avg. length (chars)	1079	563	696
Max length (chars)	4256	1288	1798
Clean Reasoning: -first format	19/30	30/30	30/30
Backtracking / re-examination	5	0	1
First-person planning	11	15	23
Hallucinated objects	3	0	0

Appendix

Table 11: Countable reasoning-trace signals across checkpoints. We analyze aligned reasoning traces from the base, mid-trained, and mid-trained + RL models on the same 30 validation scenes. Mid-training stabilizes the output format and removes hallucinated objects, while RL increases explicit planning without reintroducing the base model’s formatting failures.

Dimension	Base	$\mathcal{R}^{3}$ (mid only)	$\mathcal{R}^{3}$
Output format	Inconsistent: sometimes places the instruction before reasoning, omits the Reasoning: prefix, includes stray quotes, or fails to emit an instruction.	Clean and rigid template across all inspected traces.	Clean and rigid template across all inspected traces.
Length / verbosity	Highly variable, with occasional long rambling traces.	Shortest and most economical.	Moderate length: fuller than mid-training, but still controlled.
Reasoning style	Exploratory and deliberative, often thinking through many alternatives.	Direct and decisive, usually reaching a single-pass conclusion.	Structured: restates the goal, assesses the current state, then chooses the next step.
Self-correction / backtracking	Frequent second-guessing and occasional non-convergent loops.	No observed backtracking in the inspected traces.	Rare backtracking; when present, the model recovers and commits to an instruction.
Goal grounding	Often loses the goal or makes meta-comments about rules and format.	Briefly restates the task.	More explicitly re-derives task constraints before acting.
Spatial tracking	Weak; sometimes misreads the scene or confuses object positions.	Decent; often references the arm position or previous plan.	Strongest; tracks what has already been placed and what remains to be done.

Appendix

Table 12: Qualitative reasoning-behavior comparison. The base model is exploratory but unstable, mid-training makes the reasoning interface reliable, and RL on top of mid-training produces more deliberate, state-aware reasoning while preserving format and instruction validity.

	Agree	Cohen’s $\kappa$	Pearson
Human-maj vs. Human †	94.7/100	0.911	0.975
Human-maj vs. Qwen3.5-35B-A3B	90/100	0.837	0.984
Human-maj vs. Gemini 3.6 Flash	91/100	0.842	0.974
Human-maj vs. GPT-5.6 Sol	93/100	0.880	0.985

Appendix

Table 13: Agreement of majority-vote human labels (Human-maj) vs. each judge on 100 prompt-response pairs. † Averaged over 3 human annotators.

Figure 11: Confusion matrix of majority-vote human labels (Human-maj) vs. Qwen3.5-35B-A3B.

Figure 12: Example of expert-collected trajectory on the V task. The notation $\times n$ indicates that the expert repeats this instruction $n$ times.

Figure 13: Example expert-collected trajectory on the iip task. The notation $\times n$ indicates that the expert repeats the instruction $n$ times. At step 6, the instruction is to move the blue cube, but the robot unintentionally pushes the blue moon, resulting in task success. This example shows that expert demonstrations may also contain suboptimal behaviors.

Figure 14: Example expert-collected trajectory on the T task. The notation $\times n$ indicates that the expert repeats the instruction $n$ times. At step 8, the instruction is to move the red moon, but the robot mistakenly moves the blue moon; therefore, at step 9, the expert first corrects this behavior. At step 11, instead of moving a target block, the expert chooses to clear the yellow star, which is an oracle block for making the T-shape.

Figure 15: Example of reasoning strategy learned via $\mathcal{R}^{3}$ . The $\mathcal{R}^{3}$ model first describes the scene, assesses the current progress toward the goal, plans into the future, and finally outputs an executable instruction.

Figure 16: Example of reasoning strategy learned via $\mathcal{R}^{3}$ . The red pentagon is occluded by the robot arm. The $\mathcal{R}^{3}$ model re-examines the scene, task information, and history to correctly infer the object state.

Figure 17: Example of reasoning strategy learned via $\mathcal{R}^{3}$ . The $\mathcal{R}^{3}$ model considers several alternate plans before committing to one particular choice.

Figure 18: Example of reasoning strategy of the Qwen3.5-4B base model. The base model’s reasoning shows incomplete scene understanding, weak estimation of task progress, and under-justified planning, leading to a goal-misaligned instruction.

Figure 19: Example of reasoning strategy of the Qwen3.5-4B base model. Although the base model sometimes displays backtracking behavior, such behavior is unreliable. The model may signal re-evaluation with phrases such as “Wait” or “Let’s re-evaluate,” but the revision is often superficial or driven by hallucinated assumptions rather than grounded error correction. Consequently, the reasoning becomes inconsistent, poorly structured, and disconnected from the task goal.

Figure 20: Comparison of Gemini’s and the $\mathcal{R}^{3}$ model’s behavior on the V task. While Gemini employs more diverse instructions, the $\mathcal{R}^{3}$ model primarily relies on absolute movement instructions.

Figure 21: Example of reasoning strategy learned via $\mathcal{R}^{3}$ . The model describes the three camera views, notes that the gelatin box is still in the medium tray, and issues a transfer instruction as the next step.

Figure 22: Example of reasoning strategy learned via $\mathcal{R}^{3}$ . The model infers that the gelatin box and foam brick are already in the small tray, so the remaining stage is to pack the sugar box into the medium tray.

Figure 23: Example of reasoning strategy learned via $\mathcal{R}^{3}$ . Our model first mislocates the foam brick, then re-examines the base view, revises the scene description, and issues the correct next instruction.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 846 papers

ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning
Apr 20, 2026Tuan Van Vo, Tan Q. Nguyen, Khang Nguyen +5Robotic Manipulation VLM Reasoning
VinRobotics, Hanoi. Vietnam · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE. · University of Texas at Arlington, Texas, USA +4
RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards
Jun 24, 2026Anıl Can Ateş, Orhan Kahraman, Cihan TopalRL for Robotics Long-Horizon Robotic Manipulation
Istanbul Technical University
Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation
Jun 2, 2026Nan Sun, Yuan Zhang, Yongkun Yang +10Long-Horizon Robotic Manipulation CoT Reasoning
Tsinghua University · Xiaomi Robotics · Peking University +6

Apr 20, 2026J/K move · Enter open · S save

ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning

Tuan Van Vo, Tan Q. Nguyen, Khang Nguyen, Nhat Xuan Tran+4

VinRobotics, Hanoi. Vietnam · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE. · University of Texas at Arlington, Texas, USA+4

Robotic Manipulation VLM Reasoning Vision-Language-Action Models Reasoning for VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have gained much attention from the research community thanks to their strength in translating multimodal observations with linguistic instructions into desired robotic actions. Despite their advancements, VLAs often overlook explicit reasoning and learn the functional input-action mappings, omitting crucial logical steps, which are especially pronounced in interpretability and generalization for complex, long-horizon manipulation tasks. In this work, we propose ReFineVLA, a multimodal reasoning-aware framework that fine-tunes VLAs with teacher-guided reasons. We first augment robotic datasets with reasoning rationales generated by an expert teacher model, guiding VLA models to learn to reason about their actions. Then, we fine-tune pre-trained VLAs with the reasoning-enriched datasets with ReFineVLA, while maintaining the underlying generalization abilities and boosting reasoning capabilities. We also conduct attention map visualization to analyze the alignment among visual observation, linguistic prompts, and to-be-executed actions of ReFineVLA, reflecting the model is ability to focus on relevant tasks and actions. Through this additional step, we explore that ReFineVLA-trained models exhibit a meaningful agreement between vision-language and action domains, highlighting the enhanced multimodal understanding and generalization. Evaluated across a suite of simulated manipulation benchmarks on SimplerEnv with both WidowX and Google Robot tasks, ReFineVLA achieves state-of-the-art performance, in success rate over the second-best method on the both the WidowX benchmark and Google Robot Tasks.

Abstract

VLM Reasoning

Explore similar work

ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning

RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards

Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation