Reasoning Without Inference Cost: Latent Semantic Scaffolding for Robot VLA Policies

Andrew Ting Yan Li; Zhuo Li; Zhelin Yang; Zhipeng Dong; Quentin Rouxel; Fei Chen

cs.ROSep 4, 2026

Reasoning Without Inference Cost: Latent Semantic Scaffolding for Robot VLA Policies

Authors: Andrew Ting Yan Li, Zhuo Li, Zhelin Yang, Zhipeng Dong, Quentin Rouxel, Fei Chen

Organizations: Collaborative and Versatile Robots (CLOVER) Laboratory, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong

Abstract

Vision-language-action (VLA) models are trained by imitation and capture what action to take but not why; adding causal reasoning improves manipulation, but current methods pay for it at inference time - generating reasoning tokens or rolling out predicted future states at every step, a cost that compounds over long horizons. We ask whether this benefit can instead be captured during training and discarded before deployment. We introduce Latent Semantic Scaffolding (LSS), an auxiliary loss applied during human-demonstration pretraining that aligns a VLA's action-token representations to text embeddings of physical-reasoning rationales through a small projection head. The head is dropped at inference, leaving the unmodified base policy with zero added cost. Our central finding concerns alignment granularity: aligning each action token to the rationale of its own manipulation phase (Dense LSS) rather than to a single pooled episode-level embedding (Pooled LSS) yields representations that transfer markedly better to held-out tasks. Dense LSS attains both the best in-distribution success and the best transfer to tasks unseen during alignment, whereas pooled alignment over-specializes to the training task. A representational probe shows Dense LSS induces roughly twice the per-phase separability in the backbone, supporting that phase-local alignment is the operative mechanism.

Figures & tables

Fig. 2 : Dense LSS architecture (Stage 2, training only). A frozen DINO+SigLIP encoder and a frozen T5-XXL command encoder condition the 2B diffusion-transformer backbone, which produces $K{=}16$ action tokens $Z_{t}\in\mathbb{R}^{2176}$ supervised by the flow-matching loss $L_{\text{flow}}$ . In parallel, the LSAHead projection $g_{\phi}$ maps each action token into T5 space and a cosine-distance loss aligns it to the reasoning embedding of its own manipulation phase, $e_{\text{phase}(t)}$ (straddling tokens masked by $m_{t}$ ). The combined objective is $L_{\text{total}}=L_{\text{flow}}+\lambda L_{\text{LSS}}$ . Pooled LSS uses the identical architecture but replaces the per-phase targets with a single mean-pooled episode embedding (( 1 )). The entire LSS apparatus (orange) is discarded at inference, leaving the unmodified base policy.

Fig. 3 : Dense LSS token-to-phase alignment. Each of the 16 action tokens $Z_{t}$ is assigned to the manipulation phase spanned by its frames (tokens straddling a boundary, e.g. the greyed slot, are masked by $m_{t}$ ); color indicates the assigned phase (orange = grip, green = rotate; two representative phases shown). The per-phase reasoning text is encoded by a frozen T5 and mean-pooled into a target $e_{\text{phase}(t)}$ . The head $g_{\phi}$ projects each token into T5 space and $L_{\text{dense}}$ aligns it to the target of its own phase.

Panel of Figure 3 — Fig. 4 : Human demonstration data (top) and simulation evaluation environment (bottom).

Run	Mechanism	AVP	Reas.	Succ. (%)
A	EgoDex pretrain + finetune (baseline)	–	–	70
B	AVP kinematics, no reasoning	✓	–	79
C	Reasoning as language input	✓	✓	71
D	Reasoning via token-prediction loss	✓	✓	74
E	Pooled LSS (proposed)	✓	✓	85
–	LSS to trivial instruction text	✓	–	58

TABLE I : Ablation ladder on adjust_bottle (RoboTwin 2.0, demo_randomized , 100 rollouts). The 58%

\to

85% gap (trivial target vs. Run E) shows the alignment target matters, not merely the presence of an auxiliary loss.

Held-out task	R1	R2	R3	R4
adjust_bottle (in-dist.)	70	79	85	90
shake_bottle	34	45	39	54
move_can_pot	9.1	20	19	26

TABLE II : Transfer to held-out tasks (success rate %, 100 rollouts). Columns are Stage-2 backbones: R1 = H-RDT baseline, R2 = +human data (no LSS), R3 = +Pooled LSS, R4 = +Dense LSS. All share an identical 50-episode / 22,000-step Stage-3 finetune, so the only difference is the Stage-2 alignment objective. Dense LSS gives the best result on every task; Pooled LSS helps the in-distribution task but does not transfer.

Fig. 5 : Per-phase separability of the LSS-induced representational change on adjust_bottle . Under the same small alignment budget ( $\lambda{=}0.1$ ), Dense LSS yields ${\sim}2\times$ the silhouette of Pooled LSS; absolute values are small by design. Supporting evidence for phase-local shaping.

Reasoning for VLA Models

Reasoning for vision-language-action (VLA) models connects visual and language understanding to grounded planning and action.

See all 99 papers

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2ICL for Robotics Continual Learning for VLA Models
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)
Semantic Anchoring for Robotic Action Representations
Jul 15, 2026Yuan Xu, Youheng Shi, Chengyang Li +2Generalization in Robotic Manipulation Vision-Language-Action Models
Peking University · Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University
ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning
Apr 20, 2026Tuan Van Vo, Tan Q. Nguyen, Khang Nguyen +5Robotic Manipulation VLM Reasoning
VinRobotics, Hanoi. Vietnam · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE. · University of Texas at Arlington, Texas, USA +4

Aug 6, 2026J/K move · Enter open · S save

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

Jiarui Yang, Wen Huang, Jiale Zhang, Maowei Hu+1

Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)

ICL for Robotics Continual Learning for VLA Models Robotic Manipulation Vision-Language-Action Models VLMs for Robotics Reasoning for VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have become the dominant recipe for generalist manipulation, yet they are almost universally trained by behavior cloning: a policy imitates expert action chunks conditioned on a static image and a fixed instruction. A natural remedy is to inject explicit reasoning through textual chain-of-thought (CoT). We show, both empirically and analytically, that free-form textual CoT degrades low-level control: the reasoning it produces is ungrounded, its latency breaks closed-loop timing, and, crucially, the reasoning and action tokens are optimized against conflicting objectives so that the policy learns to narrate rather than to act. We argue that what a VLA needs is not the ability to generate language, but the ability to consume grounded language. To this end we introduce \ourmethod{}, a framework that endows a VLA with language competence through (i) in-context post-training, in which perceptual evidence is injected as structured context and the model is supervised only on actions, and (ii) an agentic tool-use interface, in which the policy queries open-vocabulary detectors, monocular depth, and a vision--language model to actively acquire task-relevant information. Rather than emitting a single templated caption, our data engine produces diverse, paraphrased, and evidence-conditioned spatial descriptions, so that the policy learns to interpret language it has never seen verbatim. Across the RoboCasa-GR1, SimplerEnv, and LIBERO simulation benchmarks, together with 8 real-world robot manipulation tasks, our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches under matched configurations.

Abstract

Reasoning for VLA Models

Explore similar work

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

Semantic Anchoring for Robotic Action Representations

ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning