Physical Words: A Vocabulary for Structuring Latent Reasoning in Robot Manipulation

Jiaming Liu; Xinghao Zhu; Yiming Zhang; Shanghang Zhang; Yinxi Wang; Yichu Yang; Zhuoyang Liu; Dantong Niu

cs.ROOct 19, 2026

Physical Words: A Vocabulary for Structuring Latent Reasoning in Robot Manipulation

Authors: Jiaming Liu, Xinghao Zhu, Yiming Zhang, Shanghang Zhang, Yinxi Wang, Yichu Yang, Zhuoyang Liu, Dantong Niu,

Organizations: NVIDIA · Peking University · UC Berkeley

Abstract

Multimodal reasoning provides robotic foundation models with predictive context needed for reliable action generation. However, language-based reasoning captures task semantics but struggles to express fine-grained physical dynamics, while latent reasoning models future state evolution but offers limited structure for organizing these dynamics into reusable reasoning primitives. To bridge this gap, we introduce PhysWord, a reasoning-before-acting foundation model built on Physical Words, which use atomic action verbs as shared semantic anchors for continuous representations of physical dynamics. Specifically, PhysWord predicts upcoming atomic action words and directly aligns their contextual hidden states with latent targets derived from the corresponding future observations. Building on these primitives, we construct a temporally structured PhysWord space with visual and optional tactile variants of each action word. Autoregressively predicting Physical Words across future timesteps captures transitions between execution stages, providing structured multimodal context for action generation. PhysWord achieves state-of-the-art success rates of 90.3% on visuotactile UniVTAC and 99.6% on vision-based LIBERO-Long, and outperforms competing methods on real-world tasks using bimanual grippers and dexterous hands. Further evaluations demonstrate that our structured latent reasoning framework generalizes to unseen scenarios and enables long-horizon manipulation.

Figures & tables

Figure 2: PhysWord framework. (a) PhysWord connects temporal perception and reasoning-and-action branches through shared attention in a MoT architecture, generating Physical Words autoregressively before denoising actions. (b) The hidden states of visual and tactile Physical Word variants receive joint action-label cross-entropy supervision and cosine alignment with future sensory latent targets. (c) Physical Words use a blockwise-causal attention mask, with full attention within each end-effector’s timestep block, causal attention across its blocks, and no cross-effector attention.

UniVTAC (Visuotactile simulator)										LIBERO-Long
Method	Lift Bottle	Insert Tube	Insert Hole	Grasp Classify	Pull Out Key	Put Bottle	Insert HDMI	Lift Can	Mean S.R. $\uparrow$	Method	S.R. $\uparrow$
ACT Tac	71	56	24	99	46	31	28	29	48.0	$\pi_{0.5}$	92.4
VITaL	72	34	25	100	47	32	6	8	40.5	OpenVLA-OFT	94.5
$\pi_{0.5}$	100	74	25	49	35	34	8	72	49.6	GR00T-N1.7	94.4
StarVLA- $\alpha$	32	69	52	68	51	88	24	65	56.1	LaST 0	95.6
InternVLA-A1	58	95	93	86	66	37	12	90	67.1	Cosmos Policy	97.6

Table 1: Comparison on UniVTAC (left) and LIBERO-Long (right) in terms of success rates (%). Bold indicates the best performance for each task.

Figure 3: Ablation study. Left: Physical Word structure ablations on two UniVTAC tasks and LIBERO-Long. Right: effect of the prediction interval in the temporally structured PhysWord space.

Figure 4: Qualitative results. (a) Real-world task execution with PhysWord. (b) Latent-to-image attention maps across simulated and real-world tasks.

Method	Pick Paper	Shuttlecock Insert	Express Packing	Screw Drive	Mean $\pm$ Avg. SD
$\pi_{0.5}$	53.3	43.3	46.7	40.0	45.8 $\pm$ 3.6
FTP-1	66.7	56.7	50.0	46.7	55.0 $\pm$ 4.6
PhysWord	93.3	60.0	73.3	60.0	71.7 $\pm$ 2.5

Table 2: Real-world evaluation. Success rates (%) on four real-world manipulation tasks and on Pick Paper and Screw Drive with unseen backgrounds (Bac), objects (Obj), and positions (Pos). Drop denotes the relative change from the in-domain mean for the same two tasks.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Duration (h)	Tactile
Human hand data	2,000+	No
DROID	350	No
OpenNeoData	5,000+	Yes
Sharpa data	100+	Yes
Total	7,400+	–

Appendix

Table 3: Pretraining data composition and volume.

Figure 5: Real-world robotic platforms and sensing. Representative hardware views of (a) the bimanual platform with Sharpa Wave dexterous hands and (b) the bimanual platform with Xense tactile grippers. Blue and green annotations identify head and wrist cameras, respectively; red insets highlight the end effectors.

Joint	i7 Pro left	i7 Pro right	Tianji
J1	$[-3.14,\,3.14]$	$[-3.14,\,3.14]$	$[-3.1067,\,3.1067]$
J2	$[-0.6977,\,3.8377]$	$[-3.8377,\,0.6977]$	$[-2.0944,\,2.0944]$
J3	$[-3.14,\,3.14]$	$[-3.14,\,3.14]$	$[-3.1067,\,3.1067]$
J4	$[-2.4248,\,2.4248]$	$[-2.4248,\,2.4248]$	$[-2.5307,\,1.0472]$
J5	$[-3.14,\,3.14]$	$[-3.14,\,3.14]$	$[-3.1067,\,3.1067]$
J6	$[-1.0467,\,1.0467]$	$[-1.0467,\,1.0467]$	$[-1.0472,\,1.0472]$

Appendix

Table 4: Arm joint position limits in radians, excluding end-effector joints.

Method	Lift Bottle	Insert Tube	Insert Hole	Grasp Classify	Pull Out Key	Put Bottle	Insert HDMI	Lift Can	Mean S.R. $\uparrow$
ACT Tac	71	56	24	99	46	31	28	29	48.0
VITaL	72	34	25	100	47	32	6	8	40.5
$\pi_{0.5}$	100	74	25	49	35	34	8	72	49.6
StarVLA- $\alpha$	32	69	52	68	51	88	24	65	56.1
InternVLA-A1	58	95	93	86	66	37	12	90	67.1
FTP-1	97	79	64	99	48	47	4	65	62.9

Appendix

Table 5: Additional results on UniVTAC, reporting task success rates (%). PhysWord-Lite removes the temporal perception branch. PhysWord-feedback replaces online tactile measurements with predicted tactile latents during evaluation. The SD row is the average of the per-task SDs.

Variant	Put Bottle	Lift Can
Full PhysWord	93	95
w/o tactile input	88	87
w/o tactile prediction	87	89
w/o both	82	81

Appendix

Table 6: Ablation of tactile input and prediction, reporting task success rates (%).

Target view	Put Bottle	Lift Can
Head view	93	95
Wrist views	85	83
Head and wrist views	84	89

Appendix

Table 7: Effect of visual latent target views on success rates (%), with policy inputs unchanged.

Physical Word design	Put Bottle	Lift Can
Shared: pick , pick	90	89
Separate: pick_v , pick_t	93	95

Appendix

Table 8: Shared versus modality-specific Physical Words. Entries report task success rates (%).

Method	Pick A4 Paper			Express Packing
Method	Original	Extended	Extended (w/o PW)	Original	Extended	Extended (w/o PW)
$\pi_{0.5}$	53.3	0.0	–	46.7	26.7	–
FTP-1	66.7	0.0	–	50.0	30.0	–
PhysWord	93.3	70.0	36.7	73.3	66.7	50.0

Appendix

Table 9: Success rates (%) on the original tasks and their extended long-horizon variants. For PhysWord, the Original and Extended columns both report results for the full model.

Method	Online tactile sensing	Pick A4 Paper	Shuttlecock Insertion
$\pi_{0.5}$	No	53.3	43.3
FTP-1	Yes	66.7	56.7
PhysWord	Yes	93.3	60.0
PhysWord w/ tactile feedback	No	90.0	56.7

Appendix

Table 10: Predictive tactile feedback on real-world dexterous tasks. Entries report success rates (%). The two PhysWord variants use the same trained checkpoint.

Action	Visual		Tactile
Action	Latent only ( $O$ )	Physical Words ( $WO$ )	Latent only ( $T$ )	Physical Words ( $WT$ )
Pick	0.0049	0.0030	0.0003	0.0001
Place	0.0050	0.0024	0.0002	0.0001
Approach	0.0044	0.0026	0.0003	0.0002
Move	0.0062	0.0041	0.0002	0.0001

Appendix

Table 11: Per-action intra-class distances (

\downarrow

) for visual and tactile representations. Physical Words yield lower distances than latent-only supervision across all evaluated actions.

Figure 6: t-SNE visualization of contextual hidden states. We compare Physical Words with latent-only supervision for visual (top) and tactile (bottom) representations. Colors denote atomic actions: pick , place , and approach , with 50 samples per action.

Figure 7: Failure case. Representative execution failures during contact-rich manipulation.

Figure 8: Attention mask for the reasoning-and-action branch. Rows denote queries and columns denote keys. Colored cells indicate allowed attention, while white cells indicate masked connections.

Row	Implementation	Median (ms)	P95 (ms)	Freq. (Hz)
	Baseline	205.4	216.5	4.9
1	+ Full CUDA Graph capture	163.2	164.0	6.1
2	+ Mask-preserving FlashAttention	145.5	146.3	6.9
3	+ GPU RGB preprocessing	131.7	132.0	7.6
4	+ Fused RMSNorm (Qwen)	116.5	119.0	8.6
5	+ Packed projections and fused RoPE	108.3	108.7	9.2

Appendix

Table 12: End-to-end inference latency on a single NVIDIA H100 GPU as runtime optimizations are applied cumulatively. Frequency is computed from median latency.

Figure 9: Real-world generalization. Examples of seen configurations and unseen variations in backgrounds, objects, and positions. Blue borders indicate configurations within the training distribution, while red borders indicate unseen configurations.

Robot Foundation Models

Robot Foundation Models are pretrained systems that combine perception, reasoning, and action to support diverse tasks and robot embodiments.

See all 128 papers

LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning
Apr 30, 2026Hao Chen, Jiaming Liu, Zhonghao Yan +10RL for Robotics Robot Foundation Models
1The Chinese University of Hong Kong · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Simplexity Robotics
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
Feb 18, 2026Zijian Song, Qichang Li, Sihan Qin +4Robotic Manipulation World Models for Robotics
Sun Yat-sen University · X-Era AI Lab · Guangdong University of Technology +1
PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models
Sep 14, 2026DeepCybo Team, Yu Bin, Haipeng Cao +51Robot Motion Generation Physical Foundation Models

Apr 30, 2026J/K move · Enter open · S save

LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning

Hao Chen, Jiaming Liu, Zhonghao Yan, Nuowei Han+9

1The Chinese University of Hong Kong · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Simplexity Robotics

RL for Robotics Robot Foundation Models Robotic Manipulation RL for VLMs

Open paper AskPDF arXiv

Robotic foundation models require reasoning over complex visual scenes to execute adaptive actions in dynamic environments. While recent studies on latent-reasoning Vision-Language-Action (VLA) models have demonstrated the capability to capture fine-grained physical dynamics, they remain predominantly confined to static imitation learning, severely limiting their adaptability and generalization. In this paper, we present LaST-R1, a novel reinforcement learning (RL) post-training framework designed to effectively harness "latent reasoning-before-acting" policies. Specifically, we propose Latent-to-Action Policy Optimization (LAPO), a core RL algorithm that jointly optimizes the latent reasoning process and the action generation. By explicitly embedding latent Chain-of-Thought (CoT) reasoning directly within the RL optimization loop, LAPO stimulates profound physical world modeling, which in turn drives robust execution in interactive environments. Furthermore, an adaptive latent CoT mechanism is introduced, allowing the policy to dynamically modulate its reasoning horizon based on diverse environment states. Experiments show that LaST-R1 achieves a near-perfect 99.9% average success rate on the LIBERO benchmark with only one-shot supervised warm-up, significantly improving convergence speed and performance over prior state-of-the-art (SOTA) methods. In real-world deployments, LaST-R1 yields up to a 22.5% average improvement over SOTA supervised fine-tuning approach across four complex tasks, including both single-arm and dual-arm settings. Finally, LaST-R1 demonstrates strong generalization across simulated and real-world environments.

Abstract

Robot Foundation Models

Explore similar work

LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning

Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation

PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models