Proprioceptive Sketches as Long-Horizon Intent for Generative Action Policies

Fangyuan Wang; Songhao Huang; Haoxiang Sun; Shipeng Lyu; Chengyang He; Anqing Duan; Peng Zhou; David Navarro-Alarcon

cs.ROOct 2, 2026

Proprioceptive Sketches as Long-Horizon Intent for Generative Action Policies

Authors: Fangyuan Wang, Songhao Huang, Haoxiang Sun, Shipeng Lyu, Chengyang He, Anqing Duan, Peng Zhou, David Navarro-Alarcon

Organizations: Hong Kong Polytechnic University · National University of Singapore · Mohamed bin Zayed University of Artificial Intelligence · Great Bay University

Abstract

Generative robot policies predict short action chunks but lack explicit long-horizon intent. Recent methods expose longer-horizon structure through language plans, subgoal images, or video forecasts, which are costly to generate and still need to be translated into robot motion. Predicting future robot motions avoids this translation, but a dense, time-indexed trajectory requires numerous parameters to cover the full remaining task, and over a short horizon it largely repeats the action chunk and adds little guidance for action generation. We propose Proprioceptive Action Models (PAM), which jointly generate a compact, timing-free sketch of the robot's remaining joint-space path and a dense executable action chunk within a single transformer denoiser. The sketch parameterizes the path by arc length rather than time, capturing geometric intent invariant to execution timing. Block-causal attention and a staggered denoising schedule maintain directed sketch-to-action dependence, ensuring the action tokens condition on a progressively cleaner sketch throughout sampling. In simulation, PAM improves over its action-only counterparts on Push-T and LIBERO-Long; on four real-world bimanual tasks, it raises success from 47.5% to 75.0%. Project page: https://nicehiro.github.io/pam_dp/

Figures & tables

Figure 1: PAM replans during a real-world dual-arm long-horizon task. At each step, PAM jointly generates a proprioceptive sketch $\mathbf{Z}_{t}$ of the future configuration-space path and a short action chunk $\mathbf{A}_{t}$ conditioned on the sketch, while only $\mathbf{A}_{t}$ is executed. For visualization, the sketch and the action chunk are mapped through forward kinematics and projected into the camera view.

Figure 2: The proprioceptive sketch. Left, B-spline coefficients

\mathbf{Z}_{t}

represent the remaining configuration path, while only the action chunk

\mathbf{A}_{t}

is executed. Right, re-timing the same continuous path changes its time-indexed representation but preserves its phase-indexed sketch.

Figure 3: Joint sketch and action generation. (a) Block-causal attention allows actions to read the sketch while preventing information flow in the reverse direction. (b) Training assigns independent noise levels to the sketch and action streams, with a shared level within each stream. (c) Staggered denoising advances the sketch ahead of the action chunk.

Figure 4: Overview of the simulation and real-world tasks. Push-T and LIBERO-Long in simulation, and four tasks on a real dual-arm platform shown at the initial state (top) and at the goal state (bottom). The Push-T goal is the green target pose; each LIBERO-Long task is specified by a language instruction (shown: put both moka pots on the stove ).

Table 5

Method	Metric	Handoff	Close lid	Fold cloth	Measure	Overall
Diffusion Policy	Success	9/20	9/20	10/20	10/20	47.5%
	Time (s)	36.2 $\pm$ 4.6	44.6 $\pm$ 9.2	41.9 $\pm$ 1.3	42.8 $\pm$ 3.6	41.4
B-spline Policy	Success	8/20	8/20	10/20	9/20	43.8%
	Time (s)	39.9 $\pm$ 7.2	42.2 $\pm$ 11.9	42.0 $\pm$ 1.2	40.6 $\pm$ 5.3	41.2
PAM-DP (ours)	Success	13/20	14/20	18/20	15/20	75.0%
	Time (s)	35.8 $\pm$ 8.0	45.3 $\pm$ 11.1	42.4 $\pm$ 2.4	45.2 $\pm$ 6.4	42.3

Table 3: Real-world task performance. Success is reported as completed episodes out of 20 trials per task. Completion time is measured in seconds over successful episodes.

Figure 5: Policy predictions on the four real-world tasks, projected into the external camera view. For B-spline Policy, orange squares and curves show the control points and the action spline.

Figure 6: Grasp retries on fold cloth . Each column shows an episode in which the first grasp missed. The large image shows the scene after the second grasp, with a translucent overlay marking the gripper at the top of its lift and an arrow indicating the lift and re-descent.

Method	Total	Miss	Slip	Coord.
Diffusion Policy	42	22	12	8
B-spline Policy	45	28	15	2
PAM-DP (ours)	20	10	9	1

Table 4: Failed episodes across 80 real-world trials per method.

Variant	Coverage
PAM-DP, zero lead	$\mathbf{0.79}\pm 0.02$
Action-only backbone	0.69 $\pm$ 0.07
Auxiliary sketch only	0.75 $\pm$ 0.04
Time-indexed sketch	0.74 $\pm$ 0.04
Bidirectional attention	0.72 $\pm$ 0.06
Shared-time training	0.74 $\pm$ 0.07

Table 5: Component ablations on image-based Push-T.

Lead $\eta$	Offset	NFE	Coverage	Delay (ms)
0	0	16	$0.779\pm 0.015$	153.0
1/16	1	17	$0.801\pm 0.018$	155.2
1/8	2	18	$0.805\pm 0.011$	158.1
1/4	4	20	$0.785\pm 0.029$	193.1
1/2	8	24	$0.788\pm 0.023$	230.0
1	16	32	$\mathbf{0.809}\pm 0.027$	280.0

Table 6: Sketch lead ablation on Push-T with a 16-step DDIM base budget.

Figure 7: Offline sketch intervention. (a) Alternative sketch routes. (b) Predicted left-gripper motion. (c) Action sensitivity to sketch and noise changes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Post-hoc sketch capacity audit on training anchors. Reconstruction RMSE is normalized by the task-specific per-step motion reference and shown on a logarithmic vertical axis. The dashed line marks a ratio of one; filled markers indicate the capacities used in the policy experiments. The reference values are

8.04

pixels for Push-T and

0.870

0.700

0.765

0.660

, and

0.665

degrees for handoff, close lid, fold cloth, measure, and LIBERO-Long, respectively.

Setting	Push-T image	Real-world
Decoder layers / heads / width	8 / 4 / 256	8 / 4 / 256
Conditioning	adaLN-zero	adaLN-zero
Attention dropout	0.01	0
Observation / prediction / execution steps	2 / 16 / 8	2 / 16 / 8
Image crop	$84\times 84$	$84\times 84$
Batch size	64	64

Appendix

Table 7: PAM-DP training settings. Real-world settings are shared by the four dual-arm tasks. Both configurations use EMA and a 100-step cosine diffusion schedule with noise prediction.

Setting	PAM-VLA
Vision-language backbone	Florence-2-large
Decoder layers / heads / width	18 / 16 / 1024
Positional encoding	RoPE
Attention / residual / MLP dropout	0.1 / 0.1 / 0.1
Observation / prediction / execution steps	1 / 10 / 10
Batch size	16

Appendix

Table 8: PAM-VLA configuration on LIBERO-Long. The sketch has

M=16

control points and predicts 15 coefficient vectors after fixing the first to zero.

Task	Available	Training	Validation
LIBERO-Long	500	500	0
Handoff	81	73	8
Close lid	81	73	8
Fold cloth	48	41	7
Measure	89	80	9

Appendix

Table 9: Demonstration counts and seed-42 splits. Push-T is not listed and follows the split of Chi et al. (2023) . LIBERO has no demonstration-level validation split.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 357 papers

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps
Apr 13, 2026Liaoyuan Fan, Zetian Xu, Chen Cao +3World Action Models Robotic Manipulation
INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China · The University of Hong Kong, Hong Kong SAR, China · Shanghai Jiao Tong University, Shanghai, China
PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation
Feb 5, 2026Yushan Liu, Jingjing Fan, Shoujie Li +3World Models for Robotics Visuomotor Policy Learning
Tsinghua University · Nanyang Technological University · Xspark AI
An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM
Sep 22, 2026Tianheng Wang, Zhou Xie, Heng Jia +3World Models for Robotics Visuomotor Policy Learning
Westlake University · Lanzhou University · Zhejiang University +2

Apr 13, 2026J/K move · Enter open · S save

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps

Liaoyuan Fan, Zetian Xu, Chen Cao, Wenyao Zhang+2

INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China · The University of Hong Kong, Hong Kong SAR, China · Shanghai Jiao Tong University, Shanghai, China

World Action Models Robotic Manipulation Long-Horizon Robotic Manipulation Robotic RL

Open paper AskPDF arXiv

Pretrained video generation models provide strong priors for robot control, but existing unified world action models still struggle to decode reliable actions without substantial robot-specific training. We attribute this limitation to a structural mismatch: while video models capture how scenes evolve, action generation requires explicit reasoning about where to interact and the underlying manipulation intent. We introduce AIM, an intent-aware unified world action model that bridges this gap via an explicit spatial interface. Instead of decoding actions directly from future visual representations, AIM predicts an aligned spatial value map that encodes task-relevant interaction structure, enabling a control-oriented abstraction of future dynamics. Built on a pretrained video generation model, AIM jointly models future observations and value maps within a shared mixture-of-transformers architecture. It employs intent-causal attention to route future information to the action branch exclusively through the value representation. We further propose a self-distillation reinforcement learning stage that freezes the video and value branches and optimizes only the action head using dense rewards derived from projected value-map responses together with sparse task-level signals. To support training and evaluation, we construct a simulation dataset of 30K manipulation trajectories with synchronized multi-view observations, actions, and value-map annotations. Experiments on RoboTwin 2.0 benchmark show that AIM achieves a 94.0% average success rate, significantly outperforming prior unified world action baselines. Notably, the improvement is more pronounced in long-horizon and contact-sensitive manipulation tasks, demonstrating the effectiveness of explicit spatial-intent modeling as a bridge between visual world modeling and robot control.

Abstract

Robot Policy Learning

Explore similar work

AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps

PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation

An Action Is Worth One Patch: Unified World-Action Modeling with PatchWAM