Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models

Min Lin; Bokai Zhou; Jianqi Lin; Kaiyuan Yang; Yicheng Chang; Junjie Yu; Xiaofan Liu; Jianhua Han

cs.ROOct 9, 2026

Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models

Authors: Min Lin, Bokai Zhou, Jianqi Lin, Kaiyuan Yang, Yicheng Chang, Junjie Yu, Xiaofan Liu, Jianhua Han, +3 more

Organizations: Sun Yat-sen University · Yinwang Intelligent Technology Co., Ltd.

Abstract

Actions specify what a robot should execute, but do not describe what those commands will achieve. Vision-language-action (VLA) policies nevertheless learn primarily by predicting action chunks from images, language, and state. We introduce Action--Future Coupled Effect Learning (AFCE), which makes a future-grounded physical Effect the policy's primary prediction target. An offline tokenizer constructs temporally ordered Effect tokens from demonstrated actions, realized robot responses, and sparse visual transitions. Reconstruction objectives encourage the representation to retain executable detail and observed physical change. A flow-matching VLA predicts Effects from current images, instruction, and robot state, while a state-conditioned decoder converts them into action chunks. Embodiment-specific interfaces allow single- and dual-arm tasks to share the Effect model. On DexJoCo, AFCE achieves $50.97\%$ overall success, improving over $π_{0.5}$ and UVT by 11.82 and 15.21 percentage points, respectively. It leads on all five bimanual tasks, highlighting the value of Effect supervision for coordinated dexterous manipulation. The project page is available at https://afceproject.github.io/AFCE-Web/.

Figures & tables

Figure 1: Comparison of policy supervision paradigms. A standard VLA predicts actions directly; world-action models use future dynamics to support control; AFCE predicts an action-decodable Effect grounded in executed actions, robot response, and world change. Future evidence constructs training targets but is unavailable to the deployed policy.

Figure 2: Overview of Action–Future Coupled Effect Learning (AFCE). Stage I fuses demonstrated actions $\mathbf{A}_{t}$ , robot response $\Delta\mathbf{S}_{t}$ , and visual transitions $\mathbf{V}_{t}$ into ordered Effect tokens $\mathbf{E}_{t}^{*}$ , supervised by action, state, and visual reconstruction. Stage II predicts the normalized target $\mathbf{Z}_{t}^{*}$ from current images, instruction, and robot state, then decodes an action chunk. The lower panel shows the shared Effect space with embodiment-specific interfaces. The dashed path denotes training targets; future evidence is not used at deployment.

Figure 3: Action and scene changes represented by the Effect. Three successive hammer-nail frames. Top: decoded robot motion; bottom: accompanying scene changes.

	Single-arm						Bimanual					Mean
Method	PT	CM	FG	HN	PB	WP	UI	HA	AS	MW	PH	SA	BA	All
UVT	$1.33$ [1pt] $\pm 1.15$	$\mathbf{80.00}$ [1pt] $\pm 2.00$	$\mathbf{64.67}$ [1pt] $\pm 5.03$	$22.00$ [1pt] $\pm 6.93$	$60.00$ [1pt] $\pm 6.00$	$\mathbf{92.67}$ [1pt] $\pm 2.31$	$0.00$ [1pt] $\pm 0.00$	$0.67$ [1pt] $\pm 1.15$	$0.00$ [1pt] $\pm 0.00$	$41.33$ [1pt] $\pm 11.02$	$30.67$ [1pt] $\pm 4.16$	$53.44$ [1pt] $\pm 2.22$	$14.53$ [1pt] $\pm 3.06$	$35.76$ [1pt] $\pm 0.56$
$\pi_{0.5}$	$\mathbf{43.33}$ [1pt] $\pm 8.08$	$60.00$ [1pt] $\pm 2.00$	$38.00$ [1pt] $\pm 5.29$	$84.67$ [1pt] $\pm 5.77$	$67.33$ [1pt] $\pm 1.15$	$88.67$ [1pt] $\pm 1.15$	$2.00$ [1pt] $\pm 0.00$	$5.33$ [1pt] $\pm 4.16$	$1.33$ [1pt] $\pm 2.31$	$32.00$ [1pt] $\pm 8.72$	$8.00$ [1pt] $\pm 3.46$	$\mathbf{63.67}$ [1pt] $\pm 1.86$	$9.73$ [1pt] $\pm 2.41$	$39.15$ [1pt] $\pm 2.11$
Ours	$5.33$ [1pt] $\pm 2.31$	$54.67$ [1pt] $\pm 7.57$	$62.00$ [1pt] $\pm 6.93$	$\mathbf{86.67}$ [1pt] $\pm 5.77$	$\mathbf{80.67}$ [1pt] $\pm 9.02$	$84.67$ [1pt] $\pm 9.02$	$\mathbf{7.33}$ [1pt] $\pm 1.15$	$\mathbf{29.33}$ [1pt] $\pm 1.15$	$\mathbf{9.33}$ [1pt] $\pm 3.06$	$\mathbf{81.33}$ [1pt] $\pm 6.43$	$\mathbf{59.33}$ [1pt] $\pm 4.16$	$62.33$ [1pt] $\pm 1.45$	$\mathbf{37.33}$ [1pt] $\pm 2.66$	$\mathbf{50.97}$ [1pt] $\pm 1.65$

Table 1: Success rates (%) on DexJoCo under rand-obj .

Figure 4: Failure rates by task family for evaluation seed 0. Arrows show changes from

\pi_{0.5}

to Ours in percentage points.

	Error (mm) $\downarrow$			GT-Effect replay $\uparrow$
$d$	Pos.	Bi-pos.	FK	SA	BA	All
256	7.794	9.507	10.519	20/60	5/50	25/110
128	8.965	11.455	11.716	21/60	2/50	23/110
64	11.585	15.272	14.980	18/60	0/50	18/110

Table 2: Effect dimensionality sensitivity.

Variant	Effect inputs			Single-arm
	$S$	$V$	$A$	PT	CM	FG	HN	PB	WP	SA
$A$ only	–	–	$\checkmark$	$1.33$ [1pt] $\pm 1.15$	$65.33$ [1pt] $\pm 3.06$	$54.67$ [1pt] $\pm 10.07$	$68.00$ [1pt] $\pm 2.00$	$69.33$ [1pt] $\pm 6.11$	$84.67$ [1pt] $\pm 2.31$	$57.22$ [1pt] $\pm 0.38$
w/o $S$	–	$\checkmark$	$\checkmark$	$5.33$ [1pt] $\pm 5.03$	$48.00$ [1pt] $\pm 5.29$	$51.33$ [1pt] $\pm 4.62$	$76.00$ [1pt] $\pm 2.00$	$84.67$ [1pt] $\pm 7.02$	$26.67$ [1pt] $\pm 5.03$	$48.67$ [1pt] $\pm 1.53$
w/o $V$	$\checkmark$	–	$\checkmark$	$14.00$ [1pt] $\pm 5.29$	$57.33$ [1pt] $\pm 3.06$	$54.67$ [1pt] $\pm 6.43$	$78.67$ [1pt] $\pm 8.08$	$76.67$ [1pt] $\pm 9.02$	$52.67$ [1pt] $\pm 12.06$	$55.67$ [1pt] $\pm 3.06$
AFCE	$\checkmark$	$\checkmark$	$\checkmark$	$5.33$ [1pt] $\pm 2.31$	$54.67$ [1pt] $\pm 7.57$	$62.00$ [1pt] $\pm 6.93$	$86.67$ [1pt] $\pm 5.77$	$80.67$ [1pt] $\pm 9.02$	$84.67$ [1pt] $\pm 9.02$	$\mathbf{62.33}$ [1pt] $\pm 1.45$

Table 3: Effect-construction input ablation on DexJoCo under rand-obj . Success rates are reported in %.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Objective
Single arm
Click Mouse	Move the mouse onto its pad and click.
Fold Glasses	Fold the glasses and place them in the case.
Hammer Nail	Drive a nail into a board with a hammer.
Pick Bucket	Place food in the bucket and lift it.
Pinch Tongs	Grasp the tongs and open and close them three times.

Appendix

Table B.1: Functional goals of the 11 DexJoCo tasks.

Figure C.1: Structure and prediction error of the effect space. (a) GT-E by task; (b) the same GT-E windows by relative trajectory progress; (c) GT-E and predicted effects in a shared embedding; (d) paired normalized effect RMSE in the original space. PCA uses 50 components (87.27% explained variance); the shared t-SNE uses perplexity 30 and seed 42.

Figure C.2: Sensitivity of the two-dimensional projection. GT-E windows after the same normalization and PCA are projected with t-SNE at perplexity/seed 30/42, 15/42, and 50/7. Task colors match Figure C.1 .

Figure C.3: Physical action-readout error on matched demonstration windows. GT-E/

D_{0}

, GT-E/

D_{1}

, and predicted-E/

D_{1}

are compared on mean Euclidean position error per hand, rotation geodesic error, and mean absolute finger joint-angle error. Means aggregate ten windows per trajectory over 110 trajectories; intervals are 95% within-task episode-bootstrap intervals (3,000 resamples).

D_{0}

is used only for this diagnostic; the deployed policy uses

D_{1}

Panel of Figure 12 — Figure C.4: Illustrative AFCE and $\pi_{0.5}$ rollouts on Fold Glasses, Hanoi, Microwave, and Photograph (top to bottom). Each task is shown at approximately 15%, 40%, 70%, and 95% of the rollout. The selected AFCE rollouts complete their task-specific interaction, whereas the corresponding $\pi_{0.5}$ rollouts fail at contact, placement, or final actuation.

Hyperparameter	Value
Stage I: Effect tokenizer
Horizon / feature widths	$H=30$ ; branch width 64; Effect width 256
Fusion / temporal / decoder layers	$2/2/4$ ; 8 heads; FFN width 1024; dropout 0
Visual sampling	Stride 5 plus terminal frame; at most 6 dynamic and 2 reference regions, with 8 slots per region
Visual descriptor / PCA	136 dimensions / 64 components; frozen DINOv3; legacy-none mask
Action reconstruction	Grouped Huber; position weight $w_{p}=2$

Appendix

Table D.1: Hyperparameters of the evaluated AFCE configuration.

Explore similar work

Oct 3, 2026

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.

Zhe Tao, Feiran Wang, Gaowen Liu +2

University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco

World Models for Robotics Vision-Language-Action Models Language-Conditioned Robot Manipulation 3D Scene Understanding

Ask PDF

Sep 30, 2026

MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies

Vision-Language-Action (VLA) models have recently incorporated world models to provide richer dynamic supervision beyond sparse action labels. However, explicitly predicting future images or videos may include control-irrelevant appearance, while guidance derived from holistic future visual representations and shared global action features may fail to establish timestep-specific correspondence between actions and local visual changes. To address this issue, we propose MotionWeave, a motion-centric future-dynamics framework for action-chunk prediction with two modules: the Action-Induced Motion Grounder (AIMG) and the Horizon Residual Composer (HRC). Specifically, AIMG conditions on action and proprioceptive representations to construct horizon-specific queries that localize interaction regions associated with each future action timestep from current visual tokens. HRC extracts differences between interaction representations at adjacent horizons, encodes them as temporal motion cues, and injects them into action tokens through a gated residual. During training, robot-arm masks rendered from future frames are used to construct KL-based motion-grounding supervision, while inference uses only the current observation. On six MetaWorld tasks, MotionWeave achieves a 75.3% average success rate, an absolute gain of 8.6% over π0 (66.7%), especially on sustained-interaction tasks. Our code is available at https://github.com/autu-mn/MotionWeave.

Jingqiu Wang, Yan Wang

School of Data Science and Engineering East China Normal University, Shanghai, China

Visuomotor Policy Learning Robotic Manipulation Vision-Language-Action Models Motion Representation Learning

Ask PDF

Aug 27, 2026

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

Vision-language-action models (VLAs) have shown strong promise for general-purpose robotic manipulation by mapping language instructions and vision observations directly to actions. However, most VLAs primarily condition action prediction on current observations and lack an explicit mechanism for reasoning over future task dynamics, which is particularly important for fine-grained, contact-rich manipulation. We present PHR-VLA, a framework that enables planning-horizon reasoning in VLAs through privileged latent representations of future dynamics. PHR-VLA introduces a lightweight auxiliary future head that, during training, aligns the VLA's internal representations with latent dynamics extracted from future observations. Evaluation results demonstrate that local, contact-centric, patch-level latent dynamics supervision from the wrist camera improves success rate on LIBERO from 84.1% to 88.4% and on real-world disassembly tasks from 63.3% to 82.5%. Patch-level supervision from a third-person camera also improves performance on Meta-World from 56.70% to 57.8%. These results demonstrate that privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies. Project website: https://davoodsz.github.io/PHR-VLA.github.io/

Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4

J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA

Vision-Language-Action Models Reasoning for VLA Models Robotic Manipulation Language-Conditioned Robot Manipulation

Ask PDF