Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models

Min Lin; Bokai Zhou; Jianqi Lin; Kaiyuan Yang; Yicheng Chang; Junjie Yu; Xiaofan Liu; Jianhua Han

cs.ROOct 9, 2026

Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models

Authors: Min Lin, Bokai Zhou, Jianqi Lin, Kaiyuan Yang, Yicheng Chang, Junjie Yu, Xiaofan Liu, Jianhua Han, +3 more

Organizations: Sun Yat-sen University · Yinwang Intelligent Technology Co., Ltd.

Abstract

Actions specify what a robot should execute, but do not describe what those commands will achieve. Vision-language-action (VLA) policies nevertheless learn primarily by predicting action chunks from images, language, and state. We introduce Action--Future Coupled Effect Learning (AFCE), which makes a future-grounded physical Effect the policy's primary prediction target. An offline tokenizer constructs temporally ordered Effect tokens from demonstrated actions, realized robot responses, and sparse visual transitions. Reconstruction objectives encourage the representation to retain executable detail and observed physical change. A flow-matching VLA predicts Effects from current images, instruction, and robot state, while a state-conditioned decoder converts them into action chunks. Embodiment-specific interfaces allow single- and dual-arm tasks to share the Effect model. On DexJoCo, AFCE achieves $50.97\%$ overall success, improving over $π_{0.5}$ and UVT by 11.82 and 15.21 percentage points, respectively. It leads on all five bimanual tasks, highlighting the value of Effect supervision for coordinated dexterous manipulation. The project page is available at https://afceproject.github.io/AFCE-Web/.

Figures & tables

Figure 1: Comparison of policy supervision paradigms. A standard VLA predicts actions directly; world-action models use future dynamics to support control; AFCE predicts an action-decodable Effect grounded in executed actions, robot response, and world change. Future evidence constructs training targets but is unavailable to the deployed policy.

Figure 2: Overview of Action–Future Coupled Effect Learning (AFCE). Stage I fuses demonstrated actions $\mathbf{A}_{t}$ , robot response $\Delta\mathbf{S}_{t}$ , and visual transitions $\mathbf{V}_{t}$ into ordered Effect tokens $\mathbf{E}_{t}^{*}$ , supervised by action, state, and visual reconstruction. Stage II predicts the normalized target $\mathbf{Z}_{t}^{*}$ from current images, instruction, and robot state, then decodes an action chunk. The lower panel shows the shared Effect space with embodiment-specific interfaces. The dashed path denotes training targets; future evidence is not used at deployment.

Figure 3: Action and scene changes represented by the Effect. Three successive hammer-nail frames. Top: decoded robot motion; bottom: accompanying scene changes.

	Single-arm						Bimanual					Mean
Method	PT	CM	FG	HN	PB	WP	UI	HA	AS	MW	PH	SA	BA	All
UVT	$1.33$ [1pt] $\pm 1.15$	$\mathbf{80.00}$ [1pt] $\pm 2.00$	$\mathbf{64.67}$ [1pt] $\pm 5.03$	$22.00$ [1pt] $\pm 6.93$	$60.00$ [1pt] $\pm 6.00$	$\mathbf{92.67}$ [1pt] $\pm 2.31$	$0.00$ [1pt] $\pm 0.00$	$0.67$ [1pt] $\pm 1.15$	$0.00$ [1pt] $\pm 0.00$	$41.33$ [1pt] $\pm 11.02$	$30.67$ [1pt] $\pm 4.16$	$53.44$ [1pt] $\pm 2.22$	$14.53$ [1pt] $\pm 3.06$	$35.76$ [1pt] $\pm 0.56$
$\pi_{0.5}$	$\mathbf{43.33}$ [1pt] $\pm 8.08$	$60.00$ [1pt] $\pm 2.00$	$38.00$ [1pt] $\pm 5.29$	$84.67$ [1pt] $\pm 5.77$	$67.33$ [1pt] $\pm 1.15$	$88.67$ [1pt] $\pm 1.15$	$2.00$ [1pt] $\pm 0.00$	$5.33$ [1pt] $\pm 4.16$	$1.33$ [1pt] $\pm 2.31$	$32.00$ [1pt] $\pm 8.72$	$8.00$ [1pt] $\pm 3.46$	$\mathbf{63.67}$ [1pt] $\pm 1.86$	$9.73$ [1pt] $\pm 2.41$	$39.15$ [1pt] $\pm 2.11$
Ours	$5.33$ [1pt] $\pm 2.31$	$54.67$ [1pt] $\pm 7.57$	$62.00$ [1pt] $\pm 6.93$	$\mathbf{86.67}$ [1pt] $\pm 5.77$	$\mathbf{80.67}$ [1pt] $\pm 9.02$	$84.67$ [1pt] $\pm 9.02$	$\mathbf{7.33}$ [1pt] $\pm 1.15$	$\mathbf{29.33}$ [1pt] $\pm 1.15$	$\mathbf{9.33}$ [1pt] $\pm 3.06$	$\mathbf{81.33}$ [1pt] $\pm 6.43$	$\mathbf{59.33}$ [1pt] $\pm 4.16$	$62.33$ [1pt] $\pm 1.45$	$\mathbf{37.33}$ [1pt] $\pm 2.66$	$\mathbf{50.97}$ [1pt] $\pm 1.65$

Table 1: Success rates (%) on DexJoCo under rand-obj .

Figure 4: Failure rates by task family for evaluation seed 0. Arrows show changes from

\pi_{0.5}

to Ours in percentage points.

	Error (mm) $\downarrow$			GT-Effect replay $\uparrow$
$d$	Pos.	Bi-pos.	FK	SA	BA	All
256	7.794	9.507	10.519	20/60	5/50	25/110
128	8.965	11.455	11.716	21/60	2/50	23/110
64	11.585	15.272	14.980	18/60	0/50	18/110

Table 2: Effect dimensionality sensitivity.

Variant	Effect inputs			Single-arm
	$S$	$V$	$A$	PT	CM	FG	HN	PB	WP	SA
$A$ only	–	–	$\checkmark$	$1.33$ [1pt] $\pm 1.15$	$65.33$ [1pt] $\pm 3.06$	$54.67$ [1pt] $\pm 10.07$	$68.00$ [1pt] $\pm 2.00$	$69.33$ [1pt] $\pm 6.11$	$84.67$ [1pt] $\pm 2.31$	$57.22$ [1pt] $\pm 0.38$
w/o $S$	–	$\checkmark$	$\checkmark$	$5.33$ [1pt] $\pm 5.03$	$48.00$ [1pt] $\pm 5.29$	$51.33$ [1pt] $\pm 4.62$	$76.00$ [1pt] $\pm 2.00$	$84.67$ [1pt] $\pm 7.02$	$26.67$ [1pt] $\pm 5.03$	$48.67$ [1pt] $\pm 1.53$
w/o $V$	$\checkmark$	–	$\checkmark$	$14.00$ [1pt] $\pm 5.29$	$57.33$ [1pt] $\pm 3.06$	$54.67$ [1pt] $\pm 6.43$	$78.67$ [1pt] $\pm 8.08$	$76.67$ [1pt] $\pm 9.02$	$52.67$ [1pt] $\pm 12.06$	$55.67$ [1pt] $\pm 3.06$
AFCE	$\checkmark$	$\checkmark$	$\checkmark$	$5.33$ [1pt] $\pm 2.31$	$54.67$ [1pt] $\pm 7.57$	$62.00$ [1pt] $\pm 6.93$	$86.67$ [1pt] $\pm 5.77$	$80.67$ [1pt] $\pm 9.02$	$84.67$ [1pt] $\pm 9.02$	$\mathbf{62.33}$ [1pt] $\pm 1.45$

Table 3: Effect-construction input ablation on DexJoCo under rand-obj . Success rates are reported in %.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Objective
Single arm
Click Mouse	Move the mouse onto its pad and click.
Fold Glasses	Fold the glasses and place them in the case.
Hammer Nail	Drive a nail into a board with a hammer.
Pick Bucket	Place food in the bucket and lift it.
Pinch Tongs	Grasp the tongs and open and close them three times.

Appendix

Table B.1: Functional goals of the 11 DexJoCo tasks.

Figure C.1: Structure and prediction error of the effect space. (a) GT-E by task; (b) the same GT-E windows by relative trajectory progress; (c) GT-E and predicted effects in a shared embedding; (d) paired normalized effect RMSE in the original space. PCA uses 50 components (87.27% explained variance); the shared t-SNE uses perplexity 30 and seed 42.

Figure C.2: Sensitivity of the two-dimensional projection. GT-E windows after the same normalization and PCA are projected with t-SNE at perplexity/seed 30/42, 15/42, and 50/7. Task colors match Figure C.1 .

Figure C.3: Physical action-readout error on matched demonstration windows. GT-E/

D_{0}

, GT-E/

D_{1}

, and predicted-E/

D_{1}

are compared on mean Euclidean position error per hand, rotation geodesic error, and mean absolute finger joint-angle error. Means aggregate ten windows per trajectory over 110 trajectories; intervals are 95% within-task episode-bootstrap intervals (3,000 resamples).

D_{0}

is used only for this diagnostic; the deployed policy uses

D_{1}

Panel of Figure 12 — Figure C.4: Illustrative AFCE and $\pi_{0.5}$ rollouts on Fold Glasses, Hanoi, Microwave, and Photograph (top to bottom). Each task is shown at approximately 15%, 40%, 70%, and 95% of the rollout. The selected AFCE rollouts complete their task-specific interaction, whereas the corresponding $\pi_{0.5}$ rollouts fail at contact, placement, or final actuation.

Hyperparameter	Value
Stage I: Effect tokenizer
Horizon / feature widths	$H=30$ ; branch width 64; Effect width 256
Fusion / temporal / decoder layers	$2/2/4$ ; 8 heads; FFN width 1024; dropout 0
Visual sampling	Stride 5 plus terminal frame; at most 6 dynamic and 2 reference regions, with 8 slots per region
Visual descriptor / PCA	136 dimensions / 64 components; frozen DINOv3; legacy-none mask
Action reconstruction	Grouped Huber; position weight $w_{p}=2$

Appendix

Table D.1: Hyperparameters of the evaluated AFCE configuration.

Explore similar work

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies
Oct 3, 2026Zhe Tao, Feiran Wang, Gaowen Liu +2World Models for Robotics Vision-Language-Action Models
University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco
MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies
Sep 30, 2026Jingqiu Wang, Yan WangVisuomotor Policy Learning Robotic Manipulation
School of Data Science and Engineering East China Normal University, Shanghai, China
PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models
Aug 27, 2026Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang +4Vision-Language-Action Models Reasoning for VLA Models
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA

Oct 3, 2026J/K move · Enter open · S save

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

Zhe Tao, Feiran Wang, Gaowen Liu, Ramana Rao Kompella+1

University of Illinois Urbana-Champaign · University of Illinois Chicago · Cisco

World Models for Robotics Vision-Language-Action Models Language-Conditioned Robot Manipulation 3D Scene Understanding

Open paper AskPDF arXiv

Robots need to anticipate how their actions will change the world, since manipulation success hinges on the resulting contacts and object motions. However, existing Vision-Language-Action (VLA) policies that predict future observations from shared features leave the forecast decoupled from the actions the policy will actually execute, and impose no physical constraints on how the scene may evolve. We introduce ForeAct3D, a framework for policy-grounded future world modeling within VLA policies. Learnable geometric queries decode depth, semantic segmentation, and camera pose from the policy representation into current and future semantic 3D scene states, and the future queries are conditioned on the policy-generated action chunk to ground the forecast in the planned interaction. A physical-consistency closure relates the two states through background staticity and instance-level rigidity, and anchors the wrist-camera pose to end-effector kinematics. These objectives shape the shared representation used for action generation during training, and no future prediction is required at inference. Without robot pretraining, ForeAct3D achieves 98.3% average success on LIBERO and an average task length of 3.73 on CALVIN, outperforming its base policy on every suite. Ablations show that semantic 3D supervision, physical consistency, and action conditioning each improve manipulation performance, and that action conditioning substantially improves future object localization. Real-world experiments on spatial placement, object insertion, and sequential manipulation further raise average success from 6.7% to 37.8% over the base policy. The project page and code are available at https://github.com/anthonytao80-crypto/ForeAct3D.

Abstract

Explore similar work

ForeAct3D: Policy-Grounded Future World Modeling for VLA Policies

MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models