What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning

Haitao Shen; Hanyu Liu; Yihan Lin; Huiping Liu; Junbo Cui; Daoheng Li; Yang Li; Jing Zhang

cs.ROOct 15, 2026

What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning

Authors: Haitao Shen, Hanyu Liu, Yihan Lin, Huiping Liu, Junbo Cui, Daoheng Li, Yang Li, Jing Zhang,

Organizations: Renmin University of China · Peking University · Beijing Jiaotong University · ModelBest ·

Abstract

Vision-language-action (VLA) policies are typically trained with action prediction as the primary supervision signal, since robots ultimately execute actions. Auxiliary objectives such as latent-action (LA) and future-vision (FV) prediction are often added to improve policy learning, yet they compete with action prediction for the same training budget. We ask whether, under a fixed budget, allocating a substantial share of training samples to such objectives, rather than to further action prediction, can improve policy learning. We build a unified VLA framework that couples a two-stream multimodal diffusion transformer (MMDiT) with a sample-level router, which assigns each training sample to either action prediction or an auxiliary objective and thus makes supervision allocation explicit. Within this framework, we instantiate three LA targets based on LAPA and its DINO-feature variants, and three FV targets based on DINOv3, V-JEPA 2, and Wan2.2 representations. Across LIBERO, CALVIN, and four real-robot tasks, suitable LA and FV supervision outperforms action-only training under the same budget, even when action prediction receives no more than half of the training samples. FV generally outperforms LA, although the strongest LA target remains competitive on CALVIN. Diagnostic analyses suggest that DINO-based LA targets emphasize relative motion, whereas FV targets emphasize absolute position and task identity. We therefore argue that what robot policies learn to predict, and how training is allocated among these targets, should be treated as explicit design choices rather than defaulting to action-dominant training.

Figures & tables

Figure 1: Overview. Left: under a fixed training budget, a router sends each sample to the action loss or one auxiliary loss (LA or FV); P $x$ routes $x$ % of samples to the action loss. Middle: two-stream VLA; FV prediction conditions on clean demonstrated actions. Right: (1) V-JEPA 2 FV vs. its source-matched P100 control; (2) LA and FV means over three targets each at P50 vs. action-only (Act.), arrows marking unrounded FV $-$ LA (real robot: % of max score); (3) per-map normalized LIBERO attention for action-only, +LA (LAPA-DINOv3), and +FV (V-JEPA 2).

Figure 2: VLM–MMDiT architecture. Action and auxiliary (LA/FV) experts interact through joint self-attention; sample-level routing selects the active loss. Future observations are used only for auxiliary training.

	Method	Param. (B)	LIBERO					CALVIN
	Method	Param. (B)	S	O	G	L	Avg. $\uparrow$	SR-5 $\uparrow$	Avg. Len. $\uparrow$
Ref.	OpenVLA	$\approx\!8$	84.7	88.4	79.2	53.7	76.5	43.5	3.270
	$\pi_{0}$	$\approx\!3$	96.8	98.8	95.8	85.2	94.2	60.1	3.840
	UniVLA	$\approx\!7$	96.5	96.8	95.6	92.0	95.2	56.5	3.800
	$\pi_{0.5}$	$\approx\!3$	98.8	98.2	98.0	92.4	96.9	–	–
Act.	Half budget	$\approx\!2$	97.2	99.4	94.6	81.6	93.20	57.7	3.971

Table 1: LIBERO and CALVIN policy performance. Mixed configurations use P50 and the full budget. Parentheses: differences from the half-budget baseline. S/O/G/L: Spatial/Object/Goal/Long; “–”: unavailable. Source-matched FV controls: Appendices B.2 , B.5 , and B.6 .

Figure 3: Allocation at the full budget. Cells show LIBERO success (%) and CALVIN Avg. Len.; darker is higher within each benchmark. Black outlines mark each model’s best allocation.

Figure 4: Performance across training budgets at P50 on (a) CALVIN and (b) LIBERO. Both panels share the line styles shown above: solid, P50; colored dashed, source-matched FV P100; gray dotted, shared action-only baseline, also the LA P100 reference.

Figure 5: Full sweep over supervision allocation and training budget. Each curve reports gains over the same-stage P100 control: the shared action-only baseline for LA and the source-matched control for FV. Rows show CALVIN

\Delta

Avg. Len. and LIBERO

\Delta

success rate (pp). Dashed zero lines indicate parity with the corresponding control. CALVIN panels share a vertical scale; LIBERO panels use target-specific vertical scales to show each trajectory.

Panel of Figure 6 — Figure 6: Real-robot evaluation: (a) representative executions; (b) partial-credit scores normalized by 160 points per task (20 trials, up to 8 points each). LA/FV use P50. Protocol: Appendix B.7 .

	AgiBotWorld-Beta unseen tasks				CALVIN environment-D validation
Encoder	ABS	$\Delta_{c=1}$	$\Delta_{c=9}$	$\Delta_{c=45}$	ABS	$\Delta_{c=1}$	$\Delta_{c=9}$	$\Delta_{c=45}$
DINOv3	76.31	25.23	13.73	14.19	52.06	69.64	75.44	58.21
V-JEPA 2	101.98	20.44	11.42	12.16	54.70	64.21	68.43	55.85
Wan2.2	105.55	32.28	17.20	16.65	53.86	53.44	59.72	50.01
LAPA-DINOv2	155.65	14.79	8.62	8.79	62.70	56.89	59.16	54.40
LAPA-DINOv3	155.61	14.68	8.71	9.23	60.14	53.23	57.41	50.48

Table 2: Frozen-source position errors (mm; lower is better) from visual-only probes;

\Delta_{c}

uses action differences at stride

c

within a 45-step segment. Bold: column minima. Protocol: Appendix C.1 .

Figure 7: Frozen-source diagnostics on AgiBotWorld-Beta seen validation. (a) Correlation between net-action-change and representation pair distances, plotted against delta-step5 NMSE. (b) Feature-cluster NMI with displacement-derived action labels and task IDs. Each point is one encoder; details in Appendix C.2 .

Panel of Figure 8 — Figure 8: Qualitative policy attention on two LIBERO observations, one per row. Columns show the input observation and the action-only, LA, and FV policies. Colors show relative intensity within each map. Both rows use the extraction protocol in Appendix C.3 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Provider and checkpoint	Configuration used
Policy VLM	OpenBMB; openbmb/MiniCPM-V-4.6 ( Yao et al., 2025 ; OpenBMB, 2026 )	Qwen3.5-0.8B language model and SigLIP2-400M vision encoder; approximately 1.30B locally loaded parameters. The text tower has hidden size 1024 and 24 layers; the vision tower has hidden size 1152 and 27 layers. The processor uses a base resolution of 448 and supports dynamic slicing.
LAPA LA	Released LARYBench asset laq_openx.pt ( Nie et al., 2026 ; Meituan LongCat, 2026 )	Image-based latent-action quantizer; four codes per transition and codebook size eight.
LAPA-DINOv2 LA	Meta FAIR; facebook/dinov2-large ( Oquab et al., 2024 ) plus laq_dinov2.pt ( Nie et al., 2026 ; Meituan LongCat, 2026 )	ViT-L/14, approximately 304M parameters; 24 layers, 16 heads, hidden size 1024. The latent-action branch uses $224\times 224$ inputs, producing a $16\times 16$ final-layer patch grid and 16 codes per transition.
LAPA-DINOv3 LA	Meta FAIR; facebook/dinov3-vitl16-pretrain-lvd1689m ( Siméoni et al., 2026 ) plus laq_dinov3.pt ( Nie et al., 2026 ; Meituan LongCat, 2026 )	ViT-L/16, approximately 303M parameters; 24 layers, 16 heads, hidden size 1024. The latent-action branch uses $224\times 224$ inputs, producing a $14\times 14$ final-layer patch grid after excluding CLS and four register tokens, and 16 codes per transition.
DINOv3 FV	Meta FAIR; facebook/dinov3-vits16-pretrain-lvd1689m ( Siméoni et al., 2026 )	ViT-S/16, approximately 21.6M parameters; 12 layers, six heads, hidden size 384. The FV branch uses $224\times 224$ inputs and final-layer continuous features.
V-JEPA 2 FV	Meta FAIR; official V-JEPA 2 release, vjepa2_vitl16.pt ( Assran et al., 2025 )	ViT-L/16 target encoder (the predictor is not loaded), approximately 304M parameters; 24 layers, 16 heads, hidden size 1024, temporal tubelet size two, and rotary position embeddings.

Appendix

Table 3: Backbones, checkpoints, and input configurations used in the reported experiments. Parameter counts describe the loaded model or encoder. LA predicts stored integer codes; FV predicts a velocity target constructed from continuous encoder outputs.

Suite	Tasks	Episodes	Frames	Episodes/task
Spatial	10	432	52,970	35–47
Object	10	454	66,984	42–50
Goal	10	428	52,042	33–50
Long	10	379	101,469	29–49
Total	40	1,693	273,465	—

Appendix

Table 4: LIBERO training demonstrations, shared by all configurations. The final column gives the minimum and maximum episode counts across the ten tasks in each suite.

Setting	Value
Action horizon	8
Effective batch size	128
Training schedule	60k optimizer updates; checkpoints at 30k, 40k, 50k, and 60k
MMDiT learning rate	Peak $10^{-4}$ ; final $5\times 10^{-6}$
VLM learning rate	Peak $10^{-5}$ ; final $5\times 10^{-7}$
Warmup	Linear increase from zero to peak over 2k updates

Appendix

Table 5: Shared optimization settings for the main LIBERO and CALVIN experiments.

Target	Route	Avg. Len.	SR-5
Action-only	Half	$4.000\pm 0.065$	$58.8\pm 1.7$
Action-only	Full	$4.335\pm 0.040$	$71.6\pm 1.8$
LAPA	P25	$3.501\pm 0.021$	$45.2\pm 0.9$
	P50	$3.486\pm 0.059$	$43.5\pm 2.0$
	P75	$4.092\pm 0.032$	$63.0\pm 1.0$
LAPA-DINOv2	P25	$4.084\pm 0.039$	$66.6\pm 2.0$

Appendix

Table 6: CALVIN ABC

\rightarrow

D results over evaluation seeds 0, 1, and 2 (mean

\pm

sample standard deviation); SR-5 is in %. Half and Full denote the half- and full-budget action-only baselines. Main-text tables and figures report seed 0.

Configuration	SR-1	SR-2	SR-3	SR-4	SR-5	Avg. Len.
Action-only (half budget)	98.1	91.5	80.3	69.5	57.7	3.971
Action-only (full budget)	99.4	95.3	87.3	79.1	70.3	4.314
LAPA	96.2	85.0	68.6	54.7	42.1	3.466
LAPA-DINOv2	94.0	86.7	78.6	70.8	63.4	3.935
LAPA-DINOv3	98.2	94.7	89.2	84.3	77.0	4.434
DINOv3 FV	97.4	93.8	88.5	82.2	75.1	4.370

Appendix

Table 7: CALVIN prefix-success metrics underlying Table 1 . SR-1 through SR-5 are cumulative success rates (%); Avg. Len. is measured in tasks per sequence.

FV source	Optimizer updates (k)	Shared action-only	P50 FV	Source-matched P100	$\Delta$ shared	$\Delta$ source-matched
DINOv3	30	93.20	91.95	96.65	-1.25	-4.70
	40	95.45	96.80	96.95	+1.35	-0.15
	50	96.35	97.55	97.90	+1.20	-0.35
	60	96.50	97.25	97.45	+0.75	-0.20
V-JEPA 2	30	93.20	96.55	96.40	+3.35	+0.15
	40	95.45	97.10	97.55	+1.65	-0.45

Appendix

Table 8: LIBERO FV comparisons with the shared action-only baseline and source-matched controls across checkpoints. Scores are success rates (%); differences are P50 FV minus the indicated reference in percentage points. All columns use the same checkpoint budget. Source-matched P100 controls hold all settings except routing fixed.

Configuration	Optimizer updates (k)	Spatial	Object	Goal	Long	Avg.
LAPA	30	97.20	98.20	90.20	73.20	89.70
LAPA	40	97.20	98.80	94.20	84.00	93.55
LAPA	50	97.00	98.60	93.40	91.40	95.10
LAPA	60	96.40	98.20	94.20	89.80	94.65
LAPA-DINOv2	30	95.80	97.60	93.60	84.60	92.90
LAPA-DINOv2	40	93.60	97.80	97.40	87.60	94.10

Appendix

Table 9: Detailed LIBERO P50 results at four optimizer-update checkpoints. P50 routes half of the training samples to action prediction in expectation. Success rates are percentages; Avg. is the mean of the four suite rates.

$\lambda_{\mathrm{LA}}$	LAPA	LAPA-DINOv2	LAPA-DINOv3
0.01	94.55	96.45	95.90
0.1	94.65	94.95	95.50
1.0	91.95	93.95	94.90

Appendix

Table 10: LA loss-weight ablation on LIBERO (P50, full budget). Entries are average success rates (Avg., %). Bold: the weight used for each target in all reported experiments.

Latent action (LA)
Target	Optimizer updates (k)
	30	40	50	60
LAPA	3.019	3.313	3.580	3.466
LAPA-DINOv2	3.120	4.194	3.812	3.935
LAPA-DINOv3	3.830	4.157	4.322	4.434
Future vision (FV)

Appendix

Table 11: CALVIN average task-chain length (Avg. Len.) under P50 routing at each tested training budget. These absolute scores underlie the P50 trajectories in Figure 4 (a).

FV source	Optimizer updates (k)	$\Delta$ Avg. Len.	$\Delta$ SR-5 (pp)
DINOv3	30	+0.243	+12.9
	40	+0.235	+9.7
	50	+0.270	+11.9
	60	+0.146	+5.0
V-JEPA 2	30	+0.770	+18.2
	40	+0.622	+19.0

Appendix

Table 12: CALVIN P50 minus source-matched P100 at all four reported checkpoints.

\Delta

Avg. Len. is in tasks per sequence;

\Delta

SR-5 is in percentage points. Each comparison is positive on both metrics.

Latent action (LA)
Target	P25	P50	P75	P100
LAPA	3.511 / 45.7	3.466 / 42.1	4.128 / 63.9	—
LAPA-DINOv2	4.126 / 68.8	3.935 / 63.4	4.264 / 72.8	—
LAPA-DINOv3	4.283 / 72.8	4.434 / 77.0	4.421 / 77.0	—
Future vision (FV)
DINOv3	2.107 / 16.6	4.370 / 75.1	4.193 / 68.7	4.224 / 70.1

Appendix

Table 13: Tested supervision allocations at the common CALVIN endpoint of 60k optimizer updates. Each cell reports Avg. Len. / SR-5 (%). P25, P50, P75, and P100 denote action-routing probabilities

p_{A}=0.25

0.50

0.75

, and

1.00

, respectively; lower P assigns a larger expected fraction of samples to auxiliary (LA/FV) prediction. FV P100 entries are source-matched controls; LA shares the full-budget action-only baseline in Table 7 , hence the dashes in its P100 column. Bold marks the best tested value for each metric within each row, including ties.

	P25		P50		P75		P100
FV source	Avg. Len.	SR-5 (%)	Avg. Len.	SR-5 (%)	Avg. Len.	SR-5 (%)	Avg. Len.	SR-5 (%)
DINOv3	1.666	10.3	4.014	64.1	4.223	69.7	3.779	54.4
V-JEPA 2	4.318	73.7	4.430	75.0	4.384	73.7	3.808	56.0
Wan2.2	4.112	64.4	3.997	64.3	4.443	80.0	3.295	46.9

Appendix

Table 14: FV supervision allocation at a fixed CALVIN training budget of 40k optimizer updates. P25, P50, P75, and P100 denote action-routing probabilities

0.25

0.50

0.75

, and

1.00

, respectively. P100 is the source-matched control. Batch size and optimization settings are fixed within each source. Bold marks the best tested value for each metric within each row.

Model	Pick-out ( $n=20$ )	Place-in ( $n=20$ )	Rearrange-cups ( $n=20$ )	Rearrange-objects ( $n=20$ )
Action-only
Action-only baseline	116 / 160	92 / 160	80 / 160	124 / 160
Latent action (LA)
LAPA (P50)	92 / 160	92 / 160	80 / 160	96 / 160
LAPA-DINOv2 (P50)	112 / 160	80 / 160	76 / 160	120 / 160
LAPA-DINOv3 (P50)	106 / 160	88 / 160	72 / 160	92 / 160

Appendix

Table 15: Real-robot manipulation outcomes on the DOS W1 left arm. Auxiliary configurations use P50; all configurations are trained for 30k optimizer updates. Scores accumulate over 20 trials per task (maximum 160; scoring in Appendix B.7 ). Bold marks the highest score in each task.

Split	Task IDs	Samples
Seen training	169	84,162
Seen validation	169	28,054
Unseen	19	12,303

Appendix

Table 16: AgiBotWorld-Beta regression splits derived from LARYBench. Counts are image-pair samples, not trajectories.

Encoder	$S^{\mathrm{dist}}_{0}$	$S^{\mathrm{trust}}_{0}$	$R^{2}_{\mathrm{pos}}$	$\mathrm{NMSE}_{\text{delta-step5}}$
LAPA-DINOv3	0.375	0.763	0.354	0.807
LAPA-DINOv2	0.189	0.628	0.190	0.809
V-JEPA 2	-0.030	0.523	0.240	0.858
DINOv3	0.047	0.525	-0.027	0.893
Wan2.2	-0.025	0.511	-0.001	0.959
LAPA	-0.089	0.479	0.004	0.959

Appendix

Table 17: Feature–action geometry and separate delta-step5 regression on AgiBotWorld-Beta seen validation (protocol in Appendix C.1 ). Regression reports normalized MSE (NMSE). Bold marks the highest geometry score and lowest regression error. The bottom-row Pearson correlations relate each geometry/readout score to NMSE across

N=6

encoders.

Encoder	$\mathrm{NMI}_{\mathrm{act}}$	$\mathrm{NMI}_{\mathrm{task}}$	Task purity
LAPA-DINOv3	0.412	0.221	0.068
LAPA-DINOv2	0.232	0.174	0.045
LAPA	0.143	0.148	0.032
DINOv3	0.183	0.828	0.653
V-JEPA 2	0.191	0.811	0.663
Wan2.2	0.155	0.577	0.370

Appendix

Table 18: Cluster associations on AgiBotWorld-Beta seen validation (protocol in Appendix C.1 ). NMI compares feature clusters with displacement-derived action labels and with task IDs.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,190 papers

Jun 25, 2026

LA4VLA: Learning to Act without Seeing via Language-Action Pretraining

Vision-Language-Action (VLA) models are commonly pretrained on robot demonstrations by jointly mapping visual observations and language instructions to actions. However, dense visual-action supervision can dominate the comparatively sparse language-action signal. As a result, policies may rely on visual shortcuts rather than learn how language conditions action execution, making them sensitive to visual variations. To address this limitation, we propose LA4VLA, a language-action pretraining framework that enables policies to acquire language-conditioned action priors without visual observations. These priors capture reusable manipulation skills shared across tasks and scenes, reducing reliance on scene-specific visual cues. Specifically, LA4VLA decomposes expert demonstration trajectories into atomic action segments and pairs each segment with a corresponding low-level action description. This yields LA-33K, a dataset of 33K Language-Action (LA) episodes derived entirely from existing demonstrations without additional robot data collection. We further develop LA4VLA-1B, a lightweight 1B-parameter VLA model, and investigate three paradigms for incorporating language-action supervision into VLA learning: LA-only pretraining, sequential LA-to-VLA pretraining, and mixed LA-VLA pretraining. Across simulation and real-world tasks, LA-pretrained policies consistently outperform matched VLA-pretrained counterparts, while combining LA and VLA supervision leads to further gains. In particular, mixed LA-VLA pretraining improves the average success rate of LA4VLA-1B over the no-pretraining baseline by up to 17.8 and 45.0 percentage points in simulation and real-world tasks, respectively. These results establish LA4VLA as an effective and complementary pretraining strategy for building stronger and more robust VLA policies.

Tao Lin, Yuxin Du, Yiran Mao +13

School of AI, Shanghai Jiao Tong University · Alibaba Group · KAUST +1

Language Model Pretraining Visuomotor Policy Learning Robotic Manipulation Vision-Language-Action Models

Ask PDF

May 26, 2026

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA, an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data, a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 11,631 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level success: FG-only improves over Raw-only by +1.4 to +8.1 success-rate points across settings. Second, fine-grained and raw instructions are complementary, following a consistent inverted-U trend peaking at FG:Raw = 1:2 to 1:1. The best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation (vs. 49.9 Raw-only). Third, fine-grained supervision improves steerable control: the largest real-world gains appear on pose (+23), color (+18), and approach direction (+18)--factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. Project page: https://finevla.xlang.ai/

Xintong Hu, Xuhong Huang, Jinyu Zhang +11

XLANG Lab, The University of Hong Kong · Qwen Team, Alibaba Inc.

Language-Conditioned Robot Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks Language-Conditioned Robot Control

Ask PDF

Sep 29, 2026

Where Predictive Supervision Goes Shapes What VLA Policies Learn

Future prediction is increasingly used to improve vision-language-action (VLA) policies, based on the premise that anticipating scene evolution encourages representations useful for control. However, forecast quality alone does not establish that a policy has learned a better representation for action. This distinction matters under distribution shift, where successful control depends on preserving spatial state and likely scene change beyond familiar configurations. We study what determines whether predictive supervision improves the visual representation used by a VLA policy. Through controlled comparisons with matched target constructions, prediction horizons, and training conditions, we find that different prediction interfaces produce markedly different forecasts and visual representations, including in the spatial, dynamics, and action information that transfers beyond familiar scenes. We trace these differences to how predictive errors shape the policy's visual stream. Consistent with this controlled finding, VLA policies trained with more direct, scene-matched future supervision show stronger robustness under simulated and physical distribution shifts. Together, our results frame future prediction as a representation-learning design problem whose value for control depends on whether its supervision reaches the representations through which the policy acts.

Hanseul Kim, Jewon Yeom, Youngjoon Jeong +2

Graduate School of Data Science Seoul National University

Visuomotor Policy Learning Vision-Language-Action Models Predictive Representation Learning Robustness of VLA Models

Ask PDF

Abstract

Vision-Language-Action Models

Explore similar work

LA4VLA: Learning to Act without Seeing via Language-Action Pretraining

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

Where Predictive Supervision Goes Shapes What VLA Policies Learn