Where Predictive Supervision Goes Shapes What VLA Policies Learn

Hanseul Kim; Jewon Yeom; Youngjoon Jeong; Minsoo Jo; Taesup Kim

cs.ROSep 29, 2026

Where Predictive Supervision Goes Shapes What VLA Policies Learn

Authors: Hanseul Kim, Jewon Yeom, Youngjoon Jeong, Minsoo Jo, Taesup Kim

Organizations: Graduate School of Data Science Seoul National University

Abstract

Future prediction is increasingly used to improve vision-language-action (VLA) policies, based on the premise that anticipating scene evolution encourages representations useful for control. However, forecast quality alone does not establish that a policy has learned a better representation for action. This distinction matters under distribution shift, where successful control depends on preserving spatial state and likely scene change beyond familiar configurations. We study what determines whether predictive supervision improves the visual representation used by a VLA policy. Through controlled comparisons with matched target constructions, prediction horizons, and training conditions, we find that different prediction interfaces produce markedly different forecasts and visual representations, including in the spatial, dynamics, and action information that transfers beyond familiar scenes. We trace these differences to how predictive errors shape the policy's visual stream. Consistent with this controlled finding, VLA policies trained with more direct, scene-matched future supervision show stronger robustness under simulated and physical distribution shifts. Together, our results frame future prediction as a representation-learning design problem whose value for control depends on whether its supervision reaches the representations through which the policy acts.

Figures & tables

Figure 1: Downstream VLA realization of future-feature prediction used in RQ3. A momentum encoder supplies main-camera targets while the pretrained VLM and action expert are shared. The special-token interface predicts from special tokens, whereas the vision-token interface predicts at main-camera patch positions and retains a same-position credit route.

Figure 2: Vision-token forecasting captures scene-specific future change and preserves transferable visual features. (a) Near-center and spatial-tail examples show decoded forecasts and change maps. (b) Decoded and actual scene-change magnitudes. (c) Their per-example spatial correlation. (d) Frozen probes of state, dynamics, actions, and geometry. Solid and dashed outlines mark near-center and spatial-tail results, split at each task’s median distance from its mean target position. Effective rank appears below each radar.

Figure 3: Direct aligned coupling recovers transferable visual features. (a) Aligned-path construction, same-region gradient share at initialization, and probe outcomes across path strengths $\gamma$ . (b) RoPE-indexed routing and probe outcomes when special-token-to-patch gradients are preserved, shuffled, or blocked under identical forward computation. Dots are seeds and black bars are means. Dashed blue and dotted red lines denote the vision-token and unindexed special-token references.

Variant	mean	Lang.	Position	Object	Task	Env.	mean	Ret.
	LIBERO	LIBERO-PRO
Baseline (no forecast)	$92.3{\scriptstyle\,\pm 1.3}$	$86.0$	$48.8{\scriptstyle\,\pm 5.9}$	$72.3$	$40.8$	$49.5$	$59.5{\scriptstyle\,\pm 1.8}$	$0.635$
Special-token ( $M=16$ )	$92.0{\scriptstyle\,\pm 1.7}$	$84.0$	$45.9{\scriptstyle\,\pm 3.2}$	$79.1$	$43.2$	$51.7$	$60.8{\scriptstyle\,\pm 1.5}$	$0.653$
Special-token ( $M=N=256$ )	$90.7{\scriptstyle\,\pm 0.8}$	$85.7$	$45.3{\scriptstyle\,\pm 3.3}$	$76.6$	$41.4$	$52.2$	$60.2{\scriptstyle\,\pm 1.8}$	$0.658$
Special-token ( $M=N=256$ ) ∗	$92.6{\scriptstyle\,\pm 0.5}$	$85.8$	$54.5{\scriptstyle\,\pm 2.4}$	$\mathbf{80.3}$	$44.3$	$52.4$	$63.5{\scriptstyle\,\pm 1.1}$	$0.678$
Vision-token	$\mathbf{94.7}{\scriptstyle\,\pm 1.9}$	$\mathbf{88.3}$	$\mathbf{60.9}{\scriptstyle\,\pm 1.0}$	$77.6$	$\mathbf{46.0}$	$\mathbf{58.8}$	$\mathbf{66.3}{\scriptstyle\,\pm 0.6}$	$\mathbf{0.696}$

Table 1: Simulation performance on LIBERO and LIBERO-PRO. LIBERO, Position, and PRO mean report mean

\pm

standard deviation over three seeds. The remaining perturbation columns report seed means. Retention is perturbed success divided by standard success.

Figure 4: Vision-token forecasting gives the strongest future-action readout and highest pooled real-robot OOD success. (a) Frozen-policy probes on 44 held-out episodes. (b–d) Inputs from the training distribution, camera blur, and an unseen pot layout. Cyan points mark training lid-grasp positions, and the ring marks the trial lid position. (e) Success rates with 95% block-bootstrap intervals.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Prediction readout	Representative methods	How supervision reaches policy vision tokens
Autoregressive output tokens	CoT-VLA ( Zhao et al., 2025 )	Sequence-mediated, no explicit same-patch attachment
Learned/query tokens	FLARE, DreamVLA, WoG, HiF-VLA ( Zheng et al., 2025 ; Zhang et al., 2025 ; Su et al., 2026 ; Lin et al., 2026 )	Attention-mediated, no explicit same-patch attachment
Predictive latent alignment	VLA-JEPA, FutureVLA ( Sun et al., 2026 ; Xu et al., 2026 )	Latent-state or intermediate alignment, no explicit same-patch attachment
Future readout + spatial auxiliary	FoMoVLA ( Li et al., 2026 )	Indirect forecast readout with a direct spatial auxiliary
Separate world model	AHEAD ( Syed et al., 2026 )	Patch-aligned outside the frozen policy
Special tokens	This study	Attention-mediated, no guaranteed same-position path

Appendix

Table A.1: Representative future-prediction interfaces and how their supervision reaches trainable policy vision tokens.

Figure B.1: Matched prediction interfaces used in RQ1. (a) The special-token interface predicts from one added token per future patch, so forecast errors reach the visual stream through contextual mixing. (b) The vision-token interface predicts from the corresponding patch token, retaining contextual mixing and adding a direct same-position route. Dashed arrows show the backward paths.

Component	Setting
Data	500 LIBERO-Object episodes from agentview_rgb , with every tenth episode held out, giving 450 training and 50 validation episodes
Training clips	Every valid clip start frame in the training episodes
Input	$128\times 128$ images, $16\times 16$ patches, and an $8\times 8$ grid of 64 visual tokens
Encoder	12-block ViT with width 192, three attention heads, and learned absolute positional embeddings
Future target	Stop-gradient patch residual at $h=24$ from a momentum encoder with $m=0.996$
Loss	Per-patch squared error weighted by the normalized norm of each target residual

Appendix

Table B.1: Controlled RQ1 training setup. All settings are shared by the special-token and vision-token interfaces.

Component	Setting
Decoder data	35 held-out episodes for fitting and 15 disjoint episodes for evaluation
Decoder input	Features from real current and future frames only
Optimization	Adam for 12,000 steps, batch size 48, learning rate $1.5\times 10^{-3}$ with cosine decay
Objective	$\ell_{1}+0.5\,\mathrm{MSE}$
Evaluation set	617 samples per seed, comprising 418 near-center and 199 spatial-tail samples
Moving pixels	Mean absolute RGB change above $0.08$ for images scaled to $[0,1]$ , excluding samples with at most 20 moving pixels

Appendix

Table B.2: Decoded-forecast evaluation setup.

	All pixels		Moving pixels
Split	Vision	Special	Vision	Special
All	$45.0\pm 1.2$	$0.3\pm 2.9$	$56.4\pm 1.6$	$3.0\pm 2.2$
Near-center	$42.8\pm 1.4$	$-0.1\pm 3.6$	$54.4\pm 1.7$	$2.9\pm 2.0$
Spatial tail	$49.0\pm 1.3$	$1.1\pm 2.0$	$60.7\pm 1.4$	$3.1\pm 2.9$

Appendix

Table B.3: Decoded-forecast results. Gap closure is reported in percent. Gap closure and change slopes report mean

\pm

standard deviation over five seeds. Change-map correlation reports the pooled median, with brackets giving the range of seed-level medians.

Target	Split	Vision-token visual	Special-token visual	Special-token stream
Position	Near	$0.939{\scriptstyle\pm 0.005}$	$0.577{\scriptstyle\pm 0.193}$	$0.508{\scriptstyle\pm 0.294}$
	Tail	$0.574{\scriptstyle\pm 0.025}$	$0.138{\scriptstyle\pm 0.134}$	$0.131{\scriptstyle\pm 0.207}$
Identity	Near	$0.931{\scriptstyle\pm 0.024}$	$0.649{\scriptstyle\pm 0.128}$	$0.494{\scriptstyle\pm 0.171}$
	Tail	$0.304{\scriptstyle\pm 0.030}$	$0.182{\scriptstyle\pm 0.043}$	$0.138{\scriptstyle\pm 0.044}$
Future displacement	Near	$0.796{\scriptstyle\pm 0.020}$	$0.529{\scriptstyle\pm 0.121}$	$0.460{\scriptstyle\pm 0.193}$
	Tail	$0.450{\scriptstyle\pm 0.117}$	$0.332{\scriptstyle\pm 0.100}$	$0.304{\scriptstyle\pm 0.167}$

Appendix

Table B.4: Frozen-feature probes underlying Figure 2(d) of the main paper. Values are mean

\pm

standard deviation over five seeds. Identity is classification accuracy, and all other targets use

R^{2}

. Near and Tail denote the near-center and spatial-tail splits. The two special-token columns read the visual and special-token grids of the same encoder.

Figure B.2: Forecast targets and positional structure jointly determine the special-token outcome. Learned-absolute-position models use the momentum target from RQ1, a frozen forecast-trained encoder, or frozen SigLIP features. The 1-D RoPE models use either the momentum or frozen-SigLIP target with a stable special-token-to-patch offset. Bars show means over five seeds and error bars show one standard deviation. Forecast gap closure, effective rank, and spatial-tail position and action probes are reported. Rank and probes read the visual-token representation learned by the corresponding prediction interface. Forecast gap closure should be compared between interfaces within a target condition, because decoder feature spaces differ across target families.

Controlled, initialization
Interface	Same-region gradient share	Centered gradient rank
Vision-token	$0.673\pm 0.041$	$5.6\pm 0.4$
Shuffled route	$0.062\pm 0.002$	$5.8\pm 0.4$
Special-token	$0.062\pm 0.000$	$1.8\pm 0.1$
$\pi_{0.5}$ , trained
Vision-token	$0.694\pm 0.070$	$26.8\pm 8.5$

Appendix

Table C.1: Spatial alignment of forecast credit. Values are mean

\pm

standard deviation. Controlled-model values use five initializations. Vision-token and shuffled-target VLA values use 400 samples from one available checkpoint. The special-token VLA row pools 1200 samples across three checkpoints.

Figure C.1: Spatial routing of forecast credit in $\pi_{0.5}$ (LIBERO-Object). Forecasts are pooled to the $4\times 4$ region grid of the special-token interface, so all conditions produce 16 forecasts over the same 256 input patches. (a) Row-normalized routing $p_{ij}$ over the input patches for two forecast regions in each condition. The square marks the corresponding input region, and the number gives the share of credit that reaches it. (b) Region-to-region routing with forecast regions as rows and input regions as columns. The rightmost plot compares the same-region share $d$ across all 16 regions with the uniform level $1/16$ .

Condition	Init. share	Eff. rank	Position $R^{2}$ (spatial tail)	Gap closed (path on, %)	Gap closed (path off, %)
Special-token ( $\gamma=0$ )	$0.062\pm 0.000$	$6.3\pm 5.2$	$0.14\pm 0.13$	$0.3\pm 2.9$	–
$\gamma=0.05$	$0.071\pm 0.004$	$9.7\pm 12.3$	$0.21\pm 0.24$	$-4.3\pm 35.5$	$0.4\pm 12.9$
$\gamma=0.2$	$0.149\pm 0.023$	$46.1\pm 4.5$	$0.52\pm 0.07$	$40.3\pm 2.7$	$19.7\pm 3.4$
$\gamma=1$	$0.484\pm 0.056$	$50.9\pm 9.3$	$0.50\pm 0.11$	$42.4\pm 1.9$	$16.4\pm 5.4$
Vision-token	$0.673\pm 0.041$	$36.0\pm 0.5$	$0.57\pm 0.03$	$45.0\pm 1.2$	–

Appendix

Table C.2: Aligned-path rescue in the controlled special-token model. Values are mean

\pm

standard deviation over five seeds. Init. share denotes the same-region gradient share at initialization. Path off evaluates the trained model after setting

\gamma=0

without further optimization.

Figure C.2: Decoded forecasts under aligned-path rescue. Rows reuse the near-center and spatial-tail examples from the RQ1 figure, and each condition uses its median gap-closure seed. The moderate aligned path predicts the future arm motion. Setting $\gamma=0$ after training leaves a weaker but visible change through the special-token pathway without retraining the model or decoder. Change maps use a common scale within each row.

Condition	Identity	Future disp.	Action chunk	Geometry
Special-token ( $\gamma=0$ )	$0.18\pm 0.04$	$0.33\pm 0.10$	$-0.03\pm 0.40$	$0.24\pm 0.44$
$\gamma=0.05$	$0.14\pm 0.08$	$0.24\pm 0.13$	$0.09\pm 0.41$	$0.40\pm 0.10$
$\gamma=0.2$	$0.27\pm 0.04$	$0.40\pm 0.09$	$0.45\pm 0.20$	$0.36\pm 0.05$
$\gamma=1$	$0.31\pm 0.10$	$0.45\pm 0.15$	$0.40\pm 0.27$	$0.10\pm 0.29$
Vision-token	$0.30\pm 0.03$	$0.45\pm 0.12$	$0.61\pm 0.03$	$0.41\pm 0.05$

Appendix

Table C.3: Additional spatial-tail frozen-feature probes for the aligned-path sensitivity analysis. All readouts use visual tokens and values are mean

\pm

standard deviation over five seeds.

Untied special-token stream
			Spatial-tail $R^{2}$
Condition	Gap closed (%)	Eff. rank	Position	Action
Native credit	$50.8\pm 1.6$	$117.4\pm 7.4$	$0.395\pm 0.060$	$0.249\pm 0.067$
Credit blocked	$42.9\pm 3.7$	$22.9\pm 3.8$	$0.301\pm 0.027$	$0.342\pm 0.105$
Credit shuffled	$14.5\pm 15.1$	$4.1\pm 2.7$	$0.067\pm 0.092$	$0.133\pm 0.206$
Shared block weights

Appendix

Table C.4: Backward-only attention-routing intervention in the RoPE-indexed

M=N

special-token model. Values are mean

\pm

standard deviation over five training seeds. Gap closed denotes decoded-forecast gap closure. The blocked shared-weight forecast uses the four runs with available decoded forecasts.

Figure C.3: Complete frozen-probe profiles for the aligned-path and backward-only interventions. Each radar reports geometry, identity, future displacement, action chunk, and position from the visual-token representation. Solid and dashed outlines denote near-center and spatial-tail evaluation, averaged over five seeds. Identity uses accuracy and the remaining probes use

R^{2}

. Effective rank and forecast gap are shown alongside each profile. (a) Aligned-path strengths and the vision-token reference. (b) Native, blocked, and shuffled attention credit in the shared-weight control, with the corresponding special-token and vision-token references.

Condition	Gap closed (%)	Eff. rank	Position $R^{2}$ (spatial tail)	Action $R^{2}$ (spatial tail)
Vision-token	$45.0\pm 1.2$	$36.0\pm 0.5$	$0.57\pm 0.03$	$0.61\pm 0.03$
Pool-16	$29.1\pm 4.6$	$37.5\pm 6.2$	$0.50\pm 0.10$	$0.26\pm 0.09$
Pool-4	$1.0\pm 1.6$	$12.6\pm 7.9$	$0.19\pm 0.09$	$0.18\pm 0.18$
Pool-1	$1.2\pm 0.7$	$15.7\pm 3.9$	$0.14\pm 0.11$	$-0.17\pm 0.32$
Shuffle	$2.0\pm 0.7$	$14.0\pm 1.9$	$0.17\pm 0.13$	$-0.08\pm 0.17$
Block	$20.2\pm 4.1$	$13.5\pm 1.1$	$0.19\pm 0.01$	$0.30\pm 0.03$

Appendix

Table C.5: Effect of spatially coarsening vision-token forecast credit. Values are mean

\pm

standard deviation over five seeds. The special-token model is a reference trained normally with

M=N=64

Figure D.1: Anchored rotary indexing removes prompt-dependent patch offsets. Native cumulative indexing places the special-token stream after the instruction, so the offset between corresponding special tokens and patches changes with prompt length. Anchoring assigns special token

k

the index

512+k

, preserving an offset of 512 for every prompt. The bottom plot shows the offsets induced by the training instructions, with the native distribution repeated for comparison.

Baseline (no forecast)
	LIBERO	LIBERO-PRO
Variant / suite	none	lang.	pos.	obj.	task	env.
Spatial	$95.7$	$94.0$	$70.0$	$69.3$	$56.7$	$51.0$
Object	$99.0$	$97.7$	$60.7$	$87.0$	$43.7$	$66.0$
Goal	$93.7$	$91.3$	$44.7$	$85.8$	$45.7$	$58.3$
LIBERO-10	$81.0$	$61.0$	$20.0$	$47.0$	$17.0$	$22.7$

Appendix

Table D.1: Per-suite

\pi_{0.5}

simulation results. Success rate (%) for every suite and perturbation, averaged over three training seeds. Averaging each column over the four suites recovers the corresponding result in Table 1 .

Variant	mean	lang.	pos.	obj.	task	env.	mean	Ret.
	LIBERO	LIBERO-PRO
Baseline	$88.2{\scriptstyle\,\pm 1.1}$	$22.1$	$0.0$	$47.3$	$13.9$	$15.1$	$19.7{\scriptstyle\,\pm 1.5}$	$0.221$
Special-token	$90.7{\scriptstyle\,\pm 2.5}$	$22.8$	$0.0$	$45.2$	$\mathbf{14.8}$	$\mathbf{17.4}$	$20.0{\scriptstyle\,\pm 0.6}$	$0.220$
Vision-token	$\mathbf{92.8}{\scriptstyle\,\pm 0.7}$	$\mathbf{32.1}$	$\mathbf{0.1}$	$\mathbf{52.4}$	$13.7$	$15.2$	$\mathbf{22.7}{\scriptstyle\,\pm 0.9}$	$\mathbf{0.243}$

Appendix

Table D.2: SmolVLA simulation results. Values are mean

\pm

standard deviation over three training seeds. Retention is perturbed success divided by unperturbed success. Best values are bold.

Interface	Prefix	L6	L11	L14	L17
Special-token, $M=16$	$0.13$	$0.10$	$0.12$	$0.12$	$0.08$
Special-token, $M=N=256$	$0.31$	$0.43$	$0.55$	$0.61$	$0.29$
Special-token, $M=N=256$ , anchored	$0.55$	$0.67$	$0.77$	$0.80$	$0.82$
Vision-token	$0.70$	$0.80$	$0.88$	$0.91$	$0.96$

Appendix

Table D.3: Spatial delivery of forecast supervision. Same-region share of the forecast-loss gradient across the

\pi_{0.5}

visual stream. Chance is

1/16

. The

M=N=256

rows average three training seeds. Gradients use the final online SigLIP features as targets because the training-time EMA target encoder was not retained.

Target	Baseline	Special-token	Vision-token	Shuffled future
Object displacement	$0.360\pm 0.024$	$0.428\pm 0.030$	$\mathbf{0.504\pm 0.003}$	$0.419\pm 0.026$
EEF displacement	$0.738\pm 0.020$	$0.759\pm 0.007$	$\mathbf{0.793\pm 0.010}$	$0.765\pm 0.004$
Object position	$0.413\pm 0.049$	$0.450\pm 0.048$	$\mathbf{0.545\pm 0.021}$	$0.539\pm 0.020$
Object-to-gripper geometry	$0.508\pm 0.029$	$0.543\pm 0.045$	$\mathbf{0.621\pm 0.046}$	$0.613\pm 0.013$

Appendix

Table D.4: Final-layer visual-token probes under object-position shift. Values are probe

R^{2}

, reported as mean

\pm

standard deviation over three training seeds and three LIBERO suites.

	Future displacement $R^{2}$			Position $R^{2}$
Variant	L10	L12	L14	L12
Variable index	$0.353\pm 0.040$	$0.410\pm 0.052$	$0.430\pm 0.035$	$0.473\pm 0.101$
Anchored index	$\mathbf{0.460\pm 0.040}$	$\mathbf{0.467\pm 0.025}$	$\mathbf{0.467\pm 0.025}$	$0.357\pm 0.025$

Appendix

Table D.5: Representation probes for the

M=N=256

controls. Values are mean

\pm

standard deviation over three matched seeds. Future displacement is reported across layers, with current object position at layer 12 as a control.

Task	Training	Held out	Median length
Pick from clutter	219	24	$7.6$ s
Pick two in order	89	10	$13.4$ s
Open pot and place	90	10	$15.7$ s
Total	398	44

Appendix

Table E.1: Real-robot training data. The held-out episodes are excluded from policy training and used for every representation probe.

Figure E.1: The three collection tasks. Each row shows the start frame and the two gripper events from one demonstration. The pot task adds a small reflective lid knob and a longer manipulation sequence to the object-placement behavior shared by the tasks.

Figure E.2: Spatial coverage of the training data. Left: staged object and container positions across the 442 demonstrations. Right: positions at which the robot grasps the pot lid, shown at the scale of the lid and its knob.

Condition	Task	Blocks	Changed factor
In distribution	bowl tasks	7	none
Blurred camera	bowl tasks	10	main-camera image
Unseen pot layout	pot task	11	layout and lid grasp

Appendix

Table E.2: Real-robot evaluation conditions. Each block is one matched instruction and layout evaluated three times by every policy.

Figure E.3: Real-robot evaluation conditions. One matched block from the in-distribution, blurred-camera, and unseen-pot conditions. The orange square marks the main-camera crop supplied to the policy.

Task	Eval. rounds	Eval. median	Training median
Pot	33	$51$ mm	$27$ mm
Bowl pick	19	$32$ mm	$37$ mm

Appendix

Table E.3: Distance of evaluation layouts from training. Distances are measured in the main-camera image and converted to millimeters.

Policy	ID (7)	Blur (10)	Pot (11)	OOD (21)
Baseline	$71\,[57,86]$	$53\,[33,73]$	$39\,[21,58]$	$46\,[32,60]$
Special, $M=16$	$86\,[71,95]$	$73\,[50,90]$	$33\,[15,52]$	$52\,[37,68]$
Special, $M=N=256$	$76\,[57,90]$	$\mathbf{83}\,[67,100]$	$30\,[9,52]$	$56\,[38,73]$
Special, $M=N=256$ , anchored	$81\,[62,95]$	$77\,[50,100]$	$42\,[27,61]$	$59\,[43,75]$
Vision-token	$\mathbf{95}\,[86,100]$	$80\,[63,97]$	$\mathbf{64}\,[42,82]$	$\mathbf{71}\,[57,84]$

Appendix

Table E.4: Blind real-robot success. Each entry gives success percentage with a 95% interval from resampling staged blocks. The number of blocks appears in parentheses, and each policy runs three trials per block. OOD pools the blurred-camera and unseen-pot conditions. Bold marks the highest point estimate in each column.

Future, $0.8$ s ahead ( $R^{2}$ )
Readout	Baseline	Special 16	Special 256	Anchored 256	Vision
Gripper width	0.435	0.606	0.561	0.566	0.688
Joint motion	0.647	0.735	0.701	0.713	0.749
Present state
Container position (mm error)	57.0	53.5	49.5	48.7	49.2
Object position (mm error)	58.0	56.1	60.5	55.2	54.1

Appendix

Table E.5: Information retained in the real-robot visual stream. Ridge readouts use main-camera tokens and episode-grouped cross-validation on the held-out episodes. Special 16 and Special 256 use native special-token indices, while Anchored 256 uses anchored indices. Higher is better for

R^{2}

, and lower is better for position error. Bold marks the best point estimate in each row. Uncertainty statements in the text use 95% bootstrap intervals over episodes.

Explore similar work

Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead
Sep 29, 2026Junghyun Kim, Ngseo Kim, ChungWoo Lee +7Latent Prediction Spurious Correlations
OpenMind, San Francisco, CA, USA · Seoul National University, Seoul, Korea · Hyundai Motors, Korea +2
FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models
Jul 16, 2026Wei Li, Peijin Jia, Yuan Ma +9Photometric Supervision Foresight
LiAuto · University of Chinese Academy of Sciences · Tsinghua University
Test-Time Training for Visual Foresight Vision-Language-Action Models
May 6, 2026Sangwu Park, Wonjoong Kim, Yeonjun In +3Diffusion-Based Vision-Language-Actions Photometric Supervision
KAIST, Seoul, South Korea.

Sep 29, 2026, cs.ROJ/K move · Enter open · S save

Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead

Junghyun Kim, Ngseo Kim, ChungWoo Lee, Seoyeon Lee+6

OpenMind, San Francisco, CA, USA · Seoul National University, Seoul, Korea · Hyundai Motors, Korea+2

Latent Prediction Spurious Correlations

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models remain brittle under visual distribution shifts, often relying on spurious correlations tied to domain-specific factors rather than task-relevant structure. We propose Domain-Invariant Latent Lookahead (DILL), a representation-learning framework that mitigates shortcut learning in VLA policies. Our key idea is to supervise policies with domain-invariant future latents learned from domain-transformed trajectory data. A Task-Domain Encoder is trained with contrastive objectives and Gaussian disentanglement regularization to separate task-relevant structure from domain-specific visual variation. The learned encoder then provides future latents for VLA policy learning through lookahead prediction and domain disentanglement, encouraging the policy to focus on task-relevant structure rather than incidental visual factors. Counterfactual task-view evaluations show that DILL reduces shortcut reliance, while LIBERO-Plus evaluations demonstrate improved visual robustness, with 69.1% average success, 11.4 percentage points above the strongest baseline. Real-world manipulation experiments further support DILL's applicability beyond controlled simulation. Complementary latent-space diagnostics show that these behavioral gains are accompanied by representations that better preserve task-consistent structure while suppressing domain-specific variation. Our project page is available at https://dill-vla.github.io/.

Abstract

Explore similar work

Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead

FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models

Test-Time Training for Visual Foresight Vision-Language-Action Models