Time-JEPA: Grounding Video World Models with Physical Time Series

Shenyi Li; Ziyang Zhang; Wen Wu; Ming Liu; Chao Zhang

cs.CVOct 14, 2026

Time-JEPA: Grounding Video World Models with Physical Time Series

Authors: Shenyi Li, Ziyang Zhang, Wen Wu, Ming Liu, Chao Zhang

Organizations: Harbin Institute of Technology · Shanghai Artificial Intelligence Laboratory · Tsinghua University

Abstract

Video world models learn how visual environments evolve by predicting missing or future content. However, physical-state information can provide useful context for predicting how environments evolve. Prior work incorporates this information through different mechanisms: domain-specific physical laws can complicate multi-domain training, while supervision on individual frames or frame pairs gives limited attention to how physical state evolves over time. We introduce Time-JEPA, which fine-tunes a video world model using low-dimensional, temporally aligned physical time series as auxiliary supervision. Physical time series connect video representations with physical state at corresponding temporal positions, encouraging the model to preserve how physical state evolves. In multiple domains including autonomous driving, robot manipulation, and human motion, Time-JEPA improves current-state decoding, multi-horizon future-state prediction, and kinematic reconstruction over matched video-fine-tuned controls across both condition and state-variable distribution shifts. Multi-domain training provides aggregate gains over matched single-domain training in all three tasks. External evaluations further suggest that time-series supervision improves sensitivity to physical anomalies and the representation of physical interactions, including in unseen scenarios. Aggregate gains on a second video backbone indicate that the method can be applied to more backbones. These results suggest that physical time series help video world models better understand the physical world.

Figures & tables

Figure 1: Time-JEPA training and evaluation. Training combines masked latent prediction with regression of aligned, pooled, and normalized physical state values from full-clip video features. The EMA target encoder supplies latent supervision. Video-only evaluation covers current-state decoding, multi-horizon future-state prediction, kinematic reconstruction, and external physical evaluations.

Figure 2: Video–state data composition. Each column summarizes one domain’s video data for fine-tuning, two-level ID/OOD settings for conditions and state variables, and representative physical time series.

Figure 3: Current-state decoding gains from multi-domain training. (a) Pearson

r

gains and (b)

R^{2}

gains of multi-domain Time-JEPA over matched single-domain training. Colors identify the four variable/condition settings in the legend. Labels report absolute gains.

			autonomous driving			robot manipulation			human motion
Condition	Variables	Metric	Original	Video [-1pt]FT	Time- [-1pt]JEPA	Original	Video [-1pt]FT	Time- [-1pt]JEPA	Original	Video [-1pt]FT	Time- [-1pt]JEPA
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.804	0.817	0.916	0.609	0.639	0.845	0.500	0.503	0.589
	$\mathrm{ID}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.698	0.713	0.849	0.389	0.424	0.725	0.228	0.231	0.345
	$\mathrm{OOD}_{\mathrm{var}}$	$r$ $\uparrow$	0.879	0.885	0.920	0.641	0.654	0.743	0.434	0.434	0.504
	$\mathrm{OOD}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.771	0.783	0.846	0.422	0.440	0.572	0.132	0.133	0.222
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.800	0.812	0.902	0.236	0.243	0.492	0.480	0.486	0.581

Table 1: Current-state decoding across variables and conditions. We compare the three V-JEPA configurations on

\mathrm{ID}_{\mathrm{var}}

and

\mathrm{OOD}_{\mathrm{var}}

state variables under

\mathrm{ID}_{\mathrm{cond}}

and

\mathrm{OOD}_{\mathrm{cond}}

conditions. We compute metrics for each variable and camera view, then average equally over views and over variables. Bold marks the highest score in each comparison. Appendix B gives variable-level and domain results.

Table 5

Condition	Domain	Metric $\downarrow$	Original	Video FT	Time-JEPA
$\mathrm{ID}_{\mathrm{cond}}$	autonomous driving	Trajectory MAE (m)	4.423 $\,\pm\,$ 0.115	3.932 $\,\pm\,$ 0.123	1.795 $\,\pm\,$ 0.027
	robot manipulation	Reach-rate MAE (cm/s)	5.523 $\,\pm\,$ 0.010	5.372 $\,\pm\,$ 0.008	2.290 $\,\pm\,$ 0.006
	human motion	Distance MAE (cm)	6.877 $\,\pm\,$ 0.093	6.815 $\,\pm\,$ 0.110	4.619 $\,\pm\,$ 0.065
$\mathrm{OOD}_{\mathrm{cond}}$	autonomous driving	Trajectory MAE (m)	4.829 $\,\pm\,$ 0.199	4.150 $\,\pm\,$ 0.146	1.858 $\,\pm\,$ 0.144
	robot manipulation	Reach-rate MAE (cm/s)	5.901 $\,\pm\,$ 0.030	5.822 $\,\pm\,$ 0.035	4.171 $\,\pm\,$ 0.046
	human motion	Distance MAE (cm)	7.229 $\,\pm\,$ 0.177	7.120 $\,\pm\,$ 0.210	4.557 $\,\pm\,$ 0.053

Table 4: Primary kinematic reconstruction errors. Mean

\pm

sample standard deviation across five probe seeds. Within each seed, trajectory, reach-rate, and distance MAE average three horizons, two arms, and four limbs, respectively. Video FT and Time-JEPA use multi-domain training. Bold marks the lowest error per row.

Figure 4: External physical evaluations. (a) GRASP Level 2 eight-property macro pair accuracy over 2,048 matched possible–impossible pairs; bars show points above 50% chance and labels give absolute accuracy. (b,c) Physion v1.5 scenario-macro Accuracy and AUROC (%) with chronological input, averaged equally over seven scenarios.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Video input 64 frames at 5 FPS,

384\!\times\!384

crop, patch size 16, tubelet size 2, 32 temporal tokens, and strict temporal sampling.

Sampling All domains use the same 64-frame, 5-FPS temporal context (12.6 s between the first and last sampled frames). Valid start times are constructed from each source timeline: a random offset within each selected autonomous-driving video, or a timestamp-aligned grid with 12.8 s spacing for robot manipulation and human motion.

Augmentation Random resized crop with scale

[0.3,1.0]

and aspect ratio

[0.75,1.35]

. We retain the original horizontal orientation.

Optimization 20 epochs, batch size 8 per GPU, AdamW, learning rate

3\!\times\!10^{-6}\rightarrow 3\!\times\!10^{-5}\rightarrow 10^{-6}

with one warmup epoch and cosine decay, weight decay 0.01, bfloat16, and EMA momentum

0.9995\rightarrow 0.9999

JEPA masks Eight block masks with spatial scale 0.15 and two with spatial scale 0.70. All use temporal scale 1.0.

Time-series supervision A separate unmasked online-encoder pass supplies the final-layer patch grid. A two-layer pointwise nonlinear auxiliary head with GELU regresses aligned states under a masked Smooth-L1 loss (

\beta=0.5

). The loss weight warms up linearly over the first 250 domain steps.

Time-series loss weight

\lambda=0.2

for autonomous driving and

\lambda=0.1

for robot manipulation and human motion. Multi-domain training retains these domain-specific values.

Appendix

Table A1: Matched V-JEPA fine-tuning recipe. Time-JEPA adds the time-series supervision branch and its loss to the shared settings. Domain steps count batches from the corresponding domain.

Condition	Variables	Metric	Original	Video [-1pt]FT	Time	Original	Video [-1pt]FT	Time	Original	Video [-1pt]FT	Time
			autonomous driving			robot manipulation			human motion
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.466	0.563	0.757	0.362	0.457	0.570	0.320	0.370	0.462
	$\mathrm{ID}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.260	0.403	0.654	0.145	0.223	0.342	0.076	0.115	0.193
	$\mathrm{OOD}_{\mathrm{var}}$	$r$ $\uparrow$	0.576	0.730	0.857	0.425	0.495	0.564	0.303	0.311	0.379
	$\mathrm{OOD}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.322	0.526	0.736	0.199	0.265	0.339	0.078	0.068	0.108
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.441	0.552	0.727	0.077	0.040	0.034	0.223	0.286	0.419

Appendix

Table B1: Current-state decoding and transfer with iVideoGPT. The three configurations use one matched probe run at native resolution and Table 1 ’s view/variable aggregation. Bold marks the best unrounded value.

			autonomous driving			robot manipulation			human motion
Condition	Variables	Metric	I-Next- L	VMAE V2-G	Time- JEPA	I-Next- L	VMAE V2-G	Time- JEPA	I-Next- L	VMAE V2-G	Time- JEPA
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.662	0.613	0.916	0.557	0.324	0.845	0.536	0.354	0.589
	$\mathrm{ID}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.475	0.429	0.849	0.345	0.128	0.725	0.279	0.113	0.345
	$\mathrm{OOD}_{\mathrm{var}}$	$r$ $\uparrow$	0.510	0.652	0.920	0.608	0.407	0.743	0.485	0.327	0.504
	$\mathrm{OOD}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.260	0.417	0.846	0.377	0.172	0.572	0.214	0.096	0.222
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.637	0.573	0.902	0.240	0.140	0.492	0.539	0.323	0.581

Appendix

Table B2: Current-state task difficulty and Time-JEPA comparison. We evaluate original pretrained InternVideo-Next-L and VideoMAE V2-G encoders alongside the Time-JEPA encoder at their native input resolutions, using Table 1 ’s splits, supervised-probe protocol, and aggregation. The column labels “I-Next-L,” “VMAE V2-G,” and “Time-JEPA” denote InternVideo-Next-L, VideoMAE V2-G, and Time-JEPA, respectively. Variable and condition notation follows Section 4.2 . Bold marks the best unrounded value among the listed encoders.

		Original			Video FT			Time-JEPA
Condition	State variable	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	Base translation speed	0.459	0.210	0.907	0.461	0.212	0.906	0.529	0.279	0.867
	Base angular speed	0.465	0.216	0.793	0.474	0.224	0.789	0.596	0.352	0.721
	Waist PC1	0.763	0.582	0.647	0.782	0.611	0.624	0.912	0.833	0.407
	Waist PC2	0.803	0.644	0.608	0.813	0.661	0.593	0.899	0.807	0.449
	Left arm position $x$	0.608	0.371	0.768	0.637	0.406	0.746	0.847	0.718	0.510

Appendix

Table B3: robot manipulation:

\mathrm{ID}_{\mathrm{var}}

variables.

		Original			Video FT			Time-JEPA
Condition	State variable	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	Left joint-position norm	0.764	0.583	0.616	0.775	0.600	0.603	0.856	0.732	0.494
	Right joint-position norm	0.715	0.511	0.685	0.736	0.541	0.664	0.850	0.722	0.517
	Left joint-velocity norm	0.723	0.521	0.705	0.756	0.569	0.669	0.907	0.821	0.432
	Right joint-velocity norm	0.783	0.613	0.610	0.806	0.649	0.581	0.920	0.845	0.386
	Left joint-effort norm	0.533	0.283	0.853	0.544	0.294	0.846	0.649	0.420	0.766

Appendix

Table B4: robot manipulation:

\mathrm{OOD}_{\mathrm{var}}

variables.

		Original			Video FT			Time-JEPA
Condition	State variable	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	Longitudinal velocity	0.972	0.942	0.213	0.975	0.949	0.200	0.990	0.979	0.127
	Longitudinal acceleration	0.477	0.225	0.886	0.510	0.256	0.868	0.771	0.594	0.642
	Yaw rate	0.963	0.926	0.280	0.967	0.934	0.266	0.986	0.973	0.170
	Variable macro	0.804	0.698	0.460	0.817	0.713	0.445	0.916	0.849	0.313
$\mathrm{OOD}_{\mathrm{cond}}$	Longitudinal velocity	0.961	0.919	0.261	0.966	0.926	0.249	0.986	0.972	0.152

Appendix

Table B5: autonomous driving:

\mathrm{ID}_{\mathrm{var}}

variables.

		Original			Video FT			Time-JEPA
Condition	State variable	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	Steering angle	0.848	0.717	0.586	0.853	0.725	0.578	0.884	0.781	0.516
	Lateral velocity	0.927	0.859	0.390	0.932	0.867	0.378	0.951	0.904	0.320
	Lateral acceleration	0.861	0.736	0.479	0.872	0.756	0.460	0.925	0.853	0.357
	Variable macro	0.879	0.771	0.485	0.885	0.783	0.472	0.920	0.846	0.398
$\mathrm{OOD}_{\mathrm{cond}}$	Steering angle	0.887	0.786	0.423	0.892	0.795	0.414	0.924	0.854	0.350

Appendix

Table B6: autonomous driving:

\mathrm{OOD}_{\mathrm{var}}

variables.

		Original			Video FT			Time-JEPA
Condition	State variable	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	Pelvis speed	0.582	0.334	0.652	0.589	0.343	0.648	0.696	0.472	0.581
	Left wrist relative speed	0.518	0.259	0.682	0.519	0.260	0.682	0.559	0.306	0.660
	Right wrist relative speed	0.519	0.267	0.613	0.522	0.269	0.613	0.557	0.304	0.598
	Left ankle relative speed	0.256	0.058	0.736	0.257	0.058	0.736	0.323	0.095	0.721
	Right ankle relative speed	0.437	0.167	0.404	0.442	0.168	0.404	0.534	0.234	0.388

Appendix

Table B7: human motion:

\mathrm{ID}_{\mathrm{var}}

variables.

		Original			Video FT			Time-JEPA
Condition	State variable	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$r\uparrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	Inter-wrist distance	0.573	0.316	0.893	0.575	0.320	0.891	0.643	0.400	0.836
	Left wrist–shoulder distance	0.287	0.000	0.669	0.290	0.000	0.669	0.385	0.074	0.644
	Right wrist–shoulder distance	0.472	0.177	0.635	0.471	0.176	0.635	0.533	0.262	0.601
	Inter-ankle distance	0.223	$-$ 0.109	0.370	0.215	$-$ 0.121	0.372	0.247	$-$ 0.089	0.367
	Pelvis support height	0.614	0.277	0.451	0.621	0.289	0.447	0.710	0.462	0.389

Appendix

Table B8: human motion:

\mathrm{OOD}_{\mathrm{var}}

variables.

autonomous driving Fine-tuning: 6,305 clips. Probe train: 1,349 clips. Probe validation: 449 clips.

\mathrm{ID}_{\mathrm{cond}}

test: 900 clips.

\mathrm{OOD}_{\mathrm{cond}}

test: 521 clips from date-disjoint caption-defined stable rain.

robot manipulation Fine-tuning: 15,000 windows. Probe train:

1{,}500\!\times\!3

. Probe validation:

500\!\times\!3

\mathrm{ID}_{\mathrm{cond}}

test:

1{,}000\!\times\!3

\mathrm{OOD}_{\mathrm{cond}}

test:

1{,}000\!\times\!2

from held-out left/right fisheye cameras.

human motion Fine-tuning: 7,000 windows. Probe train: 1,500 windows. Probe validation: 500 windows.

\mathrm{ID}_{\mathrm{cond}}

test:

1{,}000\!\times\!3

\mathrm{OOD}_{\mathrm{cond}}

test: the same 1,000 test windows from held-out exocentric view03 .

Appendix

Table D1: Data roles and evaluated views.

n\!\times\!v

denotes

n

physical windows with

v

views each. For robot manipulation, we expand probe splits over views. For human motion, we sample one seen view per training access, fix one per validation window, and evaluate all three seen views for

\mathrm{ID}_{\mathrm{cond}}

testing. Each autonomous driving clip provides one view.

\mathrm{ID}_{\mathrm{var}}

(3 variables) Longitudinal ego velocity (m/s) Longitudinal ego acceleration (m/s 2 ) Calibrated-frame yaw angular velocity (rad/s)

\mathrm{OOD}_{\mathrm{var}}

(3 variables) Steering-wheel angle (control quantity, degree) Calibrated-frame lateral velocity (m/s) Calibrated-frame lateral acceleration (m/s 2 )

Appendix

Table D2: autonomous driving state variables. Variable groups follow Section 4.2 . Both groups use supervised probes on frozen encoders.

\mathrm{ID}_{\mathrm{var}}

(18 variables) Planar base translation-speed magnitude (1D, m/s) Base angular-speed magnitude (1D, rad/s) First two frozen PCA scores of the 5D waist configuration (2D, PCA score) Signed left/right end-effector

xyz

position (6D, m) Signed left/right end-effector

xyz

velocity in the recorded frame (6D, m/s) Bilateral gripper opening state (2D, native scalar)

\mathrm{OOD}_{\mathrm{var}}

(8 variables) L2 norm of each arm’s 7D joint configuration (2D, rad) L2 norm of each arm’s joint velocity (2D, rad/s) L2 norm of bilateral joint effort (2D, native effort) Bilateral end-effector force magnitude (2D, native force)

Appendix

Table D3: robot manipulation state variables. Variable groups follow Section 4.2 . Parentheses give the number of scalar dimensions in each bilateral or vector group.

\mathrm{ID}_{\mathrm{var}}

(9 variables) Pelvis speed: speed of the midpoint between the left and right hips (1D, m/s) Left/right wrist relative speed: wrist speed after subtracting pelvis translation (2D, m/s) Left/right ankle relative speed: ankle speed after subtracting pelvis translation (2D, m/s) Left/right elbow flexion: interior shoulder–elbow–wrist angle (2D, rad) Left/right knee flexion: interior hip–knee–ankle angle (2D, rad)

\mathrm{OOD}_{\mathrm{var}}

(5 variables) Inter-wrist distance: Euclidean distance between the two wrists (1D, m) Left/right wrist–shoulder distance: Euclidean shoulder-to-wrist endpoint distance (2D, m) Inter-ankle distance: Euclidean distance between the two ankles (1D, m) Pelvis support height: pelvis height relative to the two-foot support reference (1D, m)

Appendix

Table D4: human motion state variables. Variable groups follow Section 4.2 .

Split	Horizon	Encoder	Direct $\downarrow$	Integrated $\downarrow$	AUROC $\uparrow$
$\mathrm{ID}_{\mathrm{cond}}$	1.00 s	Original	2.226 $\,\pm\,$ 0.092	2.199 $\,\pm\,$ 0.057	0.642 $\,\pm\,$ 0.002
$\mathrm{ID}_{\mathrm{cond}}$	1.00 s	Video FT	2.124 $\,\pm\,$ 0.108	2.083 $\,\pm\,$ 0.064	0.649 $\,\pm\,$ 0.003
$\mathrm{ID}_{\mathrm{cond}}$	1.00 s	Time-JEPA	1.408 $\,\pm\,$ 0.096	1.242 $\,\pm\,$ 0.050	0.707 $\,\pm\,$ 0.004
$\mathrm{ID}_{\mathrm{cond}}$	2.00 s	Original	4.728 $\,\pm\,$ 0.097	4.436 $\,\pm\,$ 0.116	0.636 $\,\pm\,$ 0.004
$\mathrm{ID}_{\mathrm{cond}}$	2.00 s	Video FT	4.528 $\,\pm\,$ 0.098	4.194 $\,\pm\,$ 0.129	0.647 $\,\pm\,$ 0.004
$\mathrm{ID}_{\mathrm{cond}}$	2.00 s	Time-JEPA	3.000 $\,\pm\,$ 0.126	2.467 $\,\pm\,$ 0.097	0.724 $\,\pm\,$ 0.004

Appendix

Table E1: autonomous driving trajectory reconstruction by horizon. Direct and integrated errors use endpoint MAE in meters. AUROC compares the correct integration law with the fixed alternatives. Values give mean

\pm

sample standard deviation across five probe seeds. Bold marks the best encoder mean.

Split	Camera	Encoder	Left $\downarrow$	Right $\downarrow$	Pair rate $\uparrow$
$\mathrm{ID}_{\mathrm{cond}}$	Top head	Original	4.130 $\,\pm\,$ 0.022	7.209 $\,\pm\,$ 0.019	0.502 $\,\pm\,$ 0.018
$\mathrm{ID}_{\mathrm{cond}}$	Top head	Video FT	4.094 $\,\pm\,$ 0.016	7.025 $\,\pm\,$ 0.026	0.492 $\,\pm\,$ 0.016
$\mathrm{ID}_{\mathrm{cond}}$	Top head	Time-JEPA	3.137 $\,\pm\,$ 0.045	3.745 $\,\pm\,$ 0.043	0.553 $\,\pm\,$ 0.007
$\mathrm{ID}_{\mathrm{cond}}$	Left hand	Original	3.805 $\,\pm\,$ 0.012	7.397 $\,\pm\,$ 0.022	0.516 $\,\pm\,$ 0.017
$\mathrm{ID}_{\mathrm{cond}}$	Left hand	Video FT	3.751 $\,\pm\,$ 0.018	7.252 $\,\pm\,$ 0.017	0.541 $\,\pm\,$ 0.030
$\mathrm{ID}_{\mathrm{cond}}$	Left hand	Time-JEPA	3.010 $\,\pm\,$ 0.027	3.863 $\,\pm\,$ 0.032	0.616 $\,\pm\,$ 0.019

Appendix

Table E2: robot manipulation reach-rate reconstruction by camera. Left/right columns give MAE in cm/s. Pair rate averages the two arms per camera. Values give mean

\pm

sample standard deviation across five probe seeds. Bold marks the best encoder mean per camera and metric.

		$\mathrm{ID}_{\mathrm{cond}}$			$\mathrm{OOD}_{\mathrm{cond}}$
Limb	Encoder	Direct $\downarrow$	Cosine $\downarrow$	AUROC $\uparrow$	Direct $\downarrow$	Cosine $\downarrow$	AUROC $\uparrow$
Elbow L	Original	7.017 $\,\pm\,$ 0.089	6.657 $\,\pm\,$ 0.103	0.561 $\,\pm\,$ 0.004	7.111 $\,\pm\,$ 0.110	6.868 $\,\pm\,$ 0.178	0.554 $\,\pm\,$ 0.006
Elbow L	Video FT	7.028 $\,\pm\,$ 0.091	6.671 $\,\pm\,$ 0.110	0.562 $\,\pm\,$ 0.004	7.108 $\,\pm\,$ 0.113	6.845 $\,\pm\,$ 0.189	0.556 $\,\pm\,$ 0.006
Elbow L	Time-JEPA	6.759 $\,\pm\,$ 0.091	6.404 $\,\pm\,$ 0.103	0.570 $\,\pm\,$ 0.006	6.790 $\,\pm\,$ 0.136	6.527 $\,\pm\,$ 0.189	0.565 $\,\pm\,$ 0.006
Elbow R	Original	6.786 $\,\pm\,$ 0.025	6.597 $\,\pm\,$ 0.020	0.568 $\,\pm\,$ 0.002	6.880 $\,\pm\,$ 0.117	6.749 $\,\pm\,$ 0.066	0.564 $\,\pm\,$ 0.004
Elbow R	Video FT	6.799 $\,\pm\,$ 0.037	6.628 $\,\pm\,$ 0.023	0.567 $\,\pm\,$ 0.002	6.884 $\,\pm\,$ 0.121	6.771 $\,\pm\,$ 0.072	0.563 $\,\pm\,$ 0.004

Appendix

Table E3: human motion cosine-law distance reconstruction. Direct and cosine-law errors use MAE in cm. AUROC compares the correct relation with three fixed alternatives. Values give mean

\pm

sample standard deviation across five probe seeds. Bold marks the best encoder mean for each limb, split, and metric.

Domain	Reference loader	Shared schedule
autonomous driving	98 steps / epoch	98 steps / epoch
robot manipulation	234 steps / epoch	234 steps / epoch
human motion	109 steps / epoch	109 steps / epoch

Appendix

Table F1: Per-domain schedule quotas. Each entry reports synchronized optimizer steps per epoch for the corresponding domain loader.

Condition	Variables	Metric	Single	Multi	$\Delta$	Single	Multi	$\Delta$	Single	Multi	$\Delta$
			autonomous driving			robot manipulation			human motion
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.916	0.988	+0.072	0.845	0.912	+0.066	0.589	0.698	+0.109
	$\mathrm{ID}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.849	0.976	+0.127	0.725	0.840	+0.115	0.345	0.506	+0.161
	$\mathrm{OOD}_{\mathrm{var}}$	$r$ $\uparrow$	0.920	0.943	+0.023	0.741	0.781	+0.040	0.504	0.669	+0.165
	$\mathrm{OOD}_{\mathrm{var}}$	$R^{2}$ $\uparrow$	0.846	0.890	+0.044	0.568	0.631	+0.062	0.222	0.456	+0.235
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$r$ $\uparrow$	0.902	0.985	+0.083	0.492	0.583	+0.091	0.581	0.702	+0.121

Appendix

Table F2: Single-domain and multi-domain Time-JEPA. Single and Multi denote the corresponding Time-JEPA encoders. We compute changes before rounding and retain the matched supervised probes and variable/view aggregation from Table 1 . Bold marks the higher score.

Figure F1: Paired domain-addition trend in human motion. Connected three-point plots trace the paired sequence trained with human motion only, autonomous driving + human motion, and all three domains. Dark blue reports Pearson

r

and orange reports

R^{2}

. Point labels give the exact scores for all four variable–condition panels.

			autonomous driving		robot manipulation		human motion
Condition	Variables	Metric	Single	Multi	Single	Multi	Single	Multi
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	nRMSE $\downarrow$	0.485	0.349	0.647	0.570	0.999	0.913
	$\mathrm{ID}_{\mathrm{var}}$	$R^{2}\uparrow$	0.719	0.851	0.506	0.587	0.244	0.375
	$\mathrm{OOD}_{\mathrm{var}}$	nRMSE $\downarrow$	0.560	0.488	0.771	0.722	0.978	0.894
	$\mathrm{OOD}_{\mathrm{var}}$	$R^{2}\uparrow$	0.667	0.748	0.379	0.444	0.190	0.321
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	nRMSE $\downarrow$	0.460	0.333	0.960	0.952	1.010	0.902

Appendix

Table F3: Single-domain and multi-domain future-state prediction. Single and Multi denote Time-JEPA encoders. We average the five horizons from 0.2 to 1.0 s equally and use the variable/view aggregation from Appendix G . Each encoder uses one probe run per variable group. Bold marks the better unrounded score.

Condition	Domain	Metric $\downarrow$	Single	Multi
$\mathrm{ID}_{\mathrm{cond}}$	autonomous driving	Trajectory MAE (m)	3.080 $\,\pm\,$ 0.114	1.795 $\,\pm\,$ 0.027
	robot manipulation	Reach-rate MAE (cm/s)	3.454 $\,\pm\,$ 0.016	2.290 $\,\pm\,$ 0.006
	human motion	Distance MAE (cm)	6.230 $\,\pm\,$ 0.080	4.619 $\,\pm\,$ 0.065
$\mathrm{OOD}_{\mathrm{cond}}$	autonomous driving	Trajectory MAE (m)	3.572 $\,\pm\,$ 0.312	1.858 $\,\pm\,$ 0.144
	robot manipulation	Reach-rate MAE (cm/s)	4.710 $\,\pm\,$ 0.025	4.171 $\,\pm\,$ 0.046
	human motion	Distance MAE (cm)	6.423 $\,\pm\,$ 0.129	4.557 $\,\pm\,$ 0.053

Appendix

Table F4: Single-domain and multi-domain kinematic reconstruction. Single and Multi denote Time-JEPA encoders. Entries give mean

\pm

sample standard deviation across five matched probe seeds. We average trajectory error over three horizons, reach-rate error over both arms, and distance error over four limbs within each seed. Bold marks the lower mean.

Domain	Variables	Dims	O	V	T	O	V	T	vs. V	vs. O
			nRMSE $\downarrow$			$R^{2}$ $\uparrow$			Time-JEPA RMSE wins
autonomous driving	$\mathrm{ID}_{\mathrm{var}}$	3	0.580	0.541	0.349	0.630	0.668	0.851	15/15	15/15
	$\mathrm{OOD}_{\mathrm{var}}$	3	0.640	0.598	0.488	0.575	0.625	0.748	15/15	15/15
robot manipulation	$\mathrm{ID}_{\mathrm{var}}$	18	0.821	0.806	0.570	0.260	0.284	0.587	90/90	90/90
	$\mathrm{OOD}_{\mathrm{var}}$	8	0.876	0.865	0.722	0.209	0.228	0.444	40/40	40/40
human motion	$\mathrm{ID}_{\mathrm{var}}$	9	1.022	1.027	0.913	0.204	0.198	0.375	44/45	43/45

Appendix

Table G1: Five-horizon future-state prediction under

\mathrm{ID}_{\mathrm{cond}}

. O/V/T denote original V-JEPA, the video-fine-tuned control, and Time-JEPA. Scores average variables and horizons equally within each domain and group. The final columns count horizon–variable cells with lower Time-JEPA RMSE than V/O (five per dimension). Each configuration and group uses one supervised joint state probe.

Horizon	Original	Video FT	Time-JEPA	Original	Video FT	Time-JEPA	vs. Video FT	vs. Original
	nRMSE $\downarrow$			$R^{2}\uparrow$			Time-JEPA RMSE wins
0.2 s	0.832	0.818	0.612	0.293	0.312	0.582	46/46	45/46
0.4 s	0.876	0.861	0.679	0.282	0.304	0.544	46/46	46/46
0.6 s	0.851	0.839	0.669	0.271	0.288	0.517	46/46	46/46
0.8 s	0.850	0.840	0.682	0.262	0.278	0.495	46/46	46/46
1.0 s	0.905	0.896	0.752	0.247	0.259	0.458	45/46	45/46

Appendix

Table G2: Three-domain

\mathrm{ID}_{\mathrm{cond}}

performance by future horizon. Domains are weighted by variable count. Final columns count variables with lower Time-JEPA RMSE than Video FT/Original; bold marks the best score.

		nRMSE $\downarrow$			$R^{2}\uparrow$
Group	Variable	O	V	T	O	V	T
$\mathrm{ID}_{\mathrm{var}}$	Longitudinal velocity	0.280	0.245	0.110	0.899	0.923	0.984
$\mathrm{ID}_{\mathrm{var}}$	Longitudinal acceleration	0.917	0.894	0.601	0.278	0.313	0.685
$\mathrm{ID}_{\mathrm{var}}$	Yaw rate	0.541	0.483	0.335	0.714	0.769	0.882
$\mathrm{OOD}_{\mathrm{var}}$	Steering angle	0.685	0.638	0.522	0.590	0.643	0.760
$\mathrm{OOD}_{\mathrm{var}}$	Lateral velocity	0.604	0.550	0.435	0.656	0.713	0.818

Appendix

Table G3: autonomous driving: five-horizon future-state prediction by variable. O/V/T denote Original/Video FT/Time-JEPA; scores average five horizons under

\mathrm{ID}_{\mathrm{cond}}

. Bold marks the best configuration.

		nRMSE $\downarrow$			$R^{2}\uparrow$
Group	Variable	O	V	T	O	V	T
$\mathrm{ID}_{\mathrm{var}}$	Pelvis speed	0.871	0.891	0.794	0.302	0.270	0.421
$\mathrm{ID}_{\mathrm{var}}$	Left wrist relative speed	1.008	1.012	1.003	0.214	0.207	0.223
$\mathrm{ID}_{\mathrm{var}}$	Right wrist relative speed	0.819	0.820	0.805	0.210	0.207	0.234
$\mathrm{ID}_{\mathrm{var}}$	Left ankle relative speed	1.844	1.853	1.821	0.092	0.082	0.121
$\mathrm{ID}_{\mathrm{var}}$	Right ankle relative speed	1.188	1.209	1.141	0.176	0.146	0.239

Appendix

Table G4: human motion: five-horizon future-state prediction by variable. O/V/T denote Original/Video FT/Time-JEPA. We average metrics equally over five horizons under

\mathrm{ID}_{\mathrm{cond}}

. Bold marks the best configuration.

		nRMSE $\downarrow$			$R^{2}\uparrow$
Group	Variable	O	V	T	O	V	T
$\mathrm{ID}_{\mathrm{var}}$	Base translation speed	0.886	0.881	0.849	0.108	0.117	0.181
$\mathrm{ID}_{\mathrm{var}}$	Base angular speed	0.754	0.757	0.722	0.104	0.096	0.177
$\mathrm{ID}_{\mathrm{var}}$	Waist PC1	0.557	0.532	0.272	0.700	0.726	0.928
$\mathrm{ID}_{\mathrm{var}}$	Waist PC2	0.574	0.569	0.402	0.697	0.703	0.851
$\mathrm{ID}_{\mathrm{var}}$	Left arm position $x$	0.803	0.796	0.346	0.253	0.266	0.860

Appendix

Table G5: robot manipulation: five-horizon future-state prediction by variable. O/V/T denote Original/Video FT/Time-JEPA; scores average five horizons under

\mathrm{ID}_{\mathrm{cond}}

. Bold marks the best configuration.

Domain	Variables	Original	Video FT	Time-JEPA	Original	Video FT	Time-JEPA
		nRMSE $\downarrow$			$R^{2}$ $\uparrow$
autonomous driving	$\mathrm{ID}_{\mathrm{var}}$	0.548	0.515	0.333	0.580	0.620	0.812
	$\mathrm{OOD}_{\mathrm{var}}$	0.590	0.570	0.449	0.512	0.543	0.711
robot manipulation	$\mathrm{ID}_{\mathrm{var}}$	0.975	0.973	0.952	$-0.012$	$-0.010$	0.006
	$\mathrm{OOD}_{\mathrm{var}}$	1.007	0.980	0.964	$-0.042$	0.013	$-0.003$
human motion	$\mathrm{ID}_{\mathrm{var}}$	1.045	1.047	0.902	0.165	0.162	0.391

Appendix

Table G6: Multi-horizon future-state prediction under

\mathrm{OOD}_{\mathrm{cond}}

by domain and variable group. We average metrics equally over the five horizons. Bold marks the best score for each domain, variable group, and metric.

Figure G1: Long-horizon persistence extension under

\mathrm{ID}_{\mathrm{cond}}

. Panels (a) and (b) compare the eight-horizon Time-JEPA probe with persistence and estimated persistence, respectively. Curves give variable-macro nRMSE over 30

\mathrm{ID}_{\mathrm{var}}

and 16

\mathrm{OOD}_{\mathrm{var}}

variables. Panel (a) uses valid current and future labels; panel (b) uses valid future labels.

Configuration	Pair accuracy (%) $\uparrow$	$\Delta$ vs. Video FT (pp)
Original V-JEPA	50.658	$-4.902$
Video FT	55.560	–
Time-JEPA	57.751	$\mathbf{+2.191}$

Appendix

Table H1: GRASP eight-property macro pair accuracy. We score all 2,048 Level 2 pairs with the fixed

C=2

M=14

protocol. The primary metric averages the eight property accuracies equally. Differences use percentage points relative to Video FT. Bold marks the highest accuracy.

Property	Pairs	Original	Video FT	Time-JEPA	Time-JEPA $-$ Video FT
Object Permanence	384	56.510	65.885	70.052	$+4.167$
Continuity	512	55.078	55.469	56.250	$+0.781$
Color Constancy	256	32.422	58.984	53.125	$-5.859$
Support	128	100.000	99.219	97.656	$-1.563$
Inertia	512	36.328	38.672	41.016	$+2.344$
Gravity	640	42.500	44.219	46.250	$+2.031$

Appendix

Table H2: GRASP pair accuracy by intuitive-physics property. Each row uses the matched pairs assigned to that property. Differences use percentage points relative to Video FT. Bold marks the highest encoder point estimate in each row.

		Chronological		Temporally shuffled
Configuration	Readout setting	Acc. $\uparrow$	AUROC $\uparrow$	Acc. $\uparrow$	AUROC $\uparrow$
Original	All-scenario	71.511	79.573	61.490	67.831
Video FT	All-scenario	75.063	81.361	64.578	72.270
Time-JEPA	All-scenario	75.062	82.543	63.470	72.523
Original	Leave-one- scenario-out	59.368	63.076	54.504	58.146
Video FT	Leave-one- scenario-out	57.952	62.028	54.415	58.647

Appendix

Table H3: Physion scenario-macro Accuracy and AUROC. We average percentages equally over seven scenarios. Chronological and temporally shuffled inputs reuse the same fitted probes. Bold marks the highest value per protocol and metric.

	All-scenario chronological		All-scenario shuffled		Leave-one-scenario-out chronological		Leave-one-scenario-out shuffled
Configuration	Acc. $\uparrow$	AUROC $\uparrow$	Acc. $\uparrow$	AUROC $\uparrow$	Acc. $\uparrow$	AUROC $\uparrow$	Acc. $\uparrow$	AUROC $\uparrow$
Original	71.401	80.100	61.739	67.737	59.034	61.430	54.106	55.592
Video FT	74.686	82.029	64.638	71.675	57.681	60.914	54.010	57.212
Time-JEPA	74.783	83.494	63.768	72.366	60.386	64.056	54.010	57.748

Appendix

Table H4: Physion overall Accuracy and AUROC. Percentages pool all 1,035 test stimuli. Larger scenarios receive more weight than in Table H3 . Chronological and temporally shuffled inputs use the same fitted probes. Bold marks the highest value per condition and metric.

	All-scenario change		Leave-one-scenario-out change		Time-JEPA leave-one-scenario-out
Scenario ( $n$ )	$\Delta$ Acc. $\uparrow$	$\Delta$ AUROC $\uparrow$	$\Delta$ Acc. $\uparrow$	$\Delta$ AUROC $\uparrow$	Acc. $\uparrow$	AUROC $\uparrow$
Collide (105)	$-$ 1.905	+0.472	+6.667	+10.704	66.667	71.662
Contain (153)	+1.961	+3.179	+5.882	+7.915	61.438	68.256
Dominoes (159)	$-$ 1.887	+0.712	+4.403	+9.525	51.572	58.244
Drop (157)	$-$ 1.274	+0.552	$-$ 4.459	$-$ 3.117	58.599	64.188
Link (157)	+1.911	$-$ 1.186	$-$ 1.274	+1.186	56.688	63.759

Appendix

Table H5: Physion results by scenario with chronological input. Changes give Time-JEPA minus Video FT in percentage points. The final two columns give Time-JEPA Accuracy and AUROC in percent. Bold marks the largest value in each column.

Figure I1: human motion auxiliary-head temporal aggregation. Grouped bars show changes from the pointwise nonlinear auxiliary head on the four variable–condition panels. Labels give each three-decimal change. The four solid colors denote the four variable–condition combinations. The left and right panels report Pearson

r

and

R^{2}

, respectively, and the gray vertical reference marks zero change. All frozen encoders use the matched pointwise nonlinear probe.

	$\Delta r\uparrow$		$\Delta R^{2}\uparrow$		nRMSE reduction $\uparrow$
Panel	Linear	Pointwise nonlinear	Linear	Pointwise nonlinear	Linear	Pointwise nonlinear
$\mathrm{ID}_{\mathrm{var}}$ , $\mathrm{ID}_{\mathrm{cond}}$	0.066	0.086	0.084	0.114	0.048	0.068
$\mathrm{ID}_{\mathrm{var}}$ , $\mathrm{OOD}_{\mathrm{cond}}$	0.074	0.096	0.095	0.133	0.055	0.076
$\mathrm{OOD}_{\mathrm{var}}$ , $\mathrm{ID}_{\mathrm{cond}}$	0.045	0.069	0.051	0.089	0.022	0.035
$\mathrm{OOD}_{\mathrm{var}}$ , $\mathrm{OOD}_{\mathrm{cond}}$	0.046	0.063	0.054	0.090	0.023	0.034

Appendix

Table I1: human motion probe-capacity check. Changes compare Time-JEPA with Video FT within each probe family. We report nRMSE reduction as Video FT minus Time-JEPA. Variable and condition notation follows Section 4.2 .

\mathrm{ID}_{\mathrm{cond}}

averages seen exocentric views, and

\mathrm{OOD}_{\mathrm{cond}}

uses held-out exocentric view03 . Bold marks the larger matched gain.

	$\mathrm{ID}_{\mathrm{var}}$ $\mathrm{ID}_{\mathrm{cond}}$		$\mathrm{ID}_{\mathrm{var}}$ $\mathrm{OOD}_{\mathrm{cond}}$		$\mathrm{OOD}_{\mathrm{var}}$ $\mathrm{ID}_{\mathrm{cond}}$		$\mathrm{OOD}_{\mathrm{var}}$ $\mathrm{OOD}_{\mathrm{cond}}$
Encoder supervision	$r\uparrow$	$R^{2}\uparrow$	$r\uparrow$	$R^{2}\uparrow$	$r\uparrow$	$R^{2}\uparrow$	$r\uparrow$	$R^{2}\uparrow$
Video FT	0.503	0.231	0.486	0.199	0.434	0.133	0.452	0.135
Full 9D aligned	0.589	0.345	0.581	0.332	0.504	0.222	0.515	0.225
Pelvis-only 1D aligned	0.546	0.279	0.534	0.256	0.453	0.156	0.469	0.157
Motion-speed 5D aligned	0.551	0.284	0.537	0.259	0.452	0.155	0.470	0.158

Appendix

Table I2: human motion supervision breadth. We average Pearson

r

and

R^{2}

equally over views, then variables. Variable and condition notation follows Section 4.2 . Bold marks the best supervision condition per panel and metric.

Reduced supervision	Variable subset	$\mathrm{ID}_{\mathrm{cond}}$ $\Delta r\uparrow$	$\mathrm{ID}_{\mathrm{cond}}$ $\Delta R^{2}\uparrow$	$\mathrm{OOD}_{\mathrm{cond}}$ $\Delta r\uparrow$	$\mathrm{OOD}_{\mathrm{cond}}$ $\Delta R^{2}\uparrow$
Pelvis-only 1D	Included	+0.013	+0.017	+0.004	$-$ 0.001
	Excluded	$-$ 0.050	$-$ 0.077	$-$ 0.054	$-$ 0.085
Motion-speed 5D	Included	+0.004	+0.005	$-$ 0.001	$-$ 0.001
	Excluded	$-$ 0.091	$-$ 0.144	$-$ 0.098	$-$ 0.164

Appendix

Table I3: Semantic decomposition of reduced supervision. Changes give reduced minus full-aligned performance. Included/Excluded indicate whether a full-model

\mathrm{ID}_{\mathrm{var}}

variable enters the reduced objective.

Condition	Metric	Video FT	Clean alignment	0.8 s offset	Cross-take
$\mathrm{ID}_{\mathrm{cond}}$	$r\uparrow$	0.503	0.589	0.575	0.510
	$R^{2}\uparrow$	0.231	0.345	0.329	0.242
	nRMSE $\downarrow$	0.742	0.674	0.683	0.735
$\mathrm{OOD}_{\mathrm{cond}}$	$r\uparrow$	0.486	0.581	0.570	0.498
	$R^{2}\uparrow$	0.199	0.332	0.320	0.219
	nRMSE $\downarrow$	0.761	0.685	0.692	0.750

Appendix

Table J1: human motion sensitivity to temporal misalignment. The pointwise nonlinear Readout uses clean labels for every frozen encoder. Metrics average views before the nine variables. Bold marks the best unrounded score in each row.

Condition	Metric	Clean	1 $\sigma$ Gaussian noise
$\mathrm{ID}_{\mathrm{cond}}$	$r\uparrow$	0.589	0.580
	$R^{2}\uparrow$	0.345	0.332
	nRMSE $\downarrow$	0.674	0.682
$\mathrm{OOD}_{\mathrm{cond}}$	$r\uparrow$	0.581	0.570
	$R^{2}\uparrow$	0.332	0.315
	nRMSE $\downarrow$	0.685	0.695

Appendix

Table J2: human motion sensitivity to Gaussian sensor noise. The pointwise nonlinear Readout uses clean labels. Metrics average views before the nine variables. Bold marks the better unrounded score in each row.

		Static		Full		Time-JEPA
Condition	Variables	$R^{2}\uparrow$	$r\uparrow$	$R^{2}\uparrow$	$r\uparrow$	$R^{2}\uparrow$	$r\uparrow$
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	0.273	0.493	0.397	0.628	0.725	0.845
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	0.270	0.512	0.373	0.602	0.568	0.741
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$-$ 0.073	0.059	$-$ 0.137	0.118	0.269	0.688
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	$-$ 0.061	0.056	$-$ 0.111	0.123	0.205	0.555

Appendix

Table K1: Current-state decoding in robot manipulation.

\mathrm{ID}_{\mathrm{cond}}

averages three seen views;

\mathrm{OOD}_{\mathrm{cond}}

averages variables across two fisheye views. Higher

R^{2}

and Pearson

r

are better; bold marks the best score.

		Static		Full		Time-JEPA
Condition	Variables	$R^{2}\uparrow$	nRMSE $\downarrow$	$R^{2}\uparrow$	nRMSE $\downarrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	0.217	0.853	0.302	0.804	0.506	0.647
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	0.167	0.900	0.243	0.857	0.379	0.771
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	$-$ 0.081	1.014	$-$ 0.116	1.029	0.100	0.919
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	$-$ 0.053	1.012	$-$ 0.077	1.023	0.047	0.951

Appendix

Table K2: Future-state prediction in robot manipulation.

\mathrm{ID}_{\mathrm{cond}}

uses the head camera;

\mathrm{OOD}_{\mathrm{cond}}

averages variables across two fisheye views. Five horizons span 0.2–1.0 s. Higher

R^{2}

and lower nRMSE are better; bold marks the best score.

		Time-JEPA		Time-JEPA+Law
Condition	Variables	$R^{2}\uparrow$	$r\uparrow$	$R^{2}\uparrow$	$r\uparrow$
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	0.849	0.916	0.809	0.890
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	0.846	0.920	0.831	0.912
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	0.824	0.902	0.780	0.878
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	0.876	0.937	0.863	0.930

Appendix

Table K3: Current-state decoding in autonomous driving. Each variable group averages three variables;

\mathrm{OOD}_{\mathrm{cond}}

is rainy weather. Higher

R^{2}

and Pearson

r

are better; bold marks the better score.

		Time-JEPA		Time-JEPA+Law
Condition	Variables	$R^{2}\uparrow$	nRMSE $\downarrow$	$R^{2}\uparrow$	nRMSE $\downarrow$
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	0.719	0.485	0.699	0.503
$\mathrm{ID}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	0.667	0.560	0.648	0.576
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{ID}_{\mathrm{var}}$	0.680	0.460	0.667	0.469
$\mathrm{OOD}_{\mathrm{cond}}$	$\mathrm{OOD}_{\mathrm{var}}$	0.604	0.529	0.594	0.536

Appendix

Table K4: Future-state prediction in autonomous driving. Scores average three variables per group and five horizons equally;

\mathrm{OOD}_{\mathrm{cond}}

is rainy weather. Higher

R^{2}

and lower nRMSE are better; bold marks the better score.

Metric	Time-JEPA	Time-JEPA+Law	Time-JEPA	Time-JEPA+Law
	$\mathrm{ID}_{\mathrm{cond}}$		$\mathrm{OOD}_{\mathrm{cond}}$
Integration-law AUROC $\uparrow$	0.694 $\,\pm\,$ 0.004	0.683 $\,\pm\,$ 0.005	0.672 $\,\pm\,$ 0.009	0.661 $\,\pm\,$ 0.010
Integrated trajectory MAE (m) $\downarrow$	3.080 $\,\pm\,$ 0.114	3.252 $\,\pm\,$ 0.128	3.572 $\,\pm\,$ 0.312	3.875 $\,\pm\,$ 0.359
Direct trajectory MAE (m) $\downarrow$	3.482 $\,\pm\,$ 0.068	3.534 $\,\pm\,$ 0.066	3.967 $\,\pm\,$ 0.565	4.072 $\,\pm\,$ 0.677
$v$ – $a$ residual, 0.5 s (m/s) $\downarrow$	0.523 $\,\pm\,$ 0.020	0.374 $\,\pm\,$ 0.016	0.658 $\,\pm\,$ 0.038	0.464 $\,\pm\,$ 0.031
$v$ – $a$ residual, 1.0 s (m/s) $\downarrow$	0.695 $\,\pm\,$ 0.027	0.531 $\,\pm\,$ 0.024	0.790 $\,\pm\,$ 0.042	0.594 $\,\pm\,$ 0.038

Appendix

Table K5: Law satisfaction and trajectory reconstruction in autonomous driving. AUROC and trajectory MAE average 1.00, 2.00, and 2.95 s; AUROC also averages five alternatives. Residuals are mean absolute values. Entries are mean

\pm

sample standard deviation over five paired probe seeds on one fixed encoder per method; bold marks the better mean.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 257 papers

May 15, 2026

Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence

Self-supervised video models are increasingly framed as world models, yet they are still evaluated almost entirely on clean video and reported as a final task score, obscuring how their representations behave under the degraded and ambiguous conditions a deployed world model must handle. We present the first systematic study of this component, analyzing four matched-capacity frontier self-supervised learning models that are strong candidates for world-model encoders -- V-JEPA 2.1, V-JEPA 2, VideoPrism, and VideoMAEv2 -- across five robustness axes relevant to their deployment as video world models: feature discriminability, corruption robustness, fine-grained discrimination, occlusion robustness, and sensitivity to temporal direction. We run the study on Something-Something-v2 and repeat every axis that ports on the egocentric EGTEA Gaze+, where the model ordering is unchanged. Our results reveal a distinct and consistent profile for latent-prediction models across all five axes. They degrade more gracefully under pixel corruption, preserve usable class structure rather than mere geometric stability under occlusion, capture fine-grained physical contact cues without reconstructing pixels, and uniquely encode the arrow of time. We finally test whether these representation-level differences matter for downstream world modeling, pairing each frozen encoder with an identical action-conditioned predictor and planner in a simulated manipulation environment. Only latent prediction yields near-complete task success, remains effective under degraded observations, and transfers to a manipulation task for which the predictor was never trained. Our results provide concrete new evidence that latent prediction is a promising foundation for robust world modeling.

Ali J Alrasheed, Aryan Yazdan Parast, Basim Azam +2

The University of Melbourne · Monash University

Video Representation Learning Video Prediction World Model Evaluation Self-Supervised Visual Representation Learning Video World Models

Ask PDF

May 30, 2026

Physical Object Understanding with a Physically Controllable World Model

A central challenge in visual intelligence is learning the physical structure of scenes from raw videos: how regions form objects and the laws that govern their interactions. Solving these tasks requires world models capable of inferring distributional states of the world from partial observations - capabilities that current architectures do not provide. We introduce a new class of probabilistic world models that support estimation of the probability of any visual variable, such as appearance and dynamics, conditioned on any other variables. Here, we identify that these models can be trained efficiently with autoregressive sequence modeling, yielding world models from which rich object understanding emerges. First, we demonstrate that our model captures the physical laws governing how objects move by generating multiple plausible future states of the world through sequential inference. Then, by analyzing motion correlations across these futures, we extract objects and articulated object subparts. Having discovered these objects, we show that our world model can manipulate them in 3D. Finally, we demonstrate how physical relationships between objects can be computed from the world model, enabling applications such as Visual Jenga.

Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen +9

Stanford University · OpenAI · Noetik Inc. +1

Physical Reasoning World Model Learning Object-Centric Representation Learning Video World Models 3D Scene Understanding

Ask PDF

Oct 12, 2026

World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models

Video world models must generate physically plausible futures, not just realistic frames. Existing approaches typically use semantic, physical, or predictive information to guide the visual generation stream, leaving a fundamental question: where inside the generator should predictions of scene evolution live? Standard video tokens jointly encode appearance and dynamics, but provide no distinct state for learning and reading predictive dynamics. We answer this question with the Physical Registry: temporally grounded registry slots inside a video DiT that are initialized from an observation-conditioned forecast, co-evolve with video tokens, and receive future-representation supervision, making predictive dynamics an explicit, readable state of the generator. In a controlled comparison of predictive-state interfaces on Physics-IQ Verified, Physical Registry improves vanilla fine-tuning by 9.9/8.7 points on I2V/V2V, the largest gain among the interfaces we compare. Across all Cosmos and WAN backbones, Physical Registry improves Physics-IQ Verified scores. With only 0.45% additional parameters, it enables Cosmos 3 Nano 16B to surpass the published I2V score of Cosmos 3 Super 64B (43.56 vs. 42.7). On WorldModelBench, it also improves physical adherence and the total score for all four evaluated backbones. Because the registry becomes candidate-specific while remaining readable, the Physical Registry Selector (PRS) ranks emerging futures by how far their intermediate registry readouts depart from the shared forecast before full denoising is complete. Across four backbones, this selection improves I2V scores by 5.6-15.9% without completed-video external scoring. Together, these results support a broader principle: dynamics prediction should live as an explicit model state that co-evolves with the video representation. Project page: https://grainsack.github.io/physical-registry/

Shinseong Kim, Jaeseok Jeong, Jianlong Fu +1

Yonsei University · Microsoft Research Asia

Video World Models Physical Consistency in Video Generation Video Diffusion Models

Ask PDF

Abstract

Video World Models

Explore similar work

Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence

Physical Object Understanding with a Physically Controllable World Model

World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models