Models of vehicle dynamics learned from logged states and commands complement physics-based models, and latent world models, which predict in a learned representation, are used to plan and train controllers in other domains. Vehicle controllers are usually specified in physical terms: costs, limits, and references depend on position, yaw angle, speed, and yaw rate, and the optimizer compares or differentiates predicted outcomes across nearby commands. A latent model placed in such a controller must therefore let these quantities be recovered and must change its predictions with commands as the vehicle does, and prediction error on its own latent targets measures neither. We contribute a measurement protocol for action-conditioned latent predictors with a physical readout that separately tests retention, physical-neighborhood organization, forecasting, and local response to command perturbations, using an untrained-encoder reference and three matched response paths that locate errors in the representation or the predictor. In a case study of a temporal joint-embedding predictive model trained on signals logged in IPG CarMaker, the representations retain the measured planar outputs, though an untrained encoder of the same architecture retains them slightly better; future-command input improves one-second forecasts with retention nearly unchanged; and responses to small command pulses diverge from the simulator already in latent coordinates, raising regret when choosing among nearby commands in all comparisons. Updating the predictor on responses corrects them locally at a cost in forecast accuracy. Measuring retention, forecasting, and local response separately is thus what qualifies a predictive latent as a candidate model for control, and the protocol provides the basis for its closed-loop evaluation.
Figures & tables
Figure 1: (a) Case study. The plant and its driver generate the recorded channels. The online encoder e and the controlled predictor ph forecast the target encoder’s embedding of the masked future window, and switches mark the masked predictor commands of arms C and D and the target mask M of arm E. (b) Measurement protocol for any encoder, target encoder, command-conditioned predictor, and frozen readouts d and g . Value lenses use logged data, and forecasting establishes predictive competence before responses are read. The response lens applies matched command pulses from a common initial condition and compares the plant output ( Y ), the decoded target embedding of the realized future window ( T ), and the decoded prediction ( P ), and history pulses perturb past commands, which changes Ht and the plant state.
Source R2
1 s
Neighbor
Latent wins
Phys. wins
Arm
lin.
nonlin.
untr.
fcst.
overlap
0.5 s
1 s
1 s
A
0.929
0.973
0.988
0.821
0.267
12/18
18/18
5/18
B
0.913
0.968
0.984
0.869
0.267
18/18
18/18
7/18
C
0.931
0.974
0.988
0.819
0.266
13/18
18/18
2/18
D
0.913
0.972
0.984
0.757
0.264
13/18
18/18
5/18
E
0.931
0.974
0.984
0.887
0.313
18/18
18/18
11/18
Table 1: Common-panel medians over 18 fits per arm. Source R2 is the current-output readout, and untr. is a nonlinear readout fitted by the same recipe on an untrained encoder (shared by arms A/C and by B/D/E). Forecast is the one-second decoded prediction. Latent and physical wins count fits that beat target-encoder and physical persistence (Section 4 ).
Arm
Family
Tgt/read. rel. err.
Pred/read. rel. err.
Pred. cosine
Pred. gain
A
future
1.878
1.028
0.172
0.320
A
history
0.510
6.785
0.075
6.723
B
future
2.202
1.022
0.197
0.472
B
history
0.623
8.193
0.057
8.154
C
history
0.536
7.007
0.054
6.769
D
history
0.561
8.761
0.066
9.001
Table 2: One-second vehicle-balanced three-path response agreement. Relative error and gain are with respect to the simulator path. Arms C and D have zero future-pulse response by construction, so only their history rows are shown.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Training
Validation
Test
Collection
1
2
3
4
1
2
1
2
Nominal
5/160
500/170
1000/190
1500/200
700/175
1750/185
1300/180
2200/190
Varied start/speed
80/185
650/195
1200/205
1850/215
1450/200
2150/210
1000/190
2300/212
Appendix
Table 3: Episode schedule per collection, as start position (m) and requested speed (km/h).
Arm
Collection
Predictor commands
Target commands
A
Nominal
Available
Available
B
Varied start/speed
Available
Available
C
Nominal
Masked
Available
D
Varied start/speed
Masked
Available
E
Varied start/speed
Available
Masked
Appendix
Table 4: Five matched training conditions, with three seeds ( 17 , 29 , 43 ) per vehicle, giving 90 fits.
Vehicle
Powertrain
Driven axles
Wheelbase
Tires (front/rear)
DemoCar Ackermann
Combustion
Front
2.54 m
195/65 R15
Audi R8 Weather
Combustion
Both
2.75 m
245/35 R19, 295/35 R19
CompanyCar EV
Battery electric
Both, per axle
2.62 m
255/35 R20, 295/30 R20
CompanyCar ModularPT P2
P2 hybrid
Both, per axle
2.62 m
255/35 R20, 295/30 R20
MB CClass350e
Parallel hybrid
Rear
2.84 m
225/50 R17, 245/45 R17
McLaren F1
Combustion
Rear
3.58 m
racing
Appendix
Table 5: Vehicle configurations. CompanyCar denotes the IPG CompanyCar models, and McLaren F1 is an open-wheel racing car. Wheelbase is the longitudinal distance between front and rear wheel centers in the vehicle data set.
Group
Signal
CarMaker quantity
Outputs ( 6 )
Fr1-origin position, yaw angle
Car.Fr1.tx , Car.Fr1.ty , Car.Yaw
body center-of-mass velocities, yaw rate
Car.vx , Car.vy , Car.YawRate
Commands ( 3 )
steering-wheel angle, pedal activities
DM.Steer.Ang , DM.Gas , DM.Brake
Auxiliary ( 13 )
body accelerations
Car.ax , Car.ay , Car.az
speed, roll and pitch rates
Car.v , Car.RollVel , Car.PitchVel
road-wheel steering angle
Steer.WhlAng
Appendix
Table 6: Input channels, all sampled at 20 ms.
Figure 2: Predictive competence and current-output retention on the common panel. Left: each fit’s 0.5 - and 1 -second latent prediction MSE divided by its own target-encoder persistence MSE. Dashed lines mark ratio one. Right: linear versus nonlinear frozen source-readout macro R2 for each fit. Orange and blue denote varied-start/speed and nominal collections.
Arm
X
Y
ψ
vx
vy
r
Current
A
0.951
0.954
0.975
0.995
0.968
0.987
B
0.942
0.956
0.974
0.994
0.965
0.986
C
0.956
0.953
0.977
0.995
0.970
0.987
D
0.950
0.962
0.973
0.995
0.966
0.986
E
0.960
0.963
0.984
0.996
0.964
0.986
1 s forecast
A
0.828
0.831
0.850
0.952
0.699
0.872
Appendix
Table 7: Median R2 per output on the common panel ( 18 fits per arm). Top: current outputs from the frozen nonlinear source readout. Bottom: one-second decoded forecasts. Position is (X,Y) , and ψ is the yaw angle.
Figure 3: Geometry diagnostics for individual fits on the common panel. Left: latent/physical neighbor overlap against the shuffled-correspondence overlap. Middle: fraction of physically distant latent-nearest neighbors against the random-pairing fraction. Right: standardized latent versus physical covariance effective rank. Orange and blue denote varied-start/speed and nominal collections.
Figure 4: Paired effects on current-output readout R2 , one-second physical forecast R2 , and neighbor overlap. Each point is the mean of six configuration-level effects after averaging seeds within configuration, and bars are descriptive 95% bootstrap intervals over the six configurations. The vertical dashed line marks zero effect.
Figure 5: Trained minus untrained encoder, same architecture, readout, and neighborhood protocol. Left: current-output readout macro R2 . Right: physical-neighbor overlap. Points are vehicle/collection/seed pairs for arms A and B on each collection’s test episodes, and diamonds are vehicle means. Negative values mean the trained encoder scores below the untrained one.
Figure 6: Current planar outputs measured in CarMaker (black) and recovered by a frozen nonlinear source-latent readout (blue), for the upper median of the 18 varied-start/speed, action-conditioned fits ranked by own-collection nonlinear decoding macro R2 , on the first test episode. The fit was chosen by rank before inspecting traces. Windows are ordered and irregularly spaced, so the horizontal axis is the window index. Training used no reconstruction objective.
Figure 7: Measured position–velocity projections (black) and jointly decoded projections (points, colored by episode order) for the fit of Figure 6 . Body velocities are rotated into the run-initial frame with measured and decoded yaw respectively, Vx=cosψvx−sinψvy and Vy=sinψvx+cosψvy , and connectors mark selected matched-point discrepancies. Agreement of projected shapes concerns values only, and injectivity and dynamics are tested in Sections 5 and 5.2 .
Figure 8: One-second decoded forecast NRMSE minus physical-persistence NRMSE by operating regime on the common panel. Negative values favor the forecast. Left: arm B (commands in targets). Right: arm E (target commands masked). Points are vehicle-level seed averages, and diamonds are medians. Regimes are fixed before scoring. Speed and absolute yaw rate are split at their training medians, braking is brake-pedal activity above 0.02 , and deceleration, neutral, and acceleration are longitudinal acceleration below −0.1 , between −0.1 and 0.1 , and above 0.1 m/s 2 . The neutral bin holds 54 to 113 windows per fit.
Family / horizon
Accepted
Causal null
Unresolved
future, 0.5 s
14/36
18
4
future, 1 s
26/36
0
10
history, 0.5 s
24/36
0
12
history, 1 s
23/36
0
13
Appendix
Table 8: Coverage of simulator reference responses. Accepted cases pass the acceptance criteria. Causal nulls are late future pulses at 0.5 s. Unresolved cases are pedal pulses below the resolution floor or responses that change by more than 15% across pulse sizes.
Figure 9: Future steering finite-pulse responses for a fixed example chosen before inspecting accuracy (Ackermann, arm B, seed 17 , early steering, 1 s endpoint). Curves show changes from the matched zero run as pulse amplitude varies, for the simulator (black), the target and readout path (green), and the predictive path (orange), with repeat standard deviations as error bars (repeated runs produced identical saved endpoint values, so the bars have zero height and are not visible). Aggregate results for all fits are in Table 2 .
Arm
Family
Lat. ratio
Tgt NRMSE
Fcst NRMSE
Pers. NRMSE
Lat. wins
Phys. wins
A
future
0.324
0.055
0.347
0.289
15/18
6/18
A
history
0.361
0.081
0.374
0.270
15/18
5/18
B
future
0.162
0.057
0.288
0.289
17/18
11/18
B
history
0.228
0.074
0.327
0.270
17/18
8/18
C
future
0.393
0.044
0.265
0.289
15/18
9/18
C
history
0.395
0.060
0.373
0.270
14/18
7/18
Appendix
Table 9: Local value competence at 1 s on the simulator-timed families. Medians across six vehicles and three seeds. A latent ratio below one beats matched target-encoder persistence. Physical wins count fits whose forecast error is below their own persistence error, so they compare paired errors.
Arm
Family
Norm ratio (pred./realized)
Rel. error
Cosine
B
future
0.055
0.995
0.312
E
future
0.174
0.990
0.355
B
history
7.664
7.536
0.263
E
history
5.487
5.418
0.387
Appendix
Table 10: Decoder-free response comparison at 1 s: predicted versus realized target-latent response to the same pulses, in training-standardized target coordinates. Medians of vehicle-level medians. A norm ratio of one means equal magnitude and says nothing about direction.
Figure 10: Decoder-free response at 1 s. Left: norm of the predicted latent response against the realized target-latent response to the same pulse (vehicle medians). The dashed line marks equal norm, and direction agreement is in Table 10 . Right: predicted-minus-realized distance to sampled training target latents against physical response error. Predicted latents are usually farther from training targets (median ratios 1.16 – 1.44 ), but the distance does not order the physical errors (within-fit rank correlations between −0.09 and 0.12 ).
Figure 11: Finite-candidate ranking. Vehicle-mean normalized regret of the model-selected command (vertical) against keeping the unperturbed template (horizontal), for arms B and E. Each point combines three seeds and the accepted cases of one vehicle. Points above the diagonal mean the model choice was worse than the template.
Figure 12: Response-targeted predictor updates for the seed- 17 fits of arms A and E. Horizontal: held-out pulse-response MSE after the update divided by before, at the independent anchor. Vertical: one-second forecast RMSE on the original test episodes after divided by before. Colors mark the update objective, and gray segments join the three updates of the same fit. The forecast-preserving variant adds a penalty on changes to the original forecasts.
Figure 13: Train and validation learning curves for the Ackermann, seed- 17 models in arms B and E, with source and target readouts shown separately. Loss values live in each fit’s own target space and are comparable within a fit only.
Criterion
Future 0.5 / 1 s
History 0.5 / 1 s
Arm B error
Arm E error
Scale 10%
14/23
24/23
0.996 / 8.193
0.995 / 5.103
Scale 15% (default)
14/26
24/23
1.022 / 8.193
0.995 / 5.103
Scale 20%
15/28
26/25
1.022 / 8.193
0.995 / 5.103
Scale 25%
15/29
26/26
1.024 / 8.193
0.995 / 5.103
Scale 30%
15/29
27/26
1.024 / 8.193
0.995 / 5.103
Norm floor 10−7 to 10−5
14/26
24/23
1.022 / 8.193
0.995 / 5.103
Appendix
Table 11: Accepted cases and vehicle-balanced one-second predictor relative error (future / history) under alternative acceptance criteria. The default is a 15% scale threshold and a 10−6 norm floor.
0.5 s
1 s
Arm
Path
Rel. err.
Cosine
Gain
Rel. err.
Cosine
Gain
A
predictor
1.004
0.102
0.155
1.030
0.167
0.203
A
target
1.557
0.711
1.675
1.081
0.549
1.458
B
predictor
1.007
0.153
0.343
1.046
0.146
0.342
B
target
1.847
0.725
2.095
1.058
0.572
1.394
C
target
1.651
0.671
1.775
1.247
0.534
1.685
Appendix
Table 12: Second-anchor response agreement for the original fits (future pulses). Predictor and target rows compare decoded responses with the simulator. Latent rows compare predicted with realized target-latent responses, and their gain is the norm ratio.
Sup.
Pred.
Coll.
Family
h
Rel. err.
Cosine
Gain
Rel. err.
Cosine
Gain
A
future
0.5
0.755
0.829
0.774
1.001
0.173
0.194
A
future
1
1.058
0.756
1.263
1.001
0.054
0.208
A
history
0.5
3.073
0.371
3.246
11.375
0.014
11.549
A
history
1
2.596
0.123
2.083
7.680
0.072
7.653
B
future
0.5
0.865
0.744
0.959
1.000
0.123
0.318
Appendix
Table 13: Pulse responses of the supervised forecaster (Sup.) and of the matched latent predictors (Pred.), seed 17 , vehicle-balanced medians.
Readout
Trained
Untrained
Paired gap
Untrained higher
GELU 2×128 , 350 epochs
0.974
0.988
0.015
12/12
GELU 2×256 , 1000 epochs
0.982
0.991
0.008
12/12
GELU 3×512 , 1000 epochs
0.985
0.992
0.005
12/12
Appendix
Table 14: Median source-readout macro R2 at three readout capacities, seed- 17 fits of arms A and B ( 12 pairs).
Arm
Trained
Untrained
Untrained higher ( k=10/30/100 )
A
0.241
0.322
6/6 , 6/6 , 6/6
B
0.247
0.328
6/6 , 6/6 , 6/6
C
0.232
0.322
6/6 , 6/6 , 6/6
D
0.243
0.328
6/6 , 6/6 , 6/6
E
0.317
0.328
3/6 , 3/6 , 5/6
Appendix
Table 15: Common-panel neighbor overlap at k=30 (seed- 17 medians) and the number of vehicles in which the untrained encoder scores higher at k=10 , 30 , and 100 .
Probe
Arm B
Arm E
B − E
Target latent from future commands
0.535 / 0.560
0.354 / 0.387
+0.194 / +0.187 , 6/6 positive
Target latent from source latent
0.954 / 0.889
0.970 / 0.941
−0.017 / −0.045 , 6/6 negative
Target latent from both
0.973 / 0.953
0.978 / 0.963
−0.006 / −0.009 , 6/6 negative
Future commands from target latent
0.929 / 0.933
0.888 / 0.889
+0.045 / +0.044 , 6/6 positive
Appendix
Table 16: Target-content probes, median R2 at 0.5 / 1 s, and the B − E vehicle-effect median with its sign count.
Latent world models are trained to predict future states in a learned representation and are then deployed inside a planner that selects actions by simulating them forward. Current practice adopts the prediction error, the single- or multi-step rollout loss on held-out data, as the training and model-selection objective, on the assumption that a lower prediction error yields better control. We show that this assumption is unreliable for a structural reason: a planner does not query the model on the training distribution but on the states that its candidate actions reach, which generally leave the data manifold, so an error averaged over the data cannot by itself govern control. We therefore reframe the objective as the discrepancy between the predicted and the true plan-cost at the plan the planner commits to, and prove that the planner's suboptimality is bounded by twice this discrepancy, whereas the data-averaged prediction error neither bounds nor tracks it. Under a linear-control premise the discrepancy separates into two terms. The first is a small on-manifold residual, on which the predicted and true dynamics agree and which a spectral tax prices through the non-normality of the latent transition operator. The second is an off-manifold divergence, on which an action carries the state off the manifold and the two dynamics diverge; this divergence is the binding term and is bounded by no data-averaged error. Synthetic operators confirm the pricing formulas, and latent model-predictive control experiments confirm the decoupling: across seeds, the single-step validation error is essentially uncorrelated with control success, whereas a fidelity score on the planner-reachable measure tracks it.
Hanzhe You, Yonggang Zhang, Maohao Ran +6
University of Science and Technology of China · The Hong Kong University of Science and Technology, HKGAI · Hong Kong Baptist University
Latent world models plan by predicting how candidate actions advance learned latent dynamics. In self-predictive models, however, the encoder and predictor are optimized jointly and can co-adapt to latent transitions that are easy to predict but weakly constrained by the physical evolution of the scene. We introduce the cross-predictive JEPA (JEPA-x), which grounds latent dynamics in privileged physical trajectories. JEPA-x treats visual observations and physical states as corresponding views of the same action-conditioned trajectory, advances both through a shared predictor, and matches each prediction to the future representations of both modalities. This encourages the action-conditioned predictor to learn a common transition rule across the two views. Privileged physical state is used only during training, leaving a visual-only model at deployment. Empirical results show that JEPA-x reduces the rollout drift of a newly fitted predictor from 0.361 to 0.104 and increases mean control success from 53.6% to 78.2% on a multi-task suite spanning six evaluation subfamilies. We additionally show that direct physical-state regression improves decodability without improving forecastability or control, indicating that the benefit comes from shaping latent dynamics rather than merely encoding physical variables.
Kehan Wen, Ziming Li, Siyuan Luo +1
Department of Electrical and Computer Engineering, NUS
Learning intrinsic dynamics---the relationship between actions and their consequences---is essential for world models. However, visual variations unrelated to the underlying control process can obscure this relationship in learned representations. We investigate the connection between action predictability, defined as the ability to reconstruct an action from consecutive latent states, and the preservation of intrinsic dynamics. Our central hypothesis is that encouraging action predictability discourages representations from discarding the distinctions between different actions and their consequences. We provide two complementary forms of evidence for this connection. Theoretically, we derive a positive lower bound on squared inverse prediction error when the encoder merges distinguishable outcomes of different actions from the same state and appearance, and show that forward prediction can still be exact on those transitions. Empirically, controlled appearance-diversity comparisons, supervision interventions, and latent probes support the role of action prediction in preserving control-relevant transition information. We instantiate action predictability with Concat IDM, which predicts actions from concatenated latent states and updates the shared encoder using action labels already available in offline trajectories, without changing the test-time planner. Across three independent training runs on Push-T with 24 appearances, Concat IDM increases planning success from 9.1% to 77.2%; its paired improvement is 52.9 percentage points larger than that observed with a single appearance. Together, these findings support action predictability as an inductive bias toward intrinsic dynamics and demonstrate the practical benefits of Concat IDM for latent planning under visual diversity.
Jiamin Shi, Yuanhong Zhang, Zhiyang Liu +2
Xi’an Jiaotong University · National University of Singapore · Beijing Zhongguancun Academy