What Do Latent Predictive Vehicle Representations Retain? Measuring State, Geometry, and Local Response

Enzo Nicolás Spotorno; Josafat Leal Filho; Antônio Augusto Fröhlich

cs.LGSep 26, 2026

What Do Latent Predictive Vehicle Representations Retain? Measuring State, Geometry, and Local Response

Authors: Enzo Nicolás Spotorno, Josafat Leal Filho, Antônio Augusto Fröhlich

Organizations: Department of Informatics and Statistics Federal University of Santa Catarina

Abstract

Models of vehicle dynamics learned from logged states and commands complement physics-based models, and latent world models, which predict in a learned representation, are used to plan and train controllers in other domains. Vehicle controllers are usually specified in physical terms: costs, limits, and references depend on position, yaw angle, speed, and yaw rate, and the optimizer compares or differentiates predicted outcomes across nearby commands. A latent model placed in such a controller must therefore let these quantities be recovered and must change its predictions with commands as the vehicle does, and prediction error on its own latent targets measures neither. We contribute a measurement protocol for action-conditioned latent predictors with a physical readout that separately tests retention, physical-neighborhood organization, forecasting, and local response to command perturbations, using an untrained-encoder reference and three matched response paths that locate errors in the representation or the predictor. In a case study of a temporal joint-embedding predictive model trained on signals logged in IPG CarMaker, the representations retain the measured planar outputs, though an untrained encoder of the same architecture retains them slightly better; future-command input improves one-second forecasts with retention nearly unchanged; and responses to small command pulses diverge from the simulator already in latent coordinates, raising regret when choosing among nearby commands in all comparisons. Updating the predictor on responses corrects them locally at a cost in forecast accuracy. Measuring retention, forecasting, and local response separately is thus what qualifies a predictive latent as a candidate model for control, and the protocol provides the basis for its closed-loop evaluation.

Figures & tables

Figure 1: (a) Case study. The plant and its driver generate the recorded channels. The online encoder $e$ and the controlled predictor $p_{h}$ forecast the target encoder’s embedding of the masked future window, and switches mark the masked predictor commands of arms C and D and the target mask $M$ of arm E. (b) Measurement protocol for any encoder, target encoder, command-conditioned predictor, and frozen readouts $d$ and $g$ . Value lenses use logged data, and forecasting establishes predictive competence before responses are read. The response lens applies matched command pulses from a common initial condition and compares the plant output ( $Y$ ), the decoded target embedding of the realized future window ( $T$ ), and the decoded prediction ( $P$ ), and history pulses perturb past commands, which changes $H_{t}$ and the plant state.

	Source $R^{2}$			$1$ s	Neighbor	Latent wins		Phys. wins
Arm	lin.	nonlin.	untr.	fcst.	overlap	$0.5$ s	$1$ s	$1$ s
A	$0.929$	$0.973$	$0.988$	$0.821$	$0.267$	$12/18$	$18/18$	$5/18$
B	$0.913$	$0.968$	$0.984$	$0.869$	$0.267$	$18/18$	$18/18$	$7/18$
C	$0.931$	$0.974$	$0.988$	$0.819$	$0.266$	$13/18$	$18/18$	$2/18$
D	$0.913$	$0.972$	$0.984$	$0.757$	$0.264$	$13/18$	$18/18$	$5/18$
E	$0.931$	$0.974$	$0.984$	$0.887$	$0.313$	$18/18$	$18/18$	$11/18$

Table 1: Common-panel medians over

18

fits per arm. Source

R^{2}

is the current-output readout, and untr. is a nonlinear readout fitted by the same recipe on an untrained encoder (shared by arms A/C and by B/D/E). Forecast is the one-second decoded prediction. Latent and physical wins count fits that beat target-encoder and physical persistence (Section 4 ).

Arm	Family	Tgt/read. rel. err.	Pred/read. rel. err.	Pred. cosine	Pred. gain
A	future	$1.878$	$1.028$	$0.172$	$0.320$
A	history	$0.510$	$6.785$	$0.075$	$6.723$
B	future	$2.202$	$1.022$	$0.197$	$0.472$
B	history	$0.623$	$8.193$	$0.057$	$8.154$
C	history	$0.536$	$7.007$	$0.054$	$6.769$
D	history	$0.561$	$8.761$	$0.066$	$9.001$

Table 2: One-second vehicle-balanced three-path response agreement. Relative error and gain are with respect to the simulator path. Arms C and D have zero future-pulse response by construction, so only their history rows are shown.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

	Training				Validation		Test
Collection	1	2	3	4	1	2	1	2
Nominal	5/160	500/170	1000/190	1500/200	700/175	1750/185	1300/180	2200/190
Varied start/speed	80/185	650/195	1200/205	1850/215	1450/200	2150/210	1000/190	2300/212

Appendix

Table 3: Episode schedule per collection, as start position (m) and requested speed (km/h).

Arm	Collection	Predictor commands	Target commands
A	Nominal	Available	Available
B	Varied start/speed	Available	Available
C	Nominal	Masked	Available
D	Varied start/speed	Masked	Available
E	Varied start/speed	Available	Masked

Appendix

Table 4: Five matched training conditions, with three seeds (

17

29

43

) per vehicle, giving

90

fits.

Vehicle	Powertrain	Driven axles	Wheelbase	Tires (front/rear)
DemoCar Ackermann	Combustion	Front	$2.54$ m	195/65 R15
Audi R8 Weather	Combustion	Both	$2.75$ m	245/35 R19, 295/35 R19
CompanyCar EV	Battery electric	Both, per axle	$2.62$ m	255/35 R20, 295/30 R20
CompanyCar ModularPT P2	P2 hybrid	Both, per axle	$2.62$ m	255/35 R20, 295/30 R20
MB CClass350e	Parallel hybrid	Rear	$2.84$ m	225/50 R17, 245/45 R17
McLaren F1	Combustion	Rear	$3.58$ m	racing

Appendix

Table 5: Vehicle configurations. CompanyCar denotes the IPG CompanyCar models, and McLaren F1 is an open-wheel racing car. Wheelbase is the longitudinal distance between front and rear wheel centers in the vehicle data set.

Group	Signal	CarMaker quantity
Outputs ( $6$ )	Fr1-origin position, yaw angle	Car.Fr1.tx , Car.Fr1.ty , Car.Yaw
	body center-of-mass velocities, yaw rate	Car.vx , Car.vy , Car.YawRate
Commands ( $3$ )	steering-wheel angle, pedal activities	DM.Steer.Ang , DM.Gas , DM.Brake
Auxiliary ( $13$ )	body accelerations	Car.ax , Car.ay , Car.az
	speed, roll and pitch rates	Car.v , Car.RollVel , Car.PitchVel
	road-wheel steering angle	Steer.WhlAng

Appendix

Table 6: Input channels, all sampled at

20

ms.

Figure 2: Predictive competence and current-output retention on the common panel. Left: each fit’s $0.5$ - and $1$ -second latent prediction MSE divided by its own target-encoder persistence MSE. Dashed lines mark ratio one. Right: linear versus nonlinear frozen source-readout macro $R^{2}$ for each fit. Orange and blue denote varied-start/speed and nominal collections.

	Arm	$X$	$Y$	$\psi$	$v_{x}$	$v_{y}$	$r$
Current	A	$0.951$	$0.954$	$0.975$	$0.995$	$0.968$	$0.987$
	B	$0.942$	$0.956$	$0.974$	$0.994$	$0.965$	$0.986$
	C	$0.956$	$0.953$	$0.977$	$0.995$	$0.970$	$0.987$
	D	$0.950$	$0.962$	$0.973$	$0.995$	$0.966$	$0.986$
	E	$0.960$	$0.963$	$0.984$	$0.996$	$0.964$	$0.986$
$1$ s forecast	A	$0.828$	$0.831$	$0.850$	$0.952$	$0.699$	$0.872$

Appendix

Table 7: Median

R^{2}

per output on the common panel (

18

fits per arm). Top: current outputs from the frozen nonlinear source readout. Bottom: one-second decoded forecasts. Position is

(X,Y)

, and

\psi

is the yaw angle.

Figure 3: Geometry diagnostics for individual fits on the common panel. Left: latent/physical neighbor overlap against the shuffled-correspondence overlap. Middle: fraction of physically distant latent-nearest neighbors against the random-pairing fraction. Right: standardized latent versus physical covariance effective rank. Orange and blue denote varied-start/speed and nominal collections.

Figure 4: Paired effects on current-output readout $R^{2}$ , one-second physical forecast $R^{2}$ , and neighbor overlap. Each point is the mean of six configuration-level effects after averaging seeds within configuration, and bars are descriptive 95% bootstrap intervals over the six configurations. The vertical dashed line marks zero effect.

Figure 5: Trained minus untrained encoder, same architecture, readout, and neighborhood protocol. Left: current-output readout macro $R^{2}$ . Right: physical-neighbor overlap. Points are vehicle/collection/seed pairs for arms A and B on each collection’s test episodes, and diamonds are vehicle means. Negative values mean the trained encoder scores below the untrained one.

Figure 6: Current planar outputs measured in CarMaker (black) and recovered by a frozen nonlinear source-latent readout (blue), for the upper median of the $18$ varied-start/speed, action-conditioned fits ranked by own-collection nonlinear decoding macro $R^{2}$ , on the first test episode. The fit was chosen by rank before inspecting traces. Windows are ordered and irregularly spaced, so the horizontal axis is the window index. Training used no reconstruction objective.

Figure 7: Measured position–velocity projections (black) and jointly decoded projections (points, colored by episode order) for the fit of Figure 6 . Body velocities are rotated into the run-initial frame with measured and decoded yaw respectively, $V_{x}=\cos\psi\,v_{x}-\sin\psi\,v_{y}$ and $V_{y}=\sin\psi\,v_{x}+\cos\psi\,v_{y}$ , and connectors mark selected matched-point discrepancies. Agreement of projected shapes concerns values only, and injectivity and dynamics are tested in Sections 5 and 5.2 .

Figure 8: One-second decoded forecast NRMSE minus physical-persistence NRMSE by operating regime on the common panel. Negative values favor the forecast. Left: arm B (commands in targets). Right: arm E (target commands masked). Points are vehicle-level seed averages, and diamonds are medians. Regimes are fixed before scoring. Speed and absolute yaw rate are split at their training medians, braking is brake-pedal activity above $0.02$ , and deceleration, neutral, and acceleration are longitudinal acceleration below $-0.1$ , between $-0.1$ and $0.1$ , and above $0.1$ m/s 2 . The neutral bin holds $54$ to $113$ windows per fit.

Family / horizon	Accepted	Causal null	Unresolved
future, $0.5$ s	$14/36$	$18$	$4$
future, $1$ s	$26/36$	$0$	$10$
history, $0.5$ s	$24/36$	$0$	$12$
history, $1$ s	$23/36$	$0$	$13$

Appendix

Table 8: Coverage of simulator reference responses. Accepted cases pass the acceptance criteria. Causal nulls are late future pulses at

0.5

s. Unresolved cases are pedal pulses below the resolution floor or responses that change by more than

15\%

across pulse sizes.

Figure 9: Future steering finite-pulse responses for a fixed example chosen before inspecting accuracy (Ackermann, arm B, seed $17$ , early steering, $1$ s endpoint). Curves show changes from the matched zero run as pulse amplitude varies, for the simulator (black), the target and readout path (green), and the predictive path (orange), with repeat standard deviations as error bars (repeated runs produced identical saved endpoint values, so the bars have zero height and are not visible). Aggregate results for all fits are in Table 2 .

Arm	Family	Lat. ratio	Tgt NRMSE	Fcst NRMSE	Pers. NRMSE	Lat. wins	Phys. wins
A	future	$0.324$	$0.055$	$0.347$	$0.289$	$15/18$	$6/18$
A	history	$0.361$	$0.081$	$0.374$	$0.270$	$15/18$	$5/18$
B	future	$0.162$	$0.057$	$0.288$	$0.289$	$17/18$	$11/18$
B	history	$0.228$	$0.074$	$0.327$	$0.270$	$17/18$	$8/18$
C	future	$0.393$	$0.044$	$0.265$	$0.289$	$15/18$	$9/18$
C	history	$0.395$	$0.060$	$0.373$	$0.270$	$14/18$	$7/18$

Appendix

Table 9: Local value competence at

1

s on the simulator-timed families. Medians across six vehicles and three seeds. A latent ratio below one beats matched target-encoder persistence. Physical wins count fits whose forecast error is below their own persistence error, so they compare paired errors.

Arm	Family	Norm ratio (pred./realized)	Rel. error	Cosine
B	future	$0.055$	$0.995$	$0.312$
E	future	$0.174$	$0.990$	$0.355$
B	history	$7.664$	$7.536$	$0.263$
E	history	$5.487$	$5.418$	$0.387$

Appendix

Table 10: Decoder-free response comparison at

1

s: predicted versus realized target-latent response to the same pulses, in training-standardized target coordinates. Medians of vehicle-level medians. A norm ratio of one means equal magnitude and says nothing about direction.

Figure 10: Decoder-free response at $1$ s. Left: norm of the predicted latent response against the realized target-latent response to the same pulse (vehicle medians). The dashed line marks equal norm, and direction agreement is in Table 10 . Right: predicted-minus-realized distance to sampled training target latents against physical response error. Predicted latents are usually farther from training targets (median ratios $1.16$ – $1.44$ ), but the distance does not order the physical errors (within-fit rank correlations between $-0.09$ and $0.12$ ).

Figure 11: Finite-candidate ranking. Vehicle-mean normalized regret of the model-selected command (vertical) against keeping the unperturbed template (horizontal), for arms B and E. Each point combines three seeds and the accepted cases of one vehicle. Points above the diagonal mean the model choice was worse than the template.

Figure 12: Response-targeted predictor updates for the seed- $17$ fits of arms A and E. Horizontal: held-out pulse-response MSE after the update divided by before, at the independent anchor. Vertical: one-second forecast RMSE on the original test episodes after divided by before. Colors mark the update objective, and gray segments join the three updates of the same fit. The forecast-preserving variant adds a penalty on changes to the original forecasts.

Figure 13: Train and validation learning curves for the Ackermann, seed- $17$ models in arms B and E, with source and target readouts shown separately. Loss values live in each fit’s own target space and are comparable within a fit only.

Criterion	Future $0.5$ / $1$ s	History $0.5$ / $1$ s	Arm B error	Arm E error
Scale $10\%$	$14/23$	$24/23$	$0.996$ / $8.193$	$0.995$ / $5.103$
Scale $15\%$ (default)	$14/26$	$24/23$	$1.022$ / $8.193$	$0.995$ / $5.103$
Scale $20\%$	$15/28$	$26/25$	$1.022$ / $8.193$	$0.995$ / $5.103$
Scale $25\%$	$15/29$	$26/26$	$1.024$ / $8.193$	$0.995$ / $5.103$
Scale $30\%$	$15/29$	$27/26$	$1.024$ / $8.193$	$0.995$ / $5.103$
Norm floor $10^{-7}$ to $10^{-5}$	$14/26$	$24/23$	$1.022$ / $8.193$	$0.995$ / $5.103$

Appendix

Table 11: Accepted cases and vehicle-balanced one-second predictor relative error (future / history) under alternative acceptance criteria. The default is a

15\%

scale threshold and a

10^{-6}

norm floor.

		$0.5$ s			$1$ s
Arm	Path	Rel. err.	Cosine	Gain	Rel. err.	Cosine	Gain
A	predictor	$1.004$	$0.102$	$0.155$	$1.030$	$0.167$	$0.203$
A	target	$1.557$	$0.711$	$1.675$	$1.081$	$0.549$	$1.458$
B	predictor	$1.007$	$0.153$	$0.343$	$1.046$	$0.146$	$0.342$
B	target	$1.847$	$0.725$	$2.095$	$1.058$	$0.572$	$1.394$
C	target	$1.651$	$0.671$	$1.775$	$1.247$	$0.534$	$1.685$

Appendix

Table 12: Second-anchor response agreement for the original fits (future pulses). Predictor and target rows compare decoded responses with the simulator. Latent rows compare predicted with realized target-latent responses, and their gain is the norm ratio.

			Sup.			Pred.
Coll.	Family	$h$	Rel. err.	Cosine	Gain	Rel. err.	Cosine	Gain
A	future	$0.5$	$0.755$	$0.829$	$0.774$	$1.001$	$0.173$	$0.194$
A	future	$1$	$1.058$	$0.756$	$1.263$	$1.001$	$0.054$	$0.208$
A	history	$0.5$	$3.073$	$0.371$	$3.246$	$11.375$	$0.014$	$11.549$
A	history	$1$	$2.596$	$0.123$	$2.083$	$7.680$	$0.072$	$7.653$
B	future	$0.5$	$0.865$	$0.744$	$0.959$	$1.000$	$0.123$	$0.318$

Appendix

Table 13: Pulse responses of the supervised forecaster (Sup.) and of the matched latent predictors (Pred.), seed

17

, vehicle-balanced medians.

Readout	Trained	Untrained	Paired gap	Untrained higher
GELU $2\times 128$ , $350$ epochs	$0.974$	$0.988$	$0.015$	$12/12$
GELU $2\times 256$ , $1000$ epochs	$0.982$	$0.991$	$0.008$	$12/12$
GELU $3\times 512$ , $1000$ epochs	$0.985$	$0.992$	$0.005$	$12/12$

Appendix

Table 14: Median source-readout macro

R^{2}

at three readout capacities, seed-

17

fits of arms A and B (

12

pairs).

Arm	Trained	Untrained	Untrained higher ( $k=10/30/100$ )
A	$0.241$	$0.322$	$6/6$ , $6/6$ , $6/6$
B	$0.247$	$0.328$	$6/6$ , $6/6$ , $6/6$
C	$0.232$	$0.322$	$6/6$ , $6/6$ , $6/6$
D	$0.243$	$0.328$	$6/6$ , $6/6$ , $6/6$
E	$0.317$	$0.328$	$3/6$ , $3/6$ , $5/6$

Appendix

Table 15: Common-panel neighbor overlap at

k=30

(seed-

17

medians) and the number of vehicles in which the untrained encoder scores higher at

k=10

30

, and

100

Probe	Arm B	Arm E	B $-$ E
Target latent from future commands	$0.535$ / $0.560$	$0.354$ / $0.387$	$+0.194$ / $+0.187$ , $6/6$ positive
Target latent from source latent	$0.954$ / $0.889$	$0.970$ / $0.941$	$-0.017$ / $-0.045$ , $6/6$ negative
Target latent from both	$0.973$ / $0.953$	$0.978$ / $0.963$	$-0.006$ / $-0.009$ , $6/6$ negative
Future commands from target latent	$0.929$ / $0.933$	$0.888$ / $0.889$	$+0.045$ / $+0.044$ , $6/6$ positive

Appendix

Table 16: Target-content probes, median

R^{2}

0.5

1

s, and the B

-

E vehicle-effect median with its sign count.

Latent Dynamics Modeling

Latent dynamics modeling learns compact hidden-state representations and transition rules to predict, explain, and control evolving systems.

See all 253 papers

A Control Theory of Predictability in Latent World Models
Jul 11, 2026Hanzhe You, Yonggang Zhang, Maohao Ran +6Model Predictive Control Latent World Models
University of Science and Technology of China · The Hong Kong University of Science and Technology, HKGAI · Hong Kong Baptist University
JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics
Aug 25, 2026Kehan Wen, Ziming Li, Siyuan Luo +1Latent Dynamics Modeling Joint-Embedding Predictive Architecture
Department of Electrical and Computer Engineering, NUS
Learning Intrinsic Dynamics in Latent World Models
Oct 15, 2026Jiamin Shi, Yuanhong Zhang, Zhiyang Liu +2Latent Dynamics Modeling Latent World Models
Xi’an Jiaotong University · National University of Singapore · Beijing Zhongguancun Academy

Jul 11, 2026J/K move · Enter open · S save

A Control Theory of Predictability in Latent World Models

Hanzhe You, Yonggang Zhang, Maohao Ran, Zhiqin Yang+5

University of Science and Technology of China · The Hong Kong University of Science and Technology, HKGAI · Hong Kong Baptist University

Model Predictive Control Latent World Models World Model Evaluation

Open paper AskPDF arXiv

Latent world models are trained to predict future states in a learned representation and are then deployed inside a planner that selects actions by simulating them forward. Current practice adopts the prediction error, the single- or multi-step rollout loss on held-out data, as the training and model-selection objective, on the assumption that a lower prediction error yields better control. We show that this assumption is unreliable for a structural reason: a planner does not query the model on the training distribution but on the states that its candidate actions reach, which generally leave the data manifold, so an error averaged over the data cannot by itself govern control. We therefore reframe the objective as the discrepancy between the predicted and the true plan-cost at the plan the planner commits to, and prove that the planner's suboptimality is bounded by twice this discrepancy, whereas the data-averaged prediction error neither bounds nor tracks it. Under a linear-control premise the discrepancy separates into two terms. The first is a small on-manifold residual, on which the predicted and true dynamics agree and which a spectral tax prices through the non-normality of the latent transition operator. The second is an off-manifold divergence, on which an action carries the state off the manifold and the two dynamics diverge; this divergence is the binding term and is bounded by no data-averaged error. Synthetic operators confirm the pricing formulas, and latent model-predictive control experiments confirm the decoupling: across seeds, the single-step validation error is essentially uncorrelated with control success, whereas a fidelity score on the planner-reachable measure tracks it.

Abstract

Latent Dynamics Modeling

Explore similar work

A Control Theory of Predictability in Latent World Models

JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics

Learning Intrinsic Dynamics in Latent World Models