Operator-Mismatch Contrastive Learning for World-Relative Trajectory Coherence

One Octadion; Firmansyah Sundana; Novanto Yudistira

cs.LGOct 18, 2026

Operator-Mismatch Contrastive Learning for World-Relative Trajectory Coherence

Authors: One Octadion, Firmansyah Sundana, Novanto Yudistira

Organizations: Faculty of Computer Science, Universitas Brawijaya, Indonesia

Abstract

World models are increasingly judged by how well they rank plausible futures, yet plausibility alone is the wrong target. The same scene can evolve under many latent rules, and a future that is smooth and physically valid under one rule can be incoherent under the rule actually in force. We formalize the resulting failure, operator-blindness, and show that it is systematically hidden by standard evaluation: a scorer can attain near-perfect generic ranking while separating current-world from other-world futures no better than chance. We introduce Operator-Mismatch Contrastive (OMC) learning, whose semantic hard negatives are valid-but-wrong-operator trajectories, and a matching protocol that measures world-relative coherence directly. On a controlled synthetic benchmark with in- and out-of-distribution operator parameters, across two trajectory-scoring backbones, and on feature-level external validation with CoPhy ballsCF, OMC raises operator-sensitive scoring far above a chance baseline that nonetheless achieves 0.99 generic top-1. We prove that the wrong-operator negative is necessary both for the protocol to detect the failure and for the objective to fix it, and position OMC as an auxiliary coherence signal (a trajectory-coherence scorer, not a generative model, planner, or control agent) that can be attached to existing world-model representations.

Figures & tables

Panel of Figure 1 — Figure 1: Operator-blindness. Under operator $A$ , both $\tau_{\mathrm{same}}$ and $\tau_{\mathrm{wrong}}$ (valid under a different operator $B$ ) are physically plausible; only $\tau_{\mathrm{same}}$ is coherent here. A scorer that merely rejects $\tau_{\mathrm{noise}}$ may still fail to separate the dashed pair—and, by Prop. 1 , generic ranking cannot reveal it.

Method family	Learns	Contrastive	Wrong-operator	World-relative
	dynamics	negatives	future	coherence
Latent-dynamics / Dreamer	$\checkmark$	$\times$	$\times$	$\times$
JEPA / I-JEPA	$\sim$	$\sim$	$\times$	$\times$
C-SWM / structured contrastive	$\checkmark$	$\checkmark$	$\times$	$\times$
CPC / InfoNCE family	$\sim$	$\checkmark$	$\times$	$\times$
Physics-informed / HNN / LNN	$\checkmark$	$\times$	$\times$	$\times$

Table 1: Positioning of OMC against related method families (citations in the text of this section). “Wrong-operator future” marks whether a method ever contrasts against a valid future from a different operator; “World-relative coherence” marks whether evaluation tests coherence under the in-force operator (not merely generic plausibility).

\checkmark

yes,

\times

no,

\sim

partial/implicit.

Panel of Figure 2 — Figure 2: OMC pipeline. A context encoder infers $\lambda_{C}$ ; an object-centric energy scorer ranks the candidate set with InfoNCE. The defining choice is the wrong-operator hard negative $\tau_{\mathrm{wrong}}$ . The operator $\times$ trajectory energy matrix / OCCL term is a diagnostic only ( $\alpha_{\text{occl}}{=}0$ ).

Metric	Candidate set	Reads as
top1_acc	$\{\tau_{\mathrm{same}}\}\cup$ easy negs	generic plausibility (diagnostic)
law_pair	full set incl. $\tau_{\mathrm{wrong}}$	$P(\tau_{\mathrm{same}}{<}\tau_{\mathrm{wrong}})$ ; n/a if $\tau_{\mathrm{wrong}}$ absent
same_lt_wrong	$\{\tau_{\mathrm{same}},\tau_{\mathrm{wrong}},\tau_{\mathrm{noise}}\}$	$P(\tau_{\mathrm{same}}{<}\tau_{\mathrm{wrong}})$ , fixed 3-way set
full_stratification	$\{\tau_{\mathrm{same}},\tau_{\mathrm{wrong}},\tau_{\mathrm{noise}}\}$	$P(\tau_{\mathrm{same}}{<}\tau_{\mathrm{wrong}}{<}\tau_{\mathrm{noise}})$
matrix_mrr	$B{\times}B$ operator $\times$ traj.	operator $\leftrightarrow$ trajectory retrieval

Table 2: Metric

\rightarrow

hypothesis map. Each operator-sensitive metric uses a different candidate set; law_pair is defined only when a wrong-operator negative is present.

Backbone	Obj.	Split	top1	law_pair	law_only	same $<$ wr	mat_mrr
LOWM	OMC	IID	0.375 $\pm$ 0.139	0.694 $\pm$ 0.005	0.411 $\pm$ 0.003	0.697 $\pm$ 0.006	0.272 $\pm$ 0.012
LOWM	OMC	OOD	0.363 $\pm$ 0.116	0.650 $\pm$ 0.006	0.359 $\pm$ 0.011	0.665 $\pm$ 0.009	0.271 $\pm$ 0.009
LOWM	no-law	IID	0.996 $\pm$ 0.001	0.000	0.196 $\pm$ 0.013	0.507 $\pm$ 0.010	0.213 $\pm$ 0.003
LOWM	no-law	OOD	0.991 $\pm$ 0.001	0.000	0.205 $\pm$ 0.004	0.514 $\pm$ 0.007	0.209 $\pm$ 0.004
direct	OMC	IID	0.574 $\pm$ 0.014	0.709 $\pm$ 0.008	0.419 $\pm$ 0.008	0.723 $\pm$ 0.002	0.262 $\pm$ 0.001
direct	OMC	OOD	0.562 $\pm$ 0.023	0.700 $\pm$ 0.012	0.409 $\pm$ 0.018	0.705 $\pm$ 0.011	0.259 $\pm$ 0.005

Table 3: Main synthetic results (

5

seeds, mean

\pm

std; matrix_size

=16

). law_pair , same_lt_wrong , matrix_mrr are primary; top1 is diagnostic.

Objective	Split	top1	law_pair	law_gap	same $<$ wr	full str.	mat_mrr
all-neg (main OMC)	IID	0.375 $\pm$ 0.139	0.694 $\pm$ 0.005	1.30 $\pm$ 0.60	0.697 $\pm$ 0.006	0.697 $\pm$ 0.006	0.272 $\pm$ 0.012
all-neg (main OMC)	OOD	0.363 $\pm$ 0.116	0.650 $\pm$ 0.006	1.21 $\pm$ 0.61	0.665 $\pm$ 0.009	0.665 $\pm$ 0.009	0.271 $\pm$ 0.009
no-law-mismatch	IID	0.996 $\pm$ 0.001	0.000	0.00	0.507 $\pm$ 0.010	0.507 $\pm$ 0.010	0.213 $\pm$ 0.003
no-law-mismatch	OOD	0.991 $\pm$ 0.001	0.000	0.00	0.514 $\pm$ 0.007	0.514 $\pm$ 0.007	0.209 $\pm$ 0.004
law-mismatch only	IID	0.416 $\pm$ 0.007	0.700 $\pm$ 0.004	2.17 $\pm$ 0.08	0.712 $\pm$ 0.004	0.676 $\pm$ 0.011	0.284 $\pm$ 0.012
law-mismatch only	OOD	0.403 $\pm$ 0.003	0.695 $\pm$ 0.003	2.11 $\pm$ 0.11	0.692 $\pm$ 0.006	0.660 $\pm$ 0.013	0.286 $\pm$ 0.013

Table 4: Negative-type ablation (LOWM,

5

seeds, matrix_size

=16

Objective	Split	top1	law_pair	same $<$ wr	mat_mrr	mat_top1
main_all (pairwise on)	IID	0.509 $\pm$ 0.010	0.692 $\pm$ 0.003	0.703 $\pm$ 0.004	0.255 $\pm$ 0.007	0.092 $\pm$ 0.008
main_all (pairwise on)	OOD	0.482 $\pm$ 0.011	0.656 $\pm$ 0.004	0.670 $\pm$ 0.003	0.257 $\pm$ 0.003	0.096 $\pm$ 0.005
no_pairwise (main OMC)	IID	0.375 $\pm$ 0.139	0.694 $\pm$ 0.005	0.697 $\pm$ 0.006	0.272 $\pm$ 0.012	0.105 $\pm$ 0.017
no_pairwise (main OMC)	OOD	0.363 $\pm$ 0.116	0.650 $\pm$ 0.006	0.665 $\pm$ 0.009	0.271 $\pm$ 0.009	0.103 $\pm$ 0.011

Table 5: Component ablation (LOWM,

5

seeds, all-negative objective, matrix_size

=16

Figure 3: Energy matrix (seed 0, IID, $16\times 16$ ), per-operator normalized ( $0=$ most coherent in each column). OMC concentrates low energy on the diagonal; the baseline does not.

Figure 4: Coherence stratification (seed 0, IID). OMC separates

\tau_{\mathrm{same}}

\tau_{\mathrm{wrong}}

\tau_{\mathrm{noise}}

; the baseline collapses same vs. wrong while placing noise far above.

$\alpha_{\text{occl}}$	top1	law_pair	same $<$ wr	mat_mrr
0.0	0.384 $\pm$ 0.125	0.694 $\pm$ 0.007	0.693 $\pm$ 0.003	0.269 $\pm$ 0.004
0.1	0.219 $\pm$ 0.016	0.703 $\pm$ 0.006	0.700 $\pm$ 0.007	0.290 $\pm$ 0.006
0.5	0.230 $\pm$ 0.033	0.706 $\pm$ 0.008	0.703 $\pm$ 0.004	0.292 $\pm$ 0.010
1.0	0.239 $\pm$ 0.043	0.710 $\pm$ 0.009	0.710 $\pm$ 0.006	0.289 $\pm$ 0.005

Table 6: Sensitivity to

\alpha_{\text{occl}}

(LOWM, no-pairwise, all-4 negatives;

3

seeds, test_iid ).

\alpha_{\text{occl}}{=}0

is the paper setting.

Negative set	top1	law_pair	same $<$ wr	mat_mrr
law-mismatch only	0.419 $\pm$ 0.003	0.707 $\pm$ 0.002	0.711 $\pm$ 0.007	0.289 $\pm$ 0.002
law-mismatch $+$ random	0.561 $\pm$ 0.001	0.708 $\pm$ 0.003	0.713 $\pm$ 0.003	0.292 $\pm$ 0.007
all-4 (main OMC)	0.384 $\pm$ 0.125	0.694 $\pm$ 0.007	0.693 $\pm$ 0.003	0.269 $\pm$ 0.004
no-law ( $\tau_{\mathrm{wrong}}$ removed)	0.996 $\pm$ 0.002	0.000	0.506 $\pm$ 0.014	0.211 $\pm$ 0.003

Table 7: Sensitivity to negative-set composition (LOWM,

\alpha_{\text{occl}}{=}0

, no-pairwise;

3

seeds, test_iid ).

Figure 5: Robustness of OMC to wrong-operator distance

\delta

(left) and InfoNCE temperature

T

(right); operator-sensitive metrics stay flat ( test_iid ,

3

seeds).

Model	Variant	top1	law_pair	same $<$ wr	mat_mrr
C-SWM-style	OMC-enhanced	0.358 $\pm$ 0.042	0.512 $\pm$ 0.006	0.491 $\pm$ 0.012	0.212 $\pm$ 0.000
C-SWM-style	random-neg	0.214 $\pm$ 0.012	0.483 $\pm$ 0.003	0.493 $\pm$ 0.006	0.211 $\pm$ 0.000
C-SWM-style	state-corrupt	0.237 $\pm$ 0.011	0.482 $\pm$ 0.007	0.492 $\pm$ 0.004	0.211 $\pm$ 0.000
C-SWM-style	temporal-neg	0.115 $\pm$ 0.017	0.513 $\pm$ 0.024	0.517 $\pm$ 0.026	0.228 $\pm$ 0.015
RSSM-lite	(no-law obj.)	0.590 $\pm$ 0.005	0.596 $\pm$ 0.005	0.609 $\pm$ 0.007	0.270 $\pm$ 0.011

Table 8: General world-model baselines (

3

seeds, IID; OOD in Appendix D ). Feature-level scorers.

Backbone	Obj.	generic top1	same $<$ wr	full str.	mat_mrr	diag-gap
LOWM	OMC	0.733 $\pm$ 0.006	0.735 $\pm$ 0.006	0.714 $\pm$ 0.014	0.344 $\pm$ 0.004	1.368 $\pm$ 0.031
LOWM	no-law	0.496 $\pm$ 0.002	0.496 $\pm$ 0.002	0.449 $\pm$ 0.041	0.211 $\pm$ 0.001	0.332 $\pm$ 0.215
direct	OMC	0.736 $\pm$ 0.014	0.736 $\pm$ 0.014	0.736 $\pm$ 0.014	0.309 $\pm$ 0.007	1.044 $\pm$ 0.014
direct	no-law	0.504 $\pm$ 0.007	0.504 $\pm$ 0.007	0.504 $\pm$ 0.007	0.210 $\pm$ 0.002	0.012 $\pm$ 0.033

Table 9: CoPhy ballsCF external validation ( test ,

5

seeds).

Figure 6: Summary across settings. OMC vs. no-law on operator-coherent ranking (left) and energy-matrix retrieval (right) for synthetic IID/OOD and CoPhy. The baseline stays near chance everywhere; OMC is consistently above it, including on external CoPhy.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

seed	split	top1	law_pair	law_gap	law_only	same $<$ wr	full str.	mat_mrr	mat_t1	diag-gap
direct_context_energy_direct_context_energy
0	IID	0.586	0.699	3.87	0.413	0.707	0.707	0.260	0.098	2.51
1	IID	0.402	0.490	-0.01	0.199	0.499	0.499	0.211	0.062	0.00
2	IID	0.590	0.705	4.10	0.409	0.703	0.703	0.255	0.090	2.66
3	IID	0.611	0.717	2.57	0.401	0.716	0.716	0.257	0.090	1.76
4	IID	0.583	0.704	3.94	0.414	0.716	0.716	0.257	0.098	2.56

Appendix

Table 16

seed	split	top1	law_pair	law_gap	law_only	same $<$ wr	full str.	mat_mrr	mat_t1	diag-gap
lowm_negs_law_mismatch_only_alpha0_lambda16
0	IID	0.406	0.700	2.26	–	0.716	0.688	0.299	0.133	1.60
1	IID	0.419	0.695	2.18	–	0.716	0.675	0.277	0.105	1.52
2	IID	0.417	0.699	2.11	–	0.707	0.667	0.266	0.094	1.47
3	IID	0.426	0.707	2.26	–	0.708	0.661	0.294	0.125	1.57
4	IID	0.412	0.697	2.04	–	0.713	0.688	0.282	0.113	1.43

Appendix

Table 17

seed	split	top1	law_pair	law_gap	law_only	same $<$ wr	full str.	mat_mrr	mat_t1	diag-gap
lowm_component_main_all
0	IID	0.508	0.691	1.61	–	0.701	0.701	0.250	0.090	1.13
1	IID	0.520	0.696	1.71	–	0.696	0.696	0.264	0.102	1.20
2	IID	0.508	0.688	1.63	–	0.705	0.705	0.252	0.086	1.15
3	IID	0.492	0.691	1.72	–	0.703	0.703	0.246	0.082	1.20
4	IID	0.518	0.692	1.64	–	0.708	0.708	0.261	0.102	1.17

Appendix

Table 18

seed	split	top1	law_pair	law_gap	law_only	same $<$ wr	full str.	mat_mrr	mat_t1	diag-gap
lowm_component_no_pairwise
0	IID	0.331	0.695	0.99	–	0.696	0.696	0.273	0.113	0.66
1	IID	0.526	0.687	2.07	–	0.692	0.692	0.268	0.102	1.43
2	IID	0.294	0.700	0.83	–	0.690	0.690	0.265	0.094	0.55
3	IID	0.514	0.698	1.98	–	0.705	0.705	0.261	0.086	1.35
4	IID	0.210	0.691	0.65	–	0.702	0.702	0.292	0.129	0.43

Appendix

Table 19

seed	split	top1	law_pair	law_gap	law_only	same $<$ wr	full str.	mat_mrr	mat_t1	diag-gap
cswm_style_energy_cswm_plus_omc
0	IID	0.348	0.505	0.01	0.194	0.485	0.485	0.212	0.062	0.00
1	IID	0.404	0.517	0.02	0.196	0.505	0.505	0.211	0.062	-0.00
2	IID	0.322	0.514	0.01	0.219	0.484	0.484	0.211	0.062	0.00
0	OOD	0.337	0.502	-0.03	0.199	0.485	0.485	0.212	0.062	0.00
1	OOD	0.364	0.516	0.01	0.205	0.500	0.500	0.211	0.062	-0.00

Appendix

Table 20

seed	split	top1	law_pair	law_gap	law_only	same $<$ wr	full str.	mat_mrr	mat_t1	diag-gap
rssm_lite_scorer_rssm_lite_mse
0	IID	0.596	0.602	0.00	0.282	0.608	0.608	0.269	0.113	0.00
1	IID	0.588	0.594	0.00	0.284	0.616	0.616	0.281	0.133	0.00
2	IID	0.587	0.592	0.00	0.281	0.602	0.602	0.259	0.102	0.00
0	OOD	0.548	0.557	0.00	0.296	0.571	0.571	0.260	0.086	0.00
1	OOD	0.548	0.557	0.00	0.293	0.567	0.567	0.262	0.090	0.00

Appendix

Table 21

seed	split	top1	law_pair	law_gap	law_only	same $<$ wr	full str.	mat_mrr	mat_t1	diag-gap
direct_context_energy_direct_context_cophy_OMC
0	test	0.730	0.730	1.07	0.406	–	–	0.314	0.137	1.23
1	test	0.740	0.740	1.08	0.424	–	–	0.310	0.125	1.23
2	test	0.728	0.730	1.05	0.402	–	–	0.310	0.125	1.23
3	test	0.756	0.758	1.08	0.460	–	–	0.334	0.145	1.28
4	test	0.732	0.732	1.07	0.386	–	–	0.329	0.152	1.26

Appendix

Table 22

World Model Evaluation

World Model Evaluation measures whether learned simulators predict consistent, physically plausible, action-responsive futures that help agents plan and act.

See all 212 papers

Operator-on-F complements value-equivalence: a planning-time diagnostic for latent world models
Jul 5, 2026Donna VakalisWorld Model-Based Planning Latent World Models
1Mila – Quebec AI Institute · 2Université de Montréal
The Intervention Gap in Latent World Models
Aug 30, 2026Donna VakalisWorld Model-Based Planning Model-Based Planning
Mila – Quebec AI Institute · University of Montreal
Learning Counterfactual World Models for Embodied Reasoning under Partial Observability
Sep 5, 2026Todd Y. Zhou, Daniel ZhangEmbodied World Models World Model-Based Planning
Harvard University

Jul 5, 2026J/K move · Enter open · S save

Operator-on-F complements value-equivalence: a planning-time diagnostic for latent world models

Donna Vakalis

1Mila – Quebec AI Institute · 2Université de Montréal

World Model-Based Planning Latent World Models World Model Evaluation Model-Based RL

Open paper AskPDF arXiv

World-model evaluation for model-based reinforcement learning typically asks whether the learned model predicts reward and value well, which can leave planning-relevant errors in the model's latent rollouts unmeasured. We introduce a complementary diagnostic, operator-on-F, that compares a model's k-step latent pushforward to the environment's on an observable subset F, using the model's own predictor. On a TD-MPC2 size sweep over cheetah-run, reward-prediction error stays within [0.028, 0.091] for every model size - only about 3x variation - so an unnormalized reward-fit check has narrow resolution to distinguish them; the (unnormalized) Bellman residual and reward error themselves have weak relationships with return (Spearman -0.10 and -0.30). Operator error spans 0.28 to 2.62 over the same sizes. At 317M the operator error is 2.62 - an order of magnitude above the 0.28-0.36 cluster - and the planning return collapses to 0.9, while reward-prediction error (0.091) is the highest of the five but stays within the same small [0.028, 0.091] range as the rest of the sweep. The rank correlation between operator error and return loss is -0.90 (anchor-bootstrap 95% CI [-0.90, -0.70] at n=5 sizes; leave-one-out removal of any single size leaves it at -0.80 or stronger). The operator also returns informative, architecture-discriminating estimates in a cross-architecture comparison between TD-MPC2 and a pure-SSL latent world model. The operator diagnostic complements value-equivalence rather than replacing it.

Abstract

World Model Evaluation

Explore similar work

Operator-on-F complements value-equivalence: a planning-time diagnostic for latent world models

The Intervention Gap in Latent World Models

Learning Counterfactual World Models for Embodied Reasoning under Partial Observability