ARC-Bench: Closed-Loop Replanning Masks Broken Action Ranking in Frozen JEPA World Models

Zhengshu Zhang; Zhiyuan Li

cs.AIAug 12, 2026

ARC-Bench: Closed-Loop Replanning Masks Broken Action Ranking in Frozen JEPA World Models

Authors: Zhengshu Zhang, Zhiyuan Li

Organizations: University of Southern California · Aalto University

Abstract

Reward-free latent world models plan by scoring candidate actions with distances in a frozen latent space: an action is preferred if its predicted future embedding lands closer to the goal embedding. This silently assumes that latent closeness is action-rankable, i.e., that ordering candidates by latent distance agrees with ordering them by true cost. We audit this assumption directly. We introduce ARC-Bench, a no-leak, fixed-candidate protocol that measures whether frozen JEPA-style objectives rank candidate actions correctly, and apply it to official released JEPA-WM checkpoints across navigation and manipulation-style control. The assumption fails, severely and structurally: on the official manipulation audits the top-scored candidate is almost always suboptimal, and the same inversion appears in the maze domains. A controlled visual-backbone extension shows that the defect persists when DINOv2 is replaced by video-pretrained V-JEPA 1 and V-JEPA 2 encoders at ViT-L/ViT-G scale. Provenance, undertraining, matched-budget backbone controls, and metric-circularity controls rule out trivial explanations. We then explain why this defect has stayed invisible: closed-loop replanning masks it. When we reduce the planner's replanning frequency, success collapses in both a navigation and a manipulation domain, and the episodes rescued by frequent replanning are enriched for severe first-plan ranking failures in the PointMaze first-plan diagnostic. Closed-loop success rates therefore systematically overstate the rankability of frozen latent representations. ARC-Bench supplies the measurement, and the masking mechanism the explanation, for methods that adapt, amortize, or replan around latent-space planners without directly auditing released JEPA-WM action rankability.

Figures & tables

Environment	Official model + scorer	Anchors (useful)	Seeds pos.	Mean top-1 regret [95% CI]	Wrong rate	Hit@1
Push-T	jepa_wm_pusht + official L2	750 (618)	5/5	0.243 [0.228, 0.257]	96.8%	3.2%
MetaWorld reach-wall	jepa_wm_metaworld + official L2	750 (750)	5/5	0.283 [0.278, 0.290]	100.0%	0.0%

Table 1: Official generated-candidate scoring audits. Candidates are generated rollouts scored by the official checkpoints with the official L2 planning objective (model unroll + representation-target distance). These are scoring audits of the official objective, not full CEM planner reruns.

Figure 1: Official JEPA-WM objectives misrank fixed candidates on manipulation-style control. Left: wrong-anchor rate and Hit@1 under the official L2 objective (750-anchor audits, five seeds). Right: mean top-1 regret with 95% bootstrap intervals.

Environment	Backbone	Useful	Mean top-1 regret [95% CI]	Wrong rate	Hit@1	Hit@3	Spearman	Dominant top-pick family
Push-T	DINOv2 control	598/750	0.239 [0.224, 0.254]	94.8%	5.2%	19.1%	+0.06	random_constant / random_iid
Push-T	V-JEPA 1 ViT-L	598/750	0.227 [0.212, 0.243]	91.1%	8.7%	24.1%	+0.05	workspace_cycle / random_constant
Push-T	V-JEPA 2 ViT-L	598/750	0.240 [0.225, 0.255]	97.3%	2.5%	15.9%	+0.05	random_constant / random_iid
Push-T	V-JEPA 2 ViT-G	598/750	0.215 [0.201, 0.230]	94.5%	5.2%	19.2%	$-0.03$	random_iid / noisy_heuristic
MetaWorld reach-wall	DINOv2 control	750/750	0.370 [0.358, 0.382]	100.0%	0.0%	0.0%	$-0.52$	zero_action / random_iid
MetaWorld reach-wall	V-JEPA 1 ViT-L	750/750	0.290 [0.289, 0.291]	100.0%	0.0%	0.0%	$-0.57$	zero_action (743/750)

Table 2: Controlled encoder-swap generated-candidate audit. The frozen visual backbone is varied while the downstream predictor, latent-distance objective, candidate protocol, and training budget are held fixed. These are controlled model variants, not official released checkpoint evaluations.

Panel of Figure 2 — Figure 2: Visual examples from the controlled encoder-swap audit. In each panel, the top row executes the action selected by the frozen latent objective, and the bottom row executes the ground-truth-best candidate from the same fixed candidate set. The selected actions can look visually plausible over short horizons while still leaving the task far from solved.

Metric	PointMaze hard	PointMaze probe	Wall hard	Wall probe
Top-1 regret	0.496	0.195	1.278	1.028
Hit@1	15.2%	48.8%	43.8%	48.4%
Hit@3	38.7%	76.6%	79.3%	82.8%
MRR	0.343	0.657	0.633	0.674
Pairwise accuracy	0.590	0.803	0.810	0.827
Spearman (score, cost)	0.247	0.713	0.766	0.790

Table 3: Localizability-probe evidence (diagnostic; not a proposed method) on the locked held-out split (256 anchors

\times

12 candidates per domain), under a validation-only selection protocol on the frozen JEPA-WM diagnostic setup. “Hard” is the strongest hard baseline per domain (refit linear dynamics on PointMaze; ridge on absolute latent differences on Wall).

Environment	Scorer	Anchors	Mean top-1 regret [95% CI]	Wrong rate	Hit@1	Spearman
U-Maze	terminal Euclidean	256	1.42 [1.33, 1.51]	100%	0.0%	$-0.58$
U-Maze	terminal Manhattan	256	1.42 [1.33, 1.51]	100%	0.0%	$-0.58$
MiniGrid/FourRooms	terminal Euclidean	640	3.14 [2.96, 3.31]	100%	0.0%	$+0.63$
MiniGrid/FourRooms	terminal Manhattan	640	2.87 [2.67, 3.07]	83.0%	17.0%	$+0.61$

Table 4: Controlled topology-stress diagnosis on public/procedural layouts (U-Maze; MiniGrid/FourRooms). Pure diagnosis (no repair, no training, no model selection): a position-level topology adapter with a breadth-first terminal-to-goal true cost audits terminal-distance ranking. Anchors are held-out (U-Maze fresh split; MiniGrid/FourRooms five-seed audit). These are supplementary topology-stress audits, not official JEPA-WM evaluations.

Figure 3: Controlled topology-stress diagnosis. Raw terminal-distance action ranking on U-Maze and MiniGrid/FourRooms under a position-level topology adapter with breadth-first true cost. Left: wrong-anchor rate and Hit@1 for terminal-Euclidean scoring (100% wrong, Hit@1 = 0 in both domains). Right: mean top-1 regret with 95% bootstrap intervals for terminal-Euclidean and Manhattan scoring. These are supplementary topology-stress audits, not official JEPA-WM evaluations.

Figure 4: CEM-style finite-budget audit on real cached scores. Increasing the sample budget $B$ improves all scorers but does not close the ranking gap: scorer quality controls selection quality at every budget.

Figure 5: Closed-loop replanning masks the ranking defect. Success rate as a function of the replan interval $k$ in the legacy-environment PointMaze route and the official Push-T online route (lines: single-seed full sweeps, 48 episodes; markers: three-seed $k=1$ vs $k=6$ confirmation, 144 paired episodes, Wilson 95% CIs). Reducing the replanning frequency exposes failure in both a navigation and a manipulation-style domain.

Domain	$k=1$ success	$k=6$ success	$\Delta$ (pp)	95% CI (pp)	Discordant	McNemar $p$
					( $k1$ -only / reverse)
PointMaze	129/144 (89.6%)	89/144 (61.8%)	+27.8	[19.4, 36.1]	44 / 4	$1.51\times 10^{-9}$
Push-T	52/144 (36.1%)	18/144 (12.5%)	+23.6	[16.7, 30.6]	35 / 1	$1.08\times 10^{-9}$

Table 5: Multi-seed closed-loop masking statistics (three seeds

\times

48 paired episodes per domain; PointMaze uses a legacy-environment route executing the official CEM configuration; Push-T uses the official JEPA-WM Push-T online evaluation route; only the replan interval is manipulated).

Environment	Backbone	$k=1$ success [95% CI]	$k=6$ success [95% CI]	Ratio
Push-T	DINOv2 control	66/120 (55.0%) [50.0, 63.3]	27/120 (22.5%) [13.3, 31.7]	2.44 $\times$
Push-T	V-JEPA 1 ViT-L	52/120 (43.3%) [32.5, 54.2]	11/120 (9.2%) [5.0, 13.3]	4.73 $\times$
Push-T	V-JEPA 2 ViT-L	39/120 (32.5%) [19.2, 45.0]	15/120 (12.5%) [7.5, 19.2]	2.60 $\times$
Push-T	V-JEPA 2 ViT-G	54/120 (45.0%) [37.5, 51.7]	20/120 (16.7%) [13.3, 20.0]	2.70 $\times$
MetaWorld reach-wall	DINOv2 control	33/60 (55.0%) [43.3, 65.0]	14/60 (23.3%) [18.3, 28.3]	2.36 $\times$
MetaWorld reach-wall	V-JEPA 1 ViT-L	4/60 (6.7%) [0.0, 16.7]	0/60 (0.0%) [0.0, 0.0]	degenerate

Table 6: Controlled encoder-swap closed-loop cadence audit using the official CEM evaluation route with encoder-swapped models. Push-T uses 120 episodes per cell (24 episodes

\times

five seeds); MetaWorld reach-wall uses 60 episodes per cell (12 episodes

\times

five seeds). Intervals are seed-level bootstrap 95% CIs, not per-episode McNemar intervals.

Figure 6: Closed-loop cadence in the controlled visual-backbone extension. Replanning every step improves success most clearly on Push-T and for the DINOv2 MetaWorld control. The MetaWorld V-JEPA 1 ViT-L and V-JEPA 2 ViT-L models are already near failure at $k=1$ , and the V-JEPA 2 ViT-G MetaWorld drop is not separated by the seed-level intervals.

Latent World Models

Latent world models learn compact predictive state representations to simulate how environments change and support planning, control, and reasoning.

See all 214 papers

LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
Sep 28, 2026Luzhe Huang, Lei Chu, Jingyi Liang +1Disentangled Representation Learning World Models for Robotics
Independent researcher
Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding
Jun 30, 2026Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan +11World Models World Model-Based Planning
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing · Institute of Information Engineering, Chinese Academy of Sciences, Beijing · School of Computer Science and Technology, Harbin Institute of Technology, Weihai +2
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Oct 14, 2026Joshua Cheung, Zhang HeLatent World Models World Model Evaluation

Sep 28, 2026J/K move · Enter open · S save

LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models

Luzhe Huang, Lei Chu, Jingyi Liang, Yuhuan Zhao

Independent researcher

Disentangled Representation Learning World Models for Robotics World Model-Based Planning Action-Conditioned World Models Model-Based Planning Latent World Models+1

Open paper AskPDF arXiv

Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and predictable visual context. This competition can entangle controllable state with high-rank nuisance appearance and degrade planning as scenes become more complex. We introduce LRC-JEPA, a lightweight end-to-end world model that routes information into a compact predictive latent

\mathbf{z}

and learned-query residual-context embeddings

\mathbf{u}

. Only

\mathbf{z}

is propagated by the dynamics model and used for planning, while

\mathbf{u}

captures temporally persistent information for cross-attention reconstruction; a differentiable residual connection encourages the latent to retain complementary dynamic content. Under explicit assumptions, we show that the resulting representation is sufficient, minimal, nuisance-invariant, and disentangled. Across four simulated control environments, LRC-JEPA improves average planning success over a parameter-matched JEPA baseline by 9 percentage points and matches or exceeds substantially larger pretrained models. On the real-world Bridge-v2 set, its 5.5M-parameter active encoder outperforms DINO-WM (22.1M) and V-JEPA2 (303.9M) encoders while also enabling faster planning. Physical-state probes, reconstruction interventions, and ablations confirm the effectiveness of LRC-JEPA's representation disentanglement.

Abstract

Latent World Models

Explore similar work

LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models