World Models with Configurable Perception

Yijun Shen; Delong Chen; Xianming Hu; Kai Zhang; Pascale Fung

cs.CVOct 20, 2026

World Models with Configurable Perception

Authors: Yijun Shen, Delong Chen, Xianming Hu, Kai Zhang, Pascale Fung

Organizations: East China Normal University · HKUST

Abstract

Real-world observations are noisy and often contain excessive task-irrelevant information. Effective world modeling and planning need to abstract observations into world state representations while filtering out irrelevant information. However, encoders in existing world models are typically non-adaptive, leading to state representations that are easily distracted by task-irrelevant information. As a result, the model may not selectively filter observations to cater to different task demands. To address this problem, we introduce ConfigWM, which realizes configurable perception through task-conditioned state extraction. Given a language description of the task at hand, ConfigWM uses it as a configuration signal to steer how visual embeddings are mapped into a task-specific state space, selectively preserving information relevant to the current task while suppressing task-irrelevant variations. Action-conditioned dynamics are then learned in this task-specific state space for planning. We evaluate ConfigWM in environments with increasing levels of distraction. While existing methods exhibit substantial declines in planning success rate as the environments become increasingly noisy, ConfigWM remains comparatively stable and outperforms the strongest competing method by 22.3% on average across diverse environments. Consistent gains in physical-quantity probing (+0.365 Pearson r) and distracted-to-clean retrieval (+75.5%) further indicate that ConfigWM learns states that preserve task-relevant information while remaining robust to distractors, supporting reliable world modeling and planning in noisy worlds. Project Page: https://configwm.github.io

Figures & tables

Figure 1: ConfigWM Architecture.

Figure 2: Constructing noisy worlds. Columns show observations with an increasing number of dynamic, target-like distractors. Rows show Two-Room, Push-T, and OGBench-Cube. Across all settings, the underlying task, action sequence, goal, and target trajectory remain unchanged.

Figure 3: Training pipeline. (a) Configurable encoder predicts the representation of the query-specified target from the noisy observation. (b) The frozen configurable encoder encodes the current and next observations under the same query, and ConfigWM learns the action-conditioned dynamics over the projected states.

Figure 4: Planning with ConfigWM. Configurable encoder encodes the initial and goal observations under the task query. ConfigWM rolls out candidate actions in the query-conditioned latent space, where the terminal distance to the goal guides iterative action optimization.

Figure 5: Matched planning performance under dynamic distractors. Planning success rates are shown across distractor levels

d_{0}

–

d_{8}

on Two-Room, Push-T, and OGBench-Cube, with each model trained and evaluated at the same level (

d_{k}\!\rightarrow\!d_{k}

). Shaded regions indicate one standard deviation across seeds, and the random policy is shown as a dotted horizontal reference. Higher is better.

Figure 6: Representation analysis. Left: linear probing of task-relevant physical quantities, reported as the macro-average of the per-quantity Pearson correlations. Right: distracted-to-clean Top-1 retrieval accuracy. The three panels in each group correspond to Two-Room, Push-T, and OGBench-Cube. The clean setting is denoted by

d_{0}

, while

d_{1}

–

d_{8}

introduce increasing levels of dynamic distractors. Higher is better.

Figure 7: Decoding query-configured predictive states. Given the same distracted observation, we vary only the color query and decode the resulting predictive state using a shared decoder. Each decoded observation recovers the queried block and task-relevant state, while preserving other task-relevant elements (e.g., the green goal block).

Figure 10: Configurable encoder across visual backbones. We compare each backbone with our configurable encoder built upon the same backbone. Across DINOv2, DINOv3, V-JEPA 2.1, and LingBot-Vision, ours suppresses task-irrelevant distractors while preserving task-relevant state information. Black dashed boxes mark distractors.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Trajectories	Avg. Steps	Source–Clean Pairs
Two-Room	10,000	92.1	10,002,339
Push-T	18,685	125.1	25,301,106
OGBench-Cube	10,000	201.0	21,780,000
Total	38,685	136.2	57,083,445

Appendix

Table 1: Scale of the constructed noisy-world datasets.

Figure 11: Progressive noisy-world construction and query-conditioned targets. Within each row, $d_{1}$ – $d_{8}$ cumulatively add distractors to the same trajectory frame. The rightmost column shows the $d_{0}$ target selected by the displayed query.

Setting	Value	Setting	Value
Learning rate	$5\times 10^{-5}$	Weight decay	$0.05$
LR schedule	100-step warmup; cosine decay	Global batch size	$512$
Temperature $\tau$	$0.07$	Gradient clipping	$1.0$
Numerical precision	BF16	Training hardware	$1\times$ A100 (80 GB)

Appendix

Table 2: Optimization settings for the configurable encoder training.

Method	Distractor Level									Average
Method	$d_{0}$	$d_{1}$	$d_{2}$	$d_{3}$	$d_{4}$	$d_{5}$	$d_{6}$	$d_{7}$	$d_{8}$	Average
Two-Room
Random Policy	$30.0$ $\pm 4.9$
LeWM	$\mathbf{100.0}$ $\pm 0.0$	$64.0$ $\pm 4.3$	$40.0$ $\pm 5.9$	$35.3$ $\pm 2.5$	$41.3$ $\pm 10.9$	$32.7$ $\pm 1.9$	$35.3$ $\pm 3.4$	$32.7$ $\pm 4.1$	$40.7$ $\pm 0.9$	$46.9$ $\pm 0.8$
Sub-JEPA	$94.7$ $\pm 3.4$	$41.3$ $\pm 2.5$	$42.0$ $\pm 5.9$	$36.7$ $\pm 4.7$	$33.3$ $\pm 8.2$	$34.0$ $\pm 8.5$	$32.7$ $\pm 5.7$	$36.0$ $\pm 4.3$	$30.7$ $\pm 2.5$	$42.4$ $\pm 3.2$
TC-LeWM	$74.7$ $\pm 2.5$	$34.0$ $\pm 5.9$	$34.0$ $\pm 2.8$	$38.0$ $\pm 4.3$	$30.0$ $\pm 8.6$	$38.7$ $\pm 3.8$	$33.3$ $\pm 5.2$	$39.3$ $\pm 5.0$	$37.3$ $\pm 0.9$	$39.9$ $\pm 2.1$

Appendix

Table 3: Matched planning performance. Models are trained and evaluated at the same distractor level (

d_{k}\!\rightarrow\!d_{k}

). Values are mean success rates (%) over seeds 40/42/45. Light-blue tags indicate standard deviation. Average is computed by first averaging

d_{0}

–

d_{8}

within each seed and then aggregating the three seed-level averages. Column-best means are bold.

Figure 12: Matched planning performance across distractor levels. Success rate (%) under the matched setting, where each model is trained and evaluated at the same distractor level (

d_{k}\!\rightarrow\!d_{k}

). Results are averaged over seeds 40/42/45, with shaded regions indicating standard deviation.

Method	Evaluation Distractor Level									Average
Method	$d_{0}$	$d_{1}$	$d_{2}$	$d_{3}$	$d_{4}$	$d_{5}$	$d_{6}$	$d_{7}$	$d_{8}$	Average
Two-Room
Random Policy	$30.0$ $\pm 4.9$
LeWM	$\mathbf{100.0}$ $\pm 0.0$	$72.7$ $\pm 1.9$	$66.0$ $\pm 4.9$	$58.7$ $\pm 5.2$	$51.3$ $\pm 5.0$	$50.7$ $\pm 3.4$	$56.7$ $\pm 6.6$	$52.0$ $\pm 7.5$	$50.0$ $\pm 5.7$	$62.0$ $\pm 3.3$
Sub-JEPA	$94.7$ $\pm 3.4$	$60.7$ $\pm 1.9$	$45.3$ $\pm 12.5$	$53.3$ $\pm 7.4$	$45.3$ $\pm 3.4$	$44.0$ $\pm 5.7$	$44.7$ $\pm 6.6$	$46.7$ $\pm 7.5$	$46.0$ $\pm 7.5$	$53.4$ $\pm 4.4$
TC-LeWM	$74.7$ $\pm 2.5$	$43.3$ $\pm 0.9$	$38.7$ $\pm 7.4$	$32.7$ $\pm 4.1$	$34.7$ $\pm 0.9$	$30.0$ $\pm 5.9$	$31.3$ $\pm 8.1$	$31.3$ $\pm 5.7$	$27.3$ $\pm 1.9$	$38.2$ $\pm 0.5$

Appendix

Table 4: Clean-trained transfer performance. Each model uses its

d_{0}

-trained checkpoint and is evaluated across

d_{0}

–

d_{8}

(

d_{0}\!\rightarrow\!d_{k}

). Values follow the same reporting conventions as Table 3 .

Figure 13: Clean-trained world-model transfer under dynamic distractors. World models are trained on

d_{0}

transitions and evaluated on

d_{0}

–

d_{8}

. For ConfigWM, this training restriction applies to the downstream state projector and dynamics model, and its configurable encoder is separately pretrained on noisy–clean pairs across distractor levels. Curves report mean success rates across seeds 40, 42, and 45. Higher is better.

Figure 14: Unified vs. Single-Domain ConfigWM. Comparison of planning success rates (top) and physical-quantity probing performance (bottom) across distractor levels. Unified ConfigWM shares a configurable encoder across environments while retaining environment-specific dynamics models.

Figure 15: Quantity-wise physical probing on Two-Room and Push-T. Each panel reports the Pearson correlation (

r

) of a linear probe trained on frozen representations for one task-relevant physical quantity across distractor levels

d_{0}

–

d_{8}

. Higher is better.

Figure 16: Quantity-wise physical probing on OGBench-Cube: cube state. Pearson correlation (

r

) is reported across distractor levels

d_{0}

–

d_{8}

for cube position, quaternion, and yaw. Higher is better.

Figure 17: Quantity-wise physical probing on OGBench-Cube. Pearson correlation (

r

) is reported across distractor levels

d_{0}

–

d_{8}

. Top: end-effector position, end-effector yaw, and gripper contact. Bottom: gripper opening, joint position, and joint velocity. Higher is better.

Figure 18: Detailed distracted-to-clean retrieval results. Top: Top-5 retrieval accuracy. Bottom: mean reciprocal rank (MRR). Each distracted state is queried against

400

clean candidates ranked by cosine similarity. Results are reported across distractor levels

d_{0}

–

d_{8}

; higher is better.

Figure 19: Additional query-conditioned decoding results. From top to bottom, the three rows show Two-Room, Push-T, and OGBench-Cube. Within each environment, the input observation is held fixed while only the queried color is varied. The decoded observations show that different queries retain different visual content in the predictive state.

Level	Correct query	Mismatched-query average	Drop
Two-Room
$d_{1}$	$96.7$ $\pm 3.4$	$45.7$ $\pm 2.3$	$51.0$ $\pm 5.5$
$d_{2}$	$96.7$ $\pm 1.9$	$41.3$ $\pm 3.7$	$55.4$ $\pm 4.3$
$d_{3}$	$97.3$ $\pm 1.9$	$39.3$ $\pm 3.0$	$58.1$ $\pm 4.9$
$d_{4}$	$97.3$ $\pm 1.9$	$33.4$ $\pm 2.6$	$63.9$ $\pm 4.5$
$d_{5}$	$96.0$ $\pm 2.8$	$32.8$ $\pm 3.7$	$63.2$ $\pm 4.6$

Appendix

Table 5: Complete query-conditioned planning results. Mean success rate (%) over three seeds at each distractor level; light-blue tags show standard deviation. Mismatched results average eight valid queries within each seed. Drop is computed from the unrounded seed-level results.

Environment	Correct query	Mismatched query	Avg. $d_{1}$ – $d_{8}$	Drop
Two-Room	Red ball	Blue ball	$29.4$ $\pm 0.7$	$67.2$ $\pm 1.7$
		Brown ball	$32.4$ $\pm 3.2$	$64.2$ $\pm 5.3$
		Cyan ball	$29.8$ $\pm 3.1$	$66.8$ $\pm 4.6$
		Green ball	$35.4$ $\pm 4.7$	$61.2$ $\pm 6.3$
		Orange ball	$46.6$ $\pm 2.6$	$50.0$ $\pm 4.7$
		Pink ball	$39.0$ $\pm 6.8$	$57.6$ $\pm 7.9$

Appendix

Table 6: Per-query planning performance under mismatched queries. Mean success rate (%) over

d_{1}

–

d_{8}

and three seeds; light-blue tags show standard deviation. Drop is computed from the unrounded results.

Figure 20: Qualitative planning rollouts on Two-Room and Push-T. Both methods complete the task under clean observations ( $d_{0}$ ). Under severe distractors ( $d_{8}$ ), ConfigWM reaches the goal, whereas the corresponding comparison model fails.

Figure 21: Qualitative planning rollouts on OGBench-Cube. Both models complete the task under clean observations ( $d_{0}$ ). Under severe distractors ( $d_{8}$ ), ConfigWM reaches the goal, whereas DINOv2-WM fails. The yellow box identifies the controlled task view.

Figure 22: Configurable encoder across visual backbones. The configurable encoder suppresses feature regions associated with the injected distractors while retaining the task-relevant entities and spatial structure visible in the clean target. Six held-out cases are shown across DINOv2-Giant, DINOv3-7B, V-JEPA2.1-Gigantic, and LingBot-Vision-Giant. Black dashed boxes mark distractors.

	Frozen backbone		The configurable encoder (Ours)
Backbone	Distractor AP $\downarrow$	Task-core AP $\uparrow$	Distractor AP $\downarrow$	Task-core AP $\uparrow$
DINOv2-Giant	0.523	0.603	0.377	0.594
DINOv3-7B	0.575	0.609	0.367	0.596
V-JEPA 2.1-Gigantic	0.544	0.605	0.376	0.602
LingBot-Vision-Giant	0.561	0.621	0.368	0.603

Appendix

Table 7: Quantitative analysis of real-world representations across large visual backbones. The configurable encoder reduces distractor decodability while largely preserving task-relevant spatial information. We report patch-level linear-probing AP across DINOv2-Giant, DINOv3-7B, V-JEPA2.1-Gigantic, and LingBot-Vision-Giant.

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 414 papers

Jun 1, 2026

IMWM: Intuition Models Complement World Models for Latent Planning

Planning with a learned latent world model is a promising route to control from raw pixels, but a strong world model alone is not enough. We show this experimentally: even with a perfect world model (operationalized by replacing the learned forward predictor with an idealized rollout of the true environment dynamics), a finite-budget sample-based planner still fails on some tasks, indicating that the bottleneck can lie in search rather than in world-model accuracy. Motivated by this gap, we propose IMWM (Intuition Model + World Model), which pairs the world model with an intuition model trained from demonstrations to recognize promising actions. The two models collaborate through three lightweight components: (i) Retrieval Initialization, which initializes the planner's action proposal from a retrieved demonstration; (ii) Hybrid Cost, which combines the intuition score with the world-model rollout cost; and (iii) a Reliability Gate, which adjusts how much the planner trusts intuition in each setting. Across four pixel-based goal-reaching tasks (Two-Room, Reacher, Push-T, and OGBench-Cube), IMWM has higher mean success than the world-model-only planner on all four, with the largest gains on Two-Room (99.2%, +11.5 percentage points) and OGBench-Cube (94.7%, +28.5 percentage points).

Baoqi Gao, Ruize Han, Miao Wang +1

Beihang University · Shenzhen University of Advanced Technology

World Model-Based Planning Goal-Conditioned Planning AI Control Model-Based Planning Latent World Models Latent-Space Planning

Ask PDF

Jul 30, 2026

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io

Xiangcheng Zhang, Yilun Du

Harvard University

World Models for Robotics World Model-Based Planning Action-Conditioned World Models Robot Policy Generalization World Models for VLA Models

Ask PDF

Sep 19, 2026

FIRM-WM: State-factorized factual-interventional recurrent modeling for reward-free visual planning

Reward-free latent world models can learn from offline videos and solve new image--goal tasks by optimizing actions through predicted latent futures. This setting places two demands on the planning state: its coordinates must be comparable with a goal image. Moreover, its dynamics must retain velocity, motion trend, contact, and other history--dependent information beyond those goal coordinates. Offline training creates a second mismatch: each recorded trajectory reveals one factual future, whereas a sampling--based planner compares many actions that were not taken from the same state. We introduce FIRM-WM (Factual--Interventional Recurrent World Model), a compact pixel world model designed around these two gaps. Its recurrent state separates a typed, goal--comparable configuration from a 128-dimensional dynamic fiber used for prediction but excluded from the terminal goal cost. Broad factual trajectories provide state coverage, while common--reset intervention branches provide observed outcomes for alternative action sequences. Before executing each branch, we reset the environment and restore the same recorded values exposed by the environment's state--setting interface. Under matched CEM planning and three independent full-pipeline seeds, FIRM-WM reaches 99.0

\pm

1.0% on TwoRoom, 92.7

\pm

2.1% on Reacher, and 88.0

\pm

3.0% on OGBench-Cube, compared with 89.0%, 88.0%, and 70.0% for LeWM. The deployed model uses 2.98--3.42M parameters and records 2.13--11.60

\times

lower planning time on these tasks.

Yilun Wu, Yunjian Zhang, Aobo Li +3

Saint Petersburg State University, Saint Petersburg, Russia · Shenzhen Kaihong Digital Industry Development Co., Ltd., Shenzhen, China · Chinese Academy of Sciences (CAS) – Shenzhen Institute of Advanced Technology +2

World Models for Robotics Latent World Models Action-Conditioned World Models Goal-Conditioned Planning

Ask PDF

Abstract

World Model Learning

Explore similar work

IMWM: Intuition Models Complement World Models for Latent Planning

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

FIRM-WM: State-factorized factual-interventional recurrent modeling for reward-free visual planning