Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning

Juntao Cheng; Jingkai Wang; Yijun Shen; Xiansheng Chen; Zhiwei Yu

cs.ROAug 23, 2026

Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning

Authors: Juntao Cheng, Jingkai Wang, Yijun Shen, Xiansheng Chen, Zhiwei Yu

Organizations: Beijing Academy of Artificial Intelligence (BAAI) · Shanghai Jiao Tong University

Abstract

World models for physical interaction are typically trained to predict future observations or latent features; however, a planning-oriented model must answer a fundamentally different question: whether a candidate action produces a task consistent future while preserving essential relations. Monolithic state representations obscure the underlying entities, while standard instance-level object slots merely identify what is present without specifying what role each entity plays in the task context. To bridge this gap, we present the Semantically Rich World Model (SR-WM), a task-conditioned world model structured around five functional roles: gripper, target, goal, relation, and phase. Within SR-WM, a visual entity encoder extracts soft entity hypotheses from pretrained patch features, allowing segmentation masks to serve as optional proposal priors without mandating them as required state representations or inference inputs. A role binder subsequently maps these hypotheses to task-specific roles, while an action conditioned dynamics model predicts role transitions alongside fine-grained semantics, including grasp/contact, predicate establishment, relation preservation, fixture state, and phase change. Crucially, this unified role state grounds downstream multi-candidate action generation, stage-aware reranking, and violation-aware suffix resampling. Our comprehensive evaluation protocol spans all four LIBERO simulation suites, cross-suite transfer, perception diagnostics, and action sensitivity analysis. Ultimately, this formulation transforms object-centric prediction into a semantic interface linking visual dynamics with planning-oriented decision making

Figures & tables

Figure 1: Systematic overview of our SR-WM framework. (a) Role-state construction pipeline: Multimodal observations (RGB history, robot state, and language instruction), optionally augmented with visual priors such as SAM proposals, are encoded by LeWM into soft entity tokens, which are organized into a structured role state. (b) Task-role binding: Exchangeable visual entities are bound to explicit task roles (e.g., Gripper , Target , and Goal ) conditioned on the task and robot state, while relation and phase states capture structured task semantics. (c) Semantic rollout: The semantic world model predicts task-critical events and predicates—such as contact, grasp, inside, and phase progression—for candidate actions, rather than high-dimensional future pixels. (d) Semantic planning: A semantic selector ranks candidate actions, executes the best prefix, and repairs violating suffixes through a receding-horizon execute-observe-replan loop.

Figure 2: From exchangeable entities to task-conditioned role states. (a) Visual observations are decomposed into task-agnostic entity hypotheses. (b) Instruction-conditioned role binding transforms entities into functional roles, where the same entity may serve different roles under different tasks. The resulting role state is used for predictive dynamics and action planning rather than auxiliary semantic classification.

Generator / State	Decision Rule	Object	Spatial	Goal	LIBERO-10	Average
LeWM latent + FM	Candidate 0	58	50	46	28	45.5
LeWM latent + FM	Monolithic selector	63	55	51	33	50.5
Instance slots + FM	Instance-slot selector	68	60	56	39	55.8
SR-WM role-conditioned FM	Candidate 0	74	67	63	47	62.8
SR-WM role-conditioned FM	Random@4	73	66	62	46	61.8
SR-WM role-conditioned FM	Best fixed ID (validation)	76	69	65	49	64.8

Table 1: Rollout success (%) by generator and decision rule. Candidate 0 and Oracle@4 compare generation and coverage; selector rows measure conversion to top-1 success. Oracle@4 is an upper bound, not selector performance.

Role	Semantic Definition and State Representation
Gripper	End-effector pose, aperture, motion, contact, and grasp status.
Target	Task-relevant object or articulated part and its geometry.
Goal	Receptacle, surface, destination, or desired fixture state.
Relation	Relative geometry and predicates among gripper, target, and goal.
Phase	Progress through approach, grasp, transport, align, release, and completion.

Table 2: Five non-exchangeable functional roles and their semantic states.

Representation	Bind.	Rel.	Phase	Rollout
	Acc.	AUROC	F1	SR
Monolithic latent	60.5	69.4	54.0	50.2
Instance slots	74.6	76.5	64.1	55.8
SR-WM (no masks)	88.3	89.1	80.8	72.6
SR-WM + soft prior	91.8	90.7	84.1	75.1
SR-WM + corrupt prior	86.2	83.9	77.4	70.8

Table 3: Role and mask-dependence diagnostics. “Rollout SR” uses the same selector and candidate budget for every visual encoder.

Variant	Role error $\downarrow$	Shuf. gap $\uparrow$	Rank acc. $\uparrow$
No semantic heads	$0.176{\pm}0.011$	$0.061{\pm}0.019$	$60.8{\pm}2.9$
No shuffled negatives	$0.149{\pm}0.009$	$0.018{\pm}0.011$	$53.7{\pm}3.3$
No preservation loss	$0.158{\pm}0.010$	$0.108{\pm}0.023$	$69.4{\pm}2.6$
No phase loss	$0.141{\pm}0.008$	$0.087{\pm}0.020$	$66.8{\pm}2.8$
Full SR-WM	$0.144{\pm}0.007$	$0.157{\pm}0.025$	$78.2{\pm}2.1$

Table 4: Action-sensitivity evaluation. The shuffled-action gap is

S(A^{+})-S(A^{-})

; larger is better. Ranking accuracy is in percent.

Representation	Success (%)
Five instance slots	$55.0{\pm}3.6$
Object slots + instruction	$62.0{\pm}3.3$
Target + goal roles	$68.0{\pm}3.1$
Target + goal + relation	$73.0{\pm}2.9$
SR-WM (+ gripper, phase)	$\mathbf{83.0{\pm}2.4}$

Table 5: Component-wise ablation of the state representation on closed-loop rollout success. Incremental addition of explicit semantic roles progressively drives physical execution performance.

Train $\rightarrow$ Test	Zero-shot	Adapted	Scratch
Object $\rightarrow$ Spatial	$58.4{\pm}3.5$	$72.1{\pm}3.2$	$72.8{\pm}3.7$
Object $\rightarrow$ Goal	$52.7{\pm}4.1$	$74.6{\pm}2.9$	$66.3{\pm}3.6$
Object $\rightarrow$ LIBERO-10	$27.9{\pm}3.0$	$58.7{\pm}4.0$	$50.6{\pm}4.4$
Joint $\rightarrow$ held-out tasks	$65.2{\pm}3.3$	$78.4{\pm}2.6$	$69.8{\pm}3.5$

Table 6: Cross-suite selected top-1 rollout success under matched candidate budgets. Adapted and Scratch use identical target-suite data and update budgets.

Model	Total	Trainable	Peak train (GB)	Gen. (ms)	Score (ms)
LeWM backbone	15.0M †	15.0M	$4.3{\pm}0.2$	n/a	$4.4{\pm}0.3$
LeWM latent + FM	35.8M	35.8M	$8.1{\pm}0.3$	$28.7{\pm}1.1$	$4.8{\pm}0.3$
Instance slots + FM	39.1M	39.1M	$8.9{\pm}0.4$	$31.0{\pm}1.4$	$7.6{\pm}0.6$
SR-WM (world only)	24.6M	24.6M	$6.6{\pm}0.3$	n/a	$7.1{\pm}0.5$
SR-WM + FM head	45.4M	45.4M	$\mathbf{9.2{\pm}0.4}$	$\mathbf{30.2{\pm}1.2}$	$\mathbf{7.3{\pm}0.5}$

Table 7: Model size and matched-hardware efficiency. † Published LeWM baseline. Latency is evaluated per replan (

K=4

). Memory and latency report mean

\pm

std over five runs on a single A100-40GB (BF16), with training batch size 32 and inference batch size 1 (post-100 warm-up replans).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Shape	Meaning	Use
$X_{t}$	$VT_{o}P\times d$	Multi-view patch tokens over observation history $T_{o}$	Soft entity extraction and optional proposal bias
$E_{t}$	$N\times d_{e}$	$N$ soft entity hypotheses with appearance, geometry, motion, and confidence	Task-conditioned role binding
$Z_{t}$	$5\times d$	Five task-conditioned state components	Predictive interface for FM generation, semantic dynamics, and selection
$A_{t}^{k}$	$H\times d_{a}$	Candidate action chunk $k$	Closed-loop execution or repair
$\widehat{Z}_{t,j}^{\,k}$	$5\times d$ for each $j$	Predicted task-conditioned state at sub-horizon boundary $j$	Transition supervision and violation localization
$\widehat{Y}_{t,j}^{\,k}$	task-dependent	Contact, grasp, predicates, preservation, phase, and fixture state	Semantic scoring and diagnostics

Appendix

Table S1: Symbolic tensor interface. Numerical dimensions are intentionally kept symbolic because they are not needed to interpret the reported comparisons; exact dimensions are checkpoint metadata in the artifact.

Property	Slots + language	SR-WM
Perception	Exchangeable entity slots	Exchangeable entity hypotheses
Language use	Global context/readout	Entity–role binding
Predictive interface	Entity set	Task-conditioned state $Z_{t}$
Action generation	Generic slot context	FM conditioned on $Z_{t}$
Dynamics	Generic future features	Action-conditioned state transitions
Planning	Scalar/readout	Establish, preserve, phase, fixture, repair

Appendix

Table S2: Operational distinction between global language conditioning over exchangeable entities and SR-WM’s task-conditioned predictive interface.

Question	Direct evidence	Empirical reading	Scope
Does state structure affect generation?	Main Table 1, Candidate 0 and Oracle@4	The complete predictive-state generator improves nominal success and sampled coverage relative to the monolithic-latent generator	Complete-system comparison at matched $K$ and rollout seeds
Does language alone explain the gain?	Main Table 5	Global instruction context improves slots, while explicit grounding, interaction state, and progress yield further gains	Complete state design with matched predictive supervision
Are predictions action-sensitive?	Main Table 4	Hard negatives and transition losses improve pairwise action discrimination	Registered positive/negative construction and score metric
Are masks required at inference?	Main Table 3	Patch-only and corrupted-prior variants remain operational without oracle masks	Evaluated LIBERO suites and named prior conditions
Does the state support transfer?	Main Table 6	Source pretraining improves target-data adaptation on Goal, LIBERO-10, and jointly pretrained held-out tasks; Spatial remains comparable	Matched target data and adaptation updates
What is the complete-system cost?	Main Table 7	The SR-WM + FM system has 45.4M parameters with the reported memory and latency	Reported hardware, precision, batch, and warm-up protocol

Appendix

Table S3: Experimental claim boundaries for the main paper. Oracle@4 measures sampled coverage; selector performance is reported by selected top-1 success.

Component	Inputs emphasized	State represented	Direct supervision
Gripper	Proprioception, gripper-region patches	End-effector pose, aperture, motion, contact, grasp	Pose/aperture targets; contact and grasp labels
Target	Instruction and candidate object entities	Task-relevant object or articulated part	Target identity, footprint, center, or part label
Goal	Instruction and receptacle/fixture entities	Destination region or desired fixture state	Goal identity, region, or terminal-state label
Relation	Gripper–target and target–goal features	Relative geometry and symbolic predicates	Predicate establishment and preservation
Phase	Temporal history of all state components	Approach, grasp, transport, align, release, complete	Phase class and valid-transition label

Appendix

Table S4: Inputs, semantics, and training supervision for the five predictive state components. Simulator state and task-derived labels supervise targets; they are not rollout inputs.

Signal	Train input/target	Rollout input	Oracle only	Scope
RGB history, instruction, proprioception	Yes	Yes	No	Common deployable observation interface
Expert actions and observed future frames	Target	No	No	FM and predictive-state supervision
Target/goal identity, footprint, center, part ID	Target	No	No	Role-binding supervision from task specification or annotation
Contact, native predicates, fixture joints, phase	Target	No	No	Semantic-transition supervision from simulator state
Soft proposal prior (Grounding DINO + SAM/SAM2)	Optional	Optional	No	Used only in explicitly labeled prior conditions
Oracle segmentation masks as model input	Diagnostic variant	No	Yes	Perceptual-headroom condition in main Table 3; distinct from training footprints

Appendix

Table S5: Information-access audit. “Target” denotes training-only supervision, not a model input. The patch-only condition consumes no segmentation proposal at rollout time.

Loss	Prediction target	Label source	Primary failure prevented
$\mathcal{L}_{\mathrm{LeWM}}$	Future latent feature	Future observation	Predictive collapse
$\mathcal{L}_{\mathrm{role}}$	Binding identity/geometry	Task spec., simulator, annotation	Wrong target or goal binding
$\mathcal{L}_{\mathrm{dyn}}$	Future predictive state	Future encoded states	Action-insensitive state
$\mathcal{L}_{\mathrm{contact}}$	Contact and stable grasp	Collision, aperture, relative motion	False grasp progress
$\mathcal{L}_{\mathrm{predicate}}$	Establishment/conjunction	Native task predicates	Partial or wrong completion
$\mathcal{L}_{\mathrm{preserve}}$	Previously true predicates remain true	Predicate history	Undoing completed subgoals

Appendix

Table S6: The eleven granular training losses and their functions.

Signal	Source	Construction	Ambiguity handling
Contact	Collision/contact state	Positive when gripper and target have valid contact	Mask invalid simulator contacts
Stable grasp	Contact, aperture, relative motion	Contact persists while target follows gripper over a temporal window	Mask transient contact
Predicate established	Native predicate evaluator	Required predicate changes from false to true	Use task argument binding
Predicate preserved	Predicate history	Predicate true at prior boundary remains true	Defined only after establishment
Conjunction	Task predicate graph	All currently required predicate leaves are true	Preserve individual leaf labels
Phase	Event-driven finite-state parser	Approach, grasp, transport, align, release, complete	Mask boundary-transition frames

Appendix

Table S7: Semantic-label sources and construction rules.

Setting reported in the main paper	Value
LeWM encoder–predictor reference	15.0M parameters
Complete SR-WM + FM system	45.4M parameters
Predictive state components	5
Primary candidate count	$K=4$
Candidate-count sweep	$K\in\{1,2,4,8\}$
Efficiency training batch	32

Appendix

Table S8: Configuration values explicitly reported by the main paper. The 15M number refers to the retained LeWM encoder–predictor, not the complete 45.4M-parameter system.

Representation package	Added structure	Success (%)
Five instance slots	Entity identity	$55.0\pm 3.6$
Instance slots + instruction	Global task context	$62.0\pm 3.3$
Target + goal binding	Entity–role grounding	$68.0\pm 3.1$
Target + goal + relation state	Interaction state	$73.0\pm 2.9$
Full SR-WM state	Gripper and temporal phase	$\mathbf{83.0\pm 2.4}$

Appendix

Table S9: Progressive predictive-state construction reproduced from main Table 5. Language supplies context; grounding, interaction state, and progress construct task variables used by dynamics and planning.

Comparison	Matched factors	Jointly varied factors	Supported reading
Main Table 1 generators	$K$ , tasks, initial states, FM family	Representation, supervision, parameters, sampled actions	Complete-system generation and coverage
Main Table 1 Candidate 0/random/fixed/selector	Same SR-WM candidates and rollout states	Selection rule	Conversion of fixed coverage to top-1 success
Main Table 5 state composition	Token interface and downstream FM/selector	State construction and matched predictive supervision	Joint contribution of the predictive-state design
Full resampling vs. suffix repair	Base model and maximum generation-call budget	Prefix pinning; realized latency	Benefit of prefix-preserving repair at matched calls
Main Table 3 mask conditions	Selector architecture and $K$	Perceptual prior and potentially its training condition	Operational mask dependence under named conditions
Adapted vs. scratch	Target data, updates, test tasks	Source pretraining data and compute	Suite-dependent target-data adaptation comparison

Appendix

Table S10: Experimental controls and scope. Each row identifies matched factors, jointly varied factors, and the corresponding system-level reading.

Result family	Required provenance key	Interpretation key
Candidate 0 / Oracle@4	Generator checkpoint, $K$ , FM noise, environment seed	Candidate 0 measures the prior; Oracle@4 measures sampled coverage
Selector / repair	Exact candidate set, selector checkpoint, extra call count	Decision and repair changes are read under matched candidate/call budgets
State composition	Variant config, supervision masks, trainable parameter count	Joint state-parameterization and predictive-supervision comparison
Mask diagnostics	Prior source, training condition, test-time condition	Operational prior dependence; oracle masks quantify perceptual headroom
Cross-suite transfer	Source data, target data, updates, checkpoint selection	Target-data adaptation after source pretraining
Efficiency	Full model graph, hardware, precision, warm-up, batch size	45.4M complete system; 15M retained-backbone reference

Appendix

Table S11: Minimum provenance needed to regenerate and interpret the reported result families.

Category	Operational definition	Candidate diagnostic	Relevant component
Binding error	Target or goal role attends to wrong entity/fixture part	Low binding accuracy	Binding loss and instruction grounding
Approach error	End effector fails to reach valid target contact	Contact false; approach phase stalls	FM generation and geometry
Grasp error	Contact occurs but a stable grasp is never achieved or is subsequently lost	Contact/grasp disagreement	Contact head and preservation
Transport error	Target departs gripper or moves away from goal	Negative preservation or relation transition	State dynamics and selector
Alignment error	Target reaches goal vicinity but required predicate fails	High geometry confidence, low establishment	Relation state and predicate head
Premature release	Gripper opens before goal predicate is valid	Invalid phase transition	Phase head and hard negatives

Appendix

Table S12: Annotation taxonomy for qualitative failure analysis. Rollout counts and rates accompany any aggregate use of these categories.

Gate	Acceptance evidence
Timestamp/camera sync	Replay report under the robot’s registered sensor tolerance
Raw context agreement	Stored-context versus online-provider field-level diff
Normalized support	No unexplained normalized dimension with $\|z\|>6$
Shadow stability	At least 100 consecutive non-actuating inference steps
Policy latency	P50/P95/P99 latency satisfies the registered control deadline
First-action clipping	No unexplained clipping event in shadow/canary logs

Appendix

Table S13: Physical deployment gates. Robot-specific numerical tolerances belong to the actuation contract and must be registered before data are inspected.

World Model-Based Planning

World model-based planning uses models of how environments change to compare imagined actions and choose goal-directed behavior.

See all 261 papers

PrismWAM: Decoupling State Representations in World Action Models
Oct 19, 2026Tianyu Xin, Wanqi Zhong, Bowen Hu +8World Models for Robotics World Models
Tsinghua University · NVIDIA
Semantic World Models
Oct 22, 2025Jacob Berg, Chuning Zhu, Yanda Bao +2World Models for Robotics World Model-Based Planning
University of Washington · Sony AI
Grounded World Model: Latent Planning with Language Goals
Apr 13, 2026Quanyi Li, Lan Feng, Haonan Zhang +4Language-Conditioned Robot Planning World Models for Robotics
EPFL · NUS

Oct 19, 2026J/K move · Enter open · S save

PrismWAM: Decoupling State Representations in World Action Models

Tianyu Xin, Wanqi Zhong, Bowen Hu, Zhangchi Guo+7

Tsinghua University · NVIDIA

World Models for Robotics World Models World Model Learning Action-Conditioned World Models Robotic Manipulation

Open paper AskPDF arXiv

World Action Models (WAMs) typically represent the world through latent embeddings from pretrained video models, which entangle heterogeneous aspects of the world, from how it looks (e.g., texture and color) to how it is physically configured (e.g., occupancy and pose). This raises an underexplored question:

\textbf{what should constitute the state of a WAM?}

We argue that the world state should separate how the world appears from how the world evolves under action: the former captures semantic context, while the latter is governed by physical state. To this end, we propose

\textbf{PrismWAM}

, which decouples physics from semantics while jointly modeling state and action. PrismWAM factorizes an embodied task into a

\mathrm{Semantic\ Planning}\rightarrow\mathrm{Physical\ Rollout}\rightarrow\mathrm{Executable\ Action}

pipeline: given only the initial RGB observation, semantic planning learns to predict the final goal image, while physical rollout predicts the fine-grained evolution toward this goal in depth space; executable actions are then generated from the depth rollout. Without any embodied action pretraining or additional data augmentation, PrismWAM achieves strong manipulation performance in both simulation and the real world. Notably, PrismWAM even surpasses a broad range of models with large-scale embodied action pretraining (e.g., OpenWAM-

α

). More broadly, we reframe World Action Models from modeling how the world looks to how it responds to action, paving the way for action-centric world models grounded in occupancy, flow, tactile sensing, proprioception, and beyond.

Abstract

World Model-Based Planning

Explore similar work

PrismWAM: Decoupling State Representations in World Action Models

Semantic World Models

Grounded World Model: Latent Planning with Language Goals