VLM-DEWM: Dynamic External World Model for Verifiable and Resilient Vision-Language Planning in Manufacturing

Guoqin Tang; Qingxuan Jia; Gang Chen; Tong Li; Zeyuan Huang; Zihang Lv; Ning Ji

cs.ROFeb 17, 2026

VLM-DEWM: Dynamic External World Model for Verifiable and Resilient Vision-Language Planning in Manufacturing

Authors: Guoqin Tang, Qingxuan Jia, Gang Chen, Tong Li, Zeyuan Huang, Zihang Lv, Ning Ji

Organizations: School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, Beijing, 100876, China

Abstract

Vision-language model (VLM) shows promise for high-level planning in smart manufacturing, yet their deployment in dynamic workcells faces two critical challenges: (1) stateless operation, they cannot persistently track out-of-view states, causing world-state drift; and (2) opaque reasoning, failures are difficult to diagnose, leading to costly blind retries. This paper presents VLM-DEWM, a cognitive architecture that decouples VLM reasoning from world-state management through a persistent, queryable Dynamic External World Model (DEWM). Each VLM decision is structured into an Externalizable Reasoning Trace (ERT), comprising action proposal, world belief, and causal assumption, which is validated against DEWM before execution. When failures occur, discrepancy analysis between predicted and observed states enables targeted recovery instead of global replanning. We evaluate VLM-DEWM on multi-station assembly, large-scale facility exploration, and real-robot recovery under induced failures. Compared to baseline memory-augmented VLM systems, VLM DEWM improves state-tracking accuracy from 56% to 93%, increases recovery success rate from below 5% to 95%, and significantly reduces computational overhead through structured memory. These results establish VLM-DEWM as a verifiable and resilient solution for long-horizon robotic operations in dynamic manufacturing environments.

Figures & tables

Figure 1: Architectural Overview of the VLM-DEWM Framework. Operating on a Database-Transaction paradigm, the system is orchestrated through three tightly coupled modules. The perception module first projects raw multimodal sensory streams into a structured instantaneous snapshot ( $Env_{t}$ ) via geometric projection and filtration. This snapshot drives the thinking module, where the VLM serves as the reasoning core to simultaneously update the external world model (DEWM)—the central knowledge hub comprising environmental ( $\Omega$ ) and internal ( $M,CS$ ) memory—and synthesize a structured External Reasoning Trace (ERT) . Crucially, the validation module treats the persistent memory in the DEWM as ground-truth evidence to rigorously verify the ERT’s logical and physical consistency before dispatching the final action sequence to the Robot Skill Library $\mathcal{A}$ for execution.

Figure 2: Structural Composition of the Environment Memory Core ( $\Omega$ ). The Spatial Network ( $\mathcal{S}$ ) maintains geometric state (left), while the Hierarchical Semantic Graph ( $\mathcal{G}$ ) encodes logical relations (right). The Grounding Link ( $\Phi_{ground}$ , red) anchors every semantic node to its physical instantiation.

Action $\pi$	Pre-Condition ( $\mathcal{G}$ )	Atomic Effects ( $\Omega_{t+1},CS_{t+1}$ )	Rollback Trigger
Pick(obj)	$\exists\texttt{Clear(obj)}$	1. $\mathcal{S}_{obj}$ : Detach $\mathcal{PC}_{obj}$ 2. $\mathcal{G}$ : Del On(obj, *) 3. $CS$ : Set Holding(obj)	Gripper width $>$ limit (Slip)
Place(loc)	$CS=\texttt{Holding(obj)}$	1. $\mathcal{S}_{obj}$ : Anchor at $loc$ 2. $\mathcal{G}$ : Add On(obj, loc) 3. $CS$ : Set Idle	Force feedback $>$ limit (Collision)
Insert(A, B)	$\exists\texttt{Aligned(A, B)}$	1. $\mathcal{G}$ : Add Inserted(A, B) 2. $CS$ : Set Idle	Torque $>$ limit / Stall

Table 1: Atomic State Transition Rules for Primitive Skills

Figure 3: The Perception-to-Memory Synchronization Pipeline. Raw observations ( $O_{t}$ ) are transformed into persistent belief updates via three stages: (1) task-focused perception producing $Env_{t}$ , (2) geometric-semantic data association, and (3) dual-layer updates (continuous S-Update and discrete G-Update with geometric verification).

Figure 4: The Progressive VLM-Guided Planning and Deep Recovery Loop (Sec. 3.3). Dynamic Context Engineering distills the DEWM state ( $\Omega,M,CS$ ) into a task-relevant prompt. The VLM then generates an External Reasoning Trace (ERT) which is mathematically validated before execution. On failure, the Deep Causal Diagnosis module computes the belief-outcome discrepancy ( $\Delta=\Omega_{t+1}\ominus\mathcal{A}_{causal}$ ) to generate targeted feedback for belief correction and informed recovery.

Figure 5: Task 1 Environment Setup. The multi-zone factory simulation scene containing blocks, spheres, and cylinders distributed across three distinct platforms. This raw visual input triggers the memory construction process.

Panel of Figure 6 — Figure 6: Initial Memory Construction in Task 1. (a-b) The Geometric Layer of DEWM separates the static background (Left) from dynamic objects (Right). (c) The Semantic Layer initializes the topological graph and its matrix representation, anchoring the scene’s logical structure before manipulation.

Panel of Figure 7 — Figure 7: Reasoning and Dynamic Update Process. (a) Based on the initial memory, the VLM decomposes the instruction “Complete Red Assembly” into a hierarchical execution tree. (b) After execution, the DEWM reflects the new physical reality. By comparing this with Fig. 6(c) , we observe the real-time topology update (e.g., edge On(cube, table) is replaced by On(cube, cylinder) ), verifying the system’s dynamic tracking capability.

Method	VLM Core	TSR (%) $\uparrow$	STA (%) $\uparrow$	PE $\downarrow$	IE (%) $\uparrow$
VLM-DEWM (Ours)	Gemini 2.5 Pro	94.00	100.00	75	100.00
SAGE [ 18 ]	Gemini 2.5 Pro	72.00	83.33	261	12.26
GEAR [ 21 ]	Gemini 2.5 Pro	0.00	–	148	–
Gemini 2.5 Pro	Gemini 2.5 Pro	28.00	39.74	1	–
GPT-5	GPT-5	12.00	34.62	1	–
Qwen3-VL	Qwen3-VL	0.00	30.77	1	–

Table 2: Quantitative Results for Task 1 (Long-Horizon Assembly)

Figure 8: Task 2 Environment Setup. Multi-modal inputs (LIDAR + Camera) + semantic segmentation acquired during the town-scale exploration. This raw sensory stream serves as the input for constructing the DEWM.

Difficulty	Count	Description
Easy	15	Basic Recognition (e.g., “Is there a bank?”)
Medium	20	Spatial Relations (e.g., “Is the bench near the park?”)
Hard	5	Multi-step Reasoning (e.g., Navigation heuristics)
High	10	Daily Scenario Q&A

Table 3: Task 2 Spatial Question Set Distribution

Panel of Figure 9 — Figure 9: Hierarchical Memory Construction in Task 2. (a) The reconstructed global semantic map serves as the metric substrate. (b) Key landmarks (e.g., hydrants) are extracted as object-centric instances. (c) The Semantic Graph abstracts these entities into topological relations (e.g., Near(bench, park) ), enabling efficient spatial querying.

Figure 10: Exploration Execution History ( $\mathcal{M}$ ). The temporal tree visualizes the robot’s 240-step exploration path. Color-coded nodes indicate memory events: Green nodes mark steps where new entities were discovered and registered into $\mathcal{G}$ ; Yellow nodes indicate steps with only re-observations, demonstrating the system’s efficiency in avoiding redundant updates.

Method	VLM Core	QSR (%) $\uparrow$	STA (%) $\uparrow$	PE $\downarrow$	IE (%) $\uparrow$
VLM-DEWM (Ours)	Gemini 2.5 Pro	94.00	92.68	509	90.14
SAGE	Gemini 2.5 Pro	52.00	56.10	2162	1.30
GEAR	Gemini 2.5 Pro	74.00	-	240	-
Gemini 2.5 Pro	Gemini 2.5 Pro	-	29.27	240	-

Table 4: Quantitative Results for Task 2 (Exploration & Q&A)

Figure 11: Real-World Deployment. The Franka Emika Panda performing the dynamic assembly task. The system successfully diagnoses and recovers from induced execution failures using the $CS$ -based causal diagnosis.

Method	VLM Core	TSR (%) $\uparrow$	CDA (%) $\uparrow$
VLM-DEWM (Ours)	Gemini 2.5 Pro	95.00	36.96
SAGE	Gemini 2.5 Pro	0.00	0.00
GEAR	Gemini 2.5 Pro	0.00	0.00
Pure VLM	Gemini 2.5 Pro	0.00	0.00

Table 5: Real-World Dynamic Recovery Results (

N=60

)

Method	TSR (%) $\uparrow$	CDA (%) $\uparrow$	STA (%) $\uparrow$	QSR (%) $\uparrow$	PE (calls) $\downarrow$
VLM-DEWM (Full)	95.00	68.97	100.00	100.00	54
w/o $\mathcal{S}$	0.00	-	0.00	60.00	-
w/o $\mathcal{G}$	0.00	-	0.00	0.00	-
w/o $\mathcal{M}$	90.00	-	100.00	100.00	266
w/o $CS$	55.00	0.00	97.50	95.50	-

Table 6: Ablation Study of Core Components in Real-World Task 3 (

N=20

)

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

VLM Judgment ( $\alpha_{t}$ )	Weight ( $\gamma$ )	Effect
High	1.0	Full observation contribution
Medium	0.5	Balanced fusion
Low	0.1	Conservative update
Bad	0.01	Near-freeze at prior

Appendix

Table 7: Mapping from VLM Reliability Judgment to Fusion Weight

Action	Signal	Threshold	Triggered Event
Grasp	Gripper Force $F_{g}$	$<5$ N (after close)	Grasp Failure
Grasp	Gripper Force $F_{g}$	$>40$ N	Over-squeeze (abort)
Transport	End-effector $\|F_{ext}\|$	$>15$ N	Collision Detected
Place	End-effector $F_{z}$	$>10$ N (sustained)	Contact Confirmed
Insert	End-effector $\|\tau_{ext}\|$	$>2$ Nm	Jamming Detected

Appendix

Table 8: Force-Torque Thresholds for State Transition Triggers

$CS_{exp}$	$CS_{t+1}$	Diagnosed Failure
Holding(X)	Idle	Grasp Slip / Gripper Failure
Holding(X)	Holding(Y)	Object Misidentification
Approaching(X)	Idle	Motion Interrupted / Collision
Placing(X, Y)	Holding(X)	Placement Rejected / Unstable

Appendix

Table 9: Rule-Based Failure Diagnosis from

\Delta_{CS}

Field	Type	Description & Example
action_proposal	String	The proposed skill call. Ex: "Grasp(blue_block_1)"
world_belief	Object	Structured assertion about state. Ex: { "pred": "On", "args": ["A", "B"] }
causal_assump	String	Expected outcome. Ex: "CS becomes Holding(A)"
confidence	Float	Self-evaluated score [0.0, 1.0].
fallback	String	Contingency plan.

Appendix

Table 10: Schema Definition for External Reasoning Trace (ERT)

Component	Complexity
Data Association (naive)	$O(N_{obs}\cdot N_{obj})$
Data Association (with HSG)	$O(N_{obs}\log N_{obj})$
Gaussian Envelope Fusion	$O(1)$ per object
Scale-aware ICP	$O(I\cdot N_{p}\log N_{p})$
Subgraph Extraction	$O(\|V_{zone}\|)$
ERT Semantic Validation	$O(1)$ per entity

Appendix

Table 11: Summary of Computational Complexity

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 657 papers

ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback
Sep 29, 2026Ahmed Nader Ahmed, Omar Moured, Mughni Irfan Mohammed Abdul +2Robot Policy Adaptation Long-Horizon Robotic Manipulation
Khalifa University, Abu Dhabi, United Arab Emirates. · Sereact GmbH, Stuttgart, Germany.
H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model
Feb 11, 2026Jinbang Huang, Wenyuan Chen, Zhiyuan Li +9Long-Horizon Robotic Manipulation World Models for Robotics
Huawei Noah’s Ark Lab · University of Toronto · University of British Columbia +1
Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
Oct 13, 2026Juntong Feng, Qingan Liu, Jingru Dong +8VLM Reasoning VLMs for Robotics
Dalian University of Technology · Nanyang Technological University

Sep 29, 2026J/K move · Enter open · S save

ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback

Ahmed Nader Ahmed, Omar Moured, Mughni Irfan Mohammed Abdul, Muhayy Ud Din+1

Khalifa University, Abu Dhabi, United Arab Emirates. · Sereact GmbH, Stuttgart, Germany.

Robot Policy Adaptation Long-Horizon Robotic Manipulation Robot Task Planning VLMs for Robotics

Open paper AskPDF arXiv

Long-horizon robotic tasks are vulnerable to unexpected environmental changes that can render planned actions ineffective or unsafe. To address this, robots must detect such changes as they occur, interpret their impact, and adjust their actions accordingly. Traditional rule-based decision-making pipelines are brittle in open-world conditions, as they are hand-tuned for specific scenarios and lack generalization. Vision-Language Models (VLMs) offer a promising alternative as they combine broad world knowledge with unified visual--text reasoning, enabling them to generalize across diverse scenarios and generate accurate, grounded task plans. However, for effective deployment in dynamic real-world settings, VLMs must be embedded into frameworks capable of handling uncertainty and environmental changes. Existing frameworks broadly address this reactively, triggering replanning only after execution failures or post-task checks, risking failed actions. Some methods verify conditions before actions, but these discrete checks miss changes occurring during execution. To address this, we present ProAct-VLM, an adaptive, physically grounded task planning framework that integrates VLMs within a real-time perception--feedback loop. ProAct-VLM continuously monitors the environment and re-plans as soon as relevant changes are detected, enabling adaptation before failure occurs. Evaluations against multiple baselines and across different VLM backbones show that our framework improves both success rates and efficiency in dynamic, long-horizon manipulation tasks. Project page: https://github.com/moured/ProAct-VLM

Abstract

VLMs for Robotics

Explore similar work

ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback

H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?