ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Models

Zhenglin Lai; Sirui Huang; Yuteng Li; Changxin Huang; Jianqiang Li; Bingzhe Wu

cs.ROMar 31, 2026

ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Models

Authors: Zhenglin Lai, Sirui Huang, Yuteng Li, Changxin Huang, Jianqiang Li, Bingzhe Wu

Organizations: Shenzhen University, Shenzhen, China.

Abstract

Reference-image-conditioned video generators are increasingly used as neural simulators for embodied planning and policy learning, yet their ability to anticipate physical hazards is rarely evaluated. We introduce ICAT, which grounds test construction in incident reports and safety guidance through structured risk memories, constrained case generation, and human verification. The resulting benchmark contains 909 evaluation units with explicit initial conditions, triggering actions, and documented consequences. Across six generation systems, severe outcomes are frequently missing or attenuated even when the initial state is preserved and the risky action is executed: the strongest system achieves 23.5% conditional strict success under best-of-eight selection. Expert auditing, image-generator sensitivity analysis, matched safe-risky pairs, and alternative-outcome scoring provide complementary checks on this system-level finding. ICAT exposes a gap between reproducing risky actions and generating their documented consequences.

Figures & tables

Figure 1: An illustrative schematic, conceptually grounded in real-world physical phenomena, showing a common failure mode of video-generative world model.

Domain	$n$ (triggering mechanism)	$c$ (consequence)	$p$ (prevention / mitigation)
Home (hot oil)	Load frozen / wet food directly into hot oil in an uncovered pot	Rapid steam generation leads to violent oil splashing; potential ignition risk	Thaw and dry food; introduce slowly; avoid overfilling; keep flammables away
Lab (reactive metal)	Bring sodium metal into contact with water (e.g., drop into beaker)	Exothermic reaction releases flammable gas; ignition / fire risk	Keep away from water/moisture; handle under inert conditions; store properly
Factory / Lab (metal fire)	Apply water to burning magnesium / metal fire	Combustion intensifies; may generate flammable gas; ineffective extinguishing	Use Class-D dry powder agent; avoid water-based suppression

Table 1: Illustrative risk memory examples (simplified). Each memory stores

(n,c,p,r)

with a traceable reference

r

(incident report, EHS protocol, or safety manual).

Category	World Model	Home			Lab			Factory			Avg. $\mathrm{RCCC}$	$\mathrm{FST}$
Category	World Model	$\mathbb{E}[\eta_{\mathrm{init}}]$	$\mathbb{E}[\eta_{\mathrm{trg}}]$	$\mathbb{E}[\boldsymbol{\eta_{\mathrm{out}}}]$	$\mathbb{E}[\eta_{\mathrm{init}}]$	$\mathbb{E}[\eta_{\mathrm{trg}}]$	$\mathbb{E}[\boldsymbol{\eta_{\mathrm{out}}}]$	$\mathbb{E}[\eta_{\mathrm{init}}]$	$\mathbb{E}[\eta_{\mathrm{trg}}]$	$\mathbb{E}[\boldsymbol{\eta_{\mathrm{out}}}]$	Avg. $\mathrm{RCCC}$	$\mathrm{FST}$
Open	SkyReels-V2 14B	0.56	0.28	0.00	0.39	0.23	0.00	0.48	0.28	0.00	0.25	0.0%
	HunyuanVideo	0.49	0.23	0.00	0.33	0.21	0.00	0.44	0.24	0.00	0.21	0.0%
	Wow-wan 14B	0.51	0.26	0.00	0.32	0.18	0.00	0.46	0.25	0.00	0.22	0.0%
Closed	Veo3.1	0.72	0.58	0.23	0.60	0.55	0.30	0.72	0.56	0.20	0.49	10.9%
	Sora2	0.72	0.43	0.08	0.58	0.36	0.22	0.72	0.40	0.11	0.41	6.9%

Table 2: Risk prediction results of video-generative world models on the ICAT benchmark. Each world model is conditioned on

(v,i)

only; the grounded explanation

e

is used solely as the human-scoring reference. For each unit, we sample three videos and report the best-of-3 under

\mathrm{RCCC}

using an identical protocol for all models.

Generator (LLM)	Condition	IGR $\uparrow$ (Eq. 4 )				UHR $\downarrow$ (Eq. 5 )	DVS $\uparrow$ (Eq. 6 )
Generator (LLM)	Condition	Home	Lab	Factory	Avg.	UHR $\downarrow$ (Eq. 5 )	DVS $\uparrow$ (Eq. 6 )
GPT-5.1	w/o risk memory	0.62	0.58	0.56	0.59	0.23	0.63
GPT-5.1	w/ risk memory	0.76	0.71	0.68	0.72	0.11	0.73
Gemini 2.5 Pro	w/o risk memory	0.59	0.54	0.53	0.55	0.26	0.59
Gemini 2.5 Pro	w/ risk memory	0.72	0.69	0.66	0.69	0.14	0.70
Claude Sonnet 4.5	w/o risk memory	0.60	0.57	0.56	0.58	0.25	0.62

Table 3: Human evaluation of incident-grounded risk-memory grounding in ICAT for risk-case generation. All settings use the same evidence base

C

for scoring; the only difference is whether

C

is provided to the generator.

Figure 3: Representative qualitative cases for risk prediction across environments. Shown are success and failure examples under identical evaluation units $d=(v,i,e)$ for Home, Lab, and Factory, each presented as a three-frame sequence (Initial $\rightarrow$ Trigger $\rightarrow$ Outcome) with human scores. Across environments, world models commonly preserve initial conditions and realize the specified trigger, while hazardous outcomes are frequently missing, ambiguous, or severity-attenuated, leading to low $\eta_{\mathrm{out}}$ .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Score	TA $\uparrow$	PA $\uparrow$	MPAD $\downarrow$
All (joint)	0.96	–	–
$\eta_{\mathrm{init}}$	–	0.97	–
$\eta_{\mathrm{trg}}$	–	0.96	–
$\eta_{\mathrm{out}}$	–	–	0.07

Appendix

Table 4: Lightweight annotation reliability statistics.

Figure 4: Dataset distribution of the risk prediction evaluation set by environment (left) and embodiment (right).

AI Safety Evaluation

AI Safety Evaluation studies how to test whether AI systems avoid harmful behavior and remain reliable across users, tasks, and deployment settings.

See all 141 papers

Rethinking World Models for Safety-Critical Embodied Systems
Sep 3, 2026Kailang Ma, Heye Huang, Inhi Kim +1Runtime Assurance Embodied World Models
Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, South Korea.
Physically Viable World Models: A Case for Query-Conditioned Embodied AI
May 28, 2026Adam J. Thorpe, Stepan Tretiakov, Cheng-Hsi Hsiao +6World Models for Robotics Causal World Models
The University of Texas at Austin
Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
Jul 30, 2026Fazhong Liu, Zhuoyan Chen, Haozhen Tan +3AI Agent Security World Model Evaluation

Sep 3, 2026J/K move · Enter open · S save

Rethinking World Models for Safety-Critical Embodied Systems

Kailang Ma, Heye Huang, Inhi Kim, Kitae Jang

Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, South Korea.

Runtime Assurance Embodied World Models

Open paper AskPDF arXiv

World models have progressed from compact latent dynamics to generative, controllable, and interactive simulators of embodied environments. However, high predictive likelihood and visual fidelity do not necessarily ensure that a model preserves the evidence required for safe decision-making. This perspective identifies three structural mismatches in current world modeling: likelihood versus risk, prediction versus intervention, and finite-horizon prediction versus accumulated consequences. We propose the Risk-Informed World Model (RIWM) as a decision-centric research direction for safety-critical embodied systems. RIWM organizes world modeling around consequences, intervention, epistemic uncertainty, and recoverability, and integrates four interdependent capabilities: decision-relevant representation, counterfactual reasoning, safety-critical episodic memory, and runtime safety assurance. It distinguishes physical, social, and operational consequences while using epistemic uncertainty to qualify the evidence supporting action. We further discuss open challenges in identifying consequential futures, validating counterfactual reasoning, maintaining revisable safety memories, translating learned consequences into executable constraints, and determining when evidence is sufficient to act. This perspective argues that future world models should move beyond predicting likely futures toward identifying which futures matter, revising judgments through experience, and recognizing when to act, revise, sense, defer, or abstain.

Abstract

AI Safety Evaluation

Explore similar work

Rethinking World Models for Safety-Critical Embodied Systems

Physically Viable World Models: A Case for Query-Conditioned Embodied AI

Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation