Safety, Security, and Cognitive Risks in World Models

Manoj Parmar

cs.CRApr 1, 2026

Safety, Security, and Cognitive Risks in World Models

Authors: Manoj Parmar

Organizations: SovereignAI Security Labs, Bengaluru, India

Abstract

World models - learned internal simulators of environment dynamics - are rapidly becoming foundational to autonomous decision-making in robotics, autonomous vehicles, and agentic AI. By predicting future states in compressed latent spaces, they enable sample-efficient planning and long-horizon imagination without direct environment interaction. Yet this predictive power introduces a distinctive set of safety, security, and cognitive risks. Adversaries can corrupt training data, poison latent representations, and exploit compounding rollout errors to cause significant degradation in safety-critical deployments. At the alignment layer, world model-equipped agents are more capable of goal misgeneralisation, deceptive alignment, and reward hacking. At the human layer, authoritative world model predictions foster automation bias, miscalibrated trust, and planning hallucination. This paper surveys the world model landscape; introduces formal definitions of trajectory persistence and representational risk; presents a five-profile attacker taxonomy; and develops a unified threat model drawing on MITRE ATLAS and the OWASP LLM Top 10. We provide an empirical proof-of-concept demonstrating trajectory-persistent adversarial attacks on a GRU-based RSSM ( $\mathcal{A}_1 = 2.26\times$ amplification, $-59.5\%$ reward reduction under adversarial fine-tuning), validate architecture-dependence via a stochastic RSSM proxy ( $\mathcal{A}_1 = 0.65\times$ ), and probe a real DreamerV3 checkpoint (non-zero action drift confirmed). We propose interdisciplinary mitigations spanning adversarial hardening, alignment engineering, NIST AI RMF and EU AI Act governance, and human-factors design, arguing that world models require the same rigour as flight-control software or medical devices.

Figures & tables

Profile	Access	Knowledge	Goal	Primary Techniques
White-box	Full stack (weights, gradients, architecture)	Complete	Targeted policy manipulation	Gradient-based encoder perturbation; dynamics poisoning
Grey-box	API-level query access; partial arch. knowledge	Partial	Model extraction; policy degradation	Black-box adversarial examples; model extraction via rollout queries
Black-box	Input channel only (sensors, data feeds)	None	Inducing unsafe states at inference	Transfer attacks; physical adversarial patches; supply-chain seeding
Insider	Training pipeline, data stores	Complete	Backdoor injection; persistent sabotage	Poisoned training batches; tampered checkpoints
Supply-chain	Pre-training corpus; upstream model providers	Partial	Encode representational bias in foundation model	Web-scale data poisoning; adversarial data seeding

Table 1: Attacker Capability Taxonomy for World Model Systems

Figure 1: Trajectory-Persistent Adversarial Attack Experiment (V3 Results): Core Results. (A) Mean latent-state error (

\ell_{2}

) over

K=30

rollout steps following a single adversarial perturbation at

t=0

(

\varepsilon=0.05

N=200

trials, GRU-based RSSM). The world model (WM, blue) amplifies the perturbation at step 1 (

\mathcal{A}_{1}=2.26\times

) before GRU contraction attenuates it; the single-step baseline (SS, orange) shows no state-mediated amplification. (B) Architecture comparison: trajectory amplification ratio

\mathcal{A}_{k}

on a log scale for the deterministic GRU world model vs. a stochastic RSSM proxy (posterior at

t=0

, prior rollout thereafter). The RSSM proxy shows lower initial amplification (

\mathcal{A}_{1}=0.65\times

) and slower decay, confirming architecture-dependence. (C) Real DreamerV3 checkpoint probe (seed 0): per-metric bar chart showing

\mathcal{A}_{1}

, normalised latent error

E_{1}

, and action drift

\|\Delta a_{1}\|

. Non-zero coupling confirms representational perturbations propagate into policy outputs. Panels (D)–(F) (mitigation effect, budget sensitivity, reward gap) appear in Figure 2 in the appendix. All error bands show

\pm 1

SE.

Area	Key Actions	Acceptance Criteria
Architecture	Document all six layers (encoder, dynamics, reward, rollout, policy, memory); define trust boundaries and data flows; identify pre-trained vs. fine-tuned layers; produce an architecture threat model using STRIDE or MITRE ATLAS [ 2 , 17 ] .	Threat model signed off by security lead; all layer owners identified.
Adversarial Hardening	Train encoders with $\varepsilon$ -ball trajectory-persistent adversarial examples (PGD, AutoAttack); deploy ensemble or MC-Dropout dynamics model; validate with PhysCond-WMA and AdvSim scenarios [ 19 , 21 , 60 ] .	$\mathcal{A}_{k}<5\times$ at $k=10$ under $\varepsilon=0.05$ ; clean-task performance degradation $<5\%$ .
Supply-Chain	Cryptographically sign training dataset batches (SHA-256 manifests); run backdoor detection (Neural Cleanse or ABS) on checkpoints; red-team pre-trained models before fine-tuning [ 51 , 24 , 52 ] .	Zero detected backdoors; provenance records retained for $\geq 2$ years.
Rollout Safety	Enforce epistemic uncertainty threshold $U_{\max}$ on rollout outputs with auto-halt; implement SafeDreamer-style constrained planning; define OOD rollback triggers [ 29 , 8 , 57 ] .	$<1\%$ of deployment rollouts exceed $U_{\max}$ without human review; constraint violation rate $<0.1\%$ in safety-critical domains.
Alignment	Audit reward heads for proxy objectives (feature attribution, TCAV); apply conservative offline RL penalties (MOPO/COMBO); monitor latent trajectories for deceptive alignment signals [ 25 , 66 , 64 ] .	Reward head causal attribution $>80\%$ on causal features; no detected evaluation-conditioned policy switching.
Privacy & Extraction	Apply DP-SGD ( $\varepsilon_{\mathrm{DP}}\leq 8$ ) during pre-training; rate-limit API queries ( $<1000$ /hr per principal); test for inversion using MI-FACE or GMI [ 63 , 61 ] .	Inversion reconstruction SSIM $<0.1$ ; extraction clone accuracy $<60\%$ of original.

Table 2: World Model Security and Safety Checklist (with acceptance criteria)

Claim	Evidence Type	Confidence	Open Validation Gap
World model encoders amplify single-step perturbations ( $\mathcal{A}_{1}=2.26\times$ )	Empirical (GRU proxy, $N=200$ )	High	Full-scale RSSM on benchmark tasks; pixel-space attacks on DreamerV3 training runs
Trajectory persistence is architecture-dependent (GRU vs. RSSM proxy)	Empirical (two architectures)	High	Comparison across $\geq 5$ architectures at scale; certified bounds
PGD-10 adversarial fine-tuning reduces $\mathcal{A}_{1}$ by 59.5%	Empirical (GRU proxy)	High	Mitigation efficacy on full DreamerV3/SafeDreamer training; downstream task reward
Non-zero action drift in real DreamerV3 checkpoint ( $\\|\Delta a_{1}\\|=0.0080$ )	Empirical (checkpoint probe)	Moderate	Statistical significance across seeds; full training-run checkpoints; task-specific evaluation
Delayed-trigger backdoors are harder to detect in MBRL than static classifiers	Literature synthesis + theory	Low–Moderate	End-to-end backdoor injection and detection experiment on pretrained world model
Foundation world model risk concentrates across downstream applications (Foundry Problem)	Theory + literature	Low–Moderate	Empirical measurement of cross-application failure propagation

Table 3: Claim–evidence map. Confidence levels: High = strong proof-of-concept evidence, direction consistent with theory; Moderate = pilot evidence, consistent with prior literature; Low–Moderate = partial empirical signal with theoretical support; Speculative = open hypothesis, no direct evidence.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Trajectory-Persistent Attack Experiment: Mitigation and Reward-Gap Results. (D) Mitigation effect: adversarial fine-tuning (PGD-10 on

t=0

) reduces

\mathcal{A}_{k}

substantially at all steps (before: solid; after: dashed). (E) Perturbation budget sensitivity before and after mitigation; the hardened model maintains lower error across the full

\varepsilon

range. (F) Absolute cumulative reward (clean vs. perturbed) as a function of planning horizon

H

; the reward gap at

H=30

0.000892\pm 0.000057

— small in absolute terms, confirming the GRU proxy operates in a regime of moderate (not catastrophic) reward damage; the world model gap remains consistently larger than the single-step gap. All error bands show

\pm 1

SE.

Adversarial Attacks on World Models

See all 17 papers

Jul 30, 2026

Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation

World models give embodied AI a predictive core: they compress observations into states, simulate action-conditioned futures, and enable planning beyond reactive control. This predictive layer, however, opens a new security boundary-compromise can propagate from data, sensors, prompts, or feedback into physical action. Rather than treating world models as an isolated component, this survey traces threats across their entire lifecycle-from data construction and representation learning, through state grounding and imagination, to trajectory evaluation, execution, and long-term adaptation via memory and tools. We show that familiar attack families: poisoning, backdoors, adversarial examples, sensor spoofing, prompt injection, trajectory manipulation, and supply-chain attacks take on distinct meanings when they corrupt world states, learned dynamics, affordance estimates, or safety costs. We also highlight a duality: world models can serve as runtime safety shields, yet when compromised or over-trusted they generate predictive safety illusions. The survey offers a lifecycle taxonomy, maps existing attacks to world-model security properties, outlines evaluation protocols for safety failures, and structures defenses across provenance, robust grounding, uncertainty-aware prediction, trajectory gating, feedback auditing, and deployment assurance.

Fazhong Liu, Zhuoyan Chen, Haozhen Tan +3

AI Agent Security World Model Evaluation Embodied World Models Adversarial Attacks on World Models

Ask PDF

Jul 25, 2026

False Prophets: On the Security of World Models in Agentic Systems

Large language models now power autonomous agents capable of complex, multi-step tasks in different environments. Accurate and reliable execution of these tasks requires the agent to predict the results of its actions. Recent research proposes to enhance predictive capabilities via specially trained environment simulators-world models. While world models can improve performance, they can also mislead agents into executing harmful actions, creating significant security and privacy risks. In this paper, we raise security concerns regarding the usage of world models in agentic systems. We discover a range of world model specific vulnerabilities, which can be exploited in terminal-based agents to execute malicious code or extract sensitive data. To facilitate future development, we introduce a security benchmark dataset designed for text-based world models. We argue that some risks are intrinsic to approximate world modeling, and show that attackers can induce mispredictions in agentic pipelines with up to 95% success rate, possibly resulting in unintended command execution, denial of service, drainage of wallet and private information extraction. Finally, we provide practical recommendations for practitioners to mitigate the discovered harms and harden agentic systems.

Erik Imgrund, Anna Wimbauer, Klim Kireev +1

BIFOLD & TU Berlin Berlin, Germany

Adversarial Attacks on World Models Adversarial Attacks AI Agent Security AI Agent Security Benchmarks

Ask PDF

Jun 15, 2026

ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control

World models are widely used in robotic and agentic engineering control systems due to their ability to learn latent dynamics for planning and decision-making. As these systems are increasingly deployed in safety-critical settings, understanding their robustness under adversarial conditions has become essential. However, existing evaluations lack a unified benchmark for testing adversarial threats across the policy, value, and latent-dynamics levels of world-model agents. To fill this gap, we present ARB4WM, a unified evaluation framework for pre-deployment robustness and risk assessment of world-model agents under visual perturbations. ARB4WM defines five white-box loss objectives across these three levels and studies their effects when combined with single-step or multi-step perturbation strategies and temporal attack modes, including full-frame, half-sequence, and sparse-frame exposure. Specifically, we evaluate four Dreamer-style agents across 20 tasks from MetaWorld and the DeepMind Control Suite under different loss objectives, perturbation strategies, and temporal attack modes. Results show that attacks targeting value estimation, latent representations, and RSSM dynamics can be as damaging as direct policy disruption, and that early or frequent perturbations are especially harmful, while input-level defenses provide limited recovery under adaptive attacks. These findings suggest that safety, risk, and reliability assessment for world models should cover multiple component-oriented attack objectives and temporal exposure protocols rather than relying solely on action-space robustness. Source code is available at https://github.com/zaoanguai/ARB4WM.

Junjian Zhang, Hao Tan, Ruonan Li +3

College of Computer Science, National University of Defense Technology, Changsha, 410073, China · National Key Laboratory of Advanced Communication Networks, Shijiazhuang, 050081, Hebei, China · College of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China +1

World Models for Robotics Adversarial Robustness World Model Evaluation Adversarial Attacks on World Models

Ask PDF

Abstract

Adversarial Attacks on World Models

Explore similar work

Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation

False Prophets: On the Security of World Models in Agentic Systems

ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control