SPOC: Safety-Aware Planning Under Partial Observability And Physical Constraints

Hyungmin Kim; Hobeom Jeon; Dohyung Kim; Minsu Jang; Jeahong Kim

cs.ROFeb 25, 2026

SPOC: Safety-Aware Planning Under Partial Observability And Physical Constraints

Authors: Hyungmin Kim, Hobeom Jeon, Dohyung Kim, Minsu Jang, Jeahong Kim

Organizations: ETRI School, University of Science and Technology, South Korean · Social Robotics Laboratory, Electronics and Telecommunication Research Institute, South Korea

Abstract

Embodied Task Planning with large language models faces safety challenges in real-world environments, where partial observability and physical constraints must be respected. Existing benchmarks often overlook these critical factors, limiting their ability to evaluate both feasibility and safety. We introduce SPOC, a benchmark for safety-aware embodied task planning, which integrates strict partial observability, physical constraints, step-by-step planning, and goal-condition-based evaluation. Covering diverse household hazards such as fire, fluid, injury, object damage, and pollution, SPOC enables rigorous assessment through both state and constraint-based online metrics. Experiments with state-of-the-art LLMs reveal that current models struggle to ensure safety-aware planning, particularly under implicit constraints. Code and dataset are available at https://github.com/khm159/SPOC

Figures & tables

Figure 1: Violations of PO (shown in (a) of the Figure) and PC (PC, shown in (b) of the Figure) can greatly reduce the feasibility. The existing accessible safety-aware ETP benchmarks [ 13 , 15 , 3 ] provide only limited support for PO and PC. In contrast, SPOC enables comprehensive handling of both PO and PC through low-level navigation and interaction actions.

Benchmark	PO	PC	GC	SSP	Env.
EARBench [ 15 ]	✗	✗	✗	✗	✗
SafeAgentBench [ 13 ]	✗	✗	$\Delta$ 2 2 2 SafeAgentBench includes GCs, but they are limited to very simple tasks. It does not provide GCs for long-horizon or abstract tasks. In addition, its evaluation focuses only on final states rather than online manner. This means it cannot detect safety constraint violations in real time through state tracking with strict GCs.	✗	$\Delta$ 3 3 3 Although SafeAgentBench supports a physical environment, it includes several unrealistic actions (e.g., cleaning, dirtying, or filling objects with water directly without using tools). Moreover, some instructions are not even executable within the environment at all, as they are generated by the LLM.
SafePlanBench [ 3 ]	✗	N/A 1 1 1 The embodiments of SafePlanBench based on a dual-arm design, which makes it unnecessary to deal with the level of PC considered in our work.	✓	✗	✓
SPOC (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of the evaluation capabilities and characteristics of previous benchmarks.

\Delta

indicates partial support. Abbreviations: Partial Observability (PO), Physical Constraints (PC), Goal Condition (GC), Step-by-Step Planning (SSP), Environment (Env).

Setting	CSR (%)	GSR (%)
PO	24.00 $\pm$ 3.27	38.56 $\pm$ 3.01
FO	36.00 $\pm$ 3.27	51.44 $\pm$ 1.50
IPC(PO)	32.00 $\pm$ 3.27	40.56 $\pm$ 1.50
IPC(FO)	45.33 $\pm$ 1.89	55.44 $\pm$ 0.57

Table 2: Ablation study on observability and physical constraints. IPC denotes Ignore Physical Constraints. Results are reported using the gpt5-mini model, averaged over three runs (mean

\pm

standard deviation) under the explicit setting.

Agent	SC	CSR (%)	GSR (%)
PreAct Agent [ 2 ]	E	29.33 $\pm$ 1.89	37.22 $\pm$ 2.20
PreAct Agent [ 2 ]	I	1.33 $\pm$ 1.89	24.33 $\pm$ 5.04
StateAct Agent [ 8 ]	E	26.67 $\pm$ 3.77	36.89 $\pm$ 6.15
StateAct Agent [ 8 ]	I	2.67 $\pm$ 1.89	22.66 $\pm$ 4.32
ReAct Agent [ 12 ]	E	24.00 $\pm$ 3.27	38.56 $\pm$ 3.01
ReAct Agent [ 12 ]	I	0.00 $\pm$ 0.00	21.00 $\pm$ 4.23

Table 3: Agent Structure Ablation on SPOC under explicit (E) and implicit (I) safety constraint representations. All results are obtained using the gpt5-mini model.

Model	SC	CSR (%)	GSR (%)
gpt5-nano [ 6 ]	E	4.00 $\pm$ 3.27	9.33 $\pm$ 4.11
gpt5-nano [ 6 ]	I	0.00 $\pm$ 0.00	6.00 $\pm$ 1.63
gpt5-mini [ 6 ]	E	24.00 $\pm$ 3.27	38.56 $\pm$ 3.01
gpt5-mini [ 6 ]	I	0.00 $\pm$ 0.00	21.00 $\pm$ 4.23
o4-mini [ 7 ]	E	28.00 $\pm$ 6.53	43.67 $\pm$ 3.60
o4-mini [ 7 ]	I	1.33 $\pm$ 1.89	27.89 $\pm$ 1.23

Table 4: Performance of the ReAct agent on SAPO under explicit (E) and implicit (I) safety constraint representations.

Robot Task Planning

Robot task planning turns goals and observations into executable actions while coordinating skills, motion, uncertainty, and safety.

See all 239 papers

Using large language models for embodied planning introduces systematic safety risks
Apr 20, 2026Tao Zhang, Kaixian Qu, Zhibin Li +4Large Language Model-Based Robot Planning Robot Safety
ETH Zurich, Zurich, Switzerland. · University College London, London, United Kingdom. · Stanford University, Stanford, California, United States. +2
SafeGen-LLM: Enhancing Safety Generalization in Task Planning for Robotic Systems
Feb 27, 2026Jialiang Fan, Weizhe Xu, Mengyu Liu +3Formal Methods for RL Large Language Model-Based Robot Planning
University of Notre Dame · Washington State University · University of Pennsylvania
SafeRun: Enabling Determinism in LLM Planning for Running
Jun 8, 2026Meilin Chen, Zepeng Zhai, Jiaxuan Zhao +1LLM Planning LLM Safety Evaluation
Xiaohongshu Inc. · Institute of Information Engineering, Chinese Academy of Sciences

Apr 20, 2026J/K move · Enter open · S save

Using large language models for embodied planning introduces systematic safety risks

Tao Zhang, Kaixian Qu, Zhibin Li, Jiajun Wu+3

ETH Zurich, Zurich, Switzerland. · University College London, London, United Kingdom. · Stanford University, Stanford, California, United States.+2

Large Language Model-Based Robot Planning Robot Safety Robot Task Planning LLM Safety Evaluation

Open paper AskPDF arXiv

Large language models are increasingly used as planners for robotic systems, yet how safely they plan remains an open question. To evaluate safe planning systematically, we introduce DESPITE, a benchmark of 12,279 tasks spanning physical and normative dangers with fully deterministic validation. Across 23 models, even near-perfect planning ability does not ensure safety: the best-planning model fails to produce a valid plan on only 0.4% of tasks but produces dangerous plans on 28.3%. Among 18 open-source models from 3B to 671B parameters, planning ability improves substantially with scale (0.4-99.3%) while safety awareness remains relatively flat (38-57%). We identify a multiplicative relationship between these two capacities, showing that larger models complete more tasks safely primarily through improved planning, not through better danger avoidance. Three proprietary reasoning models reach notably higher safety awareness (71-81%), while non-reasoning proprietary models and open-source reasoning models remain below 57%. As planning ability approaches saturation for frontier models, improving safety awareness becomes a central challenge for deploying language-model planners in robotic systems.

Abstract

Robot Task Planning

Explore similar work

Using large language models for embodied planning introduces systematic safety risks

SafeGen-LLM: Enhancing Safety Generalization in Task Planning for Robotic Systems

SafeRun: Enabling Determinism in LLM Planning for Running