SafeGen-LLM: Enhancing Safety Generalization in Task Planning for Robotic Systems

Jialiang Fan; Weizhe Xu; Mengyu Liu; Oleg Sokolsky; Insup Lee; Fanxin Kong

cs.ROFeb 27, 2026

SafeGen-LLM: Enhancing Safety Generalization in Task Planning for Robotic Systems

Authors: Jialiang Fan, Weizhe Xu, Mengyu Liu, Oleg Sokolsky, Insup Lee, Fanxin Kong

Organizations: University of Notre Dame · Washington State University · University of Pennsylvania

Abstract

Safety-critical task planning in robotic systems remains challenging: classical planners suffer from poor scalability, Reinforcement Learning (RL)-based methods generalize poorly, and base Large Language Models (LLMs) cannot guarantee safety. To address this gap, we propose safety-generalizable large language models, named SafeGen-LLM. SafeGen-LLM can not only enhance the safety satisfaction of task plans but also generalize well to novel safety properties in various domains. We first construct a multi-domain Planning Domain Definition Language 3 (PDDL3) benchmark with explicit safety constraints. Then, we introduce a two-stage post-training framework: Supervised Fine-Tuning (SFT) on a constraint-compliant planning dataset to learn planning syntax and semantics, and Group Relative Policy Optimization (GRPO) guided by fine-grained reward machines derived from formal verification to enforce safety alignment and by curriculum learning to better handle complex tasks. Extensive experiments show that SafeGen-LLM achieves strong safety generalization and outperforms frontier proprietary baselines across multi-domain planning tasks and multiple input formats (e.g., PDDLs and natural language).

Figures & tables

Fig. 1 : Overview of the proposed SafeGen-LLM framework. Stage I performs SFT on formally verified, safety-constrained plans. Stage II applies GRPO using fine-grained reward signals derived from formal verification to enforce safety alignment.

Domain	Description	Safety Constraints
Blocksworld	The agent must rearrange stacked blocks into a desired configuration using pick-and-place actions.	Blocks must be stacked in a safe and stable order, e.g., a supporting block must be positioned before the blocks it supports.
Ferry	The agent controls a ferry that transports cars between locations.	The ferry must never be overloaded, and cars must safely reach their destinations before the ferry departs or returns.
Grippers	A robot with two grippers must pick up and deliver all objects to a target room.	Robots (or manipulators) must avoid operating simultaneously in the same narrow room, and certain grippers may be restricted for safety-critical objects.
Spanner	A worker must collect spanners and tighten all nuts across connected locations.	Bolts must be tightened in a safe sequence (e.g., foundation nuts before upper ones), and access to shared tools (e.g., entering the tool shed) is limited to prevent unsafe congestion.

TABLE I : Brief introduction of the domains and their safety constraints.

Fig. 2 : Pipeline for dataset construction.

Panel of Figure 4 — Fig. 3 : Three-way benchmark comparison: GPT-5.2 vs OPTIC vs Fast Downward. Green squares = success, red circles = failure.

Fig. 4 : Error type distribution across training stages (Pretrained $\rightarrow$ SFT $\rightarrow$ GRPO) for Mistral-7B in the Blocksworld domain.

Qwen3-14B
Domain	Success Plans			Plan Format Error			Precondition Violation			Safety Constraint Violation			Goal Not Satisfied
	PT	SFT	GRPO	PT	SFT	GRPO	PT	SFT	GRPO	PT	SFT	GRPO	PT	SFT	GRPO
Blocksworld	2.0	70.0	88.0	90.0	0.0	0.0	6.0	24.0	12.0	0.0	2.0	0.0	2.0	4.0	0.0
Ferry	0.0	86.0	96.0	10.0	0.0	0.0	90.0	4.0	4.0	0.0	10.0	0.0	0.0	0.0	0.0
Grippers	2.0	92.0	98.0	0.0	0.0	0.0	46.0	4.0	0.0	46.0	4.0	2.0	6.0	0.0	0.0
Spanner	0.0	100.0	100.0	2.0	0.0	0.0	98.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

TABLE II : Error Type Percentages by Domain Across Training Stages

Domain	Success $\uparrow$			Format Error			Precondition Violation			Safety Constraint Violation			Goal Not Satisfied
	PDDL3	NL	JSON	PDDL3	NL	JSON	PDDL3	NL	JSON	PDDL3	NL	JSON	PDDL3	NL	JSON
Blocksworld	88.0	74.0	80.0	0.0	0.0	0.0	12.0	18.0	8.0	0.0	8.0	8.0	0.0	0.0	4.0
Ferry	96.0	90.0	96.0	0.0	0.0	0.0	4.0	6.0	2.0	0.0	4.0	2.0	0.0	0.0	0.0
Grippers	98.0	82.0	98.0	0.0	0.0	0.0	0.0	14.0	2.0	2.0	4.0	0.0	0.0	0.0	0.0
Spanner	100.0	90.0	96.0	0.0	2.0	0.0	0.0	8.0	4.0	0.0	0.0	0.0	0.0	0.0	0.0
Average	95.5	84.0	92.5	0.0	0.5	0.0	4.0	11.5	4.0	0.5	4.0	2.5	0.0	0.0	1.0

TABLE III : Performance comparison: Symbolic (PDDL3) vs Natural Language vs JSON input. All values in %.

Fig. 5 : Cross-model success rate comparison across four domains.

Success Rate (%) $\uparrow$
Model	Blocksworld	Ferry	Grippers	Spanner	Avg.
Pretrained	50.0	0.0	50.0	0.0	25.0
GRPO	94.0	98.0	98.0	100.0	97.5
Avg. Retries $\downarrow$
Pretrained	3.50	5.00	3.00	5.00	4.13
GRPO	1.36	1.12	1.14	1.00	1.16

TABLE IV : Comparison of Planning Success Rates and Retry Counts with SafePilot.

Panel of Figure 10 — Fig. 6 : Real-world validation in Blocksworld: (a) simulation case study comparing a classical solver (unsafe) with our safety-aware planner (safe); (b) physical deployment on a robot arm.

Hyperparameter	SFT	GRPO
Learning rate	$2\times 10^{-4}$	$1\times 10^{-5}$
LR scheduler	Cosine	Cosine
Batch size	4	8
Gradient accumulation	2	4
Effective batch size	8	32
Training duration	3 epochs	1,000 steps

TABLE V : Training hyperparameters for SFT and GRPO.

Category	Reward range
Success ( $c_{5}$ )	$+1.0$ (fixed)
Goal not satisfied ( $c_{4}$ )	$[-0.4,\;-0.1]$
Precondition violation ( $c_{3}$ )	$[-0.6,\;-0.3]$
Safety violation ( $c_{2}$ )	$[-0.9,\;-0.6]$
Format error ( $c_{1}$ )	$-1.0$ (fixed)

TABLE VI : Reward intervals for each validation category.

Phase	Progress	Easy	Medium	Hard
Early	0–30%	70%	25%	5%
Mid	30–70%	40%	40%	20%
Late	70–100%	20%	40%	40%

TABLE VII : Curriculum schedule: sampling probabilities by training phase.

Domain	Train	Test	Problem size range
Blocksworld	500	50	3–6 blocks
Ferry	500	50	3–4 locations, 2–3 cars
Grippers	500	50	1 robot, 3–4 rooms, 3 objects
Spanner	500	50	2–3 spanners, 2 nuts, 3–4 locations
Total	2,000	200	—

TABLE VIII : Dataset statistics and problem size ranges per domain.

Parameter	Local	API
Temperature	0.6	default
Top- $p$	0.9	default
Max new tokens	1,024	default
Max sequence length	5,000	—
Quantization	4-bit	—
Timeout per problem	—	300s

TABLE IX : Evaluation parameters for local and API-based models.

Explore similar work

Using large language models for embodied planning introduces systematic safety risks
Apr 20, 2026Tao Zhang, Kaixian Qu, Zhibin Li +4Large Language Model-Based Robot Planning Robot Safety
ETH Zurich, Zurich, Switzerland. · University College London, London, United Kingdom. · Stanford University, Stanford, California, United States. +2
SafeInferCom: Safe Inference-Time Compute via Verifier-Guided Mid-Generation Intervention for Robotic Task Planning
Oct 8, 2026Weizhe Xu, Jialiang Fan, Mengyu Liu +1Large Language Model-Based Robot Planning Inference-Time Guidance
University of Notre Dame · Washington State University
Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generation
Sep 25, 2025Jinbang Huang, Zhiyuan Li, Yuanzhao Hu +4Large Language Model-Based Robot Planning Synthetic Data Generation for Robotics
Huawei Noah’s Ark Lab · University of Toronto · University of British Columbia +1

Apr 20, 2026J/K move · Enter open · S save

Using large language models for embodied planning introduces systematic safety risks

Tao Zhang, Kaixian Qu, Zhibin Li, Jiajun Wu+3

ETH Zurich, Zurich, Switzerland. · University College London, London, United Kingdom. · Stanford University, Stanford, California, United States.+2

Large Language Model-Based Robot Planning Robot Safety Robot Task Planning LLM Safety Evaluation

Open paper AskPDF arXiv

Large language models are increasingly used as planners for robotic systems, yet how safely they plan remains an open question. To evaluate safe planning systematically, we introduce DESPITE, a benchmark of 12,279 tasks spanning physical and normative dangers with fully deterministic validation. Across 23 models, even near-perfect planning ability does not ensure safety: the best-planning model fails to produce a valid plan on only 0.4% of tasks but produces dangerous plans on 28.3%. Among 18 open-source models from 3B to 671B parameters, planning ability improves substantially with scale (0.4-99.3%) while safety awareness remains relatively flat (38-57%). We identify a multiplicative relationship between these two capacities, showing that larger models complete more tasks safely primarily through improved planning, not through better danger avoidance. Three proprietary reasoning models reach notably higher safety awareness (71-81%), while non-reasoning proprietary models and open-source reasoning models remain below 57%. As planning ability approaches saturation for frontier models, improving safety awareness becomes a central challenge for deploying language-model planners in robotic systems.

Abstract

Explore similar work

Using large language models for embodied planning introduces systematic safety risks

SafeInferCom: Safe Inference-Time Compute via Verifier-Guided Mid-Generation Intervention for Robotic Task Planning

Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generation