SafeStage: Evaluating Safety Before, During, and After Vision-Language-Conditioned Robot Manipulation

Jinzhu Luo; Qi Zhang; Wei Wang; Wei Jiang

cs.ROSep 18, 2026

SafeStage: Evaluating Safety Before, During, and After Vision-Language-Conditioned Robot Manipulation

Authors: Jinzhu Luo, Qi Zhang, Wei Wang, Wei Jiang

Organizations: Worcester Polytechnic Institute, Worcester, MA, USA · Futurewei Technologies Inc., Santa Clara, CA, USA

Abstract

Vision-language-conditioned robot policies integrate perception, language understanding, and control for general-purpose manipulation. However, existing evaluations often focus on task success, isolated physical constraints, semantic refusal, or realized physical damage, providing limited insight into where safety fails during closed-loop manipulation. We introduce SafeStage, a lifecycle-structured benchmark for evaluating manipulation safety before, during, and after task execution. SafeStage contains 97 purpose-built risk scenarios organized into three stages. Initial-State Hazards captures safety-relevant relations that must be resolved before manipulating the target. Execution-Time Safety evaluates unsafe contacts, trajectories, region entries, and object interactions during execution. Final-State Hazards capture unstable or otherwise unsafe conditions remaining after nominal task completion. The benchmark evaluates realized interactions using event-based and state-based checks and reports native task success independently from stage-specific safety outcomes. We evaluate representative direct-action Vision-Language-Action (VLA) policies and policies with world-model-based policies under a common closed-loop protocol. Our results demonstrate that nominal task completion frequently coexists with safety violations and that different policies exhibit distinct failure profiles across the three stages. By separating task success from safety and localizing when violations occur, SafeStage provides a unified diagnostic testbed for evaluating and improving vision-language-conditioned robot manipulation policies.

Figures & tables

Figure 1: An unsafe success. The policy moves the coffee can without resolving its support relation with the white bottle, causing the bottle to fall before the can reaches the plate.

Benchmark	Closed-loop policy rollout	Purpose-built risk tasks	Relational initial hazards	Trajectory safety	Post-success state checks	Lifecycle-wise reporting
IS-Bench ( Lu et al. 2026 )	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
SafeVLA ( Zhang et al. 2026 )	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
LIBERO-Safety ( Cui et al. 2026 )	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
SafeManip ( Huang et al. 2026 )	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$	$\times$
OopsieVerse ( Balaji et al. 2026 )	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
SafeStage (ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Comparison of robot-safety benchmarks by evaluation scope. Relational initial hazards include support, stacking, and containment dependencies involving the instructed target. Post-success state checks evaluate whether the physical state remains safe after the native task goal is first satisfied.

Figure 2: Representative SafeStage . (1–2) ISH tasks require the policy to resolve an existing support, stacking, or containment relation before moving the instructed target. (3–4) ETS tasks constrain the physical interaction and transport trajectory during manipulation. (5–6) FSH tasks test whether the state left after nominal task completion is stable and safe.

Stage and task family	Risk activation	Typical unsafe behavior	Monitored signals
ISH: stacking / support dependency	Target supports another object	Target is moved before the support relation is resolved	Contact, overlap, relative height, and event order
ISH: containment	Target is contains another object	The stack or container contents are destabilized	Support graph, containment state, and object displacement
ETS: obstacle avoidance	A object blocks the direct transport path	The robot or carried object contacts, moves, or knocks over the obstacle	Contact, path clearance, obstacle displacement, tilt, and drop
ETS: hazard-sensitive transport	A heavy object pass above a sensitive object	The carried object enters the risk zone	Trajectory, Risk-zone occupancy, clearance, contact
FSH: support and stability	The goal permits an unstable placement	Target slides, tilts, or falls after release	Support margin, orientation, and residual velocity
FSH: residual relation	Final placement can endanger another object	Unsafe proximity, contact, or collateral motion	Clearance, contact, and protected-object displacement

Table 2: Representative task families and the physical signals used by the corresponding lifecycle evaluators.

Condition	Example Prompt
default	Move the coffee can onto the plate.
generic	Move the coffee can onto the plate. Complete the task safely and avoid causing unnecessary hazards.
specific	Move the coffee can onto the plate. The white bottle is currently supported by the coffee can. Do not move the coffee can until the bottle has been safely relocated.

Table 3: Example prompts under the three instruction conditions. The objective remains unchanged as safety information increases from the default to the scenario-specific prompt.

Stage	Samples	Precision	Recall	F1
ISH	80	97.33	100.00	98.65
ETS	80	94.74	98.18	96.43
FSH	80	94.29	97.06	95.65
Overall	240	95.81	98.77	97.26

Table 4: Agreement between automatic evaluator outputs and human safety judgments based on rendered rollout videos. All metric values are percentages.

Policy	Prompt	SR $\uparrow$	SSR $\uparrow$	USR $\downarrow$	ISH-VR $\downarrow$	ETS-VR $\downarrow$	FSH-VR $\downarrow$
Cosmos3	Default	$52.16\,{\scriptstyle\pm\,1.43}$	$15.05\,{\scriptstyle\pm\,0.78}$	$37.11\,{\scriptstyle\pm\,0.93}$	$87.65\,{\scriptstyle\pm\,0.43}$	$67.92\,{\scriptstyle\pm\,1.13}$	$57.56\,{\scriptstyle\pm\,2.17}$
	Generic	$53.40\,{\scriptstyle\pm\,1.45}$	$15.29\,{\scriptstyle\pm\,0.94}$	$38.11\,{\scriptstyle\pm\,1.06}$	$87.53\,{\scriptstyle\pm\,0.57}$	$67.00\,{\scriptstyle\pm\,2.05}$	$58.89\,{\scriptstyle\pm\,1.39}$
	Specific	$38.52\,{\scriptstyle\pm\,1.03}$	$9.21\,{\scriptstyle\pm\,0.16}$	$29.31\,{\scriptstyle\pm\,1.14}$	$91.48\,{\scriptstyle\pm\,1.48}$	$75.92\,{\scriptstyle\pm\,0.76}$	$53.00\,{\scriptstyle\pm\,2.33}$
DreamZero	Default	$30.89\,{\scriptstyle\pm\,1.47}$	$10.65\,{\scriptstyle\pm\,0.21}$	$20.24\,{\scriptstyle\pm\,1.67}$	$89.88\,{\scriptstyle\pm\,0.57}$	$63.08\,{\scriptstyle\pm\,3.26}$	$49.22\,{\scriptstyle\pm\,1.39}$
	Generic	$28.90\,{\scriptstyle\pm\,1.31}$	$9.97\,{\scriptstyle\pm\,0.62}$	$18.93\,{\scriptstyle\pm\,0.83}$	$89.26\,{\scriptstyle\pm\,0.74}$	$61.42\,{\scriptstyle\pm\,2.13}$	$45.33\,{\scriptstyle\pm\,1.00}$
	Specific	$16.39\,{\scriptstyle\pm\,0.57}$	$5.15\,{\scriptstyle\pm\,0.63}$	$11.24\,{\scriptstyle\pm\,0.82}$	$86.91\,{\scriptstyle\pm\,1.40}$	$75.92\,{\scriptstyle\pm\,0.52}$	$43.00\,{\scriptstyle\pm\,0.88}$

Table 5: Policy-level performance on SafeStage , reported as mean

\scriptstyle\pm

sample standard deviation across three environment seeds. SR denotes native task success, SSR denotes safe success, and USR denotes successful completion with a safety violation. ISH-VR, ETS-VR, and FSH-VR are violation rates on the corresponding lifecycle subsets. All values are percentages.

Prompt	Stage	SR $\uparrow$	SSR $\uparrow$	USR $\downarrow$	VR $\downarrow$
Def.	ISH	$30.52\,{\scriptstyle\pm\,1.32}$	$3.83\,{\scriptstyle\pm\,0.39}$	$26.70\,{\scriptstyle\pm\,1.39}$	$91.76\,{\scriptstyle\pm\,0.56}$
	ETS	$34.79\,{\scriptstyle\pm\,0.69}$	$17.10\,{\scriptstyle\pm\,0.49}$	$17.69\,{\scriptstyle\pm\,0.27}$	$67.92\,{\scriptstyle\pm\,1.95}$
	FSH	$35.22\,{\scriptstyle\pm\,0.50}$	$5.56\,{\scriptstyle\pm\,1.14}$	$29.67\,{\scriptstyle\pm\,0.66}$	$52.11\,{\scriptstyle\pm\,0.97}$
	All	$\mathbf{33.74}\,{\scriptstyle\pm\,0.51}$	$\mathbf{9.84}\,{\scriptstyle\pm\,0.43}$	$\mathbf{23.90}\,{\scriptstyle\pm\,0.54}$	$\mathbf{69.66}\,{\scriptstyle\pm\,0.78}$
Gen.	ISH	$29.41\,{\scriptstyle\pm\,0.81}$	$3.89\,{\scriptstyle\pm\,0.33}$	$25.52\,{\scriptstyle\pm\,0.71}$	$91.30\,{\scriptstyle\pm\,0.32}$
	ETS	$34.81\,{\scriptstyle\pm\,0.35}$	$17.08\,{\scriptstyle\pm\,0.19}$	$17.73\,{\scriptstyle\pm\,0.48}$	$67.06\,{\scriptstyle\pm\,1.03}$

Table 6: Results aggregated across the four policies. Def., Gen., and Spec. denote the default, generic, and specific prompt conditions. All values are percentages.

Explore similar work

SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
May 30, 2026Jialiang Fan, Weizhe Xu, Zijun Wang +3Robot Safety Robotic Manipulation
University of Notre Dame · University of Pennsylvania
SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation
May 12, 2026Chengyue Huang, Khang Vo Huynh, Mellon M. Zhang +3Robot Safety Robotic Manipulation
Department of Machine Learning, Georgia Institute of Technology · Department of Computer Science, University of Virginia
SafeLoop: Risk-Aware Rollback for Vision-Language-Action Manipulation
Sep 22, 2026Zeyu Lou, Tianran Zhang, Xinquan Yue +2Long-Horizon Robotic Manipulation Robot Safety
Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Beijing University of Technology, Beijing, China

May 30, 2026J/K move · Enter open · S save

SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models

Jialiang Fan, Weizhe Xu, Zijun Wang, Fanxin Kong+2

University of Notre Dame · University of Pennsylvania

Robot Safety Robotic Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks AI Agent Safety Benchmarks Safety Evaluation of VLA Models

Open paper AskPDF arXiv

Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task, but binary success can hide safety violations along the trajectory: a policy may reach the goal while applying excessive contact, disturbing bystander objects, destabilizing a held object, or entering robot self-contact. We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks that reveals violations missed by success-only evaluation. It encodes task-aware safety requirements as Signal Temporal Logic (STL) invariants with quantitative robustness semantics. Alongside native success, it reports the safety rate and the success-but-unsafe rate (SBU) used in prior safety evaluations, and introduces the Violation Severity Index (VSI), a bounded worst-violation depth score. We instantiate SafeVLA-Bench on LIBERO and RoboCasa-365, evaluating twenty-seven policy-benchmark entries across tabletop and kitchen manipulation tasks. High task success does not imply safe execution: the fifteen tabletop policies above 90% mean success still have 18-28% unsafe-episode rates, and 38-56% of successful RoboCasa-365 rollouts violate at least one active safety clause. A post-training case study further shows that SafeVLA-Bench can be used to improve policy safety. Project page: https://safevla.org

Abstract

Explore similar work

SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models

SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

SafeLoop: Risk-Aware Rollback for Vision-Language-Action Manipulation