SafeStage: Evaluating Safety Before, During, and After Vision-Language-Conditioned Robot Manipulation

Jinzhu Luo; Qi Zhang; Wei Wang; Wei Jiang

cs.ROSep 18, 2026

SafeStage: Evaluating Safety Before, During, and After Vision-Language-Conditioned Robot Manipulation

Authors: Jinzhu Luo, Qi Zhang, Wei Wang, Wei Jiang

Organizations: Worcester Polytechnic Institute, Worcester, MA, USA · Futurewei Technologies Inc., Santa Clara, CA, USA

Abstract

Vision-language-conditioned robot policies integrate perception, language understanding, and control for general-purpose manipulation. However, existing evaluations often focus on task success, isolated physical constraints, semantic refusal, or realized physical damage, providing limited insight into where safety fails during closed-loop manipulation. We introduce SafeStage, a lifecycle-structured benchmark for evaluating manipulation safety before, during, and after task execution. SafeStage contains 97 purpose-built risk scenarios organized into three stages. Initial-State Hazards captures safety-relevant relations that must be resolved before manipulating the target. Execution-Time Safety evaluates unsafe contacts, trajectories, region entries, and object interactions during execution. Final-State Hazards capture unstable or otherwise unsafe conditions remaining after nominal task completion. The benchmark evaluates realized interactions using event-based and state-based checks and reports native task success independently from stage-specific safety outcomes. We evaluate representative direct-action Vision-Language-Action (VLA) policies and policies with world-model-based policies under a common closed-loop protocol. Our results demonstrate that nominal task completion frequently coexists with safety violations and that different policies exhibit distinct failure profiles across the three stages. By separating task success from safety and localizing when violations occur, SafeStage provides a unified diagnostic testbed for evaluating and improving vision-language-conditioned robot manipulation policies.

Figures & tables

Figure 1: An unsafe success. The policy moves the coffee can without resolving its support relation with the white bottle, causing the bottle to fall before the can reaches the plate.

Benchmark	Closed-loop policy rollout	Purpose-built risk tasks	Relational initial hazards	Trajectory safety	Post-success state checks	Lifecycle-wise reporting
IS-Bench ( Lu et al. 2026 )	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
SafeVLA ( Zhang et al. 2026 )	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
LIBERO-Safety ( Cui et al. 2026 )	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
SafeManip ( Huang et al. 2026 )	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$	$\times$
OopsieVerse ( Balaji et al. 2026 )	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\times$
SafeStage (ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1: Comparison of robot-safety benchmarks by evaluation scope. Relational initial hazards include support, stacking, and containment dependencies involving the instructed target. Post-success state checks evaluate whether the physical state remains safe after the native task goal is first satisfied.

Figure 2: Representative SafeStage . (1–2) ISH tasks require the policy to resolve an existing support, stacking, or containment relation before moving the instructed target. (3–4) ETS tasks constrain the physical interaction and transport trajectory during manipulation. (5–6) FSH tasks test whether the state left after nominal task completion is stable and safe.

Stage and task family	Risk activation	Typical unsafe behavior	Monitored signals
ISH: stacking / support dependency	Target supports another object	Target is moved before the support relation is resolved	Contact, overlap, relative height, and event order
ISH: containment	Target is contains another object	The stack or container contents are destabilized	Support graph, containment state, and object displacement
ETS: obstacle avoidance	A object blocks the direct transport path	The robot or carried object contacts, moves, or knocks over the obstacle	Contact, path clearance, obstacle displacement, tilt, and drop
ETS: hazard-sensitive transport	A heavy object pass above a sensitive object	The carried object enters the risk zone	Trajectory, Risk-zone occupancy, clearance, contact
FSH: support and stability	The goal permits an unstable placement	Target slides, tilts, or falls after release	Support margin, orientation, and residual velocity
FSH: residual relation	Final placement can endanger another object	Unsafe proximity, contact, or collateral motion	Clearance, contact, and protected-object displacement

Table 2: Representative task families and the physical signals used by the corresponding lifecycle evaluators.

Condition	Example Prompt
default	Move the coffee can onto the plate.
generic	Move the coffee can onto the plate. Complete the task safely and avoid causing unnecessary hazards.
specific	Move the coffee can onto the plate. The white bottle is currently supported by the coffee can. Do not move the coffee can until the bottle has been safely relocated.

Table 3: Example prompts under the three instruction conditions. The objective remains unchanged as safety information increases from the default to the scenario-specific prompt.

Stage	Samples	Precision	Recall	F1
ISH	80	97.33	100.00	98.65
ETS	80	94.74	98.18	96.43
FSH	80	94.29	97.06	95.65
Overall	240	95.81	98.77	97.26

Table 4: Agreement between automatic evaluator outputs and human safety judgments based on rendered rollout videos. All metric values are percentages.

Policy	Prompt	SR $\uparrow$	SSR $\uparrow$	USR $\downarrow$	ISH-VR $\downarrow$	ETS-VR $\downarrow$	FSH-VR $\downarrow$
Cosmos3	Default	$52.16\,{\scriptstyle\pm\,1.43}$	$15.05\,{\scriptstyle\pm\,0.78}$	$37.11\,{\scriptstyle\pm\,0.93}$	$87.65\,{\scriptstyle\pm\,0.43}$	$67.92\,{\scriptstyle\pm\,1.13}$	$57.56\,{\scriptstyle\pm\,2.17}$
	Generic	$53.40\,{\scriptstyle\pm\,1.45}$	$15.29\,{\scriptstyle\pm\,0.94}$	$38.11\,{\scriptstyle\pm\,1.06}$	$87.53\,{\scriptstyle\pm\,0.57}$	$67.00\,{\scriptstyle\pm\,2.05}$	$58.89\,{\scriptstyle\pm\,1.39}$
	Specific	$38.52\,{\scriptstyle\pm\,1.03}$	$9.21\,{\scriptstyle\pm\,0.16}$	$29.31\,{\scriptstyle\pm\,1.14}$	$91.48\,{\scriptstyle\pm\,1.48}$	$75.92\,{\scriptstyle\pm\,0.76}$	$53.00\,{\scriptstyle\pm\,2.33}$
DreamZero	Default	$30.89\,{\scriptstyle\pm\,1.47}$	$10.65\,{\scriptstyle\pm\,0.21}$	$20.24\,{\scriptstyle\pm\,1.67}$	$89.88\,{\scriptstyle\pm\,0.57}$	$63.08\,{\scriptstyle\pm\,3.26}$	$49.22\,{\scriptstyle\pm\,1.39}$
	Generic	$28.90\,{\scriptstyle\pm\,1.31}$	$9.97\,{\scriptstyle\pm\,0.62}$	$18.93\,{\scriptstyle\pm\,0.83}$	$89.26\,{\scriptstyle\pm\,0.74}$	$61.42\,{\scriptstyle\pm\,2.13}$	$45.33\,{\scriptstyle\pm\,1.00}$
	Specific	$16.39\,{\scriptstyle\pm\,0.57}$	$5.15\,{\scriptstyle\pm\,0.63}$	$11.24\,{\scriptstyle\pm\,0.82}$	$86.91\,{\scriptstyle\pm\,1.40}$	$75.92\,{\scriptstyle\pm\,0.52}$	$43.00\,{\scriptstyle\pm\,0.88}$

Table 5: Policy-level performance on SafeStage , reported as mean

\scriptstyle\pm

sample standard deviation across three environment seeds. SR denotes native task success, SSR denotes safe success, and USR denotes successful completion with a safety violation. ISH-VR, ETS-VR, and FSH-VR are violation rates on the corresponding lifecycle subsets. All values are percentages.

Prompt	Stage	SR $\uparrow$	SSR $\uparrow$	USR $\downarrow$	VR $\downarrow$
Def.	ISH	$30.52\,{\scriptstyle\pm\,1.32}$	$3.83\,{\scriptstyle\pm\,0.39}$	$26.70\,{\scriptstyle\pm\,1.39}$	$91.76\,{\scriptstyle\pm\,0.56}$
	ETS	$34.79\,{\scriptstyle\pm\,0.69}$	$17.10\,{\scriptstyle\pm\,0.49}$	$17.69\,{\scriptstyle\pm\,0.27}$	$67.92\,{\scriptstyle\pm\,1.95}$
	FSH	$35.22\,{\scriptstyle\pm\,0.50}$	$5.56\,{\scriptstyle\pm\,1.14}$	$29.67\,{\scriptstyle\pm\,0.66}$	$52.11\,{\scriptstyle\pm\,0.97}$
	All	$\mathbf{33.74}\,{\scriptstyle\pm\,0.51}$	$\mathbf{9.84}\,{\scriptstyle\pm\,0.43}$	$\mathbf{23.90}\,{\scriptstyle\pm\,0.54}$	$\mathbf{69.66}\,{\scriptstyle\pm\,0.78}$
Gen.	ISH	$29.41\,{\scriptstyle\pm\,0.81}$	$3.89\,{\scriptstyle\pm\,0.33}$	$25.52\,{\scriptstyle\pm\,0.71}$	$91.30\,{\scriptstyle\pm\,0.32}$
	ETS	$34.81\,{\scriptstyle\pm\,0.35}$	$17.08\,{\scriptstyle\pm\,0.19}$	$17.73\,{\scriptstyle\pm\,0.48}$	$67.06\,{\scriptstyle\pm\,1.03}$

Table 6: Results aggregated across the four policies. Def., Gen., and Spec. denote the default, generic, and specific prompt conditions. All values are percentages.

Explore similar work

May 30, 2026

SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models

Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task, but binary success can hide safety violations along the trajectory: a policy may reach the goal while applying excessive contact, disturbing bystander objects, destabilizing a held object, or entering robot self-contact. We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks that reveals violations missed by success-only evaluation. It encodes task-aware safety requirements as Signal Temporal Logic (STL) invariants with quantitative robustness semantics. Alongside native success, it reports the safety rate and the success-but-unsafe rate (SBU) used in prior safety evaluations, and introduces the Violation Severity Index (VSI), a bounded worst-violation depth score. We instantiate SafeVLA-Bench on LIBERO and RoboCasa-365, evaluating twenty-seven policy-benchmark entries across tabletop and kitchen manipulation tasks. High task success does not imply safe execution: the fifteen tabletop policies above 90% mean success still have 18-28% unsafe-episode rates, and 38-56% of successful RoboCasa-365 rollouts violate at least one active safety clause. A post-training case study further shows that SafeVLA-Bench can be used to improve policy safety. Project page: https://safevla.org

Jialiang Fan, Weizhe Xu, Zijun Wang +3

University of Notre Dame · University of Pennsylvania

Robot Safety Robotic Manipulation Vision-Language-Action Models Robot Manipulation Benchmarks AI Agent Safety Benchmarks Safety Evaluation of VLA Models

Ask PDF

May 12, 2026

SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

Robotic manipulation is typically evaluated by task success, but reaching the correct final state does not guarantee safe execution. A robot may succeed despite contamination, premature release, or incorrect sequencing. These failures are poorly captured by task-completion metrics or isolated state checks because safety often depends on how behavior unfolds over time. We introduce SafeManip, a benchmark for evaluating temporal safety directly from low-level manipulation rollouts across eight physical and semantic categories. SafeManip grounds symbolic manipulation predicates into reusable temporal properties over finite executions, formalized with LTLf. We further introduce ManipVerse, a property ontology enabling semantic safety rules generalize across objects, tasks, and environments. Finally, SafeManip introduces per-trigger safety metrics that normalize violations by the safety-relevant opportunities that activate each obligation, accounting for differences in how often those opportunities arise across policies and tasks. We instantiate SafeManip on RoboCasa365 and LIBERO to audit seven state-of-the-art robot foundation-model policies. Our results reveal a gap between task capability and temporal safety: higher task success does not necessarily imply safer execution, successful rollouts can still contain violations, and failure patterns vary systematically across safety categories, task horizons, and manipulation suites. SafeManip provides a reusable layer for evaluating how safely manipulation tasks are executed, not only whether they are completed.

Chengyue Huang, Khang Vo Huynh, Mellon M. Zhang +3

Department of Machine Learning, Georgia Institute of Technology · Department of Computer Science, University of Virginia

Robot Safety Robotic Manipulation Robot Manipulation Benchmarks Linear Temporal Logic

Ask PDF

Sep 22, 2026

SafeLoop: Risk-Aware Rollback for Vision-Language-Action Manipulation

Recent vision-language-action (VLA) models are promising for general-purpose manipulation, but long-horizon execution remains fragile. Small state-estimation or control errors can lead to irreversible failures (e.g., collisions and object drops). Avoiding these risks requires a proactive safety mechanism capable of anticipating hazards. In this paper, we introduce SafeLoop, a non-invasive external wrapper that adds hazard prediction and rollback-based recovery to a VLA model without changing its parameters. SafeLoop trains a risk predictor from vision and proprioception to output four values: the probability and time-to-hazard for body collisions and for object failures. A lightweight controller then chooses one of three actions based on the predicted risk: continue execution (noop), save a safety checkpoint (record), or retreat in joint space (rollback). Rollback moves the robot back to a recent safe waypoint and queries the base policy again, which may yield an alternative continuation. Across 24 LIBERO tasks (16 random seeds each) and three real-robot tasks (25 rollouts each), SafeLoop achieves a stronger overall safety-success trade-off than alternative methods, reducing hazard cases by roughly 70% while preserving task success and the base-policy control rate. Project code is available at https://github.com/Loule0-0/SafeLoop/tree/release/safeloop.

Zeyu Lou, Tianran Zhang, Xinquan Yue +2

Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Beijing University of Technology, Beijing, China

Long-Horizon Robotic Manipulation Robot Safety Robotic Manipulation Vision-Language-Action Models

Ask PDF