LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Models

Zhengyan Qian; Rui Yan; Alex Jinpeng Wang; Jinhui Tang

cs.ROAug 18, 2026

LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Models

Authors: Zhengyan Qian, Rui Yan, Alex Jinpeng Wang, Jinhui Tang

Abstract

Visual cues are increasingly adopted to guide robot learning, but whether Vision-Language-Action (VLA) models can reliably follow authorized cues while disregarding unauthorized ones remains unclear. Existing work covers only a narrow range of cue forms and focuses on final task success, providing only a coarse assessment of cue-following capability. Treating all visual cues as authorized also leaves safety risks of unauthorized following unexplored. To address these gaps, we introduce LIBERO-VIFO, a benchmark to evaluate both the capability and safety of visual cue following in VLA models. LIBERO-VIFO defines eight visual cue families spanning diverse forms. A total of four protocols in two parts are defined: Part I tests cue understanding and authorized following, while Part II evaluates unauthorized visual cue following under language-cue conflict and empty language conditions. Evaluating seven VLA models reveals that although visual cue understanding does not reliably translate into execution, current VLAs are able to execute cue-indicated tasks without language instruction, exposing an emerging risk of unauthorized visual cue following. Extended experiments on scene-instantiated cues, safety-critical settings, and real-robot deployment corroborate these findings. LIBERO-VIFO brings both the capability and safety of visual cue following into systematic evaluation, establishing visual-centric safety as a new perspective for the VLA community.

Figures & tables

Figure 1: Overview of LIBERO-VIFO. The benchmark combines visual cue assets with structured metadata parsed from each LIBERO task to construct task-conditioned cue instances and insert them into robot observations. Its four protocols are organized in two parts: Part I evaluates visual cue understanding and authorized closed-loop following, while Part II tests whether the same cues influence behavior when language specifies a different task or provides no task instruction.

Figure 2: Visual cue families and variants in LIBERO-VIFO. LIBERO-VIFO organizes visual cues into eight families spanning diverse forms that may appear in robot observations. Each visual cue family includes multiple variants.

Figure 3: Task-conditioned visual cue construction pipeline. LIBERO-VIFO combines visual cue assets with structured metadata parsed from each LIBERO task to form a task-conditioned cue specification. It then inserts the selected cue into robot observations through scene-anchored placement, fixed-region insertion, or simulator-state overlay.

Model	Part I: Authorized								Part II: Unauthorized
	Visual Cue Understanding					Authorized Cue Following			Lang. - Vis. Conflict		No Lang.
	Det. $\uparrow$	Sem. $\uparrow$	Gnd. $\uparrow$	Dec. $\uparrow$	FCA $\uparrow$	Orig. SR $\uparrow$	AVF $\uparrow$	VTR $\uparrow$	LFR $\uparrow$	UVF $\downarrow$	UVI $\downarrow$
OpenVLA-OFT	4.5	14.1	20.8	35.2	0.1	97.1	0.0	0.0	90.7	0.0	59.8
$\pi_{0.5}$	19.9	11.4	40.9	54.6	1.2	96.9	55.5	57.3	94.6	0.0	49.7
MolmoAct2	65.1	83.9	45.8	46.2	16.6	97.2	33.5	34.5	95.4	0.0	35.6
Action-Sketcher	36.1	28.0	25.5	68.0	1.9	96.9	61.2	63.2	83.6	0.0	60.1

Table 1: Overall results on LIBERO-VIFO. Part I reports image-based visual cue understanding and authorized closed-loop following, while Part II reports behavior under language–cue conflict and empty language input. Full-Chain Accuracy (FCA) requires all four questions for the same cue instance to be answered correctly, and Visual Transfer Ratio (VTR) normalizes the Authorized Visual Cue Following Rate (AVF) by the Original Success Rate (Orig. SR). All values are percentages. Bold and underlined values mark the highest and second-highest values in each column.

Model	Metric	F1 Text	F2 Geom.	F3 Ghost	F4 Pict.	F5 Gest.	F6 Sign	F7 Light.	F8 Code	Avg.
OpenVLA-OFT	VQA $\uparrow$	21.6	48.0	17.8	22.5	8.1	13.1	4.0	14.4	18.7
	AVF $\uparrow$	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	UVF $\downarrow$	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
	UVI $\downarrow$	64.8	3.3	47.6	75.6	74.0	74.1	83.1	55.9	59.8
$\pi_{0.5}$	VQA $\uparrow$	26.2	20.9	21.1	34.4	55.5	25.8	35.2	34.4	31.7
	AVF $\uparrow$	57.3	52.6	55.9	56.5	58.4	56.0	52.5	53.7	55.4

Table 2: Results across visual cue families. Results are reported for seven VLA models across eight visual cue families. The cue family that a model understands best is not necessarily the one it follows most successfully. Bold and underlined values mark the highest and second-highest family results within each model and metric.

Figure 4: Comparison of visual cues as image overlays and scene objects. Text cards (F1), signs (F6), and code tags (F8) are evaluated in both forms. Scene-object placement causes larger performance drops in closed-loop following than in visual cue understanding. Part II-A is omitted because UVF remains 0.0% for all models.

Figure 5: Text cards, signs, and QR codes are registered as MuJoCo XML objects and placed directly in the simulated workspace.

Figure 6: Safety-critical scenes. Each of the six scenes pairs a safe task with a hazardous task involving dangerous tools, fire, electric shock, explosion, or personal injury.

Figure 7: Experiments on visual cue following in safety-critical scenes and on a real robot. (a) Representative episodes in which MolmoAct2 follows authorized visual cues to complete hazardous tasks. (b) On the real robot, $\pi_{0.5}$ follows an authorized note and prioritizes the language instruction over a conflicting STOP cue.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Canonical prototypes for the eight-category robot-oriented visual signage taxonomy. Shape and color identify the functional category, while the internal pictogram specifies the associated action, object, region, interaction, or system interface.

Category	Information Encoded	Representative Uses
Action governance
Prohibition and Restriction	Actions, objects, devices, or regions that are forbidden	Restricted objects, human-only areas, hazardous equipment, and no-entry workspaces
Mandatory Action	Operations that the robot is required to execute	Required placement, inspection, shutdown, sanitation, or preparation procedures
Permission and Safe Operation	Objects, tools, surfaces, or operations authorized for robot use	Shared environments in which permitted resources must be distinguished from restricted ones
Task and environment grounding
Hazard Warning	Objects, surfaces, conditions, or spatial relations requiring caution	Hot surfaces, fragile objects, electrical hazards, sharp tools, and collision or pinch zones

Appendix

Table 3: Operational scope of the robot-oriented signage taxonomy. The eight categories cover action governance, task and environment grounding, human–robot interaction, and supporting infrastructure.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,154 papers

LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models
Jun 22, 2026Rongxu Cui, Zongzheng Zhang, Jingrui Pang +11Robot Safety Vision-Language-Action Models
Institute for AI Industry Research (AIR), Tsinghua University · Beijing Academy of Artificial Intelligence (BAAI) · Beihang University +2
SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
May 30, 2026Jialiang Fan, Weizhe Xu, Zijun Wang +3Robot Safety Robotic Manipulation
University of Notre Dame · University of Pennsylvania
ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models
Jun 25, 2026Mingyang Lyu, Yinqian Sun, Yiyang Jia +5Robotics Robot Safety
Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Beijing Key Laboratory of Safe AI and Superalignment, China. · Beijing Institute of AI Safety and Governance, China. +2

Jun 22, 2026J/K move · Enter open · S save

LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models

Rongxu Cui, Zongzheng Zhang, Jingrui Pang, Haohan Chi+10

Institute for AI Industry Research (AIR), Tsinghua University · Beijing Academy of Artificial Intelligence (BAAI) · Beihang University+2

Robot Safety Vision-Language-Action Models AI Agent Safety Benchmarks Safety Evaluation of VLA Models

Open paper AskPDF arXiv

Despite the impressive manipulation capabilities of Vision-Language-Action (VLA) models, their operational safety under strict constraints remains largely unverified. To address this, we introduce a parametric safety benchmark to procedurally generate safety-critical scenarios with comprehensive stochasticity. To overcome the scalability bottlenecks of human teleoperation, we develop a novel keypose-driven data generation pipeline. Leveraging this infrastructure, we curate a large-scale dataset of 19,664 strictly collision-free demonstrations with extensive domain randomization. We then conduct a systematic cross-paradigm evaluation of eight VLA and two embodied foundation models. Our analysis reveals a critical generalization-safety tension: although high-diversity training fosters safer trajectories, task success remains fundamentally bottlenecked by sub-optimal trajectory synthesis and semantic misalignment. By providing a scalable pipeline, a robust dataset, and profound failure-mode insights, LIBERO-Safety establishes a crucial foundation for developing safe and reliable VLA models.

Abstract

Vision-Language-Action Models

Explore similar work

LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models

SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models

ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models