ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts

Liyu Hou; Yuan Wu; Yi Chang

cs.AISep 25, 2026

ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts

Authors: Liyu Hou, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University, Changchun, China · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China

Abstract

While Vision-Language-Action (VLA) models perform strongly on manipulation tasks, their responses to invalid task premises remain underexplored. Existing evaluations of premise conflicts often focus on terminal task outcomes, yet task failure alone cannot distinguish behavioral disengagement from continued pursuit followed by an execution error. We call the latter pattern Failed Persistence. To study this phenomenon, we introduce ConflictVLA-Bench, which pairs conflict rollouts with premise-consistent reference rollouts and evaluates both outcomes and execution processes. Built on LIBERO, the benchmark contains 2,826 prompt-conditioned conflict tasks spanning four conflict families, four structural configurations, and two prompt conditions. Across all eight VLAs, invalid premises reduce original goal completion by at least 17.3 percentage points, with the reduction reaching 56.2 percentage points for OpenVLA. Crucially, even when models succeed on premise-consistent tasks and fail on their matched conflict tasks, they often continue to approach the original targets, retain early trajectory structure, and show limited action magnitude suppression. Failed Persistence therefore recurs across the evaluated models. Explicit premise checking does not consistently produce selective and coordinated behavioral changes. These findings show that terminal failure alone establishes neither behavioral disengagement nor refusal and that outcomes alone are insufficient for VLA evaluation. Experimental data and additional details are available on the project page: https://github.com/EmbodiedAISurvey/ConflictVLA-Bench

Figures & tables

Fig. 1: Task competence does not guarantee reliable premise conflict response. A VLA may execute a task successfully under valid premises yet continue pursuing the same objective after a required premise becomes invalid. The examples illustrate continued task-directed behavior under Object Grounding and Physical Feasibility conflicts.

Fig. 2: Overview of ConflictVLA-Bench. (a) We define four premise conflict families. (b) Controlled conflict tasks are constructed from LIBERO across four structural configurations and two prompt conditions. (c) Each conflict rollout is compared with a matched premise-consistent reference rollout. (d) Outcome and process evidence jointly diagnose behavioral disengagement and Failed Persistence.

Model	Outcome			Process
Model	PCSR	RGSR	$\Delta$ SR	TAR	Early20 TS	TS	AMS
OpenVLA	78.7	22.5	56.2	76.5	78.4	47.1	27.6
OpenVLA-OFT	98.3	71.1	27.1	89.4	88.6	74.4	6.7
$\pi_{0}$	93.1	57.6	35.5	86.9	83.8	59.0	8.0
$\pi_{0}$ -FAST	83.4	44.8	38.6	85.2	88.6	57.6	10.2
$\pi_{0.5}$	94.5	77.2	17.3	89.1	85.9	71.4	6.9

TABLE I: Overall outcome and process measures (%) for eight VLAs. Values that are underlined are the most favorable for each metric.

Model	Instruction						Grounding						Spatial						Physical
Model	PCSR	RGSR	$\Delta$ SR	TAR	TS	AMS	PCSR	RGSR	$\Delta$ SR	TAR	TS	AMS	PCSR	RGSR	$\Delta$ SR	TAR	TS	AMS	PCSR	RGSR	$\Delta$ SR	TAR	TS	AMS
OpenVLA	78.2	29.0	49.2	83.6	49.9	29.2	81.8	32.0	49.8	88.1	53.7	27.5	73.7	12.1	61.6	60.7	44.3	26.6	81.0	16.5	64.5	73.1	40.2	26.9
OpenVLA-OFT	97.5	84.5	13.0	97.5	84.8	5.6	98.0	79.0	19.0	94.5	78.8	4.8	100.0	53.0	47.0	80.1	63.7	10.6	97.5	68.0	29.5	86.3	70.3	5.8
$\pi_{0}$	92.1	74.5	17.6	95.1	68.5	6.9	92.0	66.0	26.0	93.9	65.8	7.4	96.0	40.1	55.9	78.1	48.7	9.3	92.5	49.4	43.1	81.5	52.6	8.7
$\pi_{0}$ -FAST	86.8	45.5	41.2	88.9	59.9	10.3	77.3	45.2	32.1	90.7	57.5	10.6	84.0	68.6	15.5	89.5	72.4	6.3	85.5	20.8	64.8	71.6	41.0	13.4
$\pi_{0.5}$	95.0	93.8	1.2	97.2	82.3	4.4	96.5	76.8	19.8	91.6	74.6	6.6	89.9	79.4	10.6	90.0	67.7	7.0	96.2	59.0	37.3	77.2	60.6	9.6

TABLE II: Outcome and process measures of eight VLAs across four premise-conflict families. Pink marks the maximum within each metric column, and blue marks the minimum. All metrics are reported in %.

Model	FR	TAR	Early20 TS	TS	AMS
OpenVLA	76.7	68.7	76.1	39.8	37.0
OpenVLA-OFT	28.7	64.0	69.4	33.0	19.3
$\pi_{0}$	41.6	68.8	74.0	34.9	13.3
$\pi_{0}$ -FAST	50.2	74.1	84.5	38.7	17.8
$\pi_{0.5}$	23.0	58.6	65.3	31.0	19.2
GR00T N1.7	34.9	67.7	75.0	35.4	19.9

TABLE III: Failure rates among capability-qualified pairs and failure-conditioned mean process measures (%). Values that are underlined are the most favorable for each metric.

Fig. 3: Task failure can coexist with substantial retention of reference task behavior. Panels (a) to (d) show TAR, Early20 TS, TS, and AMS within capability-qualified failures under P0. The distributions show that failed conflict rollouts can retain strong target approach and early trajectory similarity.

Fig. 4: Metric differences under explicit premise checking. Paired mean differences (

\mathrm{P0}-\mathrm{P1}

) across eight VLAs.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,194 papers

FATE-VLA:Failue-aware test generation for vision-language-action models
Jun 1, 2026Arusa Kanwal, Pablo Valle, Shaukat Ali +1VLM Evaluation Vision-Language-Action Models
Mondragon University Mondragon, Spain · Simula Research Laboratory Oslo, Norway
When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs
Feb 19, 2026Yu Fang, Yuchun Feng, Dong Jing +5VLM Robustness Counterfactual Evaluation of VLMs
University of North Carolina at Chapel Hill
Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks
Oct 1, 2026Sophie Higham, Riccardo Andrea Izzo, Matteo Matteucci +1Vision-Language-Action Models Robot Manipulation Benchmarks
School of Informatics, University of Edinburgh, UK · Department of Electronics, Informatics and Bioengineering, Politecnico di Milano, Italy

Jun 1, 2026J/K move · Enter open · S save

FATE-VLA:Failue-aware test generation for vision-language-action models

Arusa Kanwal, Pablo Valle, Shaukat Ali, Aitor Arrieta

Mondragon University Mondragon, Spain · Simula Research Laboratory Oslo, Norway

VLM Evaluation Vision-Language-Action Models Robustness of VLA Models Failure Detection for VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks that randomly sample task scenes. In high-dimensional embodied spaces, failures are sparse and clustered, so static benchmarking can underestimate robustness risks. We reframe VLA evaluation as an active failure-discovery problem and propose a failure-aware test-generation approach that combines diversity-driven exploration with surrogate models learned from observed executions. The method steers testing toward high-risk yet diverse scene regions. Across four state-of-the-art VLA models, it uncovers substantially more failures (up to +29.7 % over selected baselines) while revealing more diverse failure modes. This mean that, for instance, in the case of GR00T-N1.6, success rate dropped from 64.4% to 34.7%. More broadly, our findings call for a shift in VLA evaluation: from passive measurement on fixed task suites to adaptive, failure-seeking test generation that exposes the structure of model weaknesses before deployment.

Abstract

Vision-Language-Action Models

Explore similar work

FATE-VLA:Failue-aware test generation for vision-language-action models

When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks