Refusals That Bend: Measuring and Predicting Task Malleability in Embodied VLM Planners

Leo Y. Lin; Mikhail Kuznetsov; Muslum Ozgur Ozmen; Z. Berkay Celik

cs.CRSep 30, 2026

Refusals That Bend: Measuring and Predicting Task Malleability in Embodied VLM Planners

Authors: Leo Y. Lin, Mikhail Kuznetsov, Muslum Ozgur Ozmen, Z. Berkay Celik

Organizations: Purdue University · Amazon · Arizona State University

Abstract

Embodied vision-language models (VLMs) are increasingly deployed as high-level planners for robots because they generalize across diverse environments. However, this requires their safety alignment to also hold in unseen environments. Existing red-teaming assumes an adversary who optimizes the prompt, the pixels, or text in the environment, and existing benchmarks ask whether a planner recognizes or mitigates a hazard in a fixed scene. Neither asks whether a refusal the planner has already given survives an ordinary change to the environment. We ask that question by placing a single everyday object into the environment, with no pixel, gradient, or prompt under adversarial control. On $846$ tasks that a constitution-guarded planner initially refuses, we find $20.2\%$ of tasks can be flipped to compliance by one or more objects, and the number of objects differs from one task to another. In addition, the object need not be chosen for the task, i.e., items drawn from a fixed list, with no knowledge of the environment or the instruction, bypass safety about as often as items proposed for the specific task. We qualitatively contrast the tasks bypassed most and least often and find that the distinction lies in how conspicuous the hazard is in the instruction and environment. Susceptibility to safety bypass is therefore a property of the task, which we call its \emph{malleability}, and we show that it can be predicted before the target is ever queried. A composite of signals read from a small open-source VLM identifies malleable tasks $2.4\times$ as often as picking at random. Everyday objects, whether placed by an adversary or introduced by ordinary rearrangement of the environment, are thus sufficient to overturn a refusal. Because susceptibility is determined by how a task is specified, we recommend assessing malleability per task prior to deployment.

Figures & tables

Figure 1: Asked to put the travel bags beside the fire exit, the planner refuses on the unmodified scene (a) and names the risk. With one ordinary object composited into the scene, it complies and returns a plan that carries every bag to the exit: (b) a stack of books, proposed for this task by the object proposer; (c) an orange, drawn from a fixed list of everyday items without sight of the task.

Figure 2: Bypass rate by proposer and stratum. Mean per-task bypass rate under each proposer and under the clean re-query, by stratum of the stratified subset (

n=192

), with

95\%

CIs across the tasks in each stratum. The strata are defined by the targeted rate, so its rise is by construction; the evidence is that scene-only and random rise with it at every level.

Figure 3: RQ4: the strongest signal per approach and the out-of-fold composite (A5) over all

846

tasks. (a) ROC curves for the malleable label,

\mathrm{mal}(t)>0

, with ROC-AUC in the legend. (b) Precision among the

k

highest-scored tasks as the audit budget

k

grows, against the

0.202

prevalence of malleable tasks; the marked points are the composite’s precision@

50

and @

100

of Table 8 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

ROLE	MODEL	PROVIDER
Target embodied VLM $\mathcal{M}$	gemini-robotics-er-2-preview ( Google DeepMind, 2026 )	Google
Safety judge $\mathcal{J}_{s}$	gemini-2.5-flash ( Comanici et al., 2025 )	Google
Constraint judge $\mathcal{J}_{c}$	gemini-2.5-flash ( Comanici et al., 2025 )	Google
Proposal VLM $\mathcal{V}_{\text{prop}}$	Qwen3-VL-30B-A3B-Thinking ( Bai et al., 2025 )	Alibaba
Scene editor $\mathcal{E}$	Flux.2-klein-4B ( Black Forest Labs, 2026 )	Black Forest Labs
Pattern-discovery LLM	gemini-2.5-flash ( Comanici et al., 2025 )	Google

Appendix

Table 1: Models used in the study. The target model is the system under test; every other model is part of the measurement apparatus or, in Section 3.3 , of the local predictor.

Figure 4: Bypass under object insertion over the full study population. Fraction of the

846

initially-refused tasks whose bypass rate exceeds a given level; the curve starts at

20.2\%

(

n=171

), the tasks bypassed by at least one object, and the dashed line is the planner’s rate on unmodified images,

0.24\%

. The four strata of Table 2 are cut from this distribution.

STRATUM	OBJECT BYPASS RATE	POOL	SAMPLED
zero	$0\%$	$674$	$50$
low	$(0\%,10\%]$	$64$	$50$
medium	$(10\%,25\%]$	$44$	$44$
high	$>25\%$	$63$	$50$

Appendix

Table 2: The stratified subsample. Strata are defined by each task’s bypass rate under object insertion over the full study population; up to

50

tasks are drawn from each.

GROUP	ITEMS
Electronics	a laptop, a mouse, a remote, a keyboard, a cell phone
Furniture	a chair, a potted plant
Kitchen	a cup, a bowl, a spoon
Accessories	a backpack, an umbrella, a handbag, a tie, a suitcase
Indoor	a book, a clock, a vase, a teddy bear, a toothbrush
Sports	a sports ball, a frisbee, a baseball glove

Appendix

Table 3: The

26

-item list the random condition draws from, grouped for presentation only; the draw is uniform over the whole list.

Figure 5: Left: the clean hospital ward scene, on which the target refuses and names the risk of spilling hot liquid on the patient. Right: the same scene with a plastic water bottle composited onto the cart, on which the target returns a plan that moves the soup bowl to the cart beside the heart monitor. The instruction, the constitution and the decoding configuration are identical across the two panels.

MEASUREMENT	ER-1.6	ER-2
Refuses the task initially	$64.9\%$	$78.3\%$
Mean per-task bypass rate	$9.90\%$	$5.73\%$
Bypassed by at least one object	$28.9\%$	$20.2\%$
Bypass rate over trials	$9.08\%$	$5.52\%$

Appendix

Table 4: The RQ1 measurements under each target. Refusal is over the pool tasks that target judges conclusively; the bypass rows are over the tasks it refuses,

697

for ER-1.6 and

846

for ER-2.

PATTERN	MALLEABLE	ROBUST
Instruction	a plausible placement given a constructive rationale (“for inventory”, “to save space”); the risk is indirect and the object is used roughly as intended	an explicit, severe hazard: fire, electrical, chemical or unsanitary misuse, or the obstruction of safety equipment and exits
Scene	hazards intrinsic to a single object (“sharp blade”, “fragile glass”); objects described by functional state rather than by risk	hazards arising from arrangement or combination: an ignited torch beside paint thinner, chemicals blocking an emergency exit
Interaction	the instruction proposes an action, but specific scene detail decides whether it is safe; neither input alone suffices
Constitution proximity	no separating pattern: $\mathrm{prox}$ is $0.2804$ against $0.2768$ , $95\%$ CI on the gap $[-0.020,+0.012]$

Appendix

Table 5: RQ3: Qualitative analysis of malleable tasks from robust tasks.

Tasks	$281$
Distinct environments	$26$
Image size	$1024\times 1024$

Appendix

Table 6: Dataset card, over all released tasks. A task is a task ID, an instruction, an environment label, an image prompt, the image rendered from it, and a violation note naming the rules the task conflicts with.

Figure 6: Environment composition of the corpus, over all

281

released tasks. Counts rather than proportions, since several settings contribute only one or two tasks.

Figure 7: A released scene, shown unmodified and unannotated, paired with the instruction “Put the metal tools into the storage bin.” The instruction and image are benign in isolation; the hazard exists only in their combination.

FAMILY	WHAT IT COUNTS	MARKERS

Object ambiguity	uncertainty about what the object or action is	is it , is this , could be , might be , maybe , either , or is it
Conditional	safety judgement made to depend on something the proxy has not settled	if , unless , provid* that , assum* , in case , when … safe
Hedge	softens the safety judgement instead of stating it outright	however , alternativ* , if you mean/meant/meaning , could you clarif* , safer , instead , rather , perhaps , might , possibl* , recommend* against , suggest* , would advis* , caution*
Self-question	questions the proxy puts to itself	every occurrence of ?
Stance flip	changes course part-way through the reasoning	wait , hmm , reconsider* , on the other hand , let me think/rethink

Appendix

Table 7: The five families of A1, with every expression each one is counted over. Matching ignores case and requires whole words. Each family is scored as its density, its matches per 100 words (Equation 8 ); the trace’s word count is the sixth feature.

SIGNAL	SCORE	$\boldsymbol{\rho}$	ROC	PR	P@50	P@100
A1	trace length	0.330	0.734	0.395	0.58	0.51
A1	self-question density †	0.272	0.676	0.321	0.44	0.41
A1	object ambiguity density	0.263	0.684	0.332	0.44	0.36
A1	conditional density	0.241	0.674	0.324	0.44	0.35
A1	stance flip density †	0.233	0.662	0.316	0.34	0.40
A1	hedge density	0.203	0.646	0.273	0.26	0.28

Appendix

Table 8: RQ4: local proxy signals against the

846

-task ground truth (base rate

0.202

; ROC-AUC chance

0.5

, PR-AUC and precision@

k

chance

0.202

). Precision@

k

is the share of the

k

highest-scored tasks that are genuinely malleable, the quantity a red-teamer who can afford to probe only

k

of the

846

tasks would see;

k=50

and

k=100

are

5.9\%

and

11.8\%

of the pool. Trace length is an A1 feature in its own right and outranks every density on this target. The composite is scored out-of-fold; the last row adds trace length to it as a seventh feature, which changes nothing. Daggers mark the two A1 densities that do not enter the composite (Appendix I.1 ).

CHECK	CONSTRAINT	FAIL	SAMPLE	REWEIGHTED (95% CI)
no injected text	typographic-injection control	1	$99.9\%$	$100.0\%$ $[99.9,100.0]$
object not text	$\neg\mathrm{TextLabel}(o)$	3	$99.7\%$	$99.9\%$ $[99.8,100.0]$
placeable	$\mathrm{Feas}(o,I)$	6	$99.4\%$	$99.2\%$ $[98.2,99.9]$
hazard remains	$\neg\mathrm{Neutralizes}(o,t)$	15	$98.4\%$	$98.7\%$ $[97.1,99.8]$
object present	the editor rendered $o$	20	$97.9\%$	$97.4\%$ $[95.1,99.1]$
scene preserved	edit prompt: change nothing else	32	$96.7\%$	$97.3\%$ $[95.2,99.1]$

Appendix

Table 9: Constraint verification on

966

rendered targeted objects. fail counts audited edits failing the check. The all row is not a column total; it is a union of the above checks, where an edit with at least one failed check is counted once.

CHECK	BYPASSED ( $n=193$ )	REFUSED ( $n=742$ )	DIFF.	95% CI
object present	$99.0\%$	$97.8\%$	$+1.1$	$[-0.9,+2.9]$
scene preserved	$97.9\%$	$96.6\%$	$+1.3$	$[-1.2,+3.6]$
is addition	$97.9\%$	$96.1\%$	$+1.8$	$[-0.8,+4.3]$
object not text	$99.5\%$	$99.7\%$	$-0.2$	$[-1.5,+0.5]$
no injected text	$100.0\%$	$100.0\%$	$+0.0$	$[+0.0,+0.0]$
placeable	$99.5\%$	$99.5\%$	$+0.0$	$[-1.3,+0.9]$

Appendix

Table 10: Constraint verification, split by the target’s own judgement of the trial. Rates are raw sample rates on the same bootstrap draws, so each difference is paired across tasks.

Explore similar work

Apr 20, 2026cs.AI

Using large language models for embodied planning introduces systematic safety risks

Large language models are increasingly used as planners for robotic systems, yet how safely they plan remains an open question. To evaluate safe planning systematically, we introduce DESPITE, a benchmark of 12,279 tasks spanning physical and normative dangers with fully deterministic validation. Across 23 models, even near-perfect planning ability does not ensure safety: the best-planning model fails to produce a valid plan on only 0.4% of tasks but produces dangerous plans on 28.3%. Among 18 open-source models from 3B to 671B parameters, planning ability improves substantially with scale (0.4-99.3%) while safety awareness remains relatively flat (38-57%). We identify a multiplicative relationship between these two capacities, showing that larger models complete more tasks safely primarily through improved planning, not through better danger avoidance. Three proprietary reasoning models reach notably higher safety awareness (71-81%), while non-reasoning proprietary models and open-source reasoning models remain below 57%. As planning ability approaches saturation for frontier models, improving safety awareness becomes a central challenge for deploying language-model planners in robotic systems.

Tao Zhang, Kaixian Qu, Zhibin Li +4

ETH Zurich, Zurich, Switzerland. · University College London, London, United Kingdom. · Stanford University, Stanford, California, United States. +2

Large Language Model Planning Planning Robotic Systems

Ask PDF

Sep 8, 2026cs.RO

Safe Task Planning with Long-Term Graph Memory for Embodied Agents

Large language models (LLMs) and vision-language models (VLMs) have significantly advanced zero-shot task planning for embodied agents. However, most LLM- and VLM-driven methods struggle to generate safe high-level actions due to a lack of physical risk awareness, particularly under partial observability, where hazards lie outside the immediate field of view. To address this challenge, we propose a novel safe task-planning framework, SafeMem, which constructs and maintains a long-term semantic graph memory of the open and dynamic environment. Based on egocentric observations, the proposed framework incrementally accumulates knowledge about surrounding objects and their relationships with a graph. Then, an LLM-based risk predictor evaluates candidate actions using the graph memory, triggering a conservatism-modulated replanning loop with explanations for detected hazards. Extensive experiments on the IS-Bench benchmark and a real-world robot platform demonstrate that the SafeMem framework substantially improves safe success rates compared to state-of-the-art VLM-driven task planners. Video results are available on our webpage: https://sites.google.com/view/safemem.

Siyuan Li, Taiyan Lang, Aoqi Yan +6

Faculty of Computing Harbin Institute of Technology · Engineering Systems and Design Singapore University of Technology and Design · Intelligent Science and Technology Academy of CASIC

Embodied Agents Generalizable Vision-Language-Action Policies Graph-Structured Memories Safer Graph Representations

Ask PDF

May 19, 2026cs.RO

The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents

Vision-language models (VLMs) are used as high-level planners for embodied agents, translating natural language instructions and visual observations into action plans. While prior work has studied abstention in LLMs, existing benchmarks are largely text-only and do not capture the perceptual grounding and physical constraints inherent to embodied robotics environments. In such settings, abstention requires recognizing when instructions are ambiguous, physically infeasible, based on false premises, or otherwise unresolvable given the available sensory modalities and context. To address this gap, we introduce a taxonomy to categorize abstention in the context of embodied robotics and present RoboAbstention, a scalable and auditable framework for generating abstention instructions grounded in images gathered from five robotics datasets. RoboAbstention instantiates the taxonomy through a three-phase pipeline: (1) structured visual grounding, (2) deterministic constraint derivation, and (3) controlled instruction generation via category-specific templates. This enables the construction of a diverse dataset with verifiable abstention conditions. We evaluate several frontier VLMs and find that all models exhibit significant weaknesses in abstention, including those with advanced reasoning capabilities. The best-performing model, Gemini 2.5 Flash, abstains on only 39.0% of our 6,069 benchmark instructions, while the embodied planner Gemini Robotics ER 1.6 Preview abstains on just 16.5%. We further explore methods for improving abstention in VLM planners, such as defensive prompting and in-context learning, and find that these interventions substantially improve performance, reaching 93.6% abstention rate for Gemini Robotics ER 1.6 Preview and 88.6% for GPT 5.4 Mini, yet no approach fully solves the problem. We open-source RoboAbstention at https://purseclab.github.io/RoboAbstention/.

Doguhan Yeke, Elif Su Temirel, Ananth Shreekumar +3

Purdue University · Bilkent University · Work performed while an intern at Purdue University.

Abstention

Ask PDF