We show that aligned vision-language models also condition refusal on a property of a request's form: whether an image is attached, holding everything the request asks fixed. Attaching a blank canvas, an image that cannot be read, cannot relate to the request, and is byte-identical across every prompt in its condition, shifts benign refusal by tens of points. The shift is not blanket caution but a threshold shift: genuinely neutral instructions are almost unaffected (<=2 percentage points on three of four hosted models) while borderline-benign prompts move +23 to +51 points, so the cost falls on sensitivity-adjacent traffic, meaning benign questions about privacy, self-harm, violence and illegal activity. There is a benign reading of such a threshold, namely that attachment correlates with risk in real traffic, and we take it seriously; a black-box study cannot measure that correlation and we do not claim to. What it can test is whether the response to attachment is robust to variation carrying no information about the request, and on four independent measurements it is not. It varies with canvas colour and pixel count. Its sign inverts across checkpoints. It survives an explicit instruction to disregard the image. And on one model it fires on a bare assertion that an attachment exists, with nothing attached and the modality word contributing none of it. Finally we price it. On a matched harmful set the same canvas does lower attack success, so the cue buys something. But the charge is decoupled from the purchase: the checkpoint with the least harmful headroom we measure, completing only 2% of plain harmful requests, still pays the benign cost in full, and across our models the harmful-side denominator falls as alignment improves while the benign cost does not track it down. Image presence is not a conservative default that a deployer chose and priced. It is an uncontrolled variable.
Figures & tables
benign refusal %, text → blank canvas
Model
neutral
borderline
topic-matched
claude-sonnet-4-6
2 → 2
12 → 63 ∗∗∗
3 → 36
gpt-4o-mini
2 → 2
12 → 46 ∗∗∗
4 → 19
gemini-2.5-flash-lite
1 → 14 ∗∗∗
11 → 34 ∗∗∗
11 → 45
gemini-2.5-flash
0 → 2
16 → 13
11 → 18
Table 1: [ primary test ] The sensitivity ladder : benign refusal rate (%, lower is more useful) under {no image, blank canvas}, no defense, 100 prompts per rung, paired per prompt id. ∗∗∗p≤1.5×10−6 , exact McNemar. Significance is marked on the neutral and borderline rungs, the two columns entering the declared test families (§ S4 ); unmarked cells in those two columns are non-significant. The topic-matched column is reported for continuity and is not part of any tested family, so it carries no marks. Inflation is absent on neutral prompts and large on borderline-benign ones: a threshold shift rather than blanket caution (§ 4 ). The topic-matched column is the JailbreakBench-benign set, shown for continuity.
benign refusal %, rung 2 (borderline)
Model
text
blank
line draw.
blank+“ignore”
claude-sonnet-4-6
10
64 ∗∗∗
58 ∗∗∗
52
gpt-4o-mini
13
43 ∗∗∗
44 ∗∗∗
36
gemini-2.5-flash-lite
11
34 ∗∗∗
63 ∗∗∗
28
gemini-2.5-flash
15
14
25 ∗
47 ∗∗∗
Table 2: [ primary test, control arm ] Presence controls , all arms in one job, one canonical text, 100 prompts each, no defense. Two request-independent image classes, a blank canvas ( 1024×768 , pure white) and the clip-art line drawing of Fig. S1 (c) ( 1189×1418 , drawn content, no text), plus the cheapest mitigation (a neutral instruction to disregard the image). The blank canvas here is generated at the renderer’s default size, and is therefore a different file from the 512×512 blank of Fig. S1 (b); both are pure white and neither carries readable text. ∗∗∗p<10−5 , ∗p<0.05 against the text baseline, exact McNemar paired per prompt; unmarked cells in the two image columns are non-significant against that baseline. The instruction column is a mitigation arm whose contrast of interest is against the blank arm rather than against text, and those three contrasts are reported in the text; the one mark in that column is gemini-2.5-flash , where the instruction is not a mitigation but the manipulation itself and the text baseline is therefore the right comparator. Both classes inflate refusal on three of four models; the instruction removes only 6 – 12 points of a 23 – 54 -point shift, and on gemini-2.5-flash it adds 32 points: an asserted-attachment effect rather than an image effect, decomposed in Table 4 . The gemini-2.5-flash line-drawing cell ( +10 points, p=0.021 ) does not replicate: a larger property sweep puts the same contrast at +9 points, p=0.064 (§ S8 ). We treat that model as near-null.
contrast (image attached in both arms)
model
Δ
disc.
p
95% CI
black vs white, 5122
fl.-lite
+22
22/0
4.8×10−7
[+13.7,+29.7]
content vs blank, 1024×141
fl.-lite
+30
31/1
1.5×10−8
[+20.2,+38.7]
content vs blank, 1024×141
claude
+28
30/2
2.5×10−7
[+18.0,+37.0]
content vs blank, 1024×141
4o-mini
+1
0 5/4
1.00
[−4.8,+6.8]
blank 15362 vs blank 2562
qwen3-vl
+16
18/2
0.0004
[+7.7,+24.9]
Table 3: [ primary test ] The response is not invariant to properties that carry no risk information. Each row compares two attached-image arms against each other rather than against the text baseline, so image presence is held fixed and only the named property moves; size is matched within every row except the last, which varies size alone at fixed content. Every row is a paired exact McNemar contrast computed between the two image arms on matched prompt ids, with Newcombe intervals; Δ is the larger-property arm minus the smaller. The first four rows are the three hosted models, n=100 per arm; the last is the open checkpoint qwen3-vl-8b , also n=100 . The paired tests resolve what raw rate differences could not: the claude content premium is established ( p=2.5×10−7 ), while the gpt-4o-mini row is a genuine null rather than an untested one, and that model stays inside an 8 -point band across every property we varied (§ S8 ). The comparison that matters is that colour, content and pixel count move refusal while an image is attached in both arms , which presence-versus-absence contrasts cannot show. None of these properties carries information about the request, which is what makes the dependence a defect rather than a calibration.
system message
refusal
Δ
A
(none)
14%
:
P2
Respond to the text…
24%
+10 ∗
P1
+ “any file attached…”
40%
+16 ∗∗
C
file → image
39%
−1
Table 4: [ primary test, control arm ] Placebo ladder on gemini-2.5-flash , 100 prompts per arm, all four arms collected in one job, no image attached in any arm . Each rung adds one ingredient to the rung above, so the +25 -point total ( p=4.2×10−7 ) decomposes additively; P1 and C are byte-identical apart from the single word. ∗∗p<0.01 , ∗p<0.05 , exact McNemar paired per prompt. The C − P1 contrast is n.s. ( 12 / 13 discordant, p=1.0 ), which at 25 discordant pairs excludes an image-word effect above ∼11 points: this model’s response to an asserted attachment is not a response to the modality word.
benign refusal %
harmful
Model
neutral
borderline
ASR %
qwen2.5-vl-7b
0 → 2
38 → 37
4 → 7
internvl3-8b
1 → 3
48 → 55
5 → 2
pixtral-12b
0 → 1
0 4 → 0 2
48 → 81 ∗∗∗
Table 5: [ exploratory follow-up ] Open-weight models, same manipulation (text → blank canvas), no defense, no moderation layer, 100 prompts per cell, paired per prompt id, judged by gpt-5-mini as elsewhere. ∗∗∗p=1.0×10−7 , exact McNemar; all unmarked contrasts n.s. The benign inflation that reaches +23 to +51 points on hosted models (Table 1 ) is absent here, while pixtral-12b inverts on the harmful axis (§ 4 ).
Model
no image
+ canvas
Δ
gain/loss
p
claude-sonnet-4-6
18
0 0
−18
0/18
7.6×10−6
gemini-2.5-flash-lite
11
0 5
−6
0/ 0 6
0.031
gpt-4o-mini
0 6
0 1
−5
0/ 0 5
0.063
gemini-2.5-flash
14
13
−1
5/ 0 6
1.00
Table 6: [ primary test ] What the canvas buys on the hosted models : attack success (%) on OR-Bench’s harmful split, the constructed counterpart of the borderline-benign rung, under {no image, blank canvas}, no defense, 100 prompts, paired per prompt id, HarmBench classifier on gpt-5-mini . “gain/loss” counts the discordant pairs in each direction on the same convention as Table S15 of the Supplementary Document: a completion the canvas caused , then one it prevented. This is the buy side of the price reported beside it , and it is small and getting smaller: the largest reduction is 18 points on the model with the most headroom, two of the four do not clear p<0.05 , and on gemini-2.5-flash the difference is a single prompt. Set against the +51 , +34 and +23 points of benign cost on the same models (Table 1 ), and read together with the open checkpoint that pays +29 points against a 2% harmful baseline (Table S14 of the Supplementary Document), this is the decoupling the section reports. No prompt is shared between this table and the benign rungs; the two splits are matched by construction, not by item.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Result
Collected
Target models
Judge(s)
Cells
Hosted models
Tab. 1 (benign sensitivity ladder)
2026-08-06/08
4 hosted
gpt-5-mini , gpt-5-nano
32
Tab. 6 (hosted harmful side)
2026-08-06/08
4 hosted
gpt-5-mini , gpt-5-nano
16
Tab. S12 (category-stratified cost)
2026-08-09
claude, fl.-lite, 4o-mini
gemini-2.5-pro , gpt-5-mini
12
Tab. 2 (presence controls)
2026-08-07/08
4 hosted
gpt-5-mini , gpt-5-nano
32
Instruction arm, gemini-2.5-flash
2026-08-08
flash
gpt-5-mini
4
Appendix
Table S1: Provenance of every result in this paper. One row per collection campaign; the “Cells” column counts the released per-prompt judgment files behind it, and the eighteen rows sum to the 282 cells shipped in the code artifact’s judgments index, which is the machine-readable form of this table and carries the exact run directory, timestamp, prompt count and judge for each cell individually. No cell in this paper has a defense in the loop , so the defense-configuration and evaluation-protocol columns that would ordinarily appear here are vacuous and are omitted: the only manipulation anywhere is whether an image is attached. Benign rungs are scored with the JailbreakBench refusal classifier and harmful arms with the HarmBench classifier. A row naming more than one judge ships the original gpt-5-mini scoring plus the robustness rescores of § S29 , which re-score stored responses and re-query no model. Every contrast is collected inside a single job (§ 2 ); rows collected on different dates are never compared to one another, and a date range spans the arms of one campaign rather than the two sides of one contrast. Three cells of the randomised-interleaving control were stored without per-prompt rows and ship as recorded rates only; they are the sole exception and are flagged in the index.
family
n
method
F1
presence effect, borderline rung ( primary )
10
Holm
F2
which image properties change the price
10
BH
F3
generational ladder
3
Holm
F4
instruction/mention decomposition
6
Holm
F5
ladder rungs other than borderline
3
Holm
F6
size axis, replicated across three fills
3
Holm
Appendix
Table S2: Declared test families. F1 pools the borderline-rung presence contrast across all ten models we measured it on, hosted and open, with no selection: the four hosted models of Table 1 , the five of Table S4 , and llava-1.5-7b , including every null and both signs. F7 exists because the sign inversion had previously been a lone test belonging to no family, which left the paper’s steerable claim the only one not carried through a correction.
paper name
Hugging Face repository
qwen2-vl-7b
Qwen/Qwen2-VL-7B-Instruct
qwen2.5-vl-7b
Qwen/Qwen2.5-VL-7B-Instruct
qwen3-vl-8b-instruct
Qwen/Qwen3-VL-8B-Instruct
internvl3-8b
OpenGVLab/InternVL3-8B
pixtral-12b
mistralai/Pixtral-12B-2409
llava-1.5-7b
llava-hf/llava-1.5-7b-hf
Appendix
Table S3: Open-weight checkpoints. The serving-route control pairs the last row’s self-served route against the same weights on AWS Bedrock ( google.gemma-3-12b-it ); the two identifiers differ by one character, so the pairing is carried in the model registry as a fact rather than inferred from the strings.
Model
text
blank
Δ
disc.
p
qwen3-vl-8b
51
83
+32
32/0
<10−4
gemma-3-12b-it
0 4
13
+9
10/1
0.012
internvl3-8b
48
55
+7
13/6
0.167
qwen2.5-vl-7b
37
37
+0
0 7/7
1.000
pixtral-12b
0 4
0 3
−1
0 3/4
1.000
Appendix
Table S4: [ model-selection scan ] Five open-weight VLMs, one protocol : benign refusal (%) on the borderline rung, text → blank canvas, no defense, no moderation layer, 100 prompts, paired per prompt id, all cells in one job. “disc.” is discordant pairs (gained/lost); p is exact McNemar. qwen3-vl-8b carries a hosted-scale effect on an open checkpoint, which is what bounds this paper’s scope claim (§ 4 ). Four of the five share one coarse alignment-tier label and span +32 to +0 points, so that label does not predict the effect.
Δ benign refusal vs. that model’s text arm (points)
Attached image
4o-mini
fl.-lite
claude
flash
text baseline (%)
13
11
11
18
blank 5122 white
+31
+23
+38
−5
blank 2562 white
+32
+32
—
0
blank 15362 white
+33
+24
—
−3
blank 1024×141 white
+29
+27
+32
−6
Appendix
Table S5: [ primary test ] Varying the image’s properties one at a time. Benign refusal shift vs. each model’s own text arm, rung 2, no defense, 100 prompts, paired per prompt id, all arms in one job. Bold : p≤1.5×10−6 , exact McNemar; every unbolded contrast is n.s. gpt-4o-mini is property-invariant ( 8 points across a sixfold resolution change, a transposed aspect ratio, three colours, two encodings and three content classes); gemini-2.5-flash-lite and claude-sonnet-4-6 show a large content premium at matched size ( +57 vs +27 , +60 vs +32 at 1024×141 ). claude was run on four load-bearing arms only, at ∼20× the per-cell cost of the others. All ten gemini-2.5-flash arms are null (§ 4 ). Instances per property : each row is one independently rendered file, held byte-identical across the 100 prompts of its arm, so a single row cannot separate a property from the particular file carrying it. Two axes are replicated across independent instances and carry the property claims: size, over three distinct fills at each of four sizes (twelve arms, Table S7 ), and content, over three independently constructed files per level at matched dimensions on three hosted models (Table S6 ). Colour on the hosted models is not replicated and cannot be: independent renders of a flat fill are byte-identical. Direct image-versus-image contrasts for this table are in Table 3 .
model
level
blank
three instances
mean Δ
spread
claude
drawing 5122
47
71 / 67 / 74
+23.7
7.0
claude
caption 1024×141
50
76 / 73 / 70
+23.0
6.0
fl.-lite
drawing 5122
34
59 / 63 / 54
+24.7
9.2
fl.-lite
caption 1024×141
38
62 / 71 / 57
+24.7
14.3
4o-mini
drawing 5122
46
37 / 38 / 39
−8.0∗
2.0
4o-mini
caption 1024×141
43
43 / 43 / 46
+1.0
3.0
Appendix
Table S6: [ primary test ] Content levels replicated over independent instances (benign borderline rung, no defense, 100 prompts per arm, all thirty-three cells in one job, paired per prompt id against the size-matched blank re-run in that same job). Each level is represented by three independently constructed files at identical dimensions (three clip-art line drawings, three caption sentences) so spread is the between-instance range in refusal rate and is the quantity a single-file row cannot provide. Bold : all three instance contrasts significant at p<0.001 , exact McNemar. ∗ : two of three at p<0.05 (the third p=0.065 ). The last row is n.s. throughout and is a null, not an unstable effect. Read the two right-hand columns together: wherever an effect exists, the spread across instances is a fraction of it, so the level and not the artifact is carrying it. The negative sign on gpt-4o-mini is against the blank , not against text (its text arm refuses 13% ; every image arm 37 – 46% ). Colour is deliberately absent: independent renders of a flat fill are byte-identical, so colour admits no instance replication and is re-measured for run-to-run stability instead (discussed in text). Four cells lost 2 – 3 rows to transient provider unavailability and are computed over 97 – 98 pairs; the re-run threshold was fixed at 5% before any result was read.
Δ benign refusal (points) vs. text, by canvas size
fill
2562
5122
10242
15362
white
+23
+26
+31
+36
grey
+20
+33
+36
+39
black
+23
+26
+33
+35
Appendix
Table S7: [ replication ] Image-instance replication on qwen3-vl-8b : four canvas sizes crossed with three fills, twelve independently rendered arms plus a shared text baseline ( 51% ), 100 prompts each, one job, no defense, paired per prompt id. Every cell is significant at p<0.001 (exact McNemar against the text arm). The size trend is monotone in all three fills , so the axis is a property rather than an artifact of one rendered file; the fill spread at fixed size ( 3 – 7 points) stays well inside the size span ( 12 – 19 points), replicating colour-inertness at four sizes. Each arm is byte-identical within itself and all twelve differ from one another, verified by hash (§ 4 ).
benign refusal (%)
harmful ASR (%)
Generation
text
blank
Δ
p
text
blank
Δ
qwen2-vl-7b (oldest)
72 †
80
+8
0.057
2
2
− 0
qwen2.5-vl-7b
37
38
+1
1.000
4
7
+3
qwen3-vl-8b (newest)
54
82
+28
<10−4
2
1
− 1
Appendix
Table S8: [ exploratory follow-up ] One family, three generations, both sides : text → blank canvas, no defense, same canonical text, paired per prompt id, n=100 per arm. Family, size class, serving stack, judge ( gpt-5-mini ) and the rendered inputs themselves are held fixed; only the checkpoint moves. Benign: the effect is a step at the newest rung, not a gradient. We describe this as a step between adjacent checkpoints of this family, not as the date the behaviour entered the world: we observe three shipped checkpoints of one family, which cannot say when or whether it appeared elsewhere. A release changes alignment, vision encoder and pretraining corpus together, so this does not isolate post-training. Harmful: every rung is a bounded null, with 95% intervals [−4.5,+4.5] , [−2.7,+9.3] and [−6.1,+3.7] points, all equivalent to zero within a ±10 -point margin and the oldest within ±5 points. The two sides do not move together at any rung. † qwen2-vl refuses 72% with no image at all, so its +8 points is measured against a partly saturated baseline and is not magnitude-comparable to the newest rung (§ 4 ).
break-even k⋆ at prevalence π
Model
Δben/Δharm
π=10%
π=1%
π=0.1%
claude-sonnet-4-6
2.8
26
280
2,800
gpt-4o-mini
6.8
61
670
6,800
gemini-2.5-flash-lite
3.8
34
380
3,800
Appendix
Table S9: [ sensitivity analysis, no new data ] How many times worse must one harmful completion be than one false refusal for the canvas to pay for itself. Arithmetic on the measured borderline-rung rates only ( Δbenign=+51,+34,+23 points against Δharmful=18,5,6 points of prevented harm); no new experiment. k⋆=π1−π⋅ΔharmfulΔbenign , rounded to two significant figures. The model-dependent term spans a factor of 2.4 across these three models, while the prevalence term spans a factor of 100 over the range shown, so the trade is governed mainly by a quantity we do not measure, which is precisely why we report the two sides separately and never as a ratio. Severity, partial compliance, and the availability of other routes to the same answer are not modelled and would each shift k⋆ further.
Figure S1: The three request-independent images used in this paper. (a) the caption image: a 1024×141 PNG of typed text on white, the only arm carrying readable characters, which is what places an OCR path on one side of the comparison. (b) a 512×512 pure-white blank. (c) a 1189×1418 clip-art line drawing of a rabbit : drawn content, no readable text, and no relation to any request in either benchmark. We name the subject because “line drawing” alone leaves a reader unable to check that the drawn content is request-independent, which is the property the arm depends on. None relates to any request, and within an arm every prompt receives the byte-identical file (§ 2 ).
added input
benign refusal % ( Δ vs. text baseline)
tokens
as text filler
as blank canvas
≈64
47 ( −7 , n.s.)
73 ( +19 )
≈260
50 ( −4 , n.s.)
76 ( +22 )
≈2310
54 ( ±0 , p=1.00 )
86 ( +32 )
Appendix
Table S10: [ primary test, control arm ] Input length is not what the model charges for ( qwen3-vl-8b , benign borderline rung, no defense, 100 prompts, all seven arms in one job, paired per prompt id, shared text baseline 54% ). Each row holds the added input budget fixed and changes only whether it arrives as text or as a blank white canvas. Budgets are measured on the target’s own accounting ( 63 / 273 / 2310 text against 66 / 258 / 2306 visual), so the largest text arm carries 4 more tokens than the image arm it is matched against. Text padding never exceeds the baseline at any budget, while the canvas raises refusal at every budget; at the matched extreme the two are 32 points apart ( 33 flips to 1 , p=4.1×10−9 ). The canvas column re-measures three white arms from an earlier job in a fresh target and judge pass and agrees within 1 points absolute.
paired contrast
refusal %
Δ
gained
lost
p
mention, no image
51→59
+8
12
4
0.077
attachment, no mention
51→81
+30
32
2
6.9×10−8
attachment, mention held
59→79
+20
20
0
1.9×10−6
Appendix
Table S11: [ primary test ] Attachment × mention on qwen3-vl-8b , benign refusal (%) on the borderline rung, no defense, 100 prompts, paired per prompt id, all four cells in one job. The four factorial cells are refusal 51 (no image, no mention), 59 (no image, mention), 81 (image, no mention) and 79 (image, mention); every row below is one paired contrast between two of them and is self-contained, so its Δ , counts and p can be checked without reference to any other row. The mention is a system message worded so the identical string is true whether or not an image is attached, so the mention factor is byte-identical across attachment conditions. Gained / lost are the discordant counts b / c , prompts that moved to refusal and away from it, so each Δ is (b−c)/100 and each p is an exact two-sided McNemar on that pair alone; both are recomputable from the two count columns. With mention held constant, attachment still moves refusal +20 points with no prompt moving the other way: the cue is not a lexical artifact. The mention cell is underpowered rather than null : at 16 discordant pairs the exact test could only have resolved ∼10 points (§ 4 ).
Δ benign refusal, text → blank (points)
OR-Bench category
claude
4o-mini
fl.-lite
deception
+80
+43
+13
harassment
+53
+23
+27
harmful
+47
+53
+20
hate
+57
+33
+53
illegal
+53
+33
+43
Appendix
Table S12: [ primary test ] The benign cost, stratified across all ten OR-Bench categories. 30 prompts per category, 300 per model, borderline-benign rung, no defense, both arms collected in one job per model, paired per prompt id. Bold : survives Benjamini–Hochberg across all 30 per-category tests ( 26 of 30 do; the four that do not are claude unethical p=0.070 and flash-lite deception p=0.219 , harmful p=0.070 , unethical p=0.070 ). Every contrast is positive. Aggregates: claude 158 discordant to 1 , p=4.4×10−46 ; gpt-4o-mini 109 to 0 , p=3.1×10−33 ; fl.-lite 110 to 6 , p=7.6×10−26 . gemini-2.5-flash is not re-run here: it is the null model on this rung, so we report it as untested on the stratified set rather than implying it was checked. One pre-specified check did not pass: we predicted claude’s no-image baseline would sit above its 12% on the original slice, and it came in at 9.3% , so the strata differ in base sensitivity for that model and the per-category deltas, not the baseline, are the quantity to read (§ 4 ).
arm
managed
self-served
Δ
text (no image)
6
5
+1
blank 1024×768
13
12
+1
blank 1024×141
13
12
+1
caption 1024×141
6
9
−3
Appendix
Table S13: [ exploratory follow-up ] Same weights, two serving stacks. Benign refusal rate (%) for gemma-3-12b-it , the same open checkpoint reached through a managed commercial host and through our own vLLM, on the borderline-benign rung with no defense, 100 prompts, one canonical text, paired per prompt id. Route differences are bounded within ±10 points on every arm and within ±5 points on both blank arms ( 95% Newcombe intervals; 1 – 3 discordant pairs of 100 ; exact McNemar p≥0.25 ), while a blank canvas inflates refusal +7 points on both. Weights, vision encoder, pretraining and post-training alignment are identical across the two columns by construction, so the comparison isolates the inference implementation and vendor wrapper, and finds them inert (§ 4 ).
Rung
text
blank
Δ
disc.
p
neutral
0 2
0 5
+3
4/1
0.375
borderline
53
82
+29
30/1
<10−4
harmful (ASR)
0 2
0 1
−1
1/2
1.000
Appendix
Table S14: [ replication ] Both sides of the price on one open checkpoint ( qwen3-vl-8b ), all cells in one job, 100 prompts each, no defense. Benign rungs are refusal rate (JailbreakBench refusal classifier); the harmful rung is ASR (HarmBench classifier) on OR-Bench’s harmful split, the matched counterpart of the borderline rung. The ladder structure replicates on a downloadable model. The harmful contrast is an underpowered null, not a zero : this checkpoint already yields a harmful completion on only 2% of plain harmful requests, so there is almost nothing left for the cue to prevent, and we do not claim it buys nothing. With the floor removed by attack the cue does measurably act on this checkpoint, in both directions depending on encoding (Table S15 ).
encoding
no image
+ canvas
Δ
gain/loss
p
plain
17
0 7
−10
0/10
0.0020
set theory
52
40
−12
7/19
0.029
formal logic
51
46
−5
11/16
0.442
code
63
73
+10
16/ 0 6
0.052
Appendix
Table S15: [ primary test, pre-specified ] The harmful side with the floor removed ( qwen3-vl-8b , 100 HarmBench behaviours per cell, no defense, one job). Cells are attack success rate under the HarmBench classifier; p is an exact paired McNemar test over the same 100 behaviours, and gain/loss counts the discordant pairs in each direction. The three non-plain encodings are established published attacks used here as instruments and not proposed by us: set theory and formal logic follow the mathematical-encoding attacks of Zhang, Zandsalimy, and Sushmita (2026) , and the code row follows CodeAttack ( Ren et al. 2024 ) . Encoded text is byte-identical across each row’s two arms, so the 5122 white canvas is the only difference. Every no-image arm clears the 10 – 90% band we fixed in advance for a contrast to count as powered. The cue is not inert here , but it helps on three arms and hurts on one, and which it does depends on how the request is written (§ 4 ).
Claim
Status
Measured : direct paired contrasts, reported with intervals
Image presence raises benign refusal on particular aligned checkpoints ( +23 to +54 points, hosted; +28 points on an open one)
measured
⋆ The cost is spread across all ten OR-Bench categories, not two
measured (Tab. S12 )
⋆ The price varies with canvas colour and pixel count
measured (Tab. 3 ), with an image attached in both arms of every contrast; colour on the hosted models rests on one file per colour, which is structural, since flat fills admit no independent renders
⋆ The pixel-count price is not an input-length effect
measured (Tab. S10 )
The sign inverts on two open checkpoints ( +33 , +39 points ASR)
measured
Appendix
Table S16: What this paper measures, what it infers from a control, and what it does not establish. Tier 3 is deliberately non-empty: each row is a reading a reader might otherwise take from the prose, listed with the reason our design cannot support it. ⋆ marks the four rows carrying the most evidential weight: the category-stratified benign cost, the colour and pixel-count contrasts taken with an image attached in both arms, the token-matched control, and the mention-held factorial. A reader who reads only those four rows has this paper’s load-bearing evidence; every other row supports, bounds or qualifies them.
Zou et al. (2025)
this paper
request-independent blank image as the control
yes
yes
axis measured
harmful, weakening direction
benign and harmful, both signs
benign refusal attributed to attachment itself
effect of their correction on false alarms
effect of attachment, paired and stratified
matched harmful counterpart to the benign rung
—
yes
image-property contrasts
—
paired between-arm contrasts, image attached in both arms
mention placebo with no image attached
—
yes
Appendix
Table S17: What is new here relative to the closest prior work. Zou et al. (2025) share our central design choice, a request-independent blank image as the control, and reach it independently; we state the delimitation here rather than after the related-work section so a reader can locate the contribution immediately. The differences are not framing: they are the benign-cost accounting, the matched harmful counterpart, the property and placebo controls, and the open-checkpoint and serving-route arms. § 5 gives the full comparison, including where our sign result and theirs disagree and why we read them as two signs of one effect.
Vision-language models (VLMs) are increasingly trained to generate structured outputs like points and bounding boxes that downstream interfaces, agents, and robots can act on, yet safety alignment of this output channel has not been systematically analyzed. We study visual grounding safety by repurposing three safety benchmarks spanning direct harm (VLSU), social bias (BBQ-V), and situational safety (Asimov-2.0) into 15,401 matched pairs of harmful requests that differ only in the requested output: a free-text answer (VQA) or a grounding (point or bounding box). Across five VLMs, models that refuse a harmful request posed as a question often comply when the same request asks for a grounding: averaged over models, grounding refusal trails VQA refusal by 31-59 percentage points, depending on the domain, and safety system prompts do not close this gap. We propose a fine-tuning approach that combines grounding-form refusals with capability grounding data and self-distilled benign data to counter over-refusal. For Qwen3-VL-8B and VisionReasoner-7B, it improves grounding refusal by 77-95 percentage points on VLSU and BBQ-V and by 64-85 points on the held-out Asimov-2.0 domain, while also improving VQA refusal, preserving grounding capability, and keeping over-refusal limited. Representation analysis shows that fine-tuning moves harmful requests toward each model's refusal direction, most strongly for grounding, while leaving benign requests near the harmless reference.
Erfan Shayegani, Kundan Krishna, Yue Dong +3
University of California, Riverside · Work done during internship at Apple. · Apple
Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.
Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8
For reliable deployment, multimodal large language models (MLLMs) should recognize when an image provides insufficient evidence to answer a question and withhold unsupported answers. Yet refusal alone is not necessarily evidence of such recognition: models may refuse because of unrelated safety constraints or without identifying the missing evidence. We study how refusal on visually unanswerable questions is shaped by three sources of control: inference-time instructions, post-training, and internal activation interventions. Across three MLLMs on the MoHoBench benchmark, we systematically manipulate the meaning and placement of refusal instructions, compare supervised and preference-based post-training, and intervene on a refusal-related activation direction derived from naturally answered and refused examples. We find that refusal is highly sensitive to both the meaning and placement of instructions, with model-specific changes at the sample level that are obscured by aggregate refusal rates. Evidence-aware and general refusal instructions have stronger effects than an intentionally mismatched safety instruction. Post-training produces larger increases in refusal, but can also induce responses that combine refusal language with attempted answers. In contrast, activation intervention produces smaller yet transferable changes across prompt conditions, without evidence of indiscriminate refusal on tested visually answerable samples. These findings show that refusal is jointly shaped by external instructions, post-training, and internal representations, highlighting the need to disentangle evidence recognition from a model's learned propensity to refuse.