cs.CRAug 12, 2026

The Uncontrolled Variable: Vision-Language Refusal Is Conditioned on the Image-Attachment Interface, and Not Robust to Irrelevant Image Properties

Authors: Haoyu Zhang, Yi Feng, Hanwen Liu, Shibo Zheng, Zhuoxi Wang, Yang Chen, Haowen Xu, Xiangchen Guan, 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We show that aligned vision-language models also condition refusal on a property of a request's form: whether an image is attached, holding everything the request asks fixed. Attaching a blank canvas, an image that cannot be read, cannot relate to the request, and is byte-identical across every prompt in its condition, shifts benign refusal by tens of points. The shift is not blanket caution but a threshold shift: genuinely neutral instructions are almost unaffected (<=2 percentage points on three of four hosted models) while borderline-benign prompts move +23 to +51 points, so the cost falls on sensitivity-adjacent traffic, meaning benign questions about privacy, self-harm, violence and illegal activity. There is a benign reading of such a threshold, namely that attachment correlates with risk in real traffic, and we take it seriously; a black-box study cannot measure that correlation and we do not claim to. What it can test is whether the response to attachment is robust to variation carrying no information about the request, and on four independent measurements it is not. It varies with canvas colour and pixel count. Its sign inverts across checkpoints. It survives an explicit instruction to disregard the image. And on one model it fires on a bare assertion that an attachment exists, with nothing attached and the modality word contributing none of it. Finally we price it. On a matched harmful set the same canvas does lower attack success, so the cue buys something. But the charge is decoupled from the purchase: the checkpoint with the least harmful headroom we measure, completing only 2% of plain harmful requests, still pays the benign cost in full, and across our models the harmful-side denominator falls as alignment improves while the benign cost does not track it down. Image presence is not a conservative default that a deployer chose and priced. It is an uncontrolled variable.

Figures & tables

 benign refusal %, text →\to blank canvas
Modelneutralborderlinetopic-matched
claude-sonnet-4-62 →\to 212 →\to 63 ∗∗∗3 →\to 36
gpt-4o-mini2 →\to 212 →\to 46 ∗∗∗4 →\to 19
gemini-2.5-flash-lite1 →\to 14 ∗∗∗11 →\to 34 ∗∗∗11 →\to 45
gemini-2.5-flash0 →\to 216 →\to 1311 →\to 18
Table 1: [ primary test ] The sensitivity ladder : benign refusal rate (%, lower is more useful) under {no image, blank canvas}, no defense, 100100 prompts per rung, paired per prompt id. ∗∗∗p≤1.5×10−6{}^{***}p\leq 1.5\times 10^{-6} , exact McNemar. Significance is marked on the neutral and borderline rungs, the two columns entering the declared test families (§ S4 ); unmarked cells in those two columns are non-significant. The topic-matched column is reported for continuity and is not part of any tested family, so it carries no marks. Inflation is absent on neutral prompts and large on borderline-benign ones: a threshold shift rather than blanket caution (§ 4 ). The topic-matched column is the JailbreakBench-benign set, shown for continuity.
 benign refusal %, rung 2 (borderline)
Modeltextblankline draw.blank+“ignore”
claude-sonnet-4-61064 ∗∗∗58 ∗∗∗52
gpt-4o-mini1343 ∗∗∗44 ∗∗∗36
gemini-2.5-flash-lite1134 ∗∗∗63 ∗∗∗28
gemini-2.5-flash151425 ∗47 ∗∗∗
Table 2: [ primary test, control arm ] Presence controls , all arms in one job, one canonical text, 100100 prompts each, no defense. Two request-independent image classes, a blank canvas ( 1024×7681024\times 768 , pure white) and the clip-art line drawing of Fig. S1 (c) ( 1189×14181189\times 1418 , drawn content, no text), plus the cheapest mitigation (a neutral instruction to disregard the image). The blank canvas here is generated at the renderer’s default size, and is therefore a different file from the 512×512512\times 512 blank of Fig. S1 (b); both are pure white and neither carries readable text. ∗∗∗p<10−5{}^{***}p<10^{-5} , ∗p<0.05{}^{*}p<0.05 against the text baseline, exact McNemar paired per prompt; unmarked cells in the two image columns are non-significant against that baseline. The instruction column is a mitigation arm whose contrast of interest is against the blank arm rather than against text, and those three contrasts are reported in the text; the one mark in that column is gemini-2.5-flash , where the instruction is not a mitigation but the manipulation itself and the text baseline is therefore the right comparator. Both classes inflate refusal on three of four models; the instruction removes only 66 – 1212 points of a 2323 – 5454 -point shift, and on gemini-2.5-flash it adds 3232 points: an asserted-attachment effect rather than an image effect, decomposed in Table 4 . The gemini-2.5-flash line-drawing cell ( +10+10 points, p=0.021p=0.021 ) does not replicate: a larger property sweep puts the same contrast at +9+9 points, p=0.064p=0.064 (§ S8 ). We treat that model as near-null.
contrast (image attached in both arms)modelΔ\Deltadisc.pp95%95\% CI
black vs white, 5122512^{2}fl.-lite+22+2222/04.8×10−74.8{\times}10^{-7}[+13.7,+29.7][+13.7,+29.7]
content vs blank, 1024×1411024{\times}141fl.-lite+30+3031/11.5×10−81.5{\times}10^{-8}[+20.2,+38.7][+20.2,+38.7]
content vs blank, 1024×1411024{\times}141claude+28+2830/22.5×10−72.5{\times}10^{-7}[+18.0,+37.0][+18.0,+37.0]
content vs blank, 1024×1411024{\times}1414o-mini+1+10 5/41.001.00[−4.8,+6.8][-4.8,+6.8]
blank 153621536^{2} vs blank 2562256^{2}qwen3-vl+16+1618/20.00040.0004[+7.7,+24.9][+7.7,+24.9]
Table 3: [ primary test ] The response is not invariant to properties that carry no risk information. Each row compares two attached-image arms against each other rather than against the text baseline, so image presence is held fixed and only the named property moves; size is matched within every row except the last, which varies size alone at fixed content. Every row is a paired exact McNemar contrast computed between the two image arms on matched prompt ids, with Newcombe intervals; Δ\Delta is the larger-property arm minus the smaller. The first four rows are the three hosted models, n=100n=100 per arm; the last is the open checkpoint qwen3-vl-8b , also n=100n=100 . The paired tests resolve what raw rate differences could not: the claude content premium is established ( p=2.5×10−7p=2.5{\times}10^{-7} ), while the gpt-4o-mini row is a genuine null rather than an untested one, and that model stays inside an 88 -point band across every property we varied (§ S8 ). The comparison that matters is that colour, content and pixel count move refusal while an image is attached in both arms , which presence-versus-absence contrasts cannot show. None of these properties carries information about the request, which is what makes the dependence a defect rather than a calibration.
 system messagerefusalΔ\Delta
A(none)14%14\%:
P2Respond to the text…24%24\%+10 ∗
P1+ “any file attached…”40%40\%+16 ∗∗
Cfile →\rightarrow image39%39\%−1-1
Table 4: [ primary test, control arm ] Placebo ladder on gemini-2.5-flash , 100100 prompts per arm, all four arms collected in one job, no image attached in any arm . Each rung adds one ingredient to the rung above, so the +25+25 -point total ( p=4.2×10−7p=4.2\times 10^{-7} ) decomposes additively; P1 and C are byte-identical apart from the single word. ∗∗p<0.01{}^{**}p<0.01 , ∗p<0.05{}^{*}p<0.05 , exact McNemar paired per prompt. The C −- P1 contrast is n.s. ( 1212 / 1313 discordant, p=1.0p=1.0 ), which at 2525 discordant pairs excludes an image-word effect above ∼ ⁣11\sim\!11 points: this model’s response to an asserted attachment is not a response to the modality word.
 benign refusal %harmful
ModelneutralborderlineASR %
qwen2.5-vl-7b0 →\to 238 →\to 374 →\to 7
internvl3-8b1 →\to 348 →\to 555 →\to 2
pixtral-12b0 →\to 10 4 →\to 0 248 →\to 81 ∗∗∗
Table 5: [ exploratory follow-up ] Open-weight models, same manipulation (text →\to blank canvas), no defense, no moderation layer, 100100 prompts per cell, paired per prompt id, judged by gpt-5-mini as elsewhere. ∗∗∗p=1.0×10−7{}^{***}p=1.0\times 10^{-7} , exact McNemar; all unmarked contrasts n.s. The benign inflation that reaches +23+23 to +51+51 points on hosted models (Table 1 ) is absent here, while pixtral-12b inverts on the harmful axis (§ 4 ).
Modelno image++ canvasΔ\Deltagain/losspp
claude-sonnet-4-6180 0−18\mathbf{-18}0/187.6×10−6\mathbf{7.6{\times}10^{-6}}
gemini-2.5-flash-lite110 5−6\mathbf{-6}0/ 0 60.031\mathbf{0.031}
gpt-4o-mini0 60 1−5-50/ 0 50.0630.063
gemini-2.5-flash1413−1-15/ 0 61.001.00
Table 6: [ primary test ] What the canvas buys on the hosted models : attack success (%) on OR-Bench’s harmful split, the constructed counterpart of the borderline-benign rung, under {no image, blank canvas}, no defense, 100100 prompts, paired per prompt id, HarmBench classifier on gpt-5-mini . “gain/loss” counts the discordant pairs in each direction on the same convention as Table S15 of the Supplementary Document: a completion the canvas caused , then one it prevented. This is the buy side of the price reported beside it , and it is small and getting smaller: the largest reduction is 1818 points on the model with the most headroom, two of the four do not clear p<0.05p<0.05 , and on gemini-2.5-flash the difference is a single prompt. Set against the +51+51 , +34+34 and +23+23 points of benign cost on the same models (Table 1 ), and read together with the open checkpoint that pays +29+29 points against a 2%2\% harmful baseline (Table S14 of the Supplementary Document), this is the decoupling the section reports. No prompt is shared between this table and the benign rungs; the two splits are matched by construction, not by item.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

ResultCollectedTarget modelsJudge(s)Cells
Hosted models
Tab. 1 (benign sensitivity ladder)2026-08-06/084 hostedgpt-5-mini , gpt-5-nano32
Tab. 6 (hosted harmful side)2026-08-06/084 hostedgpt-5-mini , gpt-5-nano16
Tab. S12 (category-stratified cost)2026-08-09claude, fl.-lite, 4o-minigemini-2.5-pro , gpt-5-mini12
Tab. 2 (presence controls)2026-08-07/084 hostedgpt-5-mini , gpt-5-nano32
Instruction arm, gemini-2.5-flash2026-08-08flashgpt-5-mini4
Appendix
Table S1: Provenance of every result in this paper. One row per collection campaign; the “Cells” column counts the released per-prompt judgment files behind it, and the eighteen rows sum to the 282282 cells shipped in the code artifact’s judgments index, which is the machine-readable form of this table and carries the exact run directory, timestamp, prompt count and judge for each cell individually. No cell in this paper has a defense in the loop , so the defense-configuration and evaluation-protocol columns that would ordinarily appear here are vacuous and are omitted: the only manipulation anywhere is whether an image is attached. Benign rungs are scored with the JailbreakBench refusal classifier and harmful arms with the HarmBench classifier. A row naming more than one judge ships the original gpt-5-mini scoring plus the robustness rescores of § S29 , which re-score stored responses and re-query no model. Every contrast is collected inside a single job (§ 2 ); rows collected on different dates are never compared to one another, and a date range spans the arms of one campaign rather than the two sides of one contrast. Three cells of the randomised-interleaving control were stored without per-prompt rows and ship as recorded rates only; they are the sole exception and are flagged in the index.
 familynnmethod
F1presence effect, borderline rung ( primary )1010Holm
F2which image properties change the price1010BH
F3generational ladder33Holm
F4instruction/mention decomposition66Holm
F5ladder rungs other than borderline33Holm
F6size axis, replicated across three fills33Holm
Appendix
Table S2: Declared test families. F1 pools the borderline-rung presence contrast across all ten models we measured it on, hosted and open, with no selection: the four hosted models of Table 1 , the five of Table S4 , and llava-1.5-7b , including every null and both signs. F7 exists because the sign inversion had previously been a lone test belonging to no family, which left the paper’s steerable claim the only one not carried through a correction.
paper nameHugging Face repository
qwen2-vl-7bQwen/Qwen2-VL-7B-Instruct
qwen2.5-vl-7bQwen/Qwen2.5-VL-7B-Instruct
qwen3-vl-8b-instructQwen/Qwen3-VL-8B-Instruct
internvl3-8bOpenGVLab/InternVL3-8B
pixtral-12bmistralai/Pixtral-12B-2409
llava-1.5-7bllava-hf/llava-1.5-7b-hf
Appendix
Table S3: Open-weight checkpoints. The serving-route control pairs the last row’s self-served route against the same weights on AWS Bedrock ( google.gemma-3-12b-it ); the two identifiers differ by one character, so the pairing is carried in the model registry as a fact rather than inferred from the strings.
ModeltextblankΔ\Deltadisc.pp
qwen3-vl-8b5183+3232/0<10−4\mathbf{<10^{-4}}
gemma-3-12b-it0 413+910/10.012
internvl3-8b4855+713/60.167
qwen2.5-vl-7b3737+00 7/71.000
pixtral-12b0 40 3−1-10 3/41.000
Appendix
Table S4: [ model-selection scan ] Five open-weight VLMs, one protocol : benign refusal (%) on the borderline rung, text →\to blank canvas, no defense, no moderation layer, 100100 prompts, paired per prompt id, all cells in one job. “disc.” is discordant pairs (gained/lost); pp is exact McNemar. qwen3-vl-8b carries a hosted-scale effect on an open checkpoint, which is what bounds this paper’s scope claim (§ 4 ). Four of the five share one coarse alignment-tier label and span +32+32 to +0+0 points, so that label does not predict the effect.
 Δ\Delta benign refusal vs. that model’s text arm (points)
Attached image4o-minifl.-liteclaudeflash
text baseline (%)13111118
blank 5122512^{2} white+31+23+38−5-5
blank 2562256^{2} white+32+32—00
blank 153621536^{2} white+33+24—−3-3
blank 1024×1411024\times 141 white+29+27+32−6-6
Appendix
Table S5: [ primary test ] Varying the image’s properties one at a time. Benign refusal shift vs. each model’s own text arm, rung 2, no defense, 100100 prompts, paired per prompt id, all arms in one job. Bold : p≤1.5×10−6p\leq 1.5\times 10^{-6} , exact McNemar; every unbolded contrast is n.s. gpt-4o-mini is property-invariant ( 88 points across a sixfold resolution change, a transposed aspect ratio, three colours, two encodings and three content classes); gemini-2.5-flash-lite and claude-sonnet-4-6 show a large content premium at matched size ( +57+57 vs +27+27 , +60+60 vs +32+32 at 1024×1411024\times 141 ). claude was run on four load-bearing arms only, at ∼\sim 20×20\times the per-cell cost of the others. All ten gemini-2.5-flash arms are null (§ 4 ). Instances per property : each row is one independently rendered file, held byte-identical across the 100100 prompts of its arm, so a single row cannot separate a property from the particular file carrying it. Two axes are replicated across independent instances and carry the property claims: size, over three distinct fills at each of four sizes (twelve arms, Table S7 ), and content, over three independently constructed files per level at matched dimensions on three hosted models (Table S6 ). Colour on the hosted models is not replicated and cannot be: independent renders of a flat fill are byte-identical. Direct image-versus-image contrasts for this table are in Table 3 .
modellevelblankthree instancesmean Δ\Deltaspread
claudedrawing 5122512^{2}47477171 / 6767 / 7474+23.7\mathbf{+23.7}7.07.0
claudecaption 1024×1411024{\times}14150507676 / 7373 / 7070+23.0\mathbf{+23.0}6.06.0
fl.-litedrawing 5122512^{2}34345959 / 6363 / 5454+24.7\mathbf{+24.7}9.29.2
fl.-litecaption 1024×1411024{\times}14138386262 / 7171 / 5757+24.7\mathbf{+24.7}14.314.3
4o-minidrawing 5122512^{2}46463737 / 3838 / 3939−8.0∗-8.0^{*}2.02.0
4o-minicaption 1024×1411024{\times}14143434343 / 4343 / 4646+1.0+1.03.03.0
Appendix
Table S6: [ primary test ] Content levels replicated over independent instances (benign borderline rung, no defense, 100100 prompts per arm, all thirty-three cells in one job, paired per prompt id against the size-matched blank re-run in that same job). Each level is represented by three independently constructed files at identical dimensions (three clip-art line drawings, three caption sentences) so spread is the between-instance range in refusal rate and is the quantity a single-file row cannot provide. Bold : all three instance contrasts significant at p<0.001p<0.001 , exact McNemar. ∗ : two of three at p<0.05p<0.05 (the third p=0.065p=0.065 ). The last row is n.s. throughout and is a null, not an unstable effect. Read the two right-hand columns together: wherever an effect exists, the spread across instances is a fraction of it, so the level and not the artifact is carrying it. The negative sign on gpt-4o-mini is against the blank , not against text (its text arm refuses 13%13\% ; every image arm 3737 – 46%46\% ). Colour is deliberately absent: independent renders of a flat fill are byte-identical, so colour admits no instance replication and is re-measured for run-to-run stability instead (discussed in text). Four cells lost 22 – 33 rows to transient provider unavailability and are computed over 9797 – 9898 pairs; the re-run threshold was fixed at 5%5\% before any result was read.
 Δ\Delta benign refusal (points) vs. text, by canvas size
fill2562256^{2}5122512^{2}102421024^{2}153621536^{2}
white+23+26+31+36
grey+20+33+36+39
black+23+26+33+35
Appendix
Table S7: [ replication ] Image-instance replication on qwen3-vl-8b : four canvas sizes crossed with three fills, twelve independently rendered arms plus a shared text baseline ( 51%51\% ), 100100 prompts each, one job, no defense, paired per prompt id. Every cell is significant at p<0.001p<0.001 (exact McNemar against the text arm). The size trend is monotone in all three fills , so the axis is a property rather than an artifact of one rendered file; the fill spread at fixed size ( 33 – 77 points) stays well inside the size span ( 1212 – 1919 points), replicating colour-inertness at four sizes. Each arm is byte-identical within itself and all twelve differ from one another, verified by hash (§ 4 ).
 benign refusal (%)harmful ASR (%)
GenerationtextblankΔ\DeltapptextblankΔ\Delta
qwen2-vl-7b (oldest)72 †80+80.05722−- 0
qwen2.5-vl-7b3738+11.00047+3
qwen3-vl-8b (newest)5482+28<10−4\mathbf{<10^{-4}}21−- 1
Appendix
Table S8: [ exploratory follow-up ] One family, three generations, both sides : text →\to blank canvas, no defense, same canonical text, paired per prompt id, n=100n=100 per arm. Family, size class, serving stack, judge ( gpt-5-mini ) and the rendered inputs themselves are held fixed; only the checkpoint moves. Benign: the effect is a step at the newest rung, not a gradient. We describe this as a step between adjacent checkpoints of this family, not as the date the behaviour entered the world: we observe three shipped checkpoints of one family, which cannot say when or whether it appeared elsewhere. A release changes alignment, vision encoder and pretraining corpus together, so this does not isolate post-training. Harmful: every rung is a bounded null, with 95%95\% intervals [−4.5,+4.5][-4.5,+4.5] , [−2.7,+9.3][-2.7,+9.3] and [−6.1,+3.7][-6.1,+3.7] points, all equivalent to zero within a ±10\pm 10 -point margin and the oldest within ±5\pm 5 points. The two sides do not move together at any rung. † qwen2-vl refuses 72%72\% with no image at all, so its +8+8 points is measured against a partly saturated baseline and is not magnitude-comparable to the newest rung (§ 4 ).
  break-even k⋆k^{\star} at prevalence π\pi
ModelΔben/Δharm\Delta_{\text{ben}}/\Delta_{\text{harm}}π=10%\pi{=}10\%π=1%\pi{=}1\%π=0.1%\pi{=}0.1\%
claude-sonnet-4-62.82.826262802802,8002{,}800
gpt-4o-mini6.86.861616706706,8006{,}800
gemini-2.5-flash-lite3.83.834343803803,8003{,}800
Appendix
Table S9: [ sensitivity analysis, no new data ] How many times worse must one harmful completion be than one false refusal for the canvas to pay for itself. Arithmetic on the measured borderline-rung rates only ( Δbenign=+51,+34,+23\Delta_{\text{benign}}=+51,+34,+23 points against Δharmful=18,5,6\Delta_{\text{harmful}}=18,5,6 points of prevented harm); no new experiment. k⋆=1−ππ⋅ΔbenignΔharmfulk^{\star}=\frac{1-\pi}{\pi}\cdot\frac{\Delta_{\text{benign}}}{\Delta_{\text{harmful}}} , rounded to two significant figures. The model-dependent term spans a factor of 2.42.4 across these three models, while the prevalence term spans a factor of 100100 over the range shown, so the trade is governed mainly by a quantity we do not measure, which is precisely why we report the two sides separately and never as a ratio. Severity, partial compliance, and the availability of other routes to the same answer are not modelled and would each shift k⋆k^{\star} further.
Figure S1: The three request-independent images used in this paper. (a) the caption image: a 1024×1411024\times 141 PNG of typed text on white, the only arm carrying readable characters, which is what places an OCR path on one side of the comparison. (b) a 512×512512\times 512 pure-white blank. (c) a 1189×14181189\times 1418 clip-art line drawing of a rabbit : drawn content, no readable text, and no relation to any request in either benchmark. We name the subject because “line drawing” alone leaves a reader unable to check that the drawn content is request-independent, which is the property the arm depends on. None relates to any request, and within an arm every prompt receives the byte-identical file (§ 2 ).
added inputbenign refusal % ( Δ\Delta vs. text baseline)
tokensas text filleras blank canvas
≈ ⁣64\approx\!644747 ( −7-7 , n.s.)7373 ( +19+19 )
≈ ⁣260\approx\!2605050 ( −4-4 , n.s.)7676 ( +22+22 )
≈ ⁣2310\approx\!23105454 ( ±0\pm 0 , p=1.00p=1.00 )8686 ( +32+32 )
Appendix
Table S10: [ primary test, control arm ] Input length is not what the model charges for ( qwen3-vl-8b , benign borderline rung, no defense, 100100 prompts, all seven arms in one job, paired per prompt id, shared text baseline 54%54\% ). Each row holds the added input budget fixed and changes only whether it arrives as text or as a blank white canvas. Budgets are measured on the target’s own accounting ( 6363 / 273273 / 23102310 text against 6666 / 258258 / 23062306 visual), so the largest text arm carries 44 more tokens than the image arm it is matched against. Text padding never exceeds the baseline at any budget, while the canvas raises refusal at every budget; at the matched extreme the two are 3232 points apart ( 3333 flips to 11 , p=4.1×10−9p=4.1\times 10^{-9} ). The canvas column re-measures three white arms from an earlier job in a fresh target and judge pass and agrees within 11 points absolute.
paired contrastrefusal %Δ\Deltagainedlostpp
mention, no image51 ⁣→ ⁣5951\!\to\!59+8+81212440.0770.077
attachment, no mention51 ⁣→ ⁣8151\!\to\!81+30\mathbf{+30}3232226.9×10−86.9{\times}10^{-8}
attachment, mention held59 ⁣→ ⁣7959\!\to\!79+20\mathbf{+20}2020001.9×10−61.9{\times}10^{-6}
Appendix
Table S11: [ primary test ] Attachment ×\times mention on qwen3-vl-8b , benign refusal (%) on the borderline rung, no defense, 100100 prompts, paired per prompt id, all four cells in one job. The four factorial cells are refusal 5151 (no image, no mention), 5959 (no image, mention), 8181 (image, no mention) and 7979 (image, mention); every row below is one paired contrast between two of them and is self-contained, so its Δ\Delta , counts and pp can be checked without reference to any other row. The mention is a system message worded so the identical string is true whether or not an image is attached, so the mention factor is byte-identical across attachment conditions. Gained / lost are the discordant counts bb / cc , prompts that moved to refusal and away from it, so each Δ\Delta is (b−c)/100(b-c)/100 and each pp is an exact two-sided McNemar on that pair alone; both are recomputable from the two count columns. With mention held constant, attachment still moves refusal +20+20 points with no prompt moving the other way: the cue is not a lexical artifact. The mention cell is underpowered rather than null : at 1616 discordant pairs the exact test could only have resolved ∼\sim 1010 points (§ 4 ).
 Δ\Delta benign refusal, text →\to blank (points)
OR-Bench categoryclaude4o-minifl.-lite
deception+80+43+13+13
harassment+53+23+27
harmful+47+53+20+20
hate+57+33+53
illegal+53+33+43
Appendix
Table S12: [ primary test ] The benign cost, stratified across all ten OR-Bench categories. 3030 prompts per category, 300300 per model, borderline-benign rung, no defense, both arms collected in one job per model, paired per prompt id. Bold : survives Benjamini–Hochberg across all 3030 per-category tests ( 2626 of 3030 do; the four that do not are claude unethical p=0.070p=0.070 and flash-lite deception p=0.219p=0.219 , harmful p=0.070p=0.070 , unethical p=0.070p=0.070 ). Every contrast is positive. Aggregates: claude 158158 discordant to 11 , p=4.4×10−46p=4.4\times 10^{-46} ; gpt-4o-mini 109109 to 00 , p=3.1×10−33p=3.1\times 10^{-33} ; fl.-lite 110110 to 66 , p=7.6×10−26p=7.6\times 10^{-26} . gemini-2.5-flash is not re-run here: it is the null model on this rung, so we report it as untested on the stratified set rather than implying it was checked. One pre-specified check did not pass: we predicted claude’s no-image baseline would sit above its 12%12\% on the original slice, and it came in at 9.3%9.3\% , so the strata differ in base sensitivity for that model and the per-category deltas, not the baseline, are the quantity to read (§ 4 ).
armmanagedself-servedΔ\Delta
text (no image)6655+1+1
blank 1024×7681024\times 76813131212+1+1
blank 1024×1411024\times 14113131212+1+1
caption 1024×1411024\times 1416699−3-3
Appendix
Table S13: [ exploratory follow-up ] Same weights, two serving stacks. Benign refusal rate (%) for gemma-3-12b-it , the same open checkpoint reached through a managed commercial host and through our own vLLM, on the borderline-benign rung with no defense, 100100 prompts, one canonical text, paired per prompt id. Route differences are bounded within ±10\pm 10 points on every arm and within ±5\pm 5 points on both blank arms ( 95%95\% Newcombe intervals; 11 – 33 discordant pairs of 100100 ; exact McNemar p≥0.25p\geq 0.25 ), while a blank canvas inflates refusal +7+7 points on both. Weights, vision encoder, pretraining and post-training alignment are identical across the two columns by construction, so the comparison isolates the inference implementation and vendor wrapper, and finds them inert (§ 4 ).
RungtextblankΔ\Deltadisc.pp
neutral0 20 5+34/10.375
borderline5382+2930/1<10−4\mathbf{<10^{-4}}
harmful (ASR)0 20 1−1-11/21.000
Appendix
Table S14: [ replication ] Both sides of the price on one open checkpoint ( qwen3-vl-8b ), all cells in one job, 100100 prompts each, no defense. Benign rungs are refusal rate (JailbreakBench refusal classifier); the harmful rung is ASR (HarmBench classifier) on OR-Bench’s harmful split, the matched counterpart of the borderline rung. The ladder structure replicates on a downloadable model. The harmful contrast is an underpowered null, not a zero : this checkpoint already yields a harmful completion on only 2%2\% of plain harmful requests, so there is almost nothing left for the cue to prevent, and we do not claim it buys nothing. With the floor removed by attack the cue does measurably act on this checkpoint, in both directions depending on encoding (Table S15 ).
encodingno image++ canvasΔ\Deltagain/losspp
plain170 7−10\mathbf{-10}0/100.0020\mathbf{0.0020}
set theory5240−12\mathbf{-12}7/190.029\mathbf{0.029}
formal logic5146−5-511/160.442
code6373+10+1016/ 0 60.052
Appendix
Table S15: [ primary test, pre-specified ] The harmful side with the floor removed ( qwen3-vl-8b , 100100 HarmBench behaviours per cell, no defense, one job). Cells are attack success rate under the HarmBench classifier; pp is an exact paired McNemar test over the same 100100 behaviours, and gain/loss counts the discordant pairs in each direction. The three non-plain encodings are established published attacks used here as instruments and not proposed by us: set theory and formal logic follow the mathematical-encoding attacks of Zhang, Zandsalimy, and Sushmita (2026) , and the code row follows CodeAttack ( Ren et al. 2024 ) . Encoded text is byte-identical across each row’s two arms, so the 5122512^{2} white canvas is the only difference. Every no-image arm clears the 1010 – 90%90\% band we fixed in advance for a contrast to count as powered. The cue is not inert here , but it helps on three arms and hurts on one, and which it does depends on how the request is written (§ 4 ).
ClaimStatus
Measured : direct paired contrasts, reported with intervals
Image presence raises benign refusal on particular aligned checkpoints ( +23+23 to +54+54 points, hosted; +28+28 points on an open one)measured
⋆\star The cost is spread across all ten OR-Bench categories, not twomeasured (Tab. S12 )
⋆\star The price varies with canvas colour and pixel countmeasured (Tab. 3 ), with an image attached in both arms of every contrast; colour on the hosted models rests on one file per colour, which is structural, since flat fills admit no independent renders
⋆\star The pixel-count price is not an input-length effectmeasured (Tab. S10 )
The sign inverts on two open checkpoints ( +33+33 , +39+39 points ASR)measured
Appendix
Table S16: What this paper measures, what it infers from a control, and what it does not establish. Tier 3 is deliberately non-empty: each row is a reading a reader might otherwise take from the prose, listed with the reason our design cannot support it. ⋆\star marks the four rows carrying the most evidential weight: the category-stratified benign cost, the colour and pixel-count contrasts taken with an image attached in both arms, the token-matched control, and the mention-held factorial. A reader who reads only those four rows has this paper’s load-bearing evidence; every other row supports, bounds or qualifies them.
 Zou et al. (2025)this paper
request-independent blank image as the controlyesyes
axis measuredharmful, weakening directionbenign and harmful, both signs
benign refusal attributed to attachment itselfeffect of their correction on false alarmseffect of attachment, paired and stratified
matched harmful counterpart to the benign rung—yes
image-property contrasts—paired between-arm contrasts, image attached in both arms
mention placebo with no image attached—yes
Appendix
Table S17: What is new here relative to the closest prior work. Zou et al. (2025) share our central design choice, a request-independent blank image as the control, and reach it independently; we state the delimitation here rather than after the related-work section so a reader can locate the contribution immediately. The differences are not framing: they are the benign-cost accounting, the matched harmful counterpart, the property and placebo controls, and the open-checkpoint and serving-route arms. § 5 gives the full comparison, including where our sign result and theirs disagree and why we read them as two signs of one effect.
More in this topic

Refusal Behavior in Language Models

Refusal behavior in language models concerns when systems decline requests, how they detect risk, and how to balance safety with helpfulness.

See all 148 papers

Explore similar work

CardsList

  1. Visual Grounding Safety in Vision-Language Models

    Oct 5, 2026Erfan Shayegani, Kundan Krishna, Yue Dong +3VLM EvaluationVLM Safety Alignment

    University of California, Riverside · Work done during internship at Apple. · Apple

  2. From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models

    Sep 2, 2026Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8Abstention in VLMsVision-Language Models

    Oracle AI

  3. When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

    Oct 27, 2026Ananth K Suresh, Aveen Dayal, Gouthaman KV +2Refusal Mechanisms in Language ModelsAbstention in VLMs

    Indian Institute of Technology Hyderabad