Visual Persuasion: What Influences Decisions of Vision-Language Models?
Authors: Manuel Cherep, Pranav M R, Pattie Maes, Nikhil Singh
Organizations: Media Lab, Massachusetts Institute of Technology, Cambridge, USA · BITS Pilani, Goa, India · Department of Computer Science, Dartmouth College, Hanover, USA
The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.
Figures & tables
Figure 2: Estimated marginal mean probability of choice by task (columns) × optimization method (rows) and optimization stage (X-axis; original image, zero-shot modified, and final after optimization). Results are averaged across all VLMs (50% base rate).
Figure 3: Estimated marginal mean probability of choice for final optimized images produced by different optimization methods in head-to-head comparisons. Results are averaged across all VLMs (50% base rate) with error bars showing 95% confidence intervals. See Appendix F for CVPO results with k=1 .
Strategy (P(Choice) with Δ vs. best)
VLM
VTG
VFD
CVPO
Qwen-3-VL 235B
0.131 ( Δ = −0.640∗∗∗∗ )
0.601 ( Δ = −0.170∗∗∗∗ )
0.771
Llama 4 Maverick
0.138 ( Δ = −0.627∗∗∗∗ )
0.586 ( Δ = −0.179∗∗∗∗ )
0.766
GPT-5 Mini
0.190 ( Δ = −0.576∗∗∗∗ )
0.561 ( Δ = −0.205∗∗∗∗ )
0.766
Gemini 3 Flash
0.140 ( Δ = −0.621∗∗∗∗ )
0.604 ( Δ = −0.157∗∗∗∗ )
0.761
GPT-4o
0.179 ( Δ = −0.570∗∗∗∗ )
0.566 ( Δ = −0.183∗∗∗∗ )
0.749
Table 1: Model-wise choice probabilities by strategy (main value) with contrasts vs. the row-best strategy in parentheses. Asterisks indicate Benjamini-Hochberg adjusted significance ( ∗∗∗∗=p<.0001 , ∗∗∗=p<.001 , ∗∗=p<.01 , ∗=p<.05 ).
Strategy (P(Choice) with Δ vs. best)
Task
VTG
VFD
CVPO
Hotels
0.532
0.495 ( Δ = −0.038 )
0.472 ( Δ = −0.060 )
Houses
0.490 ( Δ = −0.034 )
0.487 ( Δ = −0.037 )
0.524
People
0.469 ( Δ = −0.087 )
0.487 ( Δ = −0.069 )
0.556
Products
0.538
0.460 ( Δ = −0.078 )
0.511 ( Δ = −0.027 )
Table 2: Human head-to-head choice probabilities by task and strategy. Main value is the estimated marginal mean probability; parentheses show Δ vs. the best strategy within each task.
Status (P(Choice) with Δ vs. best)
Strategy
Original
Zero-shot
Final
VTG
0.301 ( Δ = −0.320∗∗∗∗ )
0.622
0.615 ( Δ = −0.006 )
VFD
0.299 ( Δ = −0.384∗∗∗∗ )
0.556 ( Δ = −0.128∗∗ )
0.684
CVPO
0.289 ( Δ = −0.374∗∗∗∗ )
0.588 ( Δ = −0.075 )
0.663
Table 3: Human choice probabilities by strategy × status. Main value is estimated marginal mean probability; parentheses show Δ vs. best status within each strategy. All Benjamini-Hochberg adjusted ( ∗∗∗∗=p<.0001 , ∗∗∗=p<.001 , ∗∗=p<.01 , ∗=p<.05 ).
Figure 4: Effect of image normalization for κ passes on est. probability of choosing the original vs. final variants (50% base rate).
Figure 5: Effect of image normalization on human choices, compared with original vs. final trials. After 3 passes, the probability of choosing the final optimized image decreases.
Figure 6: Results testing whether distilling the discovered visual edits into zero-shot editing instructions can match the optimization that recovers them (50% base rate).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: CVPO optimization examples at different steps for each of our four datasets.
Figure 8: Results testing whether distilling the discovered visual edits into zero-shot editing instructions with unseen samples can match the optimization that recovers them (50% base rate). See Figure 6 for the in-sample results.
Method
P(Choose) [95% CIs]
Original
35.9 [33.84, 38.0] ( p<.0001 )
Zero-shot
55.9 [53.49, 58.3] ( p≈0.1 )
Distilled
59.2 [56.90, 61.5]
Appendix
Table 4: Results showing how out-of-sample distillation disentangling by product category can match the optimization that recovers them.
Figure 9: Estimated marginal mean probability of choice for final optimized images produced by different optimization methods in head-to-head comparisons with CVPO using only one candidate ( k=1 ). Results are averaged across all VLMs (50% base rate) with error bars showing 95% confidence intervals. See Figure 3 for the results with CVPO’s k=3 .
Figure 10: Similarity/distance metrics as a function of number of mitigation steps. We compare images between those in a choice set (pair), and also the mitigation-processed versions against the originals split by whether the processed image is an optimized image or an original itself (in which case it may deviate from a perfect match). Error bars show 95% confidence intervals.
Figure 11: Head-to-head experiment results disaggregated by model.
Figure 12: Head-to-head experiment results disaggregated by task.
Figure 13: Hotel experiment results disaggregated by model and strategy.
Figure 14: Hotel experiment results disaggregated by class and strategy.
Figure 15: Houses experiment results disaggregated by model and strategy.
Figure 16: People experiment results disaggregated by model and strategy.
Figure 17: Products experiment results disaggregated by model and strategy.
Figure 18: Products experiment results disaggregated by class and strategy.
Figure 19: Human mitigation experiment results disaggregated by task (dataset).
VLM
Strategy
Original
Zero-shot
Final
Claude Haiku 4.5
VTG
0.309 ( Δ = −0.375∗∗∗∗ )
0.570 ( Δ = −0.114∗∗∗∗ )
0.684
Claude Sonnet 4.5
VTG
0.316 ( Δ = −0.345∗∗∗∗ )
0.535 ( Δ = −0.126∗∗∗∗ )
0.661
Gemini 3 Flash
VTG
0.298 ( Δ = −0.378∗∗∗∗ )
0.549 ( Δ = −0.126∗∗∗∗ )
0.675
Gemini 3 Pro
VTG
0.291 ( Δ = −0.385∗∗∗∗ )
0.540 ( Δ = −0.137∗∗∗∗ )
0.676
GPT-4o
VTG
0.271 ( Δ = −0.400∗∗∗∗ )
0.594 ( Δ = −0.077∗∗∗∗ )
0.671
GPT-5 Mini
VTG
0.300 ( Δ = −0.368∗∗∗∗ )
0.570 ( Δ = −0.098∗∗∗∗ )
0.668
Appendix
Table 5: Model-wise choice probabilities by strategy and type, pooled across tasks. Main value is the estimated marginal mean probability; parentheses show Δ vs. the best type within each strategy for that model.
Task
Strategy
Original
Zero-shot
Final
Hotels
VTG
0.288 ( Δ = −0.372∗∗∗∗ )
0.635 ( Δ = −0.024 )
0.660
Hotels
VFD
0.407 ( Δ = −0.249∗ )
0.479 ( Δ = −0.176 )
0.655
Hotels
CVPO
0.339 ( Δ = −0.340∗∗∗∗ )
0.500 ( Δ = −0.179 )
0.679
Houses
VTG
0.339 ( Δ = −0.302∗∗ )
0.553 ( Δ = −0.087 )
0.641
Houses
VFD
0.299 ( Δ = −0.453∗∗∗∗ )
0.457 ( Δ = −0.296∗∗∗∗ )
0.752
Houses
CVPO
0.250 ( Δ = −0.398∗∗∗∗ )
0.648
0.648 ( Δ = −0.000 )
Appendix
Table 6: Human choice probabilities by task, strategy, and status. Main value is the estimated marginal mean probability; parentheses show Δ vs. the best status within each task-strategy. Asterisks indicate Benjamini-Hochberg adjusted significance ( ∗∗∗∗=p<.0001 , ∗∗∗=p<.001 , ∗∗=p<.01 , ∗=p<.05 ).
Dependent Var.:
chosen_flag
Constant
0.2879*** (0.0351)
statusZero-shot
0.3475*** (0.0639)
statusFinal
0.3717*** (0.0764)
strategyVFD
0.1186* (0.0552)
strategyCVPO
0.0511 (0.0495)
taskhouses
0.0508 (0.0608)
Appendix
Table 7: Regression table for human choices from experiment 3.
Dependent Var.:
chosen_flag
Constant
0.5325*** (0.0324)
strategyVFD
-0.0377 (0.0537)
strategyCVPO
-0.0602 (0.0539)
taskhouses
-0.0425 (0.0476)
taskpeople
-0.0632 (0.0467)
taskproducts
0.0053 (0.0620)
Appendix
Table 8: Regression table for human choices from experiment 4.
model_choice_miti..
Dependent Var.:
chosen_flag
Constant
0.2798*** (0.0327)
statusFinal
0.4404*** (0.0654)
taskhouses
-0.0417 (0.0520)
taskpeople
-0.1056* (0.0484)
taskproducts
0.0595 (0.0472)
Appendix
Table 9: Regression table for human choices with 3-pass mitigation results included.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
Visual persuasion uses images to shape cognition, emotion, and behavior, with its effects depending on both visual attributes and semantic context. Despite recent progress, it remains unclear whether Vision-Language Models (VLMs) understand visual persuasiveness. This motivates us to ask: can VLMs assess whether an image persuasively supports an intended message, which visual factors shape this judgment, and do they align with human judgments? Through empirical analyses on image-message pairs where human raters consistently agree on the persuasiveness judgment, we show that VLMs exhibit a recall-oriented bias: they over-predict images as persuasive while achieving high recall. We introduce Visual Persuasive Factors (VPFs), a taxonomy informed by cognitive psychology for quantifying visual cues that shape persuasive judgments. Our factor-level analysis reveals that VPFs distinguish human persuasiveness judgments, whereas VLMs only partially reproduce these patterns, often generating false positives by treating persuasion-relevant cues as sufficient evidence. Building on this insight, we evaluate VPF-guided interventions and find that properly framed VPF knowledge can improve performance, but merely specifying visual cues or adding step-by-step reasoning is insufficient. By analyzing model rationales at the level of functional reasoning steps, we further identify a central bottleneck in connecting object identification to semantic message alignment.
Gyuwon Park, Hyounghun Kim
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH
Vision language models (VLMs) are increasingly used in industrial decision-making systems, such as recruitment support and recommendation. This motivates careful analysis of how VLMs process visual and textual information. In this work, we study how VLMs interpret text rendered as an image, and investigate the influence of visual styling biases. To this end, we introduce Stealth Visual Prompts, which subtly change visual styling of text, such as color and contrast, while preserving semantic content. Using these prompts, we systematically control the visual styling of words in text and measure their impact on the analysis performed by VLMs. We further analyze how such visual perturbations affect the latent representations of the vision encoder. From our experiments, we observed that coloring positive words in green consistently shifts sentiment predictions toward a positive direction. As a result, VLMs often fail to properly account for negative words present in the text. Our analysis suggests that this behavior is correlated with changes in the latent representations of the vision encoder induced by color variations. In addition, we show that reducing text--background contrast increases reliance on visually salient cues and leads to more incorrect Visual Question Answering (VQA) outputs. These results suggest that the visual styling of rendered text can guide VLMs' interpretation in ways that diverge from human semantic understanding. Project page: https://github.com/KohsukeIde/color-bias-vlm
National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan · University of Tsukuba, Tsukuba, Japan · University of Technology Nuremberg, Nuremberg, Germany +1
Vision-Language Models (VLMs) have been shown to be blind, often underutilizing their visual inputs even on tasks that require visual reasoning. In this work, we demonstrate that VLMs are selectively blind. They modulate the amount of attention applied to visual inputs based on linguistic framing even when alternative framings demand identical visual reasoning. Using visual attention as a probe, we quantify how framing alters both the amount and distribution of attention over the image. Constrained framings, such as multiple choice and yes/no, induce substantially lower attention to image context compared to open-ended, reduce focus on task-relevant regions, and shift attention towards uninformative tokens. We further demonstrate that this attention misallocation is the principal cause of degraded accuracy and cross-framing inconsistency. Building on this mechanistic insight, we introduce a lightweight prompt-tuning method using learnable tokens that encourages the robust, visually grounded attention patterns observed in open-ended settings, improving visual grounding and improving performance across framings.
Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher +2
University of British Columbia · Vector Institute for AI · University of California, Berkeley