Visual Persuasion: What Influences Decisions of Vision-Language Models?

Manuel Cherep; Pranav M R; Pattie Maes; Nikhil Singh

cs.CVFeb 17, 2026

Visual Persuasion: What Influences Decisions of Vision-Language Models?

Authors: Manuel Cherep, Pranav M R, Pattie Maes, Nikhil Singh

Organizations: Media Lab, Massachusetts Institute of Technology, Cambridge, USA · BITS Pilani, Goa, India · Department of Computer Science, Dartmouth College, Hanover, USA

Abstract

The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.

Figures & tables

Figure 2: Estimated marginal mean probability of choice by task (columns)

\times

optimization method (rows) and optimization stage (X-axis; original image, zero-shot modified, and final after optimization). Results are averaged across all VLMs (50% base rate).

Figure 3: Estimated marginal mean probability of choice for final optimized images produced by different optimization methods in head-to-head comparisons. Results are averaged across all VLMs (50% base rate) with error bars showing 95% confidence intervals. See Appendix F for CVPO results with

k=1

VLM	VTG	VFD	CVPO
	Strategy (P(Choice) with $\Delta$ vs. best)
Qwen-3-VL 235B	0.131 ( $\Delta$ = $-0.640^{****}$ )	0.601 ( $\Delta$ = $-0.170^{****}$ )	0.771
Llama 4 Maverick	0.138 ( $\Delta$ = $-0.627^{****}$ )	0.586 ( $\Delta$ = $-0.179^{****}$ )	0.766
GPT-5 Mini	0.190 ( $\Delta$ = $-0.576^{****}$ )	0.561 ( $\Delta$ = $-0.205^{****}$ )	0.766
Gemini 3 Flash	0.140 ( $\Delta$ = $-0.621^{****}$ )	0.604 ( $\Delta$ = $-0.157^{****}$ )	0.761
GPT-4o	0.179 ( $\Delta$ = $-0.570^{****}$ )	0.566 ( $\Delta$ = $-0.183^{****}$ )	0.749

Table 1: Model-wise choice probabilities by strategy (main value) with contrasts vs. the row-best strategy in parentheses. Asterisks indicate Benjamini-Hochberg adjusted significance (

{}^{****}=p<.0001

{}^{***}=p<.001

{}^{**}=p<.01

{}^{*}=p<.05

Task	VTG	VFD	CVPO
	Strategy (P(Choice) with $\Delta$ vs. best)
Hotels	0.532	0.495 ( $\Delta$ = $-0.038$ )	0.472 ( $\Delta$ = $-0.060$ )
Houses	0.490 ( $\Delta$ = $-0.034$ )	0.487 ( $\Delta$ = $-0.037$ )	0.524
People	0.469 ( $\Delta$ = $-0.087$ )	0.487 ( $\Delta$ = $-0.069$ )	0.556
Products	0.538	0.460 ( $\Delta$ = $-0.078$ )	0.511 ( $\Delta$ = $-0.027$ )

Table 2: Human head-to-head choice probabilities by task and strategy. Main value is the estimated marginal mean probability; parentheses show

\Delta

vs. the best strategy within each task.

Strategy	Original	Zero-shot	Final
	Status (P(Choice) with $\Delta$ vs. best)
VTG	0.301 ( $\Delta$ = $-0.320^{****}$ )	0.622	0.615 ( $\Delta$ = $-0.006$ )
VFD	0.299 ( $\Delta$ = $-0.384^{****}$ )	0.556 ( $\Delta$ = $-0.128^{**}$ )	0.684
CVPO	0.289 ( $\Delta$ = $-0.374^{****}$ )	0.588 ( $\Delta$ = $-0.075$ )	0.663

Table 3: Human choice probabilities by strategy

\times

status. Main value is estimated marginal mean probability; parentheses show

\Delta

vs. best status within each strategy. All Benjamini-Hochberg adjusted (

{}^{****}=p<.0001

{}^{***}=p<.001

{}^{**}=p<.01

{}^{*}=p<.05

Figure 4: Effect of image normalization for

\kappa

passes on est. probability of choosing the original vs. final variants (50% base rate).

Figure 5: Effect of image normalization on human choices, compared with original vs. final trials. After 3 passes, the probability of choosing the final optimized image decreases.

Figure 6: Results testing whether distilling the discovered visual edits into zero-shot editing instructions can match the optimization that recovers them (50% base rate).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: CVPO optimization examples at different steps for each of our four datasets.

Figure 8: Results testing whether distilling the discovered visual edits into zero-shot editing instructions with unseen samples can match the optimization that recovers them (50% base rate). See Figure 6 for the in-sample results.

Method	P(Choose) [95% CIs]
Original	35.9 [33.84, 38.0] ( $p<.0001$ )
Zero-shot	55.9 [53.49, 58.3] ( $p\approx 0.1$ )
Distilled	59.2 [56.90, 61.5]

Appendix

Table 4: Results showing how out-of-sample distillation disentangling by product category can match the optimization that recovers them.

Figure 9: Estimated marginal mean probability of choice for final optimized images produced by different optimization methods in head-to-head comparisons with CVPO using only one candidate (

k=1

). Results are averaged across all VLMs (50% base rate) with error bars showing 95% confidence intervals. See Figure 3 for the results with CVPO’s

k=3

Figure 10: Similarity/distance metrics as a function of number of mitigation steps. We compare images between those in a choice set (pair), and also the mitigation-processed versions against the originals split by whether the processed image is an optimized image or an original itself (in which case it may deviate from a perfect match). Error bars show 95% confidence intervals.

Figure 11: Head-to-head experiment results disaggregated by model.

Figure 12: Head-to-head experiment results disaggregated by task.

Figure 13: Hotel experiment results disaggregated by model and strategy.

Figure 14: Hotel experiment results disaggregated by class and strategy.

Figure 15: Houses experiment results disaggregated by model and strategy.

Figure 16: People experiment results disaggregated by model and strategy.

Figure 17: Products experiment results disaggregated by model and strategy.

Figure 18: Products experiment results disaggregated by class and strategy.

Figure 19: Human mitigation experiment results disaggregated by task (dataset).

VLM	Strategy	Original	Zero-shot	Final
Claude Haiku 4.5	VTG	0.309 ( $\Delta$ = $-0.375^{****}$ )	0.570 ( $\Delta$ = $-0.114^{****}$ )	0.684
Claude Sonnet 4.5	VTG	0.316 ( $\Delta$ = $-0.345^{****}$ )	0.535 ( $\Delta$ = $-0.126^{****}$ )	0.661
Gemini 3 Flash	VTG	0.298 ( $\Delta$ = $-0.378^{****}$ )	0.549 ( $\Delta$ = $-0.126^{****}$ )	0.675
Gemini 3 Pro	VTG	0.291 ( $\Delta$ = $-0.385^{****}$ )	0.540 ( $\Delta$ = $-0.137^{****}$ )	0.676
GPT-4o	VTG	0.271 ( $\Delta$ = $-0.400^{****}$ )	0.594 ( $\Delta$ = $-0.077^{****}$ )	0.671
GPT-5 Mini	VTG	0.300 ( $\Delta$ = $-0.368^{****}$ )	0.570 ( $\Delta$ = $-0.098^{****}$ )	0.668

Appendix

Table 5: Model-wise choice probabilities by strategy and type, pooled across tasks. Main value is the estimated marginal mean probability; parentheses show

\Delta

vs. the best type within each strategy for that model.

Task	Strategy	Original	Zero-shot	Final
Hotels	VTG	0.288 ( $\Delta$ = $-0.372^{****}$ )	0.635 ( $\Delta$ = $-0.024$ )	0.660
Hotels	VFD	0.407 ( $\Delta$ = $-0.249^{*}$ )	0.479 ( $\Delta$ = $-0.176$ )	0.655
Hotels	CVPO	0.339 ( $\Delta$ = $-0.340^{****}$ )	0.500 ( $\Delta$ = $-0.179$ )	0.679
Houses	VTG	0.339 ( $\Delta$ = $-0.302^{**}$ )	0.553 ( $\Delta$ = $-0.087$ )	0.641
Houses	VFD	0.299 ( $\Delta$ = $-0.453^{****}$ )	0.457 ( $\Delta$ = $-0.296^{****}$ )	0.752
Houses	CVPO	0.250 ( $\Delta$ = $-0.398^{****}$ )	0.648	0.648 ( $\Delta$ = $-0.000$ )

Appendix

Table 6: Human choice probabilities by task, strategy, and status. Main value is the estimated marginal mean probability; parentheses show

\Delta

vs. the best status within each task-strategy. Asterisks indicate Benjamini-Hochberg adjusted significance (

{}^{****}=p<.0001

{}^{***}=p<.001

{}^{**}=p<.01

{}^{*}=p<.05

Dependent Var.:	chosen_flag
Constant	0.2879*** (0.0351)
statusZero-shot	0.3475*** (0.0639)
statusFinal	0.3717*** (0.0764)
strategyVFD	0.1186* (0.0552)
strategyCVPO	0.0511 (0.0495)
taskhouses	0.0508 (0.0608)

Appendix

Table 7: Regression table for human choices from experiment 3.

Dependent Var.:	chosen_flag
Constant	0.5325*** (0.0324)
strategyVFD	-0.0377 (0.0537)
strategyCVPO	-0.0602 (0.0539)
taskhouses	-0.0425 (0.0476)
taskpeople	-0.0632 (0.0467)
taskproducts	0.0053 (0.0620)

Appendix

Table 8: Regression table for human choices from experiment 4.

	model_choice_miti..
Dependent Var.:	chosen_flag
Constant	0.2798*** (0.0327)
statusFinal	0.4404*** (0.0654)
taskhouses	-0.0417 (0.0520)
taskpeople	-0.1056* (0.0484)
taskproducts	0.0595 (0.0472)

Appendix

Table 9: Regression table for human choices with 3-pass mitigation results included.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors
Nov 21, 2025Gyuwon Park, Hyounghun KimVLM Evaluation VLM Reasoning
Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH
Seeing Red, Thinking Bad: Color Bias in Vision Language Models
Aug 14, 2026Kohsuke Ide, Ryousuke Yamada, Yoshihiro Fukuhara +2VLM Robustness VLM Interpretability
National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan · University of Tsukuba, Tsukuba, Japan · University of Technology Nuremberg, Nuremberg, Germany +1
Tinted Frames: Question Framing Blinds Vision-Language Models
Mar 19, 2026Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher +2Vision-Language Models VLM Robustness
University of British Columbia · Vector Institute for AI · University of California, Berkeley

Nov 21, 2025J/K move · Enter open · S save

Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors

Gyuwon Park, Hyounghun Kim

Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH

VLM Evaluation VLM Reasoning Vision-Language Alignment

Open paper AskPDF arXiv

Visual persuasion uses images to shape cognition, emotion, and behavior, with its effects depending on both visual attributes and semantic context. Despite recent progress, it remains unclear whether Vision-Language Models (VLMs) understand visual persuasiveness. This motivates us to ask: can VLMs assess whether an image persuasively supports an intended message, which visual factors shape this judgment, and do they align with human judgments? Through empirical analyses on image-message pairs where human raters consistently agree on the persuasiveness judgment, we show that VLMs exhibit a recall-oriented bias: they over-predict images as persuasive while achieving high recall. We introduce Visual Persuasive Factors (VPFs), a taxonomy informed by cognitive psychology for quantifying visual cues that shape persuasive judgments. Our factor-level analysis reveals that VPFs distinguish human persuasiveness judgments, whereas VLMs only partially reproduce these patterns, often generating false positives by treating persuasion-relevant cues as sufficient evidence. Building on this insight, we evaluate VPF-guided interventions and find that properly framed VPF knowledge can improve performance, but merely specifying visual cues or adding step-by-step reasoning is insufficient. By analyzing model rationales at the level of functional reasoning steps, we further identify a central bottleneck in connecting object identification to semantic message alignment.

Abstract

VLM Robustness

Explore similar work

Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors

Seeing Red, Thinking Bad: Color Bias in Vision Language Models

Tinted Frames: Question Framing Blinds Vision-Language Models