"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

Naen Xu; Jiayi Sheng; Changjiang Li; Chunyi Zhou; Yuyuan Li; Tianyu Du; Jun Wang; Zhihui Fu

cs.CLApr 7, 2026

"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

Authors: Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu,

Organizations: Zhejiang University · Beihang University · Palo Alto Networks · Hangzhou Dianzi University ·

Abstract

Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative meaning simultaneously. Although Vision-Language Models (VLMs) are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied due to a scarcity of rigorous benchmarks. To address this, we first propose a multimodal pun generation pipeline. We then introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors. Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors. Moreover, we propose both prompt-level and model-level strategies to enhance pun comprehension, with an average improvement of 16.5% in F1 scores. Our findings provide valuable insights for developing future VLMs that master the subtleties of human-like humor via cross-modal reasoning.

Figures & tables

Figure 1: The recognition of multimodal pun examples from MultiPun . (a) A pun relying on phonetic similarity (“pear” and “pair”). (b) A pun based on word polysemy (double meaning of “fan”). (c) A negative sample illustrating a false positive case, where the model mistakenly interprets a non-pun as pun.

Figure 2: Overview of the MultiPun construction pipeline. Our pipeline generates both pun and non-pun samples.

Figure 3: Examples of adversarial negative samples.

Type	Model	Task	Homophonic Pun						Homographic Pun
Type	Model	Task	TPR $\uparrow$	$\Delta$ TPR $\downarrow$	TNR $\uparrow$	$\Delta$ TNR $\downarrow$	F1 $\uparrow$	$\kappa$ $\uparrow$	TPR $\uparrow$	$\Delta$ TPR $\downarrow$	TNR $\uparrow$	$\Delta$ TNR $\downarrow$	F1 $\uparrow$	$\kappa$ $\uparrow$
Closed-Source VLMs	GPT-5.1	Detection	0.933	-0.026	0.379	+0.198	0.588	0.241	0.956	-0.036	0.243	+0.201	0.551	0.146
		Localization	0.887	-0.046	0.768	+0.072	0.754	0.601	0.876	-0.108	0.695	+0.141	0.705	0.508
		Explanation	0.794	-0.062	0.910	+0.059	0.804	0.708	0.757	-0.143	0.878	+0.060	0.757	0.637
	GPT-4o	Detection	0.933	0.000	0.332	+0.144	0.571	0.202	0.956	-0.004	0.211	+0.122	0.541	0.121
		Localization	0.923	-0.015	0.582	+0.088	0.669	0.425	0.888	-0.028	0.480	+0.120	0.607	0.299

Table 1: Results of pun recognition in detection, localization, and explanation tasks. Metrics (TPR, TNR, F1,

\kappa

) are evaluated under the biased-to-pun prompt.

\Delta

measures variations when prompt bias shifts from pun to non-pun. Darker colors indicate better performance. The best results (smallest variations or highest scores) are bolded .

Model	Homophonic Pun				Homographic Pun
	Localization		Explanation		Localization		Explanation
	$w_{p}$	$w_{a}$	$w_{p}$	$w_{a}$	$w_{p}$	$w_{a}$	$w_{p}$	$w_{a}$
Closed-Source VLMs
GPT-5.1	98.8	87.8	100.0	89.0	97.7	97.7	97.9	97.9
GPT-4o	96.1	84.9	92.6	75.5	97.3	97.3	97.3	97.3
Gemini-3-Pro	97.4	86.8	97.9	88.8	98.8	98.8	98.8	98.8

Table 2: Pun component verification for pun localization and explanation. We represent the average mention ratio of the pun words

w_{p}

and alternative words

w_{a}

Figure 4: Pairwise comparison for pun explanations.

Model	Method	Homophonic Pun			Homographic Pun
Model	Method	TPR $\uparrow$	TNR $\uparrow$	F1 $\uparrow$	TPR $\uparrow$	TNR $\uparrow$	F1 $\uparrow$
GPT-5.1	Vanilla	0.794	0.910	0.804	0.757	0.878	0.757
GPT-5.1	Pun-CoT	0.840	0.915	0.836	0.813	0.894	0.803
GPT-4o	Vanilla	0.840	0.786	0.741	0.873	0.659	0.683
GPT-4o	Pun-CoT	0.876	0.835	0.794	0.888	0.727	0.730
Gemini-3 Pro	Vanilla	0.969	0.686	0.746	0.980	0.625	0.718

Table 3: Comparison of pun recognition with and without Pun-CoT across VLMs under the explanation task.

	Model	Method	TPR $\uparrow$	$\Delta$ TPR $\downarrow$	TNR $\uparrow$	$\Delta$ TNR $\downarrow$	F1 $\uparrow$
Homophonic Pun	Qwen3-VL 8B-Instruct	Vanilla	0.418	-0.268	0.881	+0.111	0.505
	Qwen3-VL 8B-Instruct	Pun-Tuning	0.577	-0.155	0.938	+0.098	0.679
	Qwen3-VL 30B-Instruct	Vanilla	0.943	-0.273	0.209	+0.469	0.535
	Qwen3-VL 30B-Instruct	Pun-Tuning	0.732	-0.062	0.948	+0.196	0.798
	LLaVA-v1.6 Vicuna-13B	Vanilla	0.031	-0.015	0.972	+0.023	0.057
	LLaVA-v1.6 Vicuna-13B	Pun-Tuning	0.495	-0.103	0.974	+0.098	0.640

Table 4: Comparison of pun recognition with and without Pun-Tuning on VLMs under the explanation task.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Homophonic	Homographic	Total
Positive Samples	194	251	445
Negative Samples:
Explicative Substitution (ES)	194	251	445
Random Substitution (RS)	194	251	445
Total Negatives	388	502	890
Total (Pos + Neg)	582	753	1335

Appendix

Table 5: Dataset statistics for MultiPun .

Category	Lexname	Description
Visual	noun.animal	Animals and distinct biological organisms
	noun.artifact	Man-made objects, tools, and instruments
	noun.body	Body parts (used restrictively)
	noun.food	Edible substances and dishes
	noun.object	Natural inanimate objects (e.g., stones)
	noun.plant	Vegetation and botanical entities

Appendix

Table 6: Classification of WordNet Lexnames into Visual Anchor Categories (retained) and Abstract Categories (filtered).

Model	API Version
OpenAI Family
GPT-5.1	gpt-5.1
GPT-4o	gpt-4o-2024-08-06
Google Gemini Family
Gemini-3-Pro	gemini-3-pro-preview
Anthropic Family

Appendix

Table 7: Closed-source VLM configurations.

Model	Checkpoint	Type
Meta Llama-4 Family
Llama-4-Scout-17B	meta-llama/Llama-4-Scout-17B-16E-Instruct	Instruct
Alibaba Qwen3-VL Family
Qwen3-VL-8B-Instruct	Qwen/Qwen3-VL-8B-Instruct	Instruct
Qwen3-VL-30B-A3B-Instruct	Qwen/Qwen3-VL-30B-A3B-Instruct	Instruct
Qwen3-VL-8B-Thinking	Qwen/Qwen3-VL-8B-Thinking	Reasoning

Appendix

Table 8: Open-source VLM configurations.

Figure 5: Pairwise comparison for pun explanations before and after Pun-Tuning.

Category	Pun Type	Train	Test	Total
Positive	Homophonic	97	97	194
Positive	Homographic	125	126	251
Negative	Homophonic	194	194	388
Negative	Homographic	250	252	502
Total		666	669	1335

Appendix

Table 9: Dataset splits for Pun-Tuning experiments.

Pun Keyword Hallucination
Homographic Pun	Random Substitution (RS)
I’m your biggest fan . claude-sonnet-4-5 $\bm{w_{p}}$ : fan $\bm{S_{p}}$ : A mechanical device with rotating blades that circulates air $\bm{w_{a}}$ : fan $\bm{S_{a}}$ : An enthusiastic admirer or supporter Explanation : The word ’fan’ has two meanings: (1) a mechanical device that circulates air, which is visually depicted in the image as an anthropomorphized electric fan, and (2) an enthusiastic supporter or admirer, as indicated by the foam finger and the phrase ’biggest fan’ which is an idiom meaning someone’s greatest supporter.	I’m your biggest lamp in the room. claude-sonnet-4-5 $\bm{w_{p}}$ : lamp $\bm{S_{p}}$ : A lighting fixture consisting of a shade and base $\bm{w_{a}}$ : fan $\bm{S_{a}}$ : An enthusiastic supporter or admirer Explanation : The word ’lamp’ creates a pun by having two meanings: literally referring to the lamp character in the image, and playing on the phrase ’biggest fan’ where ’lamp’ sounds like it’s substituting for ’fan’ in the context of being a supporter at what appears to be a stadium/arena setting.
Phonetic Hallucination
Homophonic Pun	Random Substitution (RS)
You are my sole mate. Gemini-3-Pro $\bm{w_{p}}$ : sole $\bm{S_{p}}$ : The bottom part of a shoe or foot, which is visually depicted as two shoe bottoms hugging. $\bm{w_{a}}$ : soul $\bm{S_{a}}$ : The spiritual or immaterial part of a human being, used in the phrase ’soul mate’ to mean a perfect romantic partner. Explanation : The humor relies on the phonetic similarity between ’sole’ (the bottom of a shoe, depicted in the image) and ’soul’ (as in ’soul mate’, a romantic partner). The image shows two shoe soles hugging, literalizing the pun.	You are my banana mate. Gemini-3-Pro $\bm{w_{p}}$ : banana $\bm{S_{p}}$ : A long curved fruit which grows in clusters and has soft pulpy flesh and yellow skin when ripe. $\bm{w_{a}}$ : soul $\bm{S_{a}}$ : The spiritual or immaterial part of a human being or animal, regarded as immortal. Explanation : The pun relies on the phonetic similarity between the word ’banana’ and the word ’soul’. The phrase ’banana mate’ sounds like ’soulmate’, which is a common term for a perfect romantic partner or close friend. The image depicts two bananas hugging affectionately, reinforcing the ’mate’ aspect, while the word ’banana’ directly describes the visual subjects.
Semantic Hallucination

Appendix

Table 10: Qualitative examples of false positive errors on negative samples. We illustrate four distinct hallucination patterns where VLMs fail to reject non-pun distractors: (1) Pun Keyword Hallucination (Lexical Level), where models ignore visible text to restore idiomatic keywords; (2) Phonetic Hallucination (Phonological Level), where models fabricate sound similarities between distinct terms; (3) Semantic Hallucination (Semantic Level), where models force polysemous meanings onto unambiguous synonyms; and (4) Visual Object Hallucination (Visual Level), where models misidentify visual objects to align with textual cues.

Detection Failure
Good knight .	Homophonic Pun $\bm{w_{p}}$ : knight $\bm{S_{p}}$ : originally a person of noble birth trained to arms and chivalry; today in Great Britain a person honored by the sovereign for personal merit $\bm{w_{a}}$ : night $\bm{S_{a}}$ : the time after sunset and before sunrise while it is dark outside Explanation : The visual shows a chess knight piece wearing a nightcap and lying in bed, symbolizing bedtime. The caption, "Good knight," creates a pun by substituting "knight" (the chess piece) for "night" in the common bedtime farewell, "Good night." The humor arises from the homophonic wordplay, as "knight" and "night" are pronounced the same but have different spellings and meanings.	Qwen3-VL-8B-Instruct is pun: False
Pun Words Error
Super bowl .	Homophonic Pun $\bm{w_{p}}$ : bowl $\bm{S_{p}}$ : a round vessel that is open at the top; used chiefly for holding food or liquids $\bm{w_{a}}$ : bowl $\bm{S_{a}}$ : a wooden ball (with flattened sides so that it rolls on a curved course) used in the game of lawn bowling Explanation : The visual features a soup bowl (dish) depicted as engaging in the sport of bowling, humorously anthropomorphized to throw a ball at pins. The caption, "Super bowl," creates a pun through homography, as it plays on the identical spelling of "bowl" (a dish) and "bowl" (a sporting action). The humor arises from the unexpected juxtaposition of a kitchen item with sports terminology.	gemini-3-pro $\bm{w_{p}}$ : super $\bm{S_{p}}$ : An adjective describing the bowl of soup depicted in the image, specifically highlighting its identity as ’soup’. $\bm{w_{a}}$ : soup-er $\bm{S_{a}}$ : An adjective meaning excellent or of high grade; also part of the proper noun ’Super Bowl’ referring to the NFL championship game. Explanation : The humor relies on the phonetic similarity between the phrase ’Super bowl’ (referring to the major American football championship event) and the phrase ’soup-er bowl’ (describing a bowl of soup that is excellent at bowling). The image literally depicts a bowl of soup bowling, creating a visual pun on the homophone.
Alternative Word Error
What a waist .	Homophonic Pun $\bm{w_{p}}$ : waist $\bm{S_{p}}$ : the narrowing of the body between the ribs and hips $\bm{w_{a}}$ : waste $\bm{S_{a}}$ : any materials unused and rejected as worthless or unwanted Explanation : The visual shows a garbage bin (representing "waste") wearing a belt around its middle, resembling a human waist. The caption, "What a waist," creates a homophonic pun by playing on the identical pronunciation of "waist" (a body part) and "waste" (garbage). The humor arises from the interplay between the visual of the trash bin and the double meaning of the caption.	Qwen3-VL-30B-A3B-Thinking $\bm{w_{p}}$ : waist $\bm{S_{p}}$ : The belt wrapped around the trash bin, mimicking a human waistline $\bm{w_{a}}$ : waist $\bm{S_{a}}$ : The narrow part of the human torso between the ribs and hips, typically used in compliments about body shape Explanation : The word ’waist’ in the caption refers to both the belt around the trash bin (visual sense) and the human body part (textual context), creating humor through the unexpected application of a human compliment to an inanimate object.
Cross-modal Integration Error

Appendix

Table 11: Qualitative examples of failures on positive samples (genuine puns). We identify four failure modes: (1) Detection Failure, where the pun is missed entirely; (2) Pun Words Error, where the model focuses on the wrong lexical trigger; (3) Alternative Word Error, where the model fails to retrieve the hidden meaning (

w_{a}

) of the anchor word; and (4) Cross-modal Integration Error, where the model confuses the linguistic mechanism (e.g., treating homophony as polysemy).

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

PunGraph: Retrieval-Enhanced Phonetic-Semantic Graph Reasoning for Pun Understanding
Sep 15, 2026Yuchen Su, Zijian Huang, Yaotian Shi +6Figurative Language Understanding Knowledge Augmentation for Language Models
School of Computer Science, University of Auckland, New Zealand · School of Computer and Information Technology, Shanxi University · School of Electronic Engineering and Computer Science, Queen Mary University of London
CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension
Aug 24, 2026Abhilash Nandy, Rahul Seetharaman, Aman Bansal +5LLM Reasoning with Graphs Multimodal Reasoning
Indian Institute of Technology Kharagpur, India · Microsoft Research India · LinkedIn, USA +4
PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models
Aug 10, 2026Zhanna Mukhametsharip, Vera Demberg, Varsha SureshVLM Evaluation Pragmatic Reasoning in Language Models
Saarland University, Germany · Max Planck Institute for Informatics, Germany

Sep 15, 2026J/K move · Enter open · S save

PunGraph: Retrieval-Enhanced Phonetic-Semantic Graph Reasoning for Pun Understanding

Yuchen Su, Zijian Huang, Yaotian Shi, Shaoxin Zhong+5

School of Computer Science, University of Auckland, New Zealand · School of Computer and Information Technology, Shanxi University · School of Electronic Engineering and Computer Science, Queen Mary University of London

Figurative Language Understanding Knowledge Augmentation for Language Models LLM Reasoning with Graphs

Open paper AskPDF arXiv

Puns are a challenging form of figurative language that exploit phonetic similarity and semantic ambiguity to convey multiple meanings. Although large language models (LLMs) demonstrate strong language understanding capabilities, they still struggle with pun reasoning due to limited phonetic modeling and uncontrolled end-to-end generation. We propose PunGraph, a retrieval-enhanced knowledge graph framework for pun understanding. PunGraph constructs a phonetic-semantic lexical graph using the Unisyn phonetic dictionary, IPA and G2P representations, and WordNet definitions, and retrieves candidate words or senses to constrain LLM reasoning within a structured candidate space. We further introduce WebPun, a new large-scale dataset containing 5,730 annotated heterographic and homographic puns. Experiments on SemEval-2017 and WebPun show that PunGraph consistently improves the performance of small-scale LLMs and achieves competitive results against strong proprietary models. Further analysis shows that retrieval-guided phonetic and semantic constraints effectively reduce common reasoning errors in pun interpretation, highlighting the benefits of integrating structured knowledge with LLMs. We release our code and dataset at https://github.com/ysu132/PunGraph.

Abstract

VLM Evaluation

Explore similar work

PunGraph: Retrieval-Enhanced Phonetic-Semantic Graph Reasoning for Pun Understanding

CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension

PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models