From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme

Xueyan Li; Yingyi Xue; Mengjie Jiang; Qingzi Zhu; Yazhe Niu

cs.LGDec 31, 2025

From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme

Authors: Xueyan Li, Yingyi Xue, Mengjie Jiang, Qingzi Zhu, Yazhe Niu

Organizations: Shanghai Artificial Intelligence Laboratory · School of Software Engineering Xi’an Jiaotong University · Columbia Engineering Columbia University · The Chinese University of Hong Kong MMLab

Abstract

Generating humorous memes is a challenging multimodal task that moves beyond direct image-to-caption supervision. It requires a nuanced reasoning over visual content, contextual cues, and subjective humor. To bridge this gap between visual perception and humorous punchline creation, we propose HUMOR}, a novel framework that guides VLMs through hierarchical reasoning and aligns them with group-wise human preferences. First, HUMOR employs a hierarchical, multi-path Chain-of-Thought (CoT): the model begins by identifying a template-level intent, then explores diverse reasoning paths under different contexts, and finally anchors onto a high-quality, context-specific path. This CoT supervision, which traces back from ground-truth captions, enhances reasoning diversity. We further analyze that this multi-path exploration with anchoring maintains a high expected humor quality, under the practical condition that high-quality paths retain significant probability mass. Second, to capture subjective humor, we train a pairwise reward model that operates within groups of memes sharing the same template. Following established theory, this approach ensures a consistent and robust proxy for human preference, even with subjective and noisy labels. The reward model then enables a group-wise reinforcement learning optimization, guaranteeing providing a theoretical guarantee for monotonic improvement within the trust region. Extensive experiments show that HUMOR empowers various VLMs with superior reasoning diversity, more reliable preference alignment, and higher overall meme quality. Beyond memes, our work presents a general training paradigm for open-ended, human-aligned multimodal generation, where success is guided by comparative judgment within coherent output group.

Figures & tables

Figure 1: Given a template image, HUMOR first performs hierarchical reasoning with multi-path CoT: a template-level stage infers latent intent, while a context-level stage explores multiple visually grounded reasoning paths. One high-quality path is anchored by tracing back from ground-truth captions, promoting diversity while preserving a conditional humor lower bound. A reward model then compares memes only within groups sharing the same template, preserving rank consistency and providing a proxy for human preference. This reward further enables stable group-wise RL with KL-controlled updates, so potential degradation is explicitly bounded. Together, these components combine structured reasoning, group-wise preference modeling, and stable optimization for meme generation.

Figure 2: This diagram shows the dataflow for constructing hierarchical CoT supervisions. Stage 1 explores multiple reasoning paths that bind a template to different context-specific details. Stage 2 anchors one high-quality path from ground-truth, preserving diversity while preventing collapse.

Figure 3: Training Pipeline of HUMOR. Multi-path CoT expands reasoning coverage and anchors a canonical path; the reward model translates pair data into a rank-consistent group-level signal (via EBC); GRPO then updates the generator toward higher-ranked captions.

Category / Model		Human Evaluation (0-5) $\uparrow$				Text-level Similarity $\uparrow$	Context-swap Distance $\uparrow$	Human Rate (%) $\uparrow$
Category / Model		Humor	Readability	Relevance	Originality	Text-level Similarity $\uparrow$	Context-swap Distance $\uparrow$	Human Rate (%) $\uparrow$
Open-source Models
	Qwen2.5-7B-Instruct ( Bai et al., 2025 )	2.39	3.35	2.91	2.57	0.576	0.564	75.7
	Qwen2.5-32B-Instruct ( Bai et al., 2025 )	2.54	3.52	3.09	2.76	0.564	0.566	82.2
	InternVL3-8B ( Zhu et al., 2025b )	2.39	2.79	3.04	2.79	0.545	0.564	62.7
	GLM-4.1V-9B-Thinking ( Hong et al., 2025 )	1.73	2.62	2.75	2.71	0.602	0.572	45.1

Table 1: Evaluation results across open-source models, closed-source models, and Qwen2.5-7B-Instruct fine-tuned with our proposed and different CoT methods. Metrics include context-swap distance (diversity criterion), text-level similarity (sim. to original meme text), human evaluation (Humor, Readability, Relevance, Originality), and Human Rate.

Figure 4: (a) VLM-based absolute scoring fails to distinguish meme quality. (b) Group-wise ranking produces more reliable distinctions, better aligned with human.

Model	Template 1		Template 2		Template 3		Template 4		Template 5
Model	$\tau\uparrow$	$\text{$ p $}\downarrow$	$\tau\uparrow$	$\text{$ p $}\downarrow$	$\tau\uparrow$	$\text{$ p $}\downarrow$	$\tau\uparrow$	$\text{$ p $}\downarrow$	$\tau\uparrow$	$\text{$ p $}\downarrow$
Qwen2.5-VL-7B (Base)	0.16	0.60	0.28	0.17	0.47	0.07	-0.10	0.63	0.29	0.29
Qwen2.5-VL-7B (Finetuned)	0.47	0.07	0.56	0.03*	0.42	0.11	0.14	0.50	0.47	0.07
$\Delta$ vs Base	$+0.31$	$-0.53$	$+0.28$	$-0.14$	$-0.04$	$+0.04$	$+0.25$	$-0.13$	$+0.18$	$-0.22$
Qwen2.5-VL-32B (Base)	0.16	0.61	0.16	0.44	-0.02	1.00	0.14	0.50	0.29	0.29
Qwen2.5-VL-32B (Finetuned)	0.29	0.29	0.47	0.02*	0.07	0.86	0.30	0.14	0.42	0.11

Table 2: (Reward Model) Ranking results of different baselines among distinct template images. It indicates the change after fine-tuning relative to the baseline: an increase in Kendall tau

\tau

and a decrease in p-value

p

represent improvements (highlighted in green), while the opposite indicates deterioration (shown in red). Significance levels: *

p<0.05

; **

p<0.01

; ***

p<0.001

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Comparison between direct CoT generation from the original image and our hierarchical CoT generation approach.

Figure 8: Examples of memes common on the internet

Figure 9: Examples of training datasets with different difficulty tier

Source Model	Target Score	Qwen-VL	Keye-VL	Logits (Qwen)
Qwen-VL	0	0.692	0.209	0.498
	0.2	0.642	0.229	0.452
	0.6	0.678	0.096	0.383
	0.8	0.522	0.400	0.368
	1	0.630	0.478	0.387
Keye-VL	0	0.508	0.280	0.390

Appendix

Table 3: Content reward evaluation across different target quality levels. Keye-VL as judge exhibits the clearest monotonic trend, and is therefore adopted as our content reward model in RL.

Panel of Figure 10 — Figure 10: Textual properties of meme captions in training dataset

Panel of Figure 11 — Figure 11: Analysis of Semantic Diversity. The distribution of semantic distances (defined as $1-\text{Cosine Similarity}$ ) exhibits a mean and median of 0.570. The concentration of data ( $54.5\%$ ) within the $[0.5,0.6]$ interval indicates a healthy balance: the captions are semantically related enough to share a theme, yet diverse enough to avoid trivial repetition.

Hyperparameter	Value
Finetuning Stage	sft
Finetuning Type	lora
LoRA Rank	128
LoRA Target	all
Per Device Train Batch Size	1
Gradient Accumulation Steps	8

Appendix

Table 4: Training Setup

Figure 12: The Top-5 human-ranked memes in the datasets sharing the same templates.

Figure 13: Candidate VLM evaluators’ ROC curves for AI vs. human meme classification. Curves plot TPR vs. FPR; star markers denote optimal thresholds (maximizing Youden index) and metrics. AUC (overall discrimination ability) is in the legend.

Figure 14: Case study comparing Single-path and Multi-path Hierarchical CoT supervision in meme generation using the same Mr. Bean image. The single-path model reproduces the ground-truth reasoning chain, yielding literal and less contextual humor. The multi-path model, trained with multi-scenario associative reasoning, demonstrates improved contextual understanding and humorous transferability, producing text that creatively matches new user intents.

Model		RE-classification Accuracy $\uparrow$
Model		Emotion	Intention	Theme	Style
	Qwen2.5-7B-Instruct	0.420	0.515	0.551	0.521
	Qwen2.5-32B-Instruct	0.571	0.611	0.616	0.603
	HUMOR-CoT	0.597	0.641	0.600	0.639

Appendix

Table 5: Compare qwen2.5-7B, 32B, with the results of reclassification of the model we trained on qwen2.5-7B for the sentiment, intent, theme, and style of user input

Figure 15: Qualitative examples from the persona-conditioned visual captioning pilot. For the same image-persona group, HUMOR-style multi-path reasoning explores different caption strategies before local selection.

Method	#Groups	distinct-1	distinct-2	Avg. sentence dist.
Direct	20	0.734	0.960	0.507
HUMOR-style multi-path	20	0.738	0.942	0.563

Appendix

Table 6: Candidate diversity in the cross-domain pilot. distinct-1 measures word-level diversity, distinct-2 measures local bigram-level diversity, and average pairwise sentence distance measures semantic diversity in sentence-embedding space.

Judger	#Groups	HUMOR wins	Ties	HUMOR losses
Qwen2.5-VL-7B-Instruct	20	14	0	6
MiniMax-M3	20	12	3	5
Step-3.7-Flash	20	10	9	1

Appendix

Table 7: Group-local pairwise preference results comparing the HUMOR-style selected caption against the direct selected caption.

Figure 16: Generation results of models trained with different CoT strategies.

Figure 17: Unseen Template Generation. HUMOR-CoT generalizes well to templates entirely absent from training, producing humorous and contextually aligned captions.

Figure 18: Workplace Meme Generation (Single/Multi-Panel). Real-world application examples for workplace scenarios, showing 3 cases with different emotional/contextual tags.

Figure 19: Risk example identification. Gemini-2.5-pro effectively flags politically sensitive or socially harmful meme generations.

Figure 20: Failure case analysis. When user-provided nouns are overly specific, the model may prioritize literal fit over humor, causing loss of scene coherence.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
Jul 21, 2026Tuo Liang, Zhe Hu, Disheng Liu +2Multimodal Large Language Models Computational Creativity
Case Western Reserve University · The Hong Kong Polytechnic University
Can Thinking Models Think to Detect Hateful Memes?
Mar 1, 2026Mohamed Bayan Kmainasi, Mucahid Kutlu, Ali Ezzat Shahroor +2Harmful Meme Detection Multimodal Large Language Models
Qatar University Doha, Qatar · Qatar Computing Research Institute Doha, Qatar · APAVI.AI France +1
CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension
Aug 24, 2026Abhilash Nandy, Rahul Seetharaman, Aman Bansal +5LLM Reasoning with Graphs Multimodal Reasoning
Indian Institute of Technology Kharagpur, India · Microsoft Research India · LinkedIn, USA +4

Jul 21, 2026J/K move · Enter open · S save

Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges

Tuo Liang, Zhe Hu, Disheng Liu, Jing Li+1

Case Western Reserve University · The Hong Kong Polytechnic University

Multimodal Large Language Models Computational Creativity Humor Explanation Generation Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal humor in memes, cartoons, and comics remains difficult for AI systems because intended meaning depends on non-literal mechanisms, shared cultural knowledge, and communicative intent rather than literal scene description. This survey focuses on visual humor understanding in single-image and multi-panel artifacts, while treating humor generation as an emerging downstream frontier. We position the literature against prior humor, sarcasm, and general MLLM surveys and organize it using a capability-centric hierarchy spanning recognition, interpretation and reasoning, and generation. Under this lens, we synthesize benchmark design, evaluation protocols, and modeling paradigms, tracing the field's shift from task-specific fusion models to large-model approaches based on multimodal alignment, evidence-grounded reasoning, and controlled generation. We conclude by highlighting the main barriers to progress: shortcut-prone evaluation, limited cultural and narrative coverage, weak evidence grounding, and unresolved safety and ownership concerns.

Abstract

VLM Reasoning

Explore similar work

Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges

Can Thinking Models Think to Detect Hateful Memes?

CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension