Organizations: Shanghai Artificial Intelligence Laboratory · School of Software Engineering Xi’an Jiaotong University · Columbia Engineering Columbia University · The Chinese University of Hong Kong MMLab
Generating humorous memes is a challenging multimodal task that moves beyond direct image-to-caption supervision. It requires a nuanced reasoning over visual content, contextual cues, and subjective humor. To bridge this gap between visual perception and humorous punchline creation, we propose HUMOR}, a novel framework that guides VLMs through hierarchical reasoning and aligns them with group-wise human preferences. First, HUMOR employs a hierarchical, multi-path Chain-of-Thought (CoT): the model begins by identifying a template-level intent, then explores diverse reasoning paths under different contexts, and finally anchors onto a high-quality, context-specific path. This CoT supervision, which traces back from ground-truth captions, enhances reasoning diversity. We further analyze that this multi-path exploration with anchoring maintains a high expected humor quality, under the practical condition that high-quality paths retain significant probability mass. Second, to capture subjective humor, we train a pairwise reward model that operates within groups of memes sharing the same template. Following established theory, this approach ensures a consistent and robust proxy for human preference, even with subjective and noisy labels. The reward model then enables a group-wise reinforcement learning optimization, guaranteeing providing a theoretical guarantee for monotonic improvement within the trust region. Extensive experiments show that HUMOR empowers various VLMs with superior reasoning diversity, more reliable preference alignment, and higher overall meme quality. Beyond memes, our work presents a general training paradigm for open-ended, human-aligned multimodal generation, where success is guided by comparative judgment within coherent output group.
Figures & tables
Figure 1: Given a template image, HUMOR first performs hierarchical reasoning with multi-path CoT: a template-level stage infers latent intent, while a context-level stage explores multiple visually grounded reasoning paths. One high-quality path is anchored by tracing back from ground-truth captions, promoting diversity while preserving a conditional humor lower bound. A reward model then compares memes only within groups sharing the same template, preserving rank consistency and providing a proxy for human preference. This reward further enables stable group-wise RL with KL-controlled updates, so potential degradation is explicitly bounded. Together, these components combine structured reasoning, group-wise preference modeling, and stable optimization for meme generation.
Figure 2: This diagram shows the dataflow for constructing hierarchical CoT supervisions. Stage 1 explores multiple reasoning paths that bind a template to different context-specific details. Stage 2 anchors one high-quality path from ground-truth, preserving diversity while preventing collapse.
Figure 3: Training Pipeline of HUMOR. Multi-path CoT expands reasoning coverage and anchors a canonical path; the reward model translates pair data into a rank-consistent group-level signal (via EBC); GRPO then updates the generator toward higher-ranked captions.
Category / Model
Human Evaluation (0-5) ↑
Text-level Similarity ↑
Context-swap Distance ↑
Human Rate (%) ↑
Humor
Readability
Relevance
Originality
Open-source Models
Qwen2.5-7B-Instruct ( Bai et al., 2025 )
2.39
3.35
2.91
2.57
0.576
0.564
75.7
Qwen2.5-32B-Instruct ( Bai et al., 2025 )
2.54
3.52
3.09
2.76
0.564
0.566
82.2
InternVL3-8B ( Zhu et al., 2025b )
2.39
2.79
3.04
2.79
0.545
0.564
62.7
GLM-4.1V-9B-Thinking ( Hong et al., 2025 )
1.73
2.62
2.75
2.71
0.602
0.572
45.1
Table 1: Evaluation results across open-source models, closed-source models, and Qwen2.5-7B-Instruct fine-tuned with our proposed and different CoT methods. Metrics include context-swap distance (diversity criterion), text-level similarity (sim. to original meme text), human evaluation (Humor, Readability, Relevance, Originality), and Human Rate.
Figure 4: (a) VLM-based absolute scoring fails to distinguish meme quality. (b) Group-wise ranking produces more reliable distinctions, better aligned with human.
Figure 6
Model
Template 1
Template 2
Template 3
Template 4
Template 5
τ↑
\text{p}\downarrow
τ↑
\text{p}\downarrow
τ↑
\text{p}\downarrow
τ↑
\text{p}\downarrow
τ↑
\text{p}\downarrow
Qwen2.5-VL-7B (Base)
0.16
0.60
0.28
0.17
0.47
0.07
-0.10
0.63
0.29
0.29
Qwen2.5-VL-7B (Finetuned)
0.47
0.07
0.56
0.03*
0.42
0.11
0.14
0.50
0.47
0.07
Δ vs Base
+0.31
−0.53
+0.28
−0.14
−0.04
+0.04
+0.25
−0.13
+0.18
−0.22
Qwen2.5-VL-32B (Base)
0.16
0.61
0.16
0.44
-0.02
1.00
0.14
0.50
0.29
0.29
Qwen2.5-VL-32B (Finetuned)
0.29
0.29
0.47
0.02*
0.07
0.86
0.30
0.14
0.42
0.11
Table 2: (Reward Model) Ranking results of different baselines among distinct template images. It indicates the change after fine-tuning relative to the baseline: an increase in Kendall tau τ and a decrease in p-value p represent improvements (highlighted in green), while the opposite indicates deterioration (shown in red). Significance levels: * p<0.05 ; ** p<0.01 ; *** p<0.001 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Comparison between direct CoT generation from the original image and our hierarchical CoT generation approach.
Figure 8: Examples of memes common on the internet
Figure 9: Examples of training datasets with different difficulty tier
Source Model
Target Score
Qwen-VL
Keye-VL
Logits (Qwen)
Qwen-VL
0
0.692
0.209
0.498
0.2
0.642
0.229
0.452
0.6
0.678
0.096
0.383
0.8
0.522
0.400
0.368
1
0.630
0.478
0.387
Keye-VL
0
0.508
0.280
0.390
Appendix
Table 3: Content reward evaluation across different target quality levels. Keye-VL as judge exhibits the clearest monotonic trend, and is therefore adopted as our content reward model in RL.
Figure 10: Textual properties of meme captions in training dataset
Figure 11: Analysis of Semantic Diversity. The distribution of semantic distances (defined as 1−Cosine Similarity ) exhibits a mean and median of 0.570. The concentration of data ( 54.5% ) within the [0.5,0.6] interval indicates a healthy balance: the captions are semantically related enough to share a theme, yet diverse enough to avoid trivial repetition.
Hyperparameter
Value
Finetuning Stage
sft
Finetuning Type
lora
LoRA Rank
128
LoRA Target
all
Per Device Train Batch Size
1
Gradient Accumulation Steps
8
Appendix
Table 4: Training Setup
Figure 12: The Top-5 human-ranked memes in the datasets sharing the same templates.
Figure 13: Candidate VLM evaluators’ ROC curves for AI vs. human meme classification. Curves plot TPR vs. FPR; star markers denote optimal thresholds (maximizing Youden index) and metrics. AUC (overall discrimination ability) is in the legend.
Figure 14: Case study comparing Single-path and Multi-path Hierarchical CoT supervision in meme generation using the same Mr. Bean image. The single-path model reproduces the ground-truth reasoning chain, yielding literal and less contextual humor. The multi-path model, trained with multi-scenario associative reasoning, demonstrates improved contextual understanding and humorous transferability, producing text that creatively matches new user intents.
Model
RE-classification Accuracy ↑
Emotion
Intention
Theme
Style
Qwen2.5-7B-Instruct
0.420
0.515
0.551
0.521
Qwen2.5-32B-Instruct
0.571
0.611
0.616
0.603
HUMOR-CoT
0.597
0.641
0.600
0.639
Appendix
Table 5: Compare qwen2.5-7B, 32B, with the results of reclassification of the model we trained on qwen2.5-7B for the sentiment, intent, theme, and style of user input
Figure 15: Qualitative examples from the persona-conditioned visual captioning pilot. For the same image-persona group, HUMOR-style multi-path reasoning explores different caption strategies before local selection.
Method
#Groups
distinct-1
distinct-2
Avg. sentence dist.
Direct
20
0.734
0.960
0.507
HUMOR-style multi-path
20
0.738
0.942
0.563
Appendix
Table 6: Candidate diversity in the cross-domain pilot. distinct-1 measures word-level diversity, distinct-2 measures local bigram-level diversity, and average pairwise sentence distance measures semantic diversity in sentence-embedding space.
Judger
#Groups
HUMOR wins
Ties
HUMOR losses
Qwen2.5-VL-7B-Instruct
20
14
0
6
MiniMax-M3
20
12
3
5
Step-3.7-Flash
20
10
9
1
Appendix
Table 7: Group-local pairwise preference results comparing the HUMOR-style selected caption against the direct selected caption.
Figure 16: Generation results of models trained with different CoT strategies.
Figure 17: Unseen Template Generation. HUMOR-CoT generalizes well to templates entirely absent from training, producing humorous and contextually aligned captions.
Figure 18: Workplace Meme Generation (Single/Multi-Panel). Real-world application examples for workplace scenarios, showing 3 cases with different emotional/contextual tags.
Figure 19: Risk example identification. Gemini-2.5-pro effectively flags politically sensitive or socially harmful meme generations.
Figure 20: Failure case analysis. When user-provided nouns are overly specific, the model may prioritize literal fit over humor, causing loss of scene coherence.
Multimodal humor in memes, cartoons, and comics remains difficult for AI systems because intended meaning depends on non-literal mechanisms, shared cultural knowledge, and communicative intent rather than literal scene description. This survey focuses on visual humor understanding in single-image and multi-panel artifacts, while treating humor generation as an emerging downstream frontier. We position the literature against prior humor, sarcasm, and general MLLM surveys and organize it using a capability-centric hierarchy spanning recognition, interpretation and reasoning, and generation. Under this lens, we synthesize benchmark design, evaluation protocols, and modeling paradigms, tracing the field's shift from task-specific fusion models to large-model approaches based on multimodal alignment, evidence-grounded reasoning, and controlled generation. We conclude by highlighting the main barriers to progress: shortcut-prone evaluation, limited cultural and narrative coverage, weak evidence grounding, and unresolved safety and ownership concerns.
Tuo Liang, Zhe Hu, Disheng Liu +2
Case Western Reserve University · The Hong Kong Polytechnic University
Hateful memes often require compositional multimodal reasoning: the image and text may appear benign in isolation, yet their interaction conveys harmful intent. Although thinking-based multimodal large language models (MLLMs) have recently advanced vision-language understanding, their capabilities remain underexplored for hateful meme analysis. We propose a reinforcement learning based post-training framework that improves reasoning in thinking-based MLLMs through task-specific rewards and a novel Group Relative Policy Optimization (GRPO) objective. Specifically, we (i) conduct a systematic empirical study of off-the-shelf MLLMs for hateful meme understanding, (ii) extend an existing hateful meme dataset by generating weakly or pseudo-supervised chain-of-thought rationales via distillation, and (iii) introduce a GRPO-based objective that jointly optimizes meme classification and explanation quality to encourage fine-grained, step-by-step reasoning. Experiments on the Hateful Memes benchmark show that our approach achieves state-of-the-art performance, improving accuracy and F1 by approximately 1 percent and explanation quality by approximately 3 percent. We will publicly release our code, dataset extensions, and evaluation resources to support reproducibility.
Mohamed Bayan Kmainasi, Mucahid Kutlu, Ali Ezzat Shahroor +2
Qatar University Doha, Qatar · Qatar Computing Research Institute Doha, Qatar · APAVI.AI France +1
Large-scale vision-language models (VLMs) have demonstrated remarkable versatility across a wide range of multimodal tasks. However, understanding humor remains challenging because humorous content often depends on subtle interactions among entities, events, context, and implicit relationships across image and text modalities. These interactions can involve complex chains of reasoning that are difficult to capture through conventional prompting or linear chain-of-thought reasoning. In this work, we propose CaRGo-T (Causal Reasoning Graph-of-Thought), a reasoning framework that represents the causal and contextual relationships underlying multimodal humor as a lightweight graph-based reasoning structure. The graph is serialized into a code-based representation generated by a VLM, which can subsequently be interpreted by the same or a different VLM to produce the final prediction in zero-shot or in-context learning settings. We evaluate CaRGo-T on humor understanding and humor detection across four datasets spanning diverse forms of comedic content, including satire, sarcasm, and memes. Experiments with state-of-the-art commercial and open-source VLMs show that CaRGo-T consistently improves performance over existing reasoning-based baselines, achieving gains of approximately 1-20% on humor understanding and 1-3% on humor detection. Further analysis using mutual information indicates that the reasoning representations produced by CaRGo-T contain more information relevant to the target output than those generated by baseline reasoning approaches. Code is available at https://github.com/abhi1nandy2/CaRGo-T.
Abhilash Nandy, Rahul Seetharaman, Aman Bansal +5
Indian Institute of Technology Kharagpur, India · Microsoft Research India · LinkedIn, USA +4