cs.CLMar 24, 2026
SaveI Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes
Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · National Research Council Canada, Ottawa, Canada
Abstract
Internet memes represent a popular form of multimodal online communication and often use figurative elements to convey layered meaning through the combination of text and images. However, it remains largely unclear how multimodal large language models (MLLMs) combine and interpret visual and textual information to identify figurative meaning in memes. To address this gap, we evaluate eight state-of-the-art generative MLLMs across three datasets on their ability to detect and explain six types of figurative meaning. In addition, we conduct a human evaluation of the explanations generated by these MLLMs, assessing whether the provided reasoning supports the predicted label and whether it remains faithful to the original meme content. Our findings indicate that all models exhibit a strong bias to associate a meme with figurative meaning, even when no such meaning is present. Qualitative analysis further shows that correct predictions are not always accompanied by faithful explanations.
Figures & tables
Figure 1: A meme from MET-Meme ( Xu et al., 2022 ) playfully adapts Caesar’s famous quote “I came, I saw, I conquered”. In our work, we investigate the respective contributions of image and text to models’ prediction of figurative meaning in memes.
| FigMemes | Memotion 2 | MET-Meme | ||||||||||||||
| Type | None | Allus. | Exag. | Irony | Anthrop. | Met | Contr. | Total | NS | LS | VS | ES | Total | Lit | Met | Total |
| Count | 495 | 265 | 265 | 320 | 131 | 286 | 171 | 1372 | 804 | 388 | 246 | 62 | 1500 | 1054 | 1054 | 2108 |
Table 1: Statistics of our evaluation data. Underlined types indicate negative cases (e.g. non figurative) in the binary classification task of each dataset. Total shows the number of evaluated memes in each dataset; in FigMemes , a meme can span multiple types of figurative meaning. FigMemes : Allus. = Allusion, Exag. = Exaggeration, Anthrop. = Anthropomorphism, Met. = Metaphor, Contr. = Contrast. Memotion 2: NS, LS, VS, ES indicate increasing levels of sarcasm (Not, Little, Very, Extremely). MET-Meme : Lit = Literal, Met = Metaphoric.
| Labels per Meme | 0 | 1 | 2 | 3 | 4 |
| Meme Count | 495 | 715 | 277 | 51 | 4 |
Table 2: Count of labels per meme in FigMemes . On average, each meme has 0.93 labels. 0 means no figurativeness is present.
| FigMemes | Memotion 2 | MET-Meme | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Size | Input | Allusion | Exag. | Irony | Anthrop. | Metaphor | Contrast | Sarcasm | Metaphor | Average |
| Baseline | 52.32 | 44.00 | 49.77 | 41.76 | 44.87 | 56.91 | – | – | – | ||
| Meme | 70.04 | 58.63 | 56.06 | 51.94 | 50.62 | 77.95 | 63.48 | 66.66 | 61.92 | ||
| -Text | 65.44 | 58.52 | 10.53 | 48.39 | 35.91 | 59.99 | 49.64 | 54.16 | 47.82 | ||
| 8B | -Image | 34.78 | 30.16 | 57.07 | 8.47 | 40.09 | 23.46 | 63.01 | 57.28 | 39.29 | |
| Meme | 66.87 | 61.01 | 61.31 | 54.67 | 54.97 | 70.85 | 63.37 | 66.77 | 62.48 | ||
Table 3: F1 scores (in %) of eight models on the binary classification of figurative meaning types in memes from three datasets. -Text and -Image denote the ablation of the text or image modality, respectively. The best and second-best results are shown in bold and underlined, respectively. Following Liu et al. (2022a) , in FigMemes we consider memes with 0 labels as negative cases, and take the best-performing benchmark score as baseline. Higher scores are highlighted in orange , lower scores in blue . Results are averaged over five runs.
| Model | Input | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|---|
| random | – | 50.04 | 15.57 | 49.88 | 23.73 |
| Meme | 57.60 | 24.63 | 83.54 | 38.05 | |
| -Text | 66.80 | 25.30 | 57.86 | 35.20 | |
| Aya-32B | -Image | 74.06 | 28.36 | 43.51 | 34.34 |
| Meme | 55.97 | 24.44 | 87.50 | 38.21 | |
| -Text | 62.08 | 23.79 | 65.04 | 34.84 |
Table 4: Accuracy, precision, recall and Micro-F1 (in %) of models on multi-label classification of figurative meanings in FigMemes . The highest score for each metric is highlighted in bold . Higher scores are highlighted in orange , lower scores in blue .
Figure 2: Performance of Aya-32B , Gemma-27B , and Qwen-72B on the multi-label classification task (only in FigMemes ) across modalities and meme groups. Error bars indicate standard deviation over 5 runs. Top row: Count of memes with 0, 1, or 2+ figurative types (0, 1, 2+ gold labels) that are predicted to contain from 0 to 6 figurative types. Bottom row: Count of memes containing 0, 1, or 2+ figurative types that are fully correctly predicted, where all six predicted labels match the gold labels.
Figure 3: Human evaluation results on model-generated explanations.
Figure 4: Examples of memes for which MLLMs struggle to generate high-quality explanations.
Figure 5: Two memes from Memotion 2 with gold label “not sarcastic”.
Figure 6: Difference in macro-F1 (in %) between prompting score and prompting label setting for eight models, averaged over five seeds. Bars with positive values indicate that prompting score improved performance; bars with negative values indicate a decrease. Results cover binary classification tasks across three datasets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Examples of automatic preprocessing by inpainting text. The left column shows the original meme images, while the right column displays the results after text removal using PaddleOCR and LaMa.
Figure 8: Examples of automatic and manual text removal. For cases where low image resolution hindered automated OCR detection, the original memes were manually edited to ensure a clean, text-free input for the model.
| Model | Input | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|---|
| Meme | 57.60 | 24.41 | 82.72 | 37.27 | |
| -Text | 66.80 | 25.73 | 61.90 | 34.71 | |
| Aya-32B | -Image | 74.06 | 26.90 | 39.98 | 30.23 |
| Meme | 87.50 | 37.88 | 55.97 | 24.57 | |
| -Text | 69.09 | 34.82 | 62.08 | 24.12 | |
| Gemma-27B | -Image | 51.24 | 31.44 | 68.56 | 23.70 |
Table 5: Macro-averaged accuracy, precision, recall and F1 (in %) of models on multi-label classification of figurative meanings in FigMemes .
| Parameter | Value |
|---|---|
| Temperature | 0.7 |
| Top- p | 0.1 |
| Repetition Penalty | 1.05 |
| Max Tokens | 512 |
Table 6: Sampling parameters used during inference.
| FigMemes | Memotion 2 | MET-Meme | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Size | Input | Allusion | Exag. | Irony | Anthrop. | Metaphor | Contrast | Sarcasm | Metaphor |
| Baseline | 52.32 | 44.00 | 49.77 | 41.76 | 44.87 | 56.91 | – | – | ||
| Meme | 70.04 ±0.00 | 58.63 ±0.00 | 56.06 ±0.06 | 51.94 ±0.00 | 50.62 ±0.15 | 77.95 ±0.00 | 63.48 ±0.00 | 66.66 ±0.26 | ||
| -Text | 65.44 ±0.00 | 58.52 ±0.12 | 10.53 ±0.00 | 48.39 ±0.19 | 35.91 ±0.22 | 59.99 ±0.38 | 49.64 ±0.00 | 54.16 ±0.22 | ||
| 8B | -Image | 34.78 ±0.00 | 30.16 ±0.00 | 57.07 ±0.00 | 8.47 ±0.00 | 40.09 ±0.00 | 23.46 ±0.00 | 63.01 ±0.01 | 57.28 ±0.13 | |
| Meme | 66.87 ±0.00 | 61.01 ±0.00 | 61.31 ±0.03 | 54.67 ±0.00 | 54.97 ±0.00 | 70.85 ±0.07 | 63.37 ±0.00 | 66.77 ±0.04 | ||
Table 7: F1 scores (in %) of eight models on the binary classification of figurative meaning types in memes from three datasets. -Text and -Image denote the ablation of the text or image modality, respectively. Following Liu et al. (2022a) , in FigMemes we consider memes with 0 labels as negative cases, and take the best-performing benchmark score as baseline. Results are averaged over five runs.
| FigMemes | Memotion 2 | MET-Meme | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Size | Input | Allusion | Exag. | Irony | Anthrop. | Metaphor | Contrast | Sarcasm | Metaphor |
| Baseline | 52.32 | 44.00 | 49.77 | 41.76 | 44.87 | 56.91 | – | – | ||
| Meme | 60.97 ±0.05 | 58.72 ±0.11 | 59.99 ±0.03 | 49.13 ±0.07 | 53.19 ±0.13 | 59.80 ±0.15 | 63.27 ±0.03 | 66.88 ±0.01 | ||
| -Text | 61.41 ±0.05 | 59.42 ±0.08 | 46.37 ±0.00 | 49.39 ±0.05 | 53.68 ±0.06 | 57.01 ±0.15 | 59.17 ±0.02 | 63.51 ±0.04 | ||
| 8B | -Image | 51.66 ±0.15 | 30.95 ±0.08 | 58.17 ±0.06 | 6.43 ±0.04 | 52.34 ±0.13 | 45.69 ±0.00 | 56.03 ±0.04 | 53.62 ±0.05 | |
| Meme | 69.11 ±0.00 | 57.43 ±0.04 | 60.55 ±0.00 | 54.56 ±0.10 | 57.41 ±0.05 | 53.47 ±0.04 | 63.21 ±0.00 | 67.25 ±0.00 | ||
Table 8: F1 scores (in %) of eight models on the binary classification of figurative meaning types in memes from three datasets with the second prompt template. -Text and -Image denote the ablation of the text or image modality, respectively. Following Liu et al. (2022a) , in FigMemes we consider memes with 0 labels as negative cases, and take the best-performing benchmark score as baseline. Results are averaged over five runs.
Figure 9: Distribution of predicted label counts across eight models. Each subplot shows the frequency of models predicting 0 to 6 labels, grouped by the actual number of gold labels (0, 1, and 2+).
Figure 10: Correct prediction counts for eight models across different label complexities. The bar charts represent the Mean and Standard Deviation (error bars) over five experimental seeds.
More in this topic
Figurative Language Understanding
Figurative language understanding studies how people and AI interpret nonliteral meaning in metaphors, idioms, proverbs, puns, and related forms.
See all 45 papers