cs.AIOct 22, 2026
SaveIRON-Meme: A Benchmark for Structured Understanding of Identities and Roles in Online Meme Narratives
Organizations: Department of Computer Science, Faculty of Science, Vrije Universiteit Amsterdam · Department of Communication Science, Faculty of Social Science, Vrije Universiteit Amsterdam
Abstract
As polarization grows on social media, understanding how political memes shape online narratives has become increasingly important. Previous research has relied on either qualitative frameworks that are hard to scale or computational methods that lack theoretical grounding. Both approaches treat narrative categories in isolation and focus exclusively on moral portrayals of identity. To address these limitations, we introduce a framework grounded in social science theories, encompassing three categories: social identities (such as gender, politics, and ethnicity), viewpoints (ingroup and outgroup perspectives), and narrative roles (moral character, intelligence, status, and power). Building on this framework, we present IRON-Meme, a human-annotated benchmark comprising 1,110 political memes across substantially more templates than existing benchmarks. Our analysis of IRON-Meme shows that most memes assign derogatory roles and that identities, roles, and templates are systematically interconnected. Evaluation of nine vision-language models indicates that frontier-scale and reasoning models achieve the highest performance, while accuracy on social identities and narrative roles decreases for less frequent labels. Incorporating template background knowledge does not enhance narrative role classification, suggesting that meme interpretation relies on how memes are adapted for specific communicative purposes. We make IRON-Meme and our code publicly available.
Figures & tables
Figure 1: Framework overview. An input meme is annotated by a human or VLM to produce a structured mapping of characters across the three dimensions of the IRON-Meme label schema: social identity, viewpoint, and narrative role. In this example, the white character is assigned the identity climate change denier with an ingroup viewpoint and sage role, while the grey character is classified as a climate activist with an outgroup viewpoint and fool role.
| Evaluative dimension | Moral | Intellectual | Physical | Power |
| Positive / Superior | Hero | Sage | Charmer | Winner |
| Negative / Inferior | Villain | Fool | Monster | Loser |
| Positive / Inferior | Victim |
Table 1: Narrative role labels in our framework. This set is based on two sets of binary oppositions (positive/negative and superior/inferior), and four evaluative dimensions (moral, intellectual, physical and power).
Figure 2: Final origin distribution of the IRON-Meme benchmark dataset. The inner ring illustrates the overall distribution between reused benchmarks (blue) and newly scraped data (red). The outer ring details the individual datasets and subreddits contributing to each category. The legend indicates the number of memes contributed by each source.
Figure 3: Identity–role correlation for the 10 most frequent social identities . The chart displays the frequency of social identity–narrative role combinations, with circle size and numerical values indicating frequency counts. Narrative roles (y-axis) are shown in descending order. Colors correspond to distinct roles.
| Social Identities | Viewpoints | Narrative Roles | |||||||||
| Model | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | ||
| Random (stratified) | 11.45 | 11.46 | 11.45 | 64.94 | 64.96 | 64.94 | 21.09 | 21.10 | 21.09 | ||
| DeepSeek-VL2 | 11.25 | 39.72 | 13.07 | 67.18 | 90.89 | 77.10 | 47.00 | 41.90 | 34.44 | ||
| Gemini-3.6 | 45.58 | 73.55 | 51.99 | 93.55 | 92.86 | 93.20 | 59.82 | 75.75 | 64.14 | ||
| GLM-4.1V | 45.28 | 42.81 | 38.60 | 81.45 | 77.93 | 79.61 | 41.20 | 60.49 | 46.55 | ||
| Kimi-K3 | 44.99 | 77.60 | 52.78 | 92.60 | 92.62 | 92.61 | 48.11 | 81.01 | 57.88 | ||
Table 2: Zero-shot performance of VLMs across categories (macro metrics) compared against a stratified random baseline. For each meme and each label, the baseline assigns a prediction probability equal to the prevalence of that label in the ground truth. Full model names are listed in Table 9 .
Figure 4: Narrative role error analysis heatmap.
| Prompt Condition | Precision | Recall | F1 |
| Standard | 60.50 | 77.10 | 65.20 |
| Templated | 55.60 | 75.50 | 61.90 |
Table 3: Macro-averaged narrative role performance of Gemini-3.6 with and without template knowledge.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Identity Label | ||
| Women | Men | White people |
| People of color | Citizens | National identity |
| Feminist | Anti-feminist | Sex worker |
| LGBTQ+ rights supporter | Anti-LGBTQ+ | Misogynist/male supremacist |
| Racist | Anti-racist | Nationalist/regionalist |
| Progressive | Conservative | Liberal |
Table 4: Set of social identity labels used in the dataset.
| Model | s1 (Ident.) | s2 (View.) | s3 (Roles) |
| deepseek-vl2 | 10.2% | 1.4% | 0.0% |
| gemini-3.6-flash | — | — | — |
| glm-4.1v-9b-think | 1.0% | 1.5% | 0.0% |
| kimi-k3 | 0.0% | 0.0% | 0.0% |
| molmo-7b-d | 52.7% | 2.3% | 1.4% |
| pixtral-12b | 0.0% | 0.0% | 0.0% |
Table 5: Percentage of unparseable responses per model across task steps.
| Model | Identities | Viewpoints | Roles |
| deepseek-vl2 | 7.2% | 0.0% | 5.2% |
| gemini-3.6-flash | 0.0% | 0.0% | — |
| glm-4.1v-9b-think | 1.3% | 0.0% | 0.0% |
| kimi-k3 | 0.0% | 0.0% | 0.0% |
| molmo-7b-d | 47.7% | 0.2% | 0.0% |
| pixtral-12b | 3.4% | 0.0% | 0.0% |
Table 6: Percentage of discarded labels outside task inventories per model.
Figure 6: Pairwise Jaccard agreement across annotation stages
| Category | Fleiss’ a | Avg. b | Jaccard | Perfect |
| Social identity | 0.694 | 0.693 | 0.660 | 45.82% |
| Viewpoint | 0.594 | 0.593 | 0.794 | 60.07% |
| Narrative roles | 0.509 | 0.508 | 0.532 | 26.31% |
| Overall (Triples) c | 0.426 | 0.425 | 0.403 | 30.85% |
Table 7: Inter-rater agreement metrics across annotation categories.
Figure 7: Distribution of Social Identities Across Source Sub-Datasets.
Figure 8: Viewpoints for the 10 most common social identities.
Figure 9: Examples of memes using the Steven Crowder’s “Change My Mind” Campus Sign template.
| Category | Count | % |
| Out-group only | 650 | 58.56 |
| In-group only | 251 | 22.61 |
| Both in- and out-group | 206 | 18.56 |
| No valid viewpoint | 3 | 0.27 |
| Total | 1,110 | 100.00 |
Table 8: Distribution of viewpoint categories.
Figure 10: Overview of the template-based meme curation pipeline.
| Model | Developer / Region | Family | Orientation / Capability | Parameter Scale |
| Molmo-7B | Allen Institute for AI (North America) | Molmo | Instruction | 7B |
| Qwen2.5-7B-Instruct | Alibaba Cloud (East Asia) | Qwen | Instruction | 7B |
| Qwen3-8B | Alibaba Cloud (East Asia) | Qwen | Standard | 8B |
| Qwen3-8B-Instruct | Alibaba Cloud (East Asia) | Qwen | Instruction | 8B |
| Qwen3-8B-Thinking | Alibaba Cloud (East Asia) | Qwen | Explicit Reasoning | 8B |
| GLM-4.1-9B | Zhipu AI / Tsinghua (East Asia) | GLM | Standard | 9B |
Table 9: Overview of the VLMs selected for evaluation, covering diverse model families, architectures, reasoning orientations, parameter scales, and geographic regions.
Table 10: Meme annotated examples showing agreement types and ground truth aggregation.
Figure 11: Narrative-role composition for the ten most frequently used meme templates. Stacked bars represent the number of memes assigned to each narrative role for each template. Template names correspond to their listings in Know Your Meme.
| Model | Micro Metrics | Macro Metrics | ||||
| Precision | Recall | F1 | Precision | Recall | F1 | |
| Social Identities | ||||||
| gemini-3.6-flash | 51.69 | 77.73 | 62.09 | 42.98 | 63.12 | 46.15 |
| Human annotators | 49.85 | 64.84 | 56.37 | 34.40 | 43.60 | 35.86 |
| Viewpoints | ||||||
| gemini-3.6-flash | 78.09 | 87.35 | 82.46 | 74.81 | 83.41 | 78.87 |
Table 11: Comparison of gemini-3.6-flash and human annotators in the setting without ground-truth context, evaluated on the same subset of 214 memes.
| Predictions | ||||||
| Image | Template | Template Description | Identity | GT | w/o Template | w/ Template |
| Batman Slapping Robin | Batman Slapping Robin, also known as “My Parents Are Dead,” is an exploitable image of the DC Comics superhero Batman slapping his protégé Robin in mid-conversation. Taken from a comic book published in 1965, the single-panel illustration has inspired a series of parodies featuring custom-captioned speech bubbles on 4chan, Reddit and elsewhere. | climate activist | sage | sage | fool, loser | |
| IQ Bell Curve / Midwit | IQ Bell Curve, also known as Midwit or simply Bell Curve Meme, refers to a series of memes that use the IQ distribution diagrams to mock one of the three main groups represented by the diagram, usually the largest group of people towards the center of the curve who possess average intelligence. The concept of IQ Bell Curve memes utilizes the horseshoe theory, implying that the groups with low intelligence and high intelligence often choose to follow the same goals while being guided by different reasoning. Those who possess average intelligence are referred to as “midwits.” While there were many versions of memes illustrating the concept, a variant of the meme using three different Wojaks gained particular popularity in the 2020s on X / Twitter and Reddit. | climate activist | sage | sage | fool | |
| Actual Sexual Advice Girl | Actual Sexual Advice Girl (ASAG) is an advice animal image macro series featuring a photograph of a brunette woman wearing a pair of black-rimmed glasses accompanied by captions relaying sexual advice to both men and women. | women | loser | loser | sage | |
Table 12: Qualitative comparison of model predictions with ( w/ ) and without ( w/o ) template descriptions in the prompt.
| Model | Micro Metrics | Macro Metrics | ||||
| Precision | Recall | F1 | Precision | Recall | F1 | |
| Social Identities | ||||||
| deepseek-vl2 | 11.65 | 33.23 | 17.26 | 11.25 | 39.72 | 13.07 |
| gemini-3.6-flash | 55.79 | 76.31 | 64.46 | 45.58 | 73.55 | 51.99 |
| glm-4.1v-9b-thinking | 57.02 | 45.97 | 50.90 | 45.28 | 42.81 | 38.60 |
| kimi-k3 | 55.19 | 77.50 | 64.47 | 44.99 | 77.60 | 52.78 |
Table 13: Performance evaluation of vision-language models grouped by category.
Figure 12: Label frequency and model performance across evaluation dimensions
Figure 13: Top social identity mismatches accross 9 VLMs.
Figure 14: Viewpoint error matrices for VLM classification.
Figure 15: Narrative role error matrices for VLM classification.
More in this topic
Political Discourse Analysis
Political discourse analysis studies how language expresses ideology, frames issues, persuades audiences, and shapes political debate.
See all 67 papers