MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstruction

Akshit Sharma; Prashant W. Patil

cs.CVSep 17, 2026

MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstruction

Authors: Akshit Sharma, Prashant W. Patil

Organizations: Indian Institute of Technology Guwahati

Abstract

The proliferation of harmful internet memes poses a significant societal threat, yet their automated classification remains a formidable algorithmic challenge due to the nuanced, multimodal nature of their content. To address this, we introduce MemeTAG, a novel dual-objective framework that pioneers a keyword-aware approach to meme classification. Our core innovation is a two-part semantic guidance mechanism: first, we leverage a pretrained Vision-Language Model to generate a set of descriptive keywords, that capture the high-level semantics. Second, we introduce the Aggregated Tag Inference Network (ATIN), an attention-based module that distills these keywords into a single, rich semantic embedding. This embedding serves as a target for a novel auxiliary reconstruction loss, which compels the model to learn deeply aligned visual and textual features. This approach, combined with an efficient three-stage training strategy, establishes a new state-of-the-art on the HarMeme, Hateful Memes Challenge (HMC), and PrideMM datasets, decisively outperforming existing state-of-the-art methods.

Figures & tables

Figure 1: Process of generating a set of keyword embeddings via frozen Vision-Language Model (VLM) using a neutral-language prompt. These generated keyword embeddings are used to train the Aggregated Tag Inference Network (ATIN)

Figure 2: Architecture of the proposed Aggregated Tag Inference Network (ATIN). The aggregated tag embedding from the trained ATIN serves as the ground in the auxiliary reconstruction task of MemeTAG

Figure 3: Architecture of the proposed MemeTAG model for keyword-aware meme classification. We make use of the trained Aggregated Tag Inference Network (ATIN) as shown in Figure 2

Method	Hate			Target			Stance			Humor
	Acc.	AUC.	F1	Acc.	AUC.	F1	Acc.	AUC.	F1	Acc.	AUC.	F1
CLIP Text-Only	69.030	74.990	69.110	50.720	73.210	48.160	50.990	67.910	49.950	69.940	71.230	62.880
CLIP Img-Only	70.550	80.710	73.820	61.290	81.180	58.630	61.440	77.810	58.330	76.230	82.880	73.040
CLIP	72.960	80.770	72.880	61.480	82.090	58.890	59.720	77.310	58.270	77.180	80.820	73.910
GuardHarMem [ 7 ]	70.020	75.740	67.660	65.480	75.050	49.250	52.660	71.400	52.010	71.600	73.520	79.790
CLIP-Adapter [ 31 ]	73.210	81.140	73.090	61.870	82.290	58.470	59.830	77.590	58.390	77.510	81.240	73.990

Table 1: Performance analysis of the proposed MemeTAG with existing state-of-the-art methods on PrideMM dataset

Method	Acc.	AUC.	F1
CLIP Text-Only	73.880	79.520	71.710
ATIN (Baseline)	74.240	84.150	73.260
CLIP Img-Only	79.460	89.120	78.710
CLIP	81.770	87.590	80.740
CLIP-Adapter [ 31 ]	82.490	87.810	81.130
MOMENTA [ 30 ]	82.750	88.040	81.690

Table 2: Performance analysis of the proposed MemeTAG with existing state-of-the-art methods on HarMeme dataset

Model	Acc. (%)	AUROC (%)
Image-Grid (ResNet152)	50.00	50.00
Image-Region (R-CNN)	57.00	52.00
Text-BERT	51.10	53.39
ATIN (Baseline)	57.40	61.60
Late Fusion	56.50	54.07
Concat BERT	52.10	53.94

Table 3: Performance analysis of the proposed MemeTAG with existing state-of-the-art methods on HMC dataset

Model	Supported Modalities		Input Modalities Used			Accuracy (%)	AUC (%)	F1 (%)
	Image	Text	Image	Text	Tag
Gemma-3 (27B) [ 10 ]	$\checkmark$	$\checkmark$	$\checkmark$	X	X	63.559	62.479	53.090
Gemma-3 (27B) [ 10 ]	$\checkmark$	$\checkmark$	X	$\checkmark$	X	63.559	64.337	56.270
Gemma-3 (27B) [ 10 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	X	61.010	60.708	51.748
Gemma-3 (27B) [ 10 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	64.690	65.390	57.340
Mistral Small 3.1 24B	$\checkmark$	$\checkmark$	X	$\checkmark$	X	54.520	56.820	49.840

Table 4: Comparison of MemeTAG with several state-of-the-art LLMs for Harmful Meme classification on Harmeme dataset

Metric	Without Aux Loss	With Aux Loss
Accuracy (%)	84.419	85.540
AUROC (%)	91.910	92.230
F1 Score (%)	83.575	84.810

Table 5: Impact of the auxiliary tag-embedding reconstruction loss on MemeTAG’s performance on the HarMeme dataset

Harmful Meme Detection

See all 23 papers

Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classification
Sep 22, 2026Akshit Sharma, Prashant W. PatilMultimodal Robustness Cross-Modal Alignment
CVPR Lab, MFSDSAI Indian Institute of Technology Guwahati
GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection
Feb 24, 2026Yingying Guo, Ke Zhang, Zirong ZengHarmful Meme Detection Hate Speech Detection
The Chinese University of Hong Kong, Shenzhen
MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection
Oct 11, 2026Hexiang Gu, Saihui Hou, Changlan Liu +4Retrieval-Augmented Reasoning Harmful Meme Detection
Beijing University of Posts and Telecommunications, Beijing 100876, China · the School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · the College of Electrical Engineering, Zhejiang University, Zhejiang 310027, China +2

Sep 22, 2026J/K move · Enter open · S save

Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classification

Akshit Sharma, Prashant W. Patil

CVPR Lab, MFSDSAI Indian Institute of Technology Guwahati

Multimodal Robustness Cross-Modal Alignment Multimodal Classification Harmful Meme Detection

Open paper AskPDF arXiv

Memes often derive their harmful, hateful, or sarcastic meaning from small but decisive visual, textual, or cross-modal cues. Existing multimodal classifiers can miss such evidence when relying mainly on global image-text representations. We introduce MemeCF, a cue-focused benchmark of 9,895 memes across harm, hate, and sarcasm, with annotations identifying the modality and rationale of the pivotal evidence. We also propose MemePIVOT, a local-global architecture for meme classification. MemePIVOT uses frozen CLIP features, unbalanced optimal transport to align words with image patches while allowing irrelevant evidence to remain unmatched, and an evidential fusion head to combine local grounding with global meme context under uncertainty. Experiments on HarMeme, PrideMM, and MemeCF show consistent gains over strong text-only, image-only, multimodal, and vision-language baselines. Cross-dataset and ablation results further show that explicit pivotal-evidence modeling improves robustness and contributes meaningfully beyond global multimodal representations. Our code and dataset are publicly available at https://github.com/AkshitSharma1/MemePIVOT

Abstract

Harmful Meme Detection

Explore similar work

Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classification

GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection

MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection