Retrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detection

Raihan Tanvir; Md. Golam Rabiul Alam

cs.CLFeb 22, 2026

Retrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detection

Authors: Raihan Tanvir, Md. Golam Rabiul Alam

Organizations: Department of Computer Science and Engineering, BRAC University, Dhaka 1212, Bangladesh · Department of Computer Science and Engineering, Ahsanullah University of Science and Technology, Dhaka 1208, Bangladesh

Abstract

Hateful content on social media increasingly appears as multimodal memes that combine images and text to convey harmful narratives. In low-resource languages such as Bengali, automated detection remains challenging due to limited annotated data, class imbalance, and pervasive code-mixing. To address these issues, we augment the Bengali Hateful Memes (BHM) dataset with semantically aligned samples from the Multimodal Aggression Dataset in Bengali (MIMOSA), improving both class balance and semantic diversity. We propose the Enhanced Dual Co-attention Framework (xDORA), integrating vision encoders (CLIP, DINOv2) and multilingual text encoders (XGLM, XLM-R) via weighted attention pooling to learn robust cross-modal representations. Building on these embeddings, we develop a FAISS-based k-nearest neighbor classifier for non-parametric inference and introduce RAG-Fused DORA, which incorporates retrieval-driven contextual reasoning. We further evaluate LLaVA under zero-shot, few-shot, and retrieval-augmented prompting settings. Experiments on the extended dataset show that xDORA (CLIP + XLM-R) achieves macro-average F1-scores of 0.78 for hateful meme identification and 0.71 for target entity detection, while RAG-Fused DORA improves performance to 0.79 and 0.74, yielding gains over the DORA baseline. The FAISS-based classifier performs competitively and demonstrates robustness for rare classes through semantic similarity modeling. In contrast, LLaVA exhibits limited effectiveness in few-shot settings, with only modest improvements under retrieval augmentation, highlighting constraints of pretrained vision-language models for code-mixed Bengali content without fine-tuning. These findings demonstrate the effectiveness of supervised, retrieval-augmented, and non-parametric multimodal frameworks for addressing linguistic and cultural complexities in low-resource hate speech detection.

Figures & tables

\fontspec_if_language:nTF ENG \addfontfeature Language=English Figure 1 : A sample meme from the Bengali Hateful Memes dataset, showcasing nuanced visual and textual elements that convey hateful sentiment.

	Class	Train	Valid	Test	Total
Task 1	Hate	2117	241	266	2624
Task 1	Non-Hate	3641	399	445	4485
Task 2	TI	1623	192	193	2008
	TO	160	17	27	204
	TC	249	24	37	310
	TS	85	8	9	102

\fontspec_if_language:nTF ENG \addfontfeature Language=English Table I : Number of memes in training, validation, and testing splits for the BHM dataset.

	Label	$\kappa$ -score	Average
Task 1	Hate	0.82	0.79
Task 1	Non-Hate	0.76	0.79
Task 2	TI	0.68	0.63
	TO	0.66
	TC	0.61
	TS	0.57

\fontspec_if_language:nTF ENG \addfontfeature Language=English Table II : Cohen’s

\kappa

agreement scores for BHM dataset annotations.

	Class	Train	Valid	Test	Total
Task 1	Hate	3885	486	486	4857
Task 1	Non-Hate	3588	449	448	4485
Task 2	TI	2026	253	253	2532
	TC	981	123	122	1226
	TO	786	98	98	982
	TS	94	12	11	117

\fontspec_if_language:nTF ENG \addfontfeature Language=English Table III : Number of memes in the extended BHM dataset.

\fontspec_if_language:nTF ENG \addfontfeature Language=English Figure 2 : Overview of the xDORA architecture. Visual and textual features are extracted using pretrained encoders. A dual co-attention module with multi-head attention integrates the modalities through two cross-attention flows: I2T-ACT (Image-to-Text Attention Conditioned on Text, Equation ( \fontspec_if_language:nTFENG\addfontfeatureLanguage=English3a )) and I2I-ACT (Image-to-Image Attention Conditioned on Text, Equation ( \fontspec_if_language:nTFENG\addfontfeatureLanguage=English3b )). The fused representation is aggregated via weighted attention pooling and fed to a multilayer perceptron for classification.

\fontspec_if_language:nTF ENG \addfontfeature Language=English Figure 3 : Architecture of RAG-Fused DORA. A multimodal embedding $\mathbf{Z}$ is obtained using xDORA as a feature extractor. This embedding is used in two parallel branches: (1) it is passed through an MLP classifier to produce prediction scores; and (2) it is used to retrieve the top- $k$ nearest neighbors from a FAISS index based on cosine similarity ( $\text{Top-}k[\cos(\mathbf{Z},\mathbf{Z}_{\text{db}})]$ ). The retrieved labels are aggregated using similarity-weighted label distribution. Final predictions are computed via a weighted ensemble of the classifier logits and the FAISS-derived label distribution.

\fontspec_if_language:nTF ENG \addfontfeature Language=English Figure 4 : Illustration of the RAG-Prompted LLaVA framework. Given a test image, top- $k$ semantically similar exemplars per class are retrieved from a FAISS index using xDORA-generated embeddings. The retrieved text-label pairs are formatted as few-shot exemplars and incorporated into a prompt alongside the test caption. This retrieval-augmented prompt is then fed to LLaVA, enabling few-shot multimodal classification.

Category	Model	Hateful Meme Identification (Task 1)			Target Entity Detection (Task 2)
		P	R	F1	P	R	F1
Vision Only	CLIP (ViT-B/32)	0.50 $\pm$ 0.018	0.50 $\pm$ 0.015	0.48 $\pm$ 0.015	0.41 $\pm$ 0.015	0.44 $\pm$ 0.018	0.42 $\pm$ 0.015
	DINOv2-Base	0.65 $\pm$ 0.018	0.65 $\pm$ 0.015	0.65 $\pm$ 0.018	0.48 $\pm$ 0.023	0.51 $\pm$ 0.038	0.48 $\pm$ 0.025
	ConvNeXT-Base	0.63 $\pm$ 0.018	0.62 $\pm$ 0.015	0.61 $\pm$ 0.015	0.44 $\pm$ 0.018	0.46 $\pm$ 0.018	0.45 $\pm$ 0.015
Text Only	XGLM-564M	0.75 $\pm$ 0.013	0.74 $\pm$ 0.015	0.74 $\pm$ 0.015	0.55 $\pm$ 0.018	0.61 $\pm$ 0.038	0.56 $\pm$ 0.023
	XLM-R-L	0.73 $\pm$ 0.013	0.73 $\pm$ 0.013	0.73 $\pm$ 0.013	0.65 $\pm$ 0.023	0.77 $\pm$ 0.030	0.69 $\pm$ 0.025

\fontspec_if_language:nTF ENG \addfontfeature Language=English Table IV : Performance comparison of baseline and proposed models on the test set using macro-average metrics. Models are categorized as Vision Only, Text Only, Multimodal, LVLM, Proposed Supervised Framework Variants, and Proposed Retrieval Augmented Methods. Metrics include macro-average precision (P), recall (R), and F1-score (F1) for Task 1 (hateful meme identification) and Task 2 (target entity detection), reported as mean

\pm

95% confidence interval half-width . Highest and second-highest F1-scores are in bold and underlined , respectively.

Model	Class	P	R	F1	Support
CLIP + XLM-R-L	TI	0.83	0.87	0.85	254
	TC	0.76	0.76	0.76	122
	TO	0.84	0.71	0.77	99
	TS	0.38	0.55	0.44	11
FAISS-Based k-NN	TI	0.80	0.89	0.84	254
	TC	0.81	0.70	0.75	122

\fontspec_if_language:nTF ENG \addfontfeature Language=English Table V : Class-wise performance of xDORA ( \fontspec_if_language:nTF ENG \addfontfeature Language=EnglishCLIP + XLM-R-L ), FAISS-Based k-NN, and RAG-Fused DORA for Task 2. Metrics include precision (P), recall (R), F1-score (F1), and support. Highest F1-score per class is in bold .

	(a)	(b)	(c)	(d)
Meme
Caption	Take me to somewhere expensive BRAC UNIVERSITY	\bangla জন্নাতে যখন আপনি ৭২ হুর দেখতে পান… #ধর্ম পচারক হুর হুর দাবাং দাবাং দাবাং…	\bangla মানুষের হো*য় আঙুল দেওয়ার জন্যই আমার জন্ম	\bangla Don’t think yourself dumb bro There are literally people যারা পরিক্ষার হলে রচনা দেখাতে বলে ,,
Translated Caption	Take me to somewhere expensive BRAC UNIVERSITY	When you see 72 houris in heaven… #ReligiousPreacher Hoor hoor dabangg dabangg dabangg…	I was born just to poke fingers at people’s a*se.	Don’t think yourself dumb, bro. There are literally people who ask to copy essays during exams…
Vision Only	TI	TI	TC	TI
Text Only	TS	TC	TC	TI
Multimodal	TO	TC	TI	TI

\fontspec_if_language:nTF ENG \addfontfeature Language=English Table VI : Sample evaluations of memes from the test set for Task 2 (multiclass classification). The table includes the meme image, OCR-extracted caption, translations, predictions from baseline models, proposed framework variants, and the ground truth. Correct predictions are shown in green and incorrect predictions in red .

Hate Speech Detection

Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.

See all 98 papers

Feb 25, 2026

Detecting Hate and Inflammatory Content in Bengali Memes: A New Multimodal Dataset and Co-Attention Framework

Internet memes have become a dominant form of expression on social media, including within the Bengali speaking community. While often humorous, memes can also be exploited to spread offensive, harmful, and inflammatory content targeting individuals and groups. Detecting this type of content is exceptionally challenging due to its satirical, subtle, and culturally specific nature. This problem is magnified for low-resource languages like Bengali, as existing research predominantly focuses on high-resource languages. To address this critical research gap, we introduce Bn-HIB (Bangla Hate Inflammatory Benign), a novel dataset containing 3,247 manually annotated Bengali memes categorized as Benign, Hate, or Inflammatory. Significantly, Bn- HIB is the first dataset to distinguish inflammatory content from direct hate speech in Bengali memes. Furthermore, we propose the MCFM (Multi-Modal Co-Attention Fusion Model), a simple yet effective architecture that mutually analyses both the visual and textual elements of a meme. MCFM employs a co-attention mechanism to identify and fuse the most critical features from each modality, leading to a more accurate classification. Our experiments show that MCFM significantly outperforms several state-of-the-art models on the Bn-HIB dataset, demonstrating its effectiveness in this nuanced task. To facilitate reproducibility and future research, the Bn-HIB dataset has been made publicly available through Mendeley Data. Warning: This work contains material that may be disturbing to some audience members. Viewer discretion is advised

Rakib Ullah, Mominul islam, Md Sanjid Hossain +1

Sylhet Engineering College · Computer Science And Engineering, Sylhet Engineering College, Sylhet, Bangladesh · Daffodil International University +1

Harmful Meme Detection Hate Speech Detection Toxicity Detection Multimodal Classification Low-Resource Language Processing

Ask PDF

Apr 13, 2026

MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes

Hate speech detection in Devanagari-scripted social media memes presents compounded challenges: multimodal content structure, script-specific linguistic complexity, and extreme data scarcity in low-resource settings. This paper presents our system for the CHiPSAL 2026 shared task, addressing both Subtask A (binary hate speech detection) and Subtask B (three-class sentiment classification: positive, neutral, negative). We propose a hybrid cross-modal attention fusion architecture that combines CLIP (ViT-B/32) for visual encoding with BGE-M3 for multilingual text representation, connected through 4-head self-attention and a learnable gating network that dynamically weights modality contributions on a per-sample basis. Systematic evaluation across eight model configurations demonstrates that explicit cross-modal reasoning achieves a 5.9% F1-macro improvement over text-only baselines on Subtask A, while uncovering two unexpected but critical findings: English-centric vision models exhibit near-random performance on Devanagari script, and standard ensemble methods catastrophically degrade under data scarcity (N nearly equal to 850 per fold) due to correlated overfitting. The code can be accessed at https://github.com/Tri-Yantra-Technologies/MEME-Fusion/

Samir Wagle, Reewaj Khanal, Abiral Adhikari

Department of Computer Science and Engineering, Kathmandu University, Nepal

Hate Speech Detection Multimodal Sentiment Analysis Multimodal Fusion Low-Resource Language Processing

Ask PDF

Jul 29, 2026

AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes

Hateful memes are a growing form of multimodal online harm, where hostile intent is often conveyed through the joint interpretation of images, text, cultural references, and implicit targets. While hateful meme detection has advanced in high-resource languages, Arabic remains underexplored, with existing meme resources focusing mainly on propaganda or coarse harmful-content labels. We introduce AHA-Memes (Arabic HAteful Memes), which is, to our knowledge, the first large-scale Arabic hateful meme benchmark with fine-grained, multi-label annotations. The dataset includes 5K manually annotated memes using a taxonomy that captures hate types, i.e., attack strategies. We further provide ~66K silver-labeled memes to support future studies. We benchmark text-only, image-only, and late-fusion multimodal models, as well as few-shot in-context learning (ICL) and open- and closed-weight Vision-Language Models (VLMs) under zero-shot and fine-tuning settings. Our results establish strong baselines and highlight key challenges in culturally grounded Arabic hateful meme detection. We release the dataset, annotation guidelines, and evaluation scripts to support future research. WARNING: This paper contains examples that may be disturbing to readers.

Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Abul Hasnat +3

Qatar Computing Research Institute, Qatar · 2APAVI.AI, France · 3Hamad Bin Khalifa University, Qatar +1

Arabic NLP Hate Speech Detection Multimodal Model Evaluation Harmful Meme Detection

Ask PDF

Abstract

Hate Speech Detection

Explore similar work

Detecting Hate and Inflammatory Content in Bengali Memes: A New Multimodal Dataset and Co-Attention Framework

MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes

AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes