MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection

Hexiang Gu; Saihui Hou; Changlan Liu; Min Ren; Huijia Wu; Hongxun Gu; Zhaofeng He

cs.AIOct 11, 2026

MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection

Authors: Hexiang Gu, Saihui Hou, Changlan Liu, Min Ren, Huijia Wu, Hongxun Gu, Zhaofeng He

Organizations: Beijing University of Posts and Telecommunications, Beijing 100876, China · the School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · the College of Electrical Engineering, Zhejiang University, Zhejiang 310027, China · Zhongguancun Academy, Beijing, China ·

Abstract

With the rapid proliferation of social media, harmful memes have become a growing threat to online ecosystems. Their harmfulness is often implicitly conveyed through complex interactions among visual, textual, and contextual cues, making accurate detection challenging. Recent studies have introduced Retrieval-Augmented Generation (RAG) and external knowledge into harmful meme detection. However, existing methods mainly retrieve similar instances or auxiliary information, limiting their ability to capture reusable harmfulness knowledge and underlying reasoning mechanisms, and thus constraining both detection performance and interpretability. To address these challenges, we propose MemeLogos, a knowledge-driven retrieval-augmented reasoning framework for harmful meme detection. MemeLogos abstracts recurring harmfulness patterns and reasoning structures from diverse harmful cases into reusable knowledge units and organizes them into a dedicated star-shaped knowledge graph. By combining structured knowledge retrieval with an explicit multi-stage reasoning process inspired by cognitive mechanisms, MemeLogos leverages retrieved knowledge to guide harmfulness reasoning, improving both detection reliability and interpretability. Extensive experiments show that MemeLogos consistently outperforms existing harmful meme detection methods while generating more coherent and interpretable reasoning, demonstrating the effectiveness of structured harmfulness knowledge for multimodal content understanding.

Figures & tables

Fig. 1: Overview of MemeLogos. The framework abstracts reusable harmfulness rules from multimodal meme cases, organizes them into a structured harmfulness knowledge graph, and performs graph-grounded retrieval and cognitive reasoning for accurate and interpretable harmful meme detection.

Nodes									Edges
Type	Case	Rule	Anch.	Rhet.	Targ.	Cons.	Verd.	Total	Relation	Align.	Trig.	Guard	Supp.	Concl.	Total
Count	4477	2541	2195	2056	2002	1829	2541	17641	Count	15706	10164	8084	4477	2541	40972

TABLE I: Statistics of the harmfulness knowledge graph.

Fig. 2: Pipeline of Formalized Rule Extraction. MemeLogos first distills each meme into structured harmfulness primitives, then performs rhetoric-based clustering, verdict-aware separation, and semantic cluster refinement to aggregate recurring patterns, which are finally generalized into reusable harmfulness rule units.

Fig. 3: Overview of Cognitive RAG. System 1 independently interprets the input meme without retrieved knowledge, while System 2 extracts preliminary semantic cues, retrieves adversarial harmful and harmless rules from the knowledge graph, and integrates the initial interpretation with retrieved evidence for knowledge-grounded reasoning and final harmfulness judgment.

	Method	Accuracy	Precision	Recall	F1-score
Single-Modality	CLIP Text-Only [ 12 ]	74.14	75.54	56.92	64.92
Single-Modality	CLIP Image-Only [ 12 ]	80.25	79.47	71.74	75.41
Multi-Modality	CLIP Image+Text [ 12 ]	80.81	80.81	71.06	75.62
	VisualBERT_COCO [ 27 ]	63.21	63.95	63.48	63.72
	Pro-Cap [ 28 ]	75.58	78.09	78.76	78.44
	ISSUES [ 29 ]	82.54	83.09	74.80	78.72

TABLE II: Performance comparison on the MemeMind dataset. We compare MemeLogos with representative baselines. MemeLogos denotes the training-free framework, while MemeLogos_Align further incorporates keyword and reasoning alignment. ⋄ indicates direct evaluation of vision-language models without task-specific fine-tuning. Bold indicates the best result, while underlining indicates the second-best result.

Method	Access (s)	Inference (s)	Total (s)
Rule-RAG	0.076	0.092	0.168
Graph-RAG	0.170	0.107	0.277
Instance-RAG	0.139	0.278	0.417

TABLE III: Computational efficiency of different knowledge representations. Access denotes knowledge retrieval, while inference denotes downstream prediction without full reasoning.

Category	Model Variant	Accuracy	F1
Representation	Instance-RAG	82.96	80.44
	Rule-RAG	82.76	80.41
Variants	w/o Aligned edges	86.13	84.47
	w/o Guards edges	85.32	83.51
Our Method	MemeLogos	86.96	85.22

TABLE IV: Ablation on graph structure.

Variant	Accuracy	F1
w/o Keywords Extraction	83.49	83.13
w/o Adversarial Context	85.11	82.64
w/o Cognitive Reasoning	64.55	39.47
MemeLogos	86.96	85.22

TABLE V: Ablation of components in Cognitive RAG.

Backbone	Variant	Accuracy	F1
Qwen3-VL-8B	Zero-shot	73.24	66.32
Qwen3-VL-8B	MemeLogos	86.96	85.22
InternVL3.5-8B	Zero-shot	69.75	63.81
InternVL3.5-8B	MemeLogos	85.43	84.76
MiniCPM-V-4.5	Zero-shot	75.62	75.67
MiniCPM-V-4.5	MemeLogos	81.95	82.54

TABLE VI: Performance of MemeLogos across different multimodal backbones.

Strategy	Accuracy	F1
MemeLogos	86.96	85.22
Keywords Alignment	87.20	85.68
Reasoning Alignment	88.57	86.88
MemeLogos_Align	89.50	88.03

TABLE VII: Ablation of Training Alignment.

Panel of Figure 11 — Fig. 4: Qualitative illustration of the reasoning process in MemeLogos. The framework extracts key semantic cues, retrieves relevant harmfulness rules, and performs dual-system reasoning to progressively verify the evidence and derive the final harmfulness verdict.

Method	Faith.	Suff.	Rel.
Qwen3-VL-8B	3.73	3.42	—
MemeLogos	4.43	4.19	4.61

TABLE VIII: Human evaluation results on interpretability.

Panel of Figure 13 — Fig. 5: Failure case of MemeLogos. The model captures the surface-level visual juxtaposition but retrieves and over-relies on an inappropriate harmless rule, failing to recognize the implicit sexualized meaning conveyed by the slang expression and consequently producing an incorrect verdict.

In-domain Supervised Evaluation
Method	Accuracy	F1
CLIP	72.39	72.33
MemeCLIP	76.06	75.09
U-CoT+	71.60	71.37
RGCL	76.30	76.50
RA-HMD	78.10	78.40

TABLE IX: Cross-domain zero-shot evaluation of MemeLogos on PrideMM.

Fig. 6: The sensitivity experiments of MemeLogos to the rule-clustering threshold $\tau_{r}$ conducted on a fixed 5,000/200 diagnostic subset. Error bars denote 95% confidence intervals estimated using 5,000 class-stratified bootstrap resamples.

Fig. 7: The sensitivity experiments of MemeLogos to the combinations of the Semantically Aligned edge threshold $\tau_{a}$ and Guards Against edge threshold $\tau_{g}$ conducted on a fixed 5,000/200 diagnostic subset.

Rules per Polarity ( $k$ )	Accuracy	F1
1	86.65	84.93
3	86.96	85.22
5	86.62	84.91

TABLE X: Sensitivity analysis of the number of retrieved rules for each polarity.

Hexiang Gu received the B.E. degree from Beijing University of Posts and Telecommunications, Beijing, China, in 2025. He is currently pursuing the Ph.D. degree at the School of Artificial Intelligence, Beijing University of Posts and Telecommunications. His research interests include multimodal large language model, and multimodal content safety.

Table 18

Saihui Hou received the B.E. and Ph.D. degrees from the University of Science and Technology of China in 2014 and 2019, respectively. He is currently an Associate Professor at the School of Artificial Intelligence, Beijing Normal University. His research interests include computer vision and machine learning, with a focus on gait recognition, which aims to identify individuals based on their walking patterns.

Table 19

Changlan Liu received the B.S. degree from Northeast Agricultural University in 2026. He is currently pursuing the direct Ph.D. degree with the College of Electrical Engineering, Zhejiang University. His research interests include LLM security.

Table 20

Min Ren received the B.E. and Ph.D. degrees from the National University of Defense Technology and University of Chinese Academy of Sciences in 2013 and 2023, respectively. He is currently an Associate Professor at the School of Artificial Intelligence, Beijing University of Posts and Telecommunications. His research interests include embodied intelligence and computer vision.

Table 21

Huijia Wu received the B.Sc. degree in e-commerce from Dalian Maritime University, China in 2007, the M.Sc. degree in e-commerce from Shanghai University of Finance and Economics, China in 2011, and the Ph.D. degree from the Institute of Automation, Chinese Academy of Sciences (CASIA), China. He worked as a Senior Researcher at the WeChat Pattern Recognition Center and as an Advertising Algorithm Expert at Alibaba’s Alimama, with 8 years of experience and achievements in both academia and industry. He has published over 10 papers in leading international conferences such as ACL, AAAI, EMNLP, and COLING. His research interests include natural language processing, personalized recommendation systems and machine learning.

Table 22

Hongxun Gu received the BSc and MSc degrees from Huazhong University of Science and Technology, Wuhan, China, in 1994 and 1997 respectively, and the MSc degree in Management from The Hong Kong Polytechnic University in 2016. He is currently the General Manager of the Network and Information Security Management Department in China Telecommunications Corporation. He is an Adjunct Professor at Huazhong University of Science and Technology. He is a Board Member of the China Cyber Security Society, a Fellow of the China Institute of Communications. His research interests cover cyberspace security, artificial intelligence security, and related fields.

Table 23

Zhaofeng He received the B.S. degree in Electronic Engineering and Information Science from the University of Science and Technology of China, Hefei, China, in 2005, and the Ph.D. degree in Computer Applied Technology from the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 2010. He is currently a Professor at the School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing. His research interests include computer vision, biometrics, and intelligent decision-making in games.

Table 24

Retrieval-Augmented Reasoning

See all 27 papers

SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection
Sep 12, 2026Hanling Wang, Chenlong Wei, Yingjuan Li +4Harmful Meme Detection Multimodal Content Moderation
University of California, San Diego · Xi’an Jiaotong-Liverpool University · University of Washington +2
PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Framework
May 1, 2026Zihan Ding, Ziyuan Yang, Yi ZhangInterpretable ML Harmful Meme Detection
Sichuan University, Chengdu, Sichuan, China · Nanyang Technological University, Singapore
Distinguishing Right from Wrong in Debates: Attribution Analysis of Chinese Harmful Memes
May 23, 2026Weiming Wang, Junyu Lu, Han Wang +5Hate Speech Detection Harmful Meme Detection
Key Laboratory of Social Computing and Cognitive Intelligence, Dalian University of Technology, Dalian, China

Sep 12, 2026J/K move · Enter open · S save

SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection

Hanling Wang, Chenlong Wei, Yingjuan Li, Di Wu+3

University of California, San Diego · Xi’an Jiaotong-Liverpool University · University of Washington+2

Harmful Meme Detection Multimodal Content Moderation Multimodal Reasoning

Open paper AskPDF arXiv

Detecting harmful memes is critical for maintaining safe online communities. However, harmful intent is often implicit, arising from visual-textual incongruity and cultural stereotypes, which challenges existing multimodal detectors. We propose SyRHM, a framework that decomposes harmful meme detection into meaning-grounded retrieval and symbolic-language-enhanced multi-stage reasoning. SyRHM retrieves semantically related memes by parsing multimodal content into textual elements and descriptions, providing grounded context beyond surface-level similarity. Building on the retrieved context, SyRHM uses a translator stage to convert multimodal inputs into symbolic intermediate representations, and then performs multi-stage reasoning via planner and solver stages, enabling expressive and interpretable analysis of harmful intent. Experiments on FHM, HarM, and MultiOff demonstrate the effectiveness of SyRHM, achieving superior performance on most evaluation settings against multimodal and reasoning-based baselines, while providing reasoning traces for harmful content. The code is available at: https://github.com/Scabbards1500/SyRHM

Abstract

Retrieval-Augmented Reasoning

Explore similar work

SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection

PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Framework

Distinguishing Right from Wrong in Debates: Attribution Analysis of Chinese Harmful Memes