SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection

Hanling Wang; Chenlong Wei; Yingjuan Li; Di Wu; Yuchao Zhang; Xiaohui Zhu; Yao Zhu

cs.CLSep 12, 2026

SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection

Authors: Hanling Wang, Chenlong Wei, Yingjuan Li, Di Wu, Yuchao Zhang, Xiaohui Zhu, Yao Zhu

Organizations: University of California, San Diego · Xi’an Jiaotong-Liverpool University · University of Washington · The Australian National University ·

Abstract

Detecting harmful memes is critical for maintaining safe online communities. However, harmful intent is often implicit, arising from visual-textual incongruity and cultural stereotypes, which challenges existing multimodal detectors. We propose SyRHM, a framework that decomposes harmful meme detection into meaning-grounded retrieval and symbolic-language-enhanced multi-stage reasoning. SyRHM retrieves semantically related memes by parsing multimodal content into textual elements and descriptions, providing grounded context beyond surface-level similarity. Building on the retrieved context, SyRHM uses a translator stage to convert multimodal inputs into symbolic intermediate representations, and then performs multi-stage reasoning via planner and solver stages, enabling expressive and interpretable analysis of harmful intent. Experiments on FHM, HarM, and MultiOff demonstrate the effectiveness of SyRHM, achieving superior performance on most evaluation settings against multimodal and reasoning-based baselines, while providing reasoning traces for harmful content. The code is available at: https://github.com/Scabbards1500/SyRHM

Figures & tables

Figure 1: Similar memes can convey different meanings: the left examples are neutral, while the right example conveys harmful stereotypes targeting individuals with Down syndrome.

Figure 3: Visualization of the Parser step

Figure 4: Illustration of the Translator process. Visualizations of the other reasoning steps are provided in the Appendix.

Open-source LMM (Zero-shot)
Method	FHM			MultiOff
	ACC	BACC	MCC	ACC	BACC	MCC
LLaVA-1.5-13B	55.01	55.13	10.65	48.15	56.23	14.41
Qwen2.5-VL-7B	63.20	63.01	26.53	64.43	59.94	21.70
LLaVA-v1.6-Vicuna-13B	56.90	56.66	13.72	61.07	65.94	33.46
LLaVA-OneVision-Qwen2-7B	56.60	56.25	13.37	59.73	60.16	19.81

Table 1: Results on the binary classification datasets FHM and MultiOff.

Figure 5: Qualitative examples of SyRHM and the baseline. Additional cases are provided in the Appendix.

Table 2: Performance comparison on HarM datasets: (a) HarM-c with binary harmful classification, harmfulness level, and target type tasks; (b) HarM-p with the same tasks.

Figure 6: Performance variation with the number of retrieved memes $k$ on HarM.

Setting ID	$\lambda_{0}$	$\lambda_{1}$	$\lambda_{2}$	ACC
S1	1.0	0.0	0.0	67.80
S2	0.0	1.0	0.0	64.97
S3	0.0	0.0	1.0	68.08
S4	0.5	0.5	0.0	68.93
S5	0.5	0.0	0.5	69.77
S6	0.0	0.5	0.5	71.75

Table 3: Ablation of modality weights for associative retrieval.

\lambda_{0}

\lambda_{1}

, and

\lambda_{2}

denote the weights of image embeddings, symbolic elements, and generated descriptions.

Figure 7: Ablation study: impact of removing individual reasoning stages on model accuracy.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Train Samples	Test Samples
FHM	8500	1000
MultiOff	445	149
HarM-c	3013	354
HarM-p	2938	355

Appendix

Table 4: Dataset statistics (train/test) used in our experiments.

Dataset	Task 1		Task 2			Task 3
Dataset	0	1	0	1	2	0	1	2	3	4
FHM	510	490	–	–	–	–	–	–	–	–
MultiOff	91	58	–	–	–	–	–	–	–	–
HarM-c	230	124	230	21	103	230	119	0	3	2
HarM-p	184	171	184	159	12	184	81	69	15	6

Appendix

Table 5: Label distribution for each dataset across the three tasks.

Dataset	Output Schema	Label Mapping
FHM	{"is_hateful": <boolean>}	0 not harmful, 1 harmful
MultiOff	{"is_offensive": <boolean>}	0 not harmful, 1 harmful
HarM	{"label": [Harm_Level, Target_Type]}	Harm Level: 0 not harmful, 1 somewhat harmful, 2 very harmful; Target Type: 0 none, 1 individual, 2 organization, 3 community, 4 society.

Appendix

Table 6: Schema-constrained output formats and label mappings for different datasets.

Figure 8: Definition and instructions for FHM.

Figure 9: Definition and instructions for MultiOff.

Figure 10: Definition and instructions for HarM.

Item	Setting
Retrieval setting	Top- $k{=}3$ ; $\lambda_{0}{:}\lambda_{1}{:}\lambda_{2}{=}0.34{:}0.33{:}0.33$ for image, elements, and descriptions.
Decoding setting	Context length 4096, temperature 0.2, top- $p{=}1$ .
Repetition	Three runs with fixed random seeds; mean performance is reported.
Main packages	PyTorch, Transformers, scikit-learn
Hardware	2 NVIDIA 4090 GPUs

Appendix

Table 7: Implementation details of SyRHM.

Figure 11: Illustration of the Planner Stage

Method		Harmful or Not			Harmfulness Level	Target type
VLM	LLM	ACC	BACC	MCC	M-F1	M-F1
Qwen2.5-VL-7B	–	67.23	53.97	18.23	35.92	3.82
Qwen2.5-VL-7B	Qwen2.5-7B	67.80	67.78	34.13	47.68	11.02
Qwen2.5-VL-7B	Qwen2.5-14B	70.34	71.78	41.59	51.90	15.94
Qwen2.5-VL-7B	Qwen2.5-32B	68.08	70.79	39.73	46.98	16.01
InternVL3.5-8B	–	64.41	69.82	38.99	42.86	12.25

Appendix

Table 8: Effect of LLM scaling across different backbones on HarM.

Figure 12: Illustration of the Solver Stage

Retrieval Embedding Model	ACC	BACC	MCC
Without Retrieval	68.93	69.21	36.83
CLIP-ViT-B/32	70.34	71.78	41.59
CLIP-ViT-L/14	69.38	71.53	42.06
SigLIP-BASE-PATCH16	68.80	72.99	44.86

Appendix

Table 9: Effect of retrieval embedding models on HarM.

Dataset	Task1 ACC	Task2 M-F1	Task3 M-F1
FHM	68.93	51.57	12.98
MultiOff	75.14	50.89	14.55
HarM-c	70.34	51.90	15.94
HarM-p	70.34	55.22	12.64

Appendix

Table 10: Cross-dataset retrieval results on three tasks.

Stage	Input	Output	Total	Time(s)
Parser	2,138.5	246.9	2,385.4	3.50
Insertion	–	–	–	0.04
Retrieval	–	–	–	0.05
Translator	2,743.1	796.0	3,539.2	34.97
Planner	1,938.0	337.3	2,275.3	15.30
Solver	2,673.1	308.7	2,981.7	7.68

Appendix

Table 11: Average inference cost of SyRHM on HarM. We report per-sample token usage and wall-clock time for each stage.

Figure 13: Process Visualization of 01634 from FHM. The meme depicts a man shearing a sheep, which may be misinterpreted as violent due to its visual content. While the baseline model incorrectly predicts it as harmful, SyRHM leverages retrieved humorous contexts and symbolic reasoning to identify the meme as non-harmful.

Figure 14: Process Visualization of covid_memes_5560 from HarM. The meme criticizes delayed COVID-19 test kit delivery by Trump’s administration. While the baseline model fails to detect harm, SyRHM identifies the harmful intent by leveraging retrieved contexts and analyzing sarcasm and blame-oriented framing.

Harmful Meme Detection

See all 23 papers

MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection
Oct 11, 2026Hexiang Gu, Saihui Hou, Changlan Liu +4Retrieval-Augmented Reasoning Harmful Meme Detection
Beijing University of Posts and Telecommunications, Beijing 100876, China · the School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · the College of Electrical Engineering, Zhejiang University, Zhejiang 310027, China +2
PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Framework
May 1, 2026Zihan Ding, Ziyuan Yang, Yi ZhangInterpretable ML Harmful Meme Detection
Sichuan University, Chengdu, Sichuan, China · Nanyang Technological University, Singapore
Can Thinking Models Think to Detect Hateful Memes?
Mar 1, 2026Mohamed Bayan Kmainasi, Mucahid Kutlu, Ali Ezzat Shahroor +2Harmful Meme Detection Multimodal Large Language Models
Qatar University Doha, Qatar · Qatar Computing Research Institute Doha, Qatar · APAVI.AI France +1

Oct 11, 2026J/K move · Enter open · S save

MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection

Hexiang Gu, Saihui Hou, Changlan Liu, Min Ren+3

Beijing University of Posts and Telecommunications, Beijing 100876, China · the School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · the College of Electrical Engineering, Zhejiang University, Zhejiang 310027, China+2

Retrieval-Augmented Reasoning Harmful Meme Detection Multimodal Reasoning

Open paper AskPDF arXiv

With the rapid proliferation of social media, harmful memes have become a growing threat to online ecosystems. Their harmfulness is often implicitly conveyed through complex interactions among visual, textual, and contextual cues, making accurate detection challenging. Recent studies have introduced Retrieval-Augmented Generation (RAG) and external knowledge into harmful meme detection. However, existing methods mainly retrieve similar instances or auxiliary information, limiting their ability to capture reusable harmfulness knowledge and underlying reasoning mechanisms, and thus constraining both detection performance and interpretability. To address these challenges, we propose MemeLogos, a knowledge-driven retrieval-augmented reasoning framework for harmful meme detection. MemeLogos abstracts recurring harmfulness patterns and reasoning structures from diverse harmful cases into reusable knowledge units and organizes them into a dedicated star-shaped knowledge graph. By combining structured knowledge retrieval with an explicit multi-stage reasoning process inspired by cognitive mechanisms, MemeLogos leverages retrieved knowledge to guide harmfulness reasoning, improving both detection reliability and interpretability. Extensive experiments show that MemeLogos consistently outperforms existing harmful meme detection methods while generating more coherent and interpretable reasoning, demonstrating the effectiveness of structured harmfulness knowledge for multimodal content understanding.

Abstract

Harmful Meme Detection

Explore similar work

MemeLogos: Knowledge-Driven Retrieval-Augmented Reasoning for Harmful Meme Detection

PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Framework

Can Thinking Models Think to Detect Hateful Memes?