HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes

Yujia Li; Yiqun Zhang; Zihan Cheng; Yijie Huang; Tenglong Ye; Zihan Wang; Xiaocui Yang; Shi Feng

cs.CVAug 17, 2026

HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes

Authors: Yujia Li, Yiqun Zhang, Zihan Cheng, Yijie Huang, Tenglong Ye, Zihan Wang, Xiaocui Yang, Shi Feng,

Organizations: School of Computer Science and Engineering, Northeastern University Shenyang 110819, China · Apple Inc., Beijing 100006, China

Abstract

Multimodal harmful meme detection is typically formulated as image--text harmfulness classification. A model may correctly predict harmfulness while misidentifying the attacked target or its supporting evidence. We therefore extend harmful meme detection with fine-grained target identification, asking what type of target is attacked, who is targeted, and where the target appears in the meme. The model predicts harmfulness for every meme and, for harmful memes, outputs the target category, target entity, textual mention, and visual region. To support this task, we introduce Meme3W, which unifies multiple public harmful meme datasets and provides human-verified annotations for harmful instances. We further introduce Joint Record Accuracy (JRA), a strict record-level metric requiring the harmfulness label and all target-identification fields to be jointly correct. Experiments with representative multimodal large language models reveal a substantial gap between harmfulness accuracy and JRA. To narrow this gap, we propose HarmTrace, an anchor-calibrated decoupled optimization framework. HarmTrace strengthens target-entity supervision through entity-aware supervised fine-tuning. It then applies Conditional Target-identification Policy Optimization (CTPO) to decouple harmfulness and target-identification advantages, restricting target-identification optimization to label-correct responses for harmful examples. CTPO uses a Virtual Positive Anchor (VPA) as a fully correct reference for target-identification advantage normalization. HarmTrace improves both JRA and harmfulness accuracy across the evaluated backbones, with JRA on the Qwen3-VL-8B backbone increasing from 17.58% to 52.51%. Our code is publicly available at https://github.com/llly1234/HarmTrace-for-Harmful-Memes.

Figures & tables

Figure 1: JRA requires the harmfulness label and all target-identification fields to be jointly correct. Under this criterion, representative MLLMs show a clear gap between harmfulness accuracy and fine-grained target identification.

Figure 2: A representative Meme3W annotation, showing the analysis process and a final structured output for fine-grained target identification, answering where the target appears, who is targeted, and which target category applies.

Figure 3: Overview of HarmTrace. The left panel shows an input and its required output schema. E-SFT upweights target-entity tokens, while CTPO normalizes label rewards over all rollouts and target-identification rewards over the label-correct harmful subset $\mathcal{C}$ . The bottom panel shows VPA for unequal and tied rewards among incomplete responses.

Closed-source MLLMs
Model	Label		JRA	Category	Entity	Text Mention		Visual Region
Model	Acc.	F1	JRA	EM	$\mathrm{F1}_{\mathrm{rel}}$	EM	F1	IoU50	IoU75
Gemini3-Flash	74.88	73.66	47.95	79.91	73.29	61.87	63.01	65.98	59.59
GPT-5.2	75.92	69.01	28.31	62.56	60.73	43.38	46.80	38.58	33.33
GLM-5V-Turbo	73.38	72.28	44.75	74.89	68.26	60.96	62.56	62.79	57.31
Open-source Large MLLMs

Table 1: Main results of different models on Meme3W. Best results are bolded, and second-best results are underlined.

Setting	Label		JRA	Category	Entity	Text Mention		Visual Region
Setting	Acc.	F1	JRA	EM	$\mathrm{F1}_{\mathrm{rel}}$	EM	F1	IoU50	IoU75
Qwen3-VL-8B	73.75	69.03	17.58	60.05	46.35	34.02	37.67	53.20	39.95
+SFT	77.80	73.00	43.15	65.98	61.42	52.05	56.42	57.53	53.88
+E-SFT ( $\lambda_{\mathrm{ent}}=5$ )	77.33	72.95	44.75	67.81	62.79	55.94	59.92	58.90	54.34
+E-SFT ( $\lambda_{\mathrm{ent}}=10$ )	78.65	74.26	45.89	67.58	63.70	54.79	58.67	61.19	57.31
+E-SFT ( $\lambda_{\mathrm{ent}}=15$ )	77.14	72.35	44.98	66.67	64.16	55.02	58.67	57.53	54.11

Table 2: Ablation study of SFT and E-SFT with different entity-token weights

\lambda_{\mathrm{ent}}

Init.	RL	Label		JRA
Init.	RL	Acc.	F1	JRA
w/o SFT	GRPO	74.51	71.52	18.72
	+Dec.	74.41	71.43	20.32
	+VPA	75.16	71.80	22.37
+SFT	GRPO	77.99	74.89	45.21
	+Dec.	77.42	73.74	47.26

Table 3: RL-stage ablation of HarmTrace under different starting points. w/o SFT denotes initializing RL from the base model; +Dec. adds decoupled reward optimization; and +VPA further adds VPA to +Dec.

Figure 4: Target-identification results under E-SFT initialization. The left panel shows GRPO scores, and the right shows gains from decoupled optimization and VPA over GRPO.

Figure 5: Effect of VPA under E-SFT initialization. The left panel shows the zero-advantage group ratio, while the right tracks positive advantage of the best incomplete responses.

Method	Label		JRA
Method	Acc.	F1	JRA
PPO	77.61	72.33	46.80
GRPO	78.74	74.49	49.09
DAPO	78.08	72.94	47.03
PAPO	78.46	74.61	48.40
HarmTrace	80.15	76.77	52.51

Table 4: Performance comparison of different RL methods under the same E-SFT-initialized Qwen3-VL-8B backbone.

Method	Category	Entity	Text		Visual
Method	EM	$\mathrm{F1}_{\mathrm{rel}}$	EM	F1	IoU50	IoU75
PPO	63.93	61.42	54.57	58.09	58.68	55.02
GRPO	68.72	65.98	57.99	61.70	59.82	56.39
DAPO	65.30	61.19	55.02	59.10	60.27	57.08
PAPO	70.55	66.21	57.76	61.12	63.93	60.73
HarmTrace	72.60	68.72	60.73	64.93	66.67	62.56

Table 5: Performance comparison of RL methods on target-identification fields under the same E-SFT initialization.

Dataset	Topic	#Total	#Harm.	Harm.	Tgt. cat.	Tgt. ent.	Text	Box
				Original annotations
PrideMM ( Shah et al. 2024 )	LGBTQ+	5,063	2,482	✓	✓	–	–	–
MAMI ( Fersini et al. 2022 )	Misogyny	11,000	5,500	✓	✓	–	–	–
Hateful Memes ( Kiela et al. 2020 )	General hate	10,000	5,000	✓	–	–	–	–
Harm-C ( Pramanick et al. 2021 )	COVID-19	3,544	1,249	✓	✓	–	–	–
Meme3W	Mixed	10,662	4,418	✓	✓	✓	✓	✓

Table 6: Annotation fields in the source datasets and Meme3W. Harm. denotes harmfulness labels. Tgt. cat. denotes source-specific target-category annotations defined under different taxonomies. Tgt. ent., Text, and Box denote target entities, textual mentions, and visual bounding boxes, respectively. A check mark indicates field availability.

Attack target	Count	Share (%)
Group background	1,463	41.23
Religion and caste	1,090	30.72
Gender	677	19.08
Health condition	318	8.96
Total	3,548	100.00

Table 7: Attack-target distribution of the retained FHM harmful subset before augmentation with MAMI, PrideMM, and Harm-C. Counts use the same filtering criteria as the final Meme3W dataset.

Figure 6: Candidate-assisted human-annotation interface. Annotators inspect the source text and three anonymized MLLM candidates, revise all structured fields, and draw or adjust image-target boxes before saving the annotation.

Entity A	Entity B	Same Form
women	woman	Yes
Muslim people	Muslims	Yes
Jewish people	Jews	Yes
LGBTQ people	LGBTQ community	Yes
immigrants	refugees	No
Asian people	Chinese people	No

Table 8: Examples of target-entity canonicalization. Yes indicates that two mentions map to the same canonical form; No indicates that they remain distinct. Final target-entity correctness is evaluated using

F1_{\mathrm{rel}}

Figure 7: Candidate-blind annotation interface without access to MLLM candidates.

Field	Metric	Blind vs. Gold
Target category	Category agreement	0.900
Target entity	Normalized F1	0.797
Textual mention	Token-F1	0.808
Visual region	IoU	0.803
Unweighted mean	–	0.827

Table 9: Candidate-blind audit against the final gold annotations.

Figure 8: Attack-target distribution and target-modality composition among the 4,418 harmful memes in Meme3W.

Split	Harmful	Non-harmful	Total
Training	3,759	5,307	9,066
Validation	221	312	533
Test	438	625	1,063
Total	4,418	6,244	10,662

Table 10: Numbers of harmful and non-harmful memes in each Meme3W split.

Parameter	Value
LoRA rank / alpha	64 / 128
Epochs	3
Learning rate	$1\times 10^{-4}$
Max sequence length	4096
Per-device train batch size	8
Gradient accumulation steps	2

Table 11: SFT hyperparameter settings.

Parameter	Value
LoRA rank / alpha	64 / 128
Epochs	3
Train batch size	32
PPO mini-batch size	32
Rollout group size	8
Clip ratio	0.2

Table 12: RL hyperparameter settings.

Figure 9: Sensitivity of the JRA improvement to the target-entity threshold $\tau_{e}$ and textual-mention threshold $\tau_{s}$ . Each cell reports the JRA difference between HarmTrace and GRPO in percentage points.

Model	IoU 0.5	IoU 0.6	IoU 0.7	IoU 0.75	IoU 0.8
GRPO	49.09	47.49	46.35	46.35	44.52
HarmTrace	52.51	52.05	50.46	50.23	50.00
$\Delta$	+3.42	+4.57	+4.11	+3.88	+5.48

Table 13: Sensitivity of JRA to the minimum-enclosing-rectangle IoU threshold with

\tau_{e}=\tau_{s}=0.7

. Values are percentages on the 438 gold-harmful test memes, and

\Delta

denotes HarmTrace minus GRPO.

Visual rule	GRPO	HarmTrace	$\Delta$ JRA
Enclosing-rectangle IoU	49.09	52.51	+3.42
Bipartite set matching	46.12	50.00	+3.88
Geometric set IoU	49.77	52.28	+2.51

Table 14: JRA under alternative visual-region matching rules. HarmTrace remains above GRPO under all three rules.

Comparator	JRA	HarmTrace	$\Delta$ JRA	Paired 95% CI
E-SFT	$45.89$	$52.51$	$+6.62$	$[2.51,10.96]$
GRPO	$49.09$	$52.51$	$+3.42$	$[0.23,6.85]$

Table 15: Paired comparisons of JRA on the 438 gold-harmful test memes. All values are percentages. Confidence intervals are percentile intervals from 10,000 paired bootstrap resamples of test instances.

Cat.	#	E-SFT	GRPO	Ours
Group/bg.	146	46.58	50.00	56.16
Religion/caste	111	60.36	63.96	62.16
Gender	103	44.66	47.57	56.31
Health cond.	78	25.64	28.21	26.92
Overall	438	45.89	49.09	52.51

Table 16: Category-wise JRA for E-SFT, GRPO, and HarmTrace on gold-harmful test memes.

Harmful Meme Detection

See all 23 papers

SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection
Sep 12, 2026Hanling Wang, Chenlong Wei, Yingjuan Li +4Harmful Meme Detection Multimodal Content Moderation
University of California, San Diego · Xi’an Jiaotong-Liverpool University · University of Washington +2
AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes
Jul 29, 2026Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Abul Hasnat +3Arabic NLP Hate Speech Detection
Qatar Computing Research Institute, Qatar · 2APAVI.AI, France · 3Hamad Bin Khalifa University, Qatar +1
Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detection
Sep 17, 2026Bo Xu, Chenyuan Wang, Xinyu Chen +5Multimodal Large Language Models Hate Speech Detection
School of Software, Dalian University of Technology · School of Computer Science and Technology, Dalian University of Technology · School of Computing Technologies, RMIT University

Sep 12, 2026J/K move · Enter open · S save

SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection

Hanling Wang, Chenlong Wei, Yingjuan Li, Di Wu+3

University of California, San Diego · Xi’an Jiaotong-Liverpool University · University of Washington+2

Harmful Meme Detection Multimodal Content Moderation Multimodal Reasoning

Open paper AskPDF arXiv

Detecting harmful memes is critical for maintaining safe online communities. However, harmful intent is often implicit, arising from visual-textual incongruity and cultural stereotypes, which challenges existing multimodal detectors. We propose SyRHM, a framework that decomposes harmful meme detection into meaning-grounded retrieval and symbolic-language-enhanced multi-stage reasoning. SyRHM retrieves semantically related memes by parsing multimodal content into textual elements and descriptions, providing grounded context beyond surface-level similarity. Building on the retrieved context, SyRHM uses a translator stage to convert multimodal inputs into symbolic intermediate representations, and then performs multi-stage reasoning via planner and solver stages, enabling expressive and interpretable analysis of harmful intent. Experiments on FHM, HarM, and MultiOff demonstrate the effectiveness of SyRHM, achieving superior performance on most evaluation settings against multimodal and reasoning-based baselines, while providing reasoning traces for harmful content. The code is available at: https://github.com/Scabbards1500/SyRHM

Abstract

Harmful Meme Detection

Explore similar work

SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection

AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes

Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detection