CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detection

Yuchen Zhang; Shuang Dai; Zeyu Fu; Yunfei Long; Ravi Shekhar; Haralambos Mouratidis

cs.CVAug 16, 2026

CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detection

Authors: Yuchen Zhang, Shuang Dai, Zeyu Fu, Yunfei Long, Ravi Shekhar, Haralambos Mouratidis

Organizations: Institute for Analytics and Data Science, University of Essex, Colchester, United Kingdom · Department of Engineering, University of Exeter, Exeter, United Kingdom · Department of Computer Science, University of Exeter, Exeter, United Kingdom · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom

Abstract

Hateful video detection has become increasingly important with the rapid growth of video-centric social media platforms, given the serious risks that hate speech poses to both individual well-being and social cohesion. Compared with text or static multimodal content, hateful video detection remains underexplored and significantly more challenging, as hateful meaning often arises from complex interactions among multimodal cues, including speech, audio, and visual content. Moreover, such signals are often brief, implicit, and temporally dependent, making them difficult to capture using conventional video-level representations. In this work, we propose CLARA, a clip-level multimodal framework for hateful video detection. Instead of treating a video as a single instance, CLARA models it as a sequence of fine-grained clips, enabling more precise capture of temporally localized hateful signals. We introduce a Mixture-of-Experts clip encoder for adaptive multimodal alignment, a local-global segment contrastive objective to jointly model short-term cues and long-range temporal dependencies, and VLM-derived rationales integrated via a gated Transformer to provide high-level semantic guidance. Extensive experiments on three hateful video datasets demonstrate that CLARA consistently outperforms state-of-the-art methods. Further ablation studies and parameter analyses validate the effectiveness of each component.

Figures & tables

Figure 1. An example of hateful video. The hate speech is not fully revealed by single frame or utterance, but emerges through contextual buildup across multiple clips.

Figure 2. Overall framework of CLARA. An input video is first segmented into utterance-aligned clips, from which multimodal features are extracted and fused into clip-level representations via a MoE encoder. Local-global segment contrastive learning is then applied to enhance temporal consistency across clips. Meanwhile, video-level rationales are generated by a VLM and integrated with the clip representations through a rationale-gated Transformer to produce the final video representation, which is finally used for hateful video classification.

	HateMM	MHC_CN	MHC_EN	DeHate
Hateful videos	427	298	296	2,120
Non-hateful Videos	638	608	607	4,568
Total videos	1,065	906	903	6,688
Platforms	BitChute	Bilibili	YouTube	BitChute, TikTok

Table 1. Dataset Statistics.

Method	HateMM		MHC_CN		MHC_EN		DeHate
Method	Acc	M-F1	Acc	M-F1	Acc	M-F1	Acc	M-F1
BERT	0.724 $\pm$ 0.028	0.709 $\pm$ 0.032	0.696 $\pm$ 0.006	0.635 $\pm$ 0.019	0.674 $\pm$ 0.010	0.417 $\pm$ 0.020	0.651 $\pm$ 0.006	0.522 $\pm$ 0.030
Roberta	0.729 $\pm$ 0.014	0.718 $\pm$ 0.025	0.671 $\pm$ 0.003	0.402 $\pm$ 0.001	0.672 $\pm$ 0.002	0.402 $\pm$ 0.001	0.683 $\pm$ 0.000	0.406 $\pm$ 0.000
MFCC	0.653 $\pm$ 0.048	0.598 $\pm$ 0.066	0.646 $\pm$ 0.014	0.511 $\pm$ 0.020	0.656 $\pm$ 0.018	0.463 $\pm$ 0.031	0.677 $\pm$ 0.003	0.441 $\pm$ 0.030
Wav2vec	0.696 $\pm$ 0.013	0.678 $\pm$ 0.016	0.675 $\pm$ 0.003	0.403 $\pm$ 0.001	0.674 $\pm$ 0.008	0.403 $\pm$ 0.003	0.683 $\pm$ 0.001	0.407 $\pm$ 0.001
ViT	0.700 $\pm$ 0.031	0.699 $\pm$ 0.031	0.701 $\pm$ 0.007	0.553 $\pm$ 0.024	0.678 $\pm$ 0.014	0.438 $\pm$ 0.026	0.685 $\pm$ 0.010	0.542 $\pm$ 0.018

Table 2. Overall Performance in terms of Acc, M-F1, M-Pre and M-Rec. Best in bold and second-best is underlined .

Panel of Figure 3 — Figure 3. Parameter study of CLARA on HateMM. (a) the effect of the MoE configuration, including the number of experts and top-k routing, (b) the effect of the local and global temporal ratios used in contrastive learning, (c) the effect of the contrastive learning weight $\lambda_{\text{CL}}$ , and (d) the effect of the training budget.

Method	HateMM		MHC_CN		MHC_EN		DeHate
Method	Acc	M-F1	Acc	M-F1	Acc	M-F1	Acc	M-F1
w/o A	0.751	0.735	0.704	0.663	0.701	0.667	0.662	0.557
w/o V	0.743	0.739	0.705	0.672	0.688	0.581	0.651	0.510
w/o T	0.712	0.707	0.690	0.653	0.669	0.468	0.668	0.532
w/o MoE	0.789	0.780	0.738	0.679	0.735	0.669	0.708	0.637
w/o CL	0.821	0.802	0.757	0.707	0.749	0.679	0.716	0.641

Table 3. Ablation Results in terms of Accuracy and M-F1.

Panel of Figure 4 — Figure 4. Expert routing statistics of the MoE. : (a) the distribution measured by gate probability ratio. (b) the distribution of experts measured by top-k selection ratio.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Videos	Total Clips	Clips per Video
Dataset	Videos	Total Clips	Mean	P25	Median	P75	Range
DeHate	6,688	191,554	28.6	6	18	43	1–223
HateMM	1,065	43,564	40.9	8	27	53	1–434
MHC_CN	906	10,158	11.2	3	9	17	1–59
MHC_EN	903	9,676	10.7	4	9	15	1–47

Appendix

Table 4. Distribution of the number of clips per video.

Dataset	Total Clips	Clip Duration (s)
Dataset	Total Clips	Mean	P25	Median	P75	Range
DeHate	191,554	3.4	1.3	2.2	4.1	0.0 ∗ –254.1
HateMM	43,564	3.4	1.3	2.0	4.0	0.0 ∗ –480.0
MHC_CN	10,158	2.8	1.3	2.0	2.7	0.0 ∗ –58.7
MHC_EN	9,676	3.2	1.5	2.4	4.0	0.0 ∗ –56.4
∗ Values below 0.05 s are rounded to 0.0.

Appendix

Table 5. Distribution of clip durations.

STEP 1 PROMPT (Objective Analysis)

# Role

You are a professional video content verifier. Your task is to accurately summarize what the video shows and what the accompanying text conveys in an objective and neutral way.

# Instructions

You may use:

(1) 20 frames sampled uniformly in temporal order.

(2) Video title (if available) and full transcription.

Appendix

Table 6. Prompt used in Step 1 for objective video analysis.

STEP 2 PROMPT (Hateful Content Verification)

# Role

You are a professional content verifier conducting hateful content analysis. Your task is to make a careful, evidence-based judgment using the information provided.

# Instructions

You may use:

(1) 20 frames sampled uniformly in temporal order.

(2) video title and full transcription.

Appendix

Table 7. Prompt used in Step 2 for hateful content verification.

Element	Description
Content summary	The Content_summary from Step 1.
Visual description	The visual_description from Step 1.
Textual description	The textual_description from Step 1.
Cross-modal relation	A sentence combining cross_modal_relation and cross_modal_explanation from Step 1 in the form “ The relation between textual and visual content is { cross_modal_relation } and { cross_modal_explanation }”. If only one is available, we use it directly.
Contextual elements	If contextually_elements from Step 1 is a non-empty list, we convert it into a sentence of the form “ Contextually elements include: $e_{1},e_{2},\dots,e_{k}$ ”, where $e_{i}$ are the elements in contextually_elements . Otherwise, this field is set to an empty string.
Final decision	A sentence constructed from Label , Explicitness , and Confidence from Step 2. If the Label is non-hate, the sentence “ The video is considered non-hateful with { confidence } confidence. ” If the label is hate, the sentence is “ The video is considered hateful ( { explicitness } ) with { confidence } confidence. ”

Appendix

Table 8. Elements of textual rationale.

Dataset	Metric	CLARA–MoRE	CLARA–HVGuard	CLARA–MM-HSD
HateMM	Acc	4.00e-3	1.63e-2	6.50e-3
	M-F1	6.10e-3	2.42e-2	7.50e-3
	M-P	2.80e-3	2.42e-2	1.34e-2
	M-R	9.40e-3	3.19e-2	1.17e-2
MHC_CN	Acc	6.00e-4	6.50e-3	1.40e-3
	M-F1	1.64e-2	3.00e-4	1.24e-2

Appendix

Table 9. Paired t-test p-values comparing CLARA with strong baselines across all datasets.

Panel of Figure 5 — Figure 5. Parameter study of CLARA on MHC_CN. (a) the effect of the MoE configuration, including the number of experts and top-k routing, (b) the effect of the local and global temporal ratios used in contrastive learning, (c) the effect of the contrastive learning weight $\lambda_{\text{CL}}$ , and (d) the effect of the training budget.

Panel of Figure 6 — Figure 6. Parameter study of CLARA on MHC_EN. (a) the effect of the MoE configuration, including the number of experts and top-k routing, (b) the effect of the local and global temporal ratios used in contrastive learning, (c) the effect of the contrastive learning weight $\lambda_{\text{CL}}$ , and (d) the effect of the training budget.

Panel of Figure 7 — Figure 7. Parameter study of CLARA on DeHate. (a) the effect of the MoE configuration, including the number of experts and top-k routing, (b) the effect of the local and global temporal ratios used in contrastive learning, (c) the effect of the contrastive learning weight $\lambda_{\text{CL}}$ , and (d) the effect of the training budget.

Panel of Figure 8 — Figure 8. Expert routing statistics of the MoE on MHC_CN : (a) the distribution measured by gate probability ratio. (b) the distribution of experts measured by top-k selection ratio.

Panel of Figure 9 — Figure 9. Expert routing statistics of the MoE on MHC_EN : (a) the distribution measured by gate probability ratio. (b) the distribution of experts measured by top-k selection ratio.

Panel of Figure 10 — Figure 10. Expert routing statistics of the MoE on DeHate : (a) the distribution measured by gate probability ratio. (b) the distribution of experts measured by top-k selection ratio.

Dataset	Text Encoder	Acc	M-F1	M-P	M-R
HateMM	qwen_0.6	0.864	0.855	0.870	0.846
	qwen_8	0.878	0.871	0.882	0.864
	bert	0.879	0.872	0.874	0.872
MHC_CN	qwen_0.6	0.780	0.736	0.767	0.730
	qwen_8	0.772	0.729	0.764	0.722
	bert	0.779	0.734	0.760	0.726

Appendix

Table 10. Impact of text encoder on CLARA across all datasets.

Dataset	VLM	Acc	M-F1	M-P	M-R
HateMM	LLaVA	0.8575	0.8443	0.8577	0.8379
HateMM	Qwen3VL	0.8790	0.8720	0.8740	0.8720
MHC_CN	LLaVA	0.7084	0.6673	0.6715	0.6514
MHC_CN	Qwen3VL	0.7790	0.7340	0.7600	0.7260
MHC_EN	LLaVA	0.7372	0.6898	0.7067	0.6859
MHC_EN	Qwen3VL	0.7630	0.7270	0.7320	0.7230

Appendix

Table 11. Impact of rationale generator on CLARA across all datasets.

Dataset	Segmentation	Acc	M-F1	M-P	M-R
HateMM	Fix-length	0.8366	0.8285	0.8345	0.8288
HateMM	Utterance-aligned	0.8790	0.8720	0.8740	0.8720
MHC_CN	Fix-length	0.7017	0.6547	0.7371	0.6452
MHC_CN	Utterance-aligned	0.7790	0.7340	0.7600	0.7260
MHC_EN	Fix-length	0.7497	0.7073	0.7248	0.7041
MHC_EN	Utterance-aligned	0.7630	0.7270	0.7320	0.7230

Appendix

Table 12. Impact of clip segmentation strategy.

Method	Params (M)	Training Cost		Inference Cost
Method	Params (M)	Time (s/epoch)	Peak Mem. (MB)	FLOPs (G/video)	Latency (ms/video)
MoRE	110.444	3.961	2,766.45	0.642	1.983
MMHSD	4.626	0.329	352.29	0.008	1.052
HVGuard	3.312	0.205	133.96	0.007	0.552
CLARA	44.950	3.515	2,765.14	2.033	3.720

Appendix

Table 13. Training and inference computational costs on HateMM.

Hate Speech Detection

Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.

See all 97 papers

GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection
Feb 24, 2026Yingying Guo, Ke Zhang, Zirong ZengHarmful Meme Detection Hate Speech Detection
The Chinese University of Hong Kong, Shenzhen
Decoding Multimodal Cues: Unveiling the Implicit Meaning Behind Hateful Videos
Jun 10, 2026Junyu Lu, Deyi Ji, Liqun Liu +9Multimodal Reasoning Hate Speech Detection
Dalian University of Technology Dalian, China · Tencent Beijing, China · Singapore University of Technology and Design Singapore, Singapore
More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detection
Mar 22, 2026Runze Sun, Yu Zheng, Zexuan Xiong +4Hate Speech Detection Multimodal Reasoning
Department of Automation, Tsinghua University

Feb 24, 2026J/K move · Enter open · S save

GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection

Yingying Guo, Ke Zhang, Zirong Zeng

The Chinese University of Hong Kong, Shenzhen

Harmful Meme Detection Hate Speech Detection VLM Adaptation Multimodal Fusion

Open paper AskPDF arXiv

Detecting hateful content in multimodal memes presents unique challenges, as harmful messages often emerge from the complex interplay between benign images and text. We propose GatedCLIP, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection. Our approach introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment. Experiments on the Hateful Memes dataset demonstrate that GatedCLIP achieves an AUROC of 0.66, substantially outperforming the CLIP baseline (AUROC 0.49) while maintaining computational efficiency with only 350K trainable parameters.

Abstract

Hate Speech Detection

Explore similar work

GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection

Decoding Multimodal Cues: Unveiling the Implicit Meaning Behind Hateful Videos

More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detection