Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention

Cheng Ye; Weidong Chen; Peipei Song; Zhendong Mao

cs.CVSep 29, 2026

Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention

Authors: Cheng Ye, Weidong Chen, Peipei Song, Zhendong Mao

Organizations: University of Science and Technology of China, Hefei

Abstract

Emotional Video Captioning aims to generate factually accurate and emotionally empathetic descriptions. While recent methods have recognized the importance of visual causes to guide emotion perception and caption generation, they fundamentally rely on simple attention matching, which inevitably suffers from {causal redundancy and spurious correlations} in co-occurrence bias (e.g., misclassifying sadness'' as joy'' on a sunny beach), leading to severe shortcut learning from confusing backgrounds. Furthermore, existing evaluations fail to verify whether models have genuinely mastered causal reasoning or merely exploited background confounders. To address these limitations, we first construct {EVC-CauseGround}, a comprehensive benchmark with dense spatio-temporal causal annotations. Crucially, it introduces a carefully selected {Causal-Faithfulness Subset} to explicitly quantify genuine emotion-cause attribution. Second, we propose {Causal-EVC}, an emotion-grounding captioning framework, which introduces a Motion-guided Causal Spatiotemporal Localization module to precisely decouple causal triggers from background confounders. Besides, we introduce an Interpretable Sparse Emotion Routing module. By synthesizing counterfactual representations and formulating a novel counterfactual contrastive objective, we enforce the model to anchor its emotion predictions strictly on authentic causal triggers instead of confusing background. Extensive experiments show that Causal-EVC not only achieves the best performance on semantic metrics but also exhibits significant advantages in the causal-faithfulness subset, which demonstrates that our model could mine emotional cues from genuine visual causes and mitigate co-occurrence bias for interpretable multimodal emotion understanding.

Figures & tables

Figure 1: Motivation. Traditional methods suffer from (a) Causal Redundancy and (b) Spurious Correlations, where models exploit shortcut learning and emotional hallucinations due to background confounders. (c) Paradigm Comparison between Causal-EVC and previous methods demonstrate the superiority for genuine emotion-cause reasoning of Causal-EVC.

Figure 2: Performance Comparison on the caption generation and causal reasoning.

Figure 3: The construction pipeline of the EVC-CauseGround benchmark, which is consist of (A) Data Source Collection, (B) Spatiotemporal Visual Cause Annotation and (C) Human-led Sample Check and Partition, respectively.

Figure 4: Overview of Causal-EVC. Given the video and emotion dictionary, we perform (A) Motion-Guided Causal Spatiotemporal Localization to extract visual cause masks, and (B) Interpretable Sparse Emotion Routing to mine visual emotion cues for cause-grounding captioning.

Methods	Emotion		Semantic					Hybrid
Methods	${\rm Acc}_{sw}$ $\uparrow$	${\rm Acc}_{c}$ $\uparrow$	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$	ROUGE $\uparrow$	CIDEr $\uparrow$	BFS $\uparrow$	CFS $\uparrow$
EVC-MSVD
SA-LSTM ( 2018 )	68.8	67.2	80.7	45.5	33.0	68.2	72.1	59.0	71.3
FT ( 55 )	69.4	67.1	77.2	36.3	29.0	63.4	62.5	52.5	63.7
SGN ( 2021 )	73.9	73.1	77.5	41.1	30.6	63.6	71.0	56.4	71.5
CANet ( 2022 )	78.7	76.8	78.5	41.8	30.8	65.7	74.4	57.9	75.1

Table 1: Performance for emotional video captioning task on two benchmarks. The best results are highlighted in bold.

Methods	$mIoU$	$R@0.3$	$R@0.5$	${\rm Acc_{sw}}$
MM-ECPE ( 2025b )	18.6	28.7	12.6	53.1
Momentor ( 2024 )	21.5	35.8	20.4	58.5
VTimeLLM ( 2024 )	23.7	42.1	22.0	61.4
VideoChat-T ( 2025 )	37.6	61.9	40.8	78.4
Qwen2.5-VL-7B ( 2025 )	39.8	62.5	41.3	80.8
Gemini-2.5-Flash ( 2025 )	30.7	50.9	27.1	67.0

Table 2: Results on EVC-CauseGround.

Components			${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
Motion	Temporal	Spatial	${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
$\times$	$\times$	$\times$	92.9	67.8	154.4	142.0
	$\checkmark$	$\times$	89.4	68.6	156.9	143.2
	$\times$	$\checkmark$	91.2	69.7	158.5	144.9
	$\checkmark$	$\checkmark$	90.8	70.3	160.2	146.2
$\checkmark$	$\times$	$\times$	90.5	67.2	152.8	140.2

Table 3: The ablation study for dense MCSL.

Components			${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
Motion	Temporal	Spatial	${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
$\times$	$\times$	$\times$	92.9	67.8	154.4	142.0
	$\checkmark$	$\times$	89.4	68.6	156.9	143.2
	$\times$	$\checkmark$	91.2	69.7	158.5	144.9
	$\checkmark$	$\checkmark$	90.8	70.3	160.2	146.2
$\checkmark$	$\times$	$\times$	90.5	67.2	152.8	140.2

Table 3: The ablation study for dense MCSL.

Components			${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
$\mathcal{L}_{cf}$	$\mathcal{L}_{temp}$	$\mathcal{L}_{spa}$	${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
$\times$	$\times$	$\times$	87.3	66.7	153.2	139.9
	$\checkmark$	$\times$	87.8	68.5	157.4	143.4
	$\times$	$\checkmark$	87.4	68.9	158.3	144.0
	$\checkmark$	$\checkmark$	88.4	70.1	164.0	148.8
$\checkmark$	$\times$	$\times$	90.8	67.2	155.9	142.8

Table 4: The ablation study for proposed losses.

Components		${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
MCSL	ISER	${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
$\times$	$\times$	82.1	65.3	150.9	137.0
$\checkmark$	$\times$	85.7	70.2	165.4	149.4
$\times$	$\checkmark$	92.9	67.8	154.4	142.0
$\checkmark$	$\checkmark$	94.2	71.2	169.8	154.6

Table 5: The ablation study for proposed modules.

Top-K	${\rm Acc}_{sw}$	BLEU-4	CIDEr	CFS
1	49.6	51.5	115.9	102.5
5	88.6	67.4	149.8	137.4
10	94.2	71.2	169.8	154.6
20	92.9	70.1	161.8	147.8
50	92.2	68.9	156.3	143.3

Table 6: Results of different Top-K routing.

Figure 5: Comparison of caption generation between MM-ECPE and our Causal-EVC.

Figure 6: Case study for cause localization.

Figure 7: t-SNE visualization of cause representation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Modality	Emotion Labels	Emotion Captions	Temporal Grounding	Spatial (Dense Tubes)	Counterfactual Subset
Emotion Classification & Captioning
V-Emo ( 2014 )	V, A	✓	$\times$	$\times$	$\times$	$\times$
EmVidCap ( 55 )	V, T	✓	✓	$\times$	$\times$	$\times$
Emotion-Cause Pair Extraction (ECPE)
ECF ( 2022 )	V, A, T	✓	$\times$	$\checkmark$ (Utterance-level)	$\times$	$\times$
MECESD ( 2024 )	V, A, T	✓	✓	$\checkmark$ (Utterance-level)	$\times$	$\times$

Appendix

Table A.1: Comparison of EVC-CauseGround with existing datasets across different domains. “V”, “A”, and “T” stand for video, audio, and text modalities, respectively. Our dataset is the first to simultaneously provide fine-grained spatio-temporal causal grounding and a rigorously diagnosed counterfactual subset for real-world emotional videos.

Metric	Accuracy	Relevance	Coherence	Usability
VEIN ( 2024 )	5.47	5.18	6.27	5.04
EPAN ( 2023 )	6.24	5.97	7.13	5.76
DCGN ( 2024 )	6.88	6.43	7.68	6.32
VideoBLIP ( 2024 )	5.91	5.54	7.20	4.56
MM-ECPE ( 2025b )	6.40	6.75	8.04	7.36
GPT-4o ( 2024 )	6.98	6.04	7.24	6.62

Appendix

Table A.2: The comparison with SOTA methods on human evaluations.

Explore similar work

Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction
Jun 7, 2026Weidong Chen, Cheng Ye, Zhendong Mao +3Video Captioning Fine-Grained Video Understanding
School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China
Faithful Action-unit Causal Reasoning for Counterfactually Faithful Emotion Explanations
Jun 14, 2026Van Thong Huynh, Hong Hai Nguyen, Thuy Pham +2Faithful Explanation Free-Form Textual Rationales
Faculty of CSE, Ho Chi Minh City University of Technology (HCMUT), VNUHCM, Vietnam · Dept. of AI, FPT University, Vietnam · Faculty of DSAI, College of Technology, National Economic University, Vietnam +1
Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning
Jul 31, 2026Junbo Wang, Liangyu Fu, Yuke Li +2Emotion Recognition Video Captioning
School of Software, Northwestern Polytechnical University, Xi’an 710129, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China · School of Computer Science, The University of Sydney, NSW 2006, Australia

Jun 7, 2026, cs.CVJ/K move · Enter open · S save

Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction

Weidong Chen, Cheng Ye, Zhendong Mao, Liping Wang+2

School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China

Video Captioning Fine-Grained Video Understanding Emotion Fine-Grained Perception

Open paper AskPDF arXiv

Emotional Video Captioning (EVC) is a challenging task that aims to generate factually accurate and emotionally rich descriptions for videos. Existing EVC methods leverage holistic visual features to mine global emotional cues, and then aggregate multimodal features to guide the emotional caption generation, which ignores the critical characteristic of the EVC task. Visual emotions are evoked by specific motivational causes, which are usually only implied in core video segments. The holistic mining brings significant information redundancy and inaccurate emotional cues. Thus, fine-grained visual cause extraction has a facilitative effect on both emotion perception and emotion-attributed caption generation. To this end, we propose a fine-grained emotion-cause pair extraction framework for emotion-attributed video captioning. Specifically, we learn pair-wise emotion and cause features in two rounds: 1) We propose a Concept-aware Visual Semantic Decomposition module to augment visual features by exploring scene, object, and motion concepts. Besides, to enhance emotional features, we propose a Visual-guided Emotion Interpretable Learning module, which guides emotion refinement with visual temporal dynamics, and augments the interpretable refinement process by reliable VAD-vector constraints. 2) We achieve emotion-cause pair extraction by cross-coupling the visual and emotional features before and after refinement, and leverage contrastive loss to achieve semantic forced alignment. Overall, our approach optimizes complex semantic understanding and emotion perception of videos, leading to a promising performance in emotional captioning. Extensive experiments on three challenging datasets demonstrate the superiority of our approach and each proposed module, e.g., achieving the best performances with +4.4% and +5.4% w.r.t. BLEU-2 and ROUGE-L, respectively, on the EVC-MSVD dataset.

Abstract

Explore similar work

Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction

Faithful Action-unit Causal Reasoning for Counterfactually Faithful Emotion Explanations

Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning