Organizations: Hefei University of Technology · MBZUAI · NUS · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · The Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology
Point-level weakly-supervised temporal sentiment localization (P-WTSL) aims to detect sentiment-relevant segments in untrimmed multimodal videos using timestamp sentiment annotations, which greatly reduces the costly frame-level labeling. To further tackle the challenges of imprecise sentiment boundaries in P-WTSL, we propose the Face-guided Sentiment Boundary Enhancement Network (FSENet), a unified framework that leverages fine-grained facial features to guide sentiment localization. Specifically, our approach first introduces the Face-guided Sentiment Discovery (FSD) module, which integrates facial features into multimodal interaction via dual-branch modeling for effective sentiment stimuli clues; We then propose the Point-aware Sentiment Semantics Contrast (PSSC) strategy to discriminate sentiment semantics of candidate points (frame-level) near annotation points via contrastive learning, thereby enhancing the model's ability to recognize sentiment boundaries. At last, we design the Boundary-aware Sentiment Pseudo-label Generation (BSPG) approach to convert sparse point annotations into temporally smooth supervisory pseudo-labels. Extensive experiments and visualizations on the benchmark demonstrate the effectiveness of our framework, achieving state-of-the-art performance under full supervision, video-level, and point-level weak supervision, thereby showcasing the strong generalization ability of our FSENet across different annotation settings.
Figures & tables
Figure 1: Illustration of the task and motivation. (a) Given the point-level sentiment points in the untrimmed video, the model needs to achieve sentiment classification and boundary localization. (b) We explored the facial-centric method to guide sentiment clues discovery. (c) To alleviate sentiment boundary jitter and discontinuity under weak supervision, we propose a novel pseudo-label smoothing strategy.
Figure 2: Overview of our FSENet framework. (a)FSD module introduces facial features into the temporal sentiment localization task through two branches to advance discovery of sentiment clues: one focuses on facial-centric interaction, while the other employs global sentiment perception weighting score. (b)PSSC aims to identify similar and dissimilar semantic points in the temporal axis to enhance the point-level sentiment discrimination by contrastive learning. (c)BSPG transforms sentiment point-level annotations into segment-level sentiment boundaries pseudo-label using the threshold filtering and step-by-step boundary-aware scheme, where sentiment scores decay progressively based on parameters β and w .
Setting
Method
mAP@IoU(%)
Average
0.1
0.15
0.2
0.25
0.3
Recall
F2 Score
mAP
Full Supervision
VAANet [ 60 ]
28.01
24.95
20.68
16.45
12.50
69.82
39.59
20.51
† AFSD [ 61 ]
33.44
27.09
21.78
16.64
11.76
75.10
31.78
22.14
FSENet(ours)
28.02
25.45
22.63
18.93
14.33
79.58
30.37
21.87
Weak Supervision (Video-level)
CoLA [ 62 ]
11.87
10.14
8.11
6.83
4.98
64.07
21.19
8.38
UM [ 34 ]
14.77
13.28
10.77
8.41
6.54
55.33
23.94
10.75
Table 1: Performance comparison with SOTA methods under Fully and Weakly supervised settings on TSL300 . “ † ” indicates methods reproduced by ourselves. The best and second-best results are bolded and underlined , respectively.
Method
mAP@IoU(%)
Average
0.1
0.2
0.3
Recall
F2 Score
mAP
HR-pro stage1
11.83
7.26
4.44
18.81
18.46
7.65
HR-pro stage2
16.27
11.49
6.76
26.61
24.71
11.47
TSL
15.18
10.71
7.46
27.24
24.86
11.00
FSENet (Ours)
22.7
22.49
13.24
33.08
24.89
16.54
Table 2: Performance results on CMU-MOSEI dataset under point-level settings.
Method
mAP@IoU (%)
Avg
0.1
0.2
0.3
mAP
Qwen3-Omni(zero-shot)
7.67
5.03
2.67
5.07
LLaMA-2-7B (LoRA)
10.98
10.98
8.45
9.97
LLaMA-2-7B (LoRA+BSPG)
27.76
11.82
5.16
11.75
Ours (small model)
29.31
22.49
13.24
21.45
Table 3: Comparison with large language models (LLMs) on the TSL300 dataset. Zero-shot methods are evaluated without fine-tuning. while LoRA-based models are fine-tuned with point-level supervision.
Method
mAP@IoU (%)
Avg
0.1
0.15
0.2
0.25
0.3
mAP
AV Features Only
HR-pro stage1
16.64
13.95
11.86
9.74
8.01
12.04
HR-pro stage2
26.40
21.80
16.91
13.27
9.96
17.67
TSL
28.14
23.02
18.68
14.36
10.46
18.93
FSENet(ours)
25.91
24.70
20.08
16.85
11.53
19.41
Table 4: Fair evaluation under identical feature settings on TSL300 . Group ➀ uses only audio-visual features, and group ➁ adds local facial features. All △ adopt the same facial extractor.
#id
FCI
GSP
mAP@IoU (%)
Avg
first stage
second stage
0.1
0.2
0.3
mAP
a
–
–
✓
24.95
18.63
11.52
18.42
b
✓
–
✓
27.19
18.93
11.16
19.10
c
–
✓
✓
27.46
18.31
10.37
18.78
d
✓
✓
–
28.69
20.83
11.26
20.22
e
✓
✓
✓
29.31
22.49
13.24
21.45
Table 5: Ablation study on the main network component in FSD. We report the result of mAPs, and ‘–’ denotes not used.
Pseudo-Label
Threshold
Boundary
mAP@IoU(%)
Avg
0.1
0.2
0.3
mAP
None
–
–
27.51
15.31
8.72
16.92
Threshold Only
✓
–
28.06
20.98
11.55
20.29
Step Only
–
✓
25.76
21.59
13.10
20.31
Ours (full model)
✓
✓
29.31
22.49
13.24
21.45
Table 6: Ablation of pseudo-label generation strategies.
#id
Lframe
Lframeglo
Lsc
mAP@IoU(%)
Avg
0.1
0.2
0.3
mAP
a
✓
✓
✓
29.31
22.49
13.24
21.45
b
–
✓
✓
28.28
19.52
10.33
20.02
c
✓
–
✓
27.67
20.40
9.99
20.65
d
✓
✓
–
29.17
21.18
12.49
20.94
e
–
–
✓
19.15
11.91
7.12
12.53
Table 7: Ablation of weakly-supervised optimization losses.
Figure 3: Visualization of the predicted results for temporal sentiment localization. The ‘pink’ segments are positive sentiment, ‘blue’ means negative, and ‘gray’ means no valid location output. △ indicates that methods incorporate facial features settings.
Figure 4: T-SNE visualization comparing our FSENet with SOTA. Each point represents a segment-level feature (‘p/n/n-sent’ denotes positive, negative, and non-sentiment categories, respectively).
BSPG
w
5
6
7
8
9
mAP (%)
19.22
19.77
21.45
19.66
19.26
β
0.1
0.3
0.5
0.6
0.7
mAP (%)
19.64
18.88
21.29
21.45
21.20
Table 8: Ablation study of BSPG parameters (confidence decay factor β and smoothing width w ) on mAP performance.
Figure 5: Ablation study of Top-K parameter k in PSSC on mAP performance.
Figure 6: Ablation study of loss function weights λ1 and λ2 on mAP performance.
Figure 7: Proportion of proposal type for all prediction results of models on TSL300. Error categories: Bg (background), Loc (localization), Dbl (double detection), Conf (confusion), WLbl (wrong label), and TP (true positive).
Modality
mAP@IoU (%)
Avg
0.1
0.15
0.2
0.25
0.3
mAP
Single Modality
A (Audio)
18.98
15.47
12.96
9.75
7.09
12.85
V (Visual)
19.27
16.14
12.06
9.14
6.45
12.61
F (Local Face)
22.11
15.97
11.33
8.40
6.62
12.88
Bimodal Combinations
Table 9: Performance comparison of different modality combinations on the TSL task. A: Audio, V: Visual, F: Local Face. Metrics are reported as mAP@IoU (%) across various thresholds and average mAP.
Method
mAP@IoU (%)
Avg
0.1
0.15
0.2
0.25
0.3
mAP
L2
28.06
24.40
20.98
16.49
11.55
20.29
L1
26.53
23.66
20.70
16.36
12.97
20.05
Dot
27.23
24.90
21.21
17.42
14.22
21.00
Cosine(Ours)
29.31
25.47
22.49
16.76
13.24
21.45
Table 10: Ablation of feature distance metrics for Point-aware Sentiment Semantics Contrast modeling.
Temporal Forgery Localization (TFL) aims to precisely identify manipulated segments within videos or audio streams, providing interpretable evidence for multimedia forensics and security. While most existing TFL methods rely on dense frame-level labels in a fully supervised manner, Weakly Supervised TFL (WS-TFL) reduces labeling cost by learning only from binary video-level labels. However, current WS-TFL approaches suffer from mismatched training and inference objectives, limited supervision from binary labels, gradient blockage caused by non-differentiable top-k aggregation, and the absence of explicit modeling of inter-proposal relationships. To address these issues, we propose GEM-TFL (Graph-based EM-powered Temporal Forgery Localization), a two-phase classification-regression framework that effectively bridges the supervision gap between training and inference. Built upon this foundation, (1) we enhance weak supervision by reformulating binary labels into multi-dimensional latent attributes through an EM-based optimization process; (2) we introduce a training-free temporal consistency refinement that realigns frame-level predictions for smoother temporal dynamics; and (3) we design a graph-based proposal refinement module that models temporal-semantic relationships among proposals for globally consistent confidence estimation. Extensive experiments on benchmark datasets demonstrate that GEM-TFL achieves more accurate and robust temporal forgery localization, substantially narrowing the gap with fully supervised methods.
Xiaodong Zhu, Yuanming Zheng, Suting Wang +4
NERCMS, School of Computer Science, Wuhan University
Multimodal sentiment analysis often remains text-dominant due to raw-video noise and insufficient temporal modeling. Using CH-SIMS v2.0S, this study proposes three improvements: the NAPS pipeline---a seven-stage system integrating face tracking,identity embedding, and normalized lip-motion analysis to reduce visual noise;DS-TANet, combining an EfficientNetB2 static stream, RAFT optical-flow motion stream, motion-guided attention, and Bi-GRU temporal modeling; and DS-TAFNet, fusing visual and MacBERT-Base textual representations via concatenation fusion. With NAPS, the static visual baseline achieves 80.98% Macro F1, comparable to the text baseline of 80.55%; DS-TANet improves visual Macro F1 to 82.58%;and DS-TAFNet achieves 87.49% accuracy and 87.48% Macro F1. These results demonstrate that improving visual input quality and temporal representation is more effective than increasing fusion complexity under limited-data conditions.
Su Li, Yigong Zhang, Lei Xiong +1
School of Information Engineering, Kunming University, No. 2 Puxin Road, Guandu District, Yangpu Campus, Kunming, 650214, Yunnan, China
Weakly supervised video anomaly detection (WSVAD) has predominantly focused on temporal localization, identifying when anomalies occur while largely neglecting their spatial extent within frames. Yet, spatial localization is essential for interpretability and practical deployment in real-world settings. We introduce a patch-based spatiotemporal framework for weakly supervised anomaly localization that jointly models where and when anomalies occur. Our approach operates on grid-level patch features and learns region-level anomaly scores under a multiple instance learning paradigm. We further propose a Proximity-Aware Top-k spatiotemporal selection strategy that enables the model to generate fine-grained spatial anomaly maps without requiring bounding-box supervision during training. Our method surpasses existing state-of-the-art approaches across multiple benchmarks, yielding substantial gains in spatiotemporal localization accuracy. In addition, we release frame-level bounding-box annotations for the test sets of two widely used datasets, along with our code and pretrained models, providing new resources to facilitate future research in spatially grounded WSVAD.