Organizations: Hefei University of Technology · MBZUAI · NUS · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · The Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology
Point-level weakly-supervised temporal sentiment localization (P-WTSL) aims to detect sentiment-relevant segments in untrimmed multimodal videos using timestamp sentiment annotations, which greatly reduces the costly frame-level labeling. To further tackle the challenges of imprecise sentiment boundaries in P-WTSL, we propose the Face-guided Sentiment Boundary Enhancement Network (FSENet), a unified framework that leverages fine-grained facial features to guide sentiment localization. Specifically, our approach first introduces the Face-guided Sentiment Discovery (FSD) module, which integrates facial features into multimodal interaction via dual-branch modeling for effective sentiment stimuli clues; We then propose the Point-aware Sentiment Semantics Contrast (PSSC) strategy to discriminate sentiment semantics of candidate points (frame-level) near annotation points via contrastive learning, thereby enhancing the model's ability to recognize sentiment boundaries. At last, we design the Boundary-aware Sentiment Pseudo-label Generation (BSPG) approach to convert sparse point annotations into temporally smooth supervisory pseudo-labels. Extensive experiments and visualizations on the benchmark demonstrate the effectiveness of our framework, achieving state-of-the-art performance under full supervision, video-level, and point-level weak supervision, thereby showcasing the strong generalization ability of our FSENet across different annotation settings.
Figures & tables
Figure 1: Illustration of the task and motivation. (a) Given the point-level sentiment points in the untrimmed video, the model needs to achieve sentiment classification and boundary localization. (b) We explored the facial-centric method to guide sentiment clues discovery. (c) To alleviate sentiment boundary jitter and discontinuity under weak supervision, we propose a novel pseudo-label smoothing strategy.
Figure 2: Overview of our FSENet framework. (a)FSD module introduces facial features into the temporal sentiment localization task through two branches to advance discovery of sentiment clues: one focuses on facial-centric interaction, while the other employs global sentiment perception weighting score. (b)PSSC aims to identify similar and dissimilar semantic points in the temporal axis to enhance the point-level sentiment discrimination by contrastive learning. (c)BSPG transforms sentiment point-level annotations into segment-level sentiment boundaries pseudo-label using the threshold filtering and step-by-step boundary-aware scheme, where sentiment scores decay progressively based on parameters β and w .
Setting
Method
mAP@IoU(%)
Average
0.1
0.15
0.2
0.25
0.3
Recall
F2 Score
mAP
Full Supervision
VAANet [ 60 ]
28.01
24.95
20.68
16.45
12.50
69.82
39.59
20.51
† AFSD [ 61 ]
33.44
27.09
21.78
16.64
11.76
75.10
31.78
22.14
FSENet(ours)
28.02
25.45
22.63
18.93
14.33
79.58
30.37
21.87
Weak Supervision (Video-level)
CoLA [ 62 ]
11.87
10.14
8.11
6.83
4.98
64.07
21.19
8.38
UM [ 34 ]
14.77
13.28
10.77
8.41
6.54
55.33
23.94
10.75
Table 1: Performance comparison with SOTA methods under Fully and Weakly supervised settings on TSL300 . “ † ” indicates methods reproduced by ourselves. The best and second-best results are bolded and underlined , respectively.
Method
mAP@IoU(%)
Average
0.1
0.2
0.3
Recall
F2 Score
mAP
HR-pro stage1
11.83
7.26
4.44
18.81
18.46
7.65
HR-pro stage2
16.27
11.49
6.76
26.61
24.71
11.47
TSL
15.18
10.71
7.46
27.24
24.86
11.00
FSENet (Ours)
22.7
22.49
13.24
33.08
24.89
16.54
Table 2: Performance results on CMU-MOSEI dataset under point-level settings.
Method
mAP@IoU (%)
Avg
0.1
0.2
0.3
mAP
Qwen3-Omni(zero-shot)
7.67
5.03
2.67
5.07
LLaMA-2-7B (LoRA)
10.98
10.98
8.45
9.97
LLaMA-2-7B (LoRA+BSPG)
27.76
11.82
5.16
11.75
Ours (small model)
29.31
22.49
13.24
21.45
Table 3: Comparison with large language models (LLMs) on the TSL300 dataset. Zero-shot methods are evaluated without fine-tuning. while LoRA-based models are fine-tuned with point-level supervision.
Method
mAP@IoU (%)
Avg
0.1
0.15
0.2
0.25
0.3
mAP
AV Features Only
HR-pro stage1
16.64
13.95
11.86
9.74
8.01
12.04
HR-pro stage2
26.40
21.80
16.91
13.27
9.96
17.67
TSL
28.14
23.02
18.68
14.36
10.46
18.93
FSENet(ours)
25.91
24.70
20.08
16.85
11.53
19.41
Table 4: Fair evaluation under identical feature settings on TSL300 . Group ➀ uses only audio-visual features, and group ➁ adds local facial features. All △ adopt the same facial extractor.
#id
FCI
GSP
mAP@IoU (%)
Avg
first stage
second stage
0.1
0.2
0.3
mAP
a
–
–
✓
24.95
18.63
11.52
18.42
b
✓
–
✓
27.19
18.93
11.16
19.10
c
–
✓
✓
27.46
18.31
10.37
18.78
d
✓
✓
–
28.69
20.83
11.26
20.22
e
✓
✓
✓
29.31
22.49
13.24
21.45
Table 5: Ablation study on the main network component in FSD. We report the result of mAPs, and ‘–’ denotes not used.
Pseudo-Label
Threshold
Boundary
mAP@IoU(%)
Avg
0.1
0.2
0.3
mAP
None
–
–
27.51
15.31
8.72
16.92
Threshold Only
✓
–
28.06
20.98
11.55
20.29
Step Only
–
✓
25.76
21.59
13.10
20.31
Ours (full model)
✓
✓
29.31
22.49
13.24
21.45
Table 6: Ablation of pseudo-label generation strategies.
#id
Lframe
Lframeglo
Lsc
mAP@IoU(%)
Avg
0.1
0.2
0.3
mAP
a
✓
✓
✓
29.31
22.49
13.24
21.45
b
–
✓
✓
28.28
19.52
10.33
20.02
c
✓
–
✓
27.67
20.40
9.99
20.65
d
✓
✓
–
29.17
21.18
12.49
20.94
e
–
–
✓
19.15
11.91
7.12
12.53
Table 7: Ablation of weakly-supervised optimization losses.
Figure 3: Visualization of the predicted results for temporal sentiment localization. The ‘pink’ segments are positive sentiment, ‘blue’ means negative, and ‘gray’ means no valid location output. △ indicates that methods incorporate facial features settings.
Figure 4: T-SNE visualization comparing our FSENet with SOTA. Each point represents a segment-level feature (‘p/n/n-sent’ denotes positive, negative, and non-sentiment categories, respectively).
BSPG
w
5
6
7
8
9
mAP (%)
19.22
19.77
21.45
19.66
19.26
β
0.1
0.3
0.5
0.6
0.7
mAP (%)
19.64
18.88
21.29
21.45
21.20
Table 8: Ablation study of BSPG parameters (confidence decay factor β and smoothing width w ) on mAP performance.
Figure 5: Ablation study of Top-K parameter k in PSSC on mAP performance.
Figure 6: Ablation study of loss function weights λ1 and λ2 on mAP performance.
Figure 7: Proportion of proposal type for all prediction results of models on TSL300. Error categories: Bg (background), Loc (localization), Dbl (double detection), Conf (confusion), WLbl (wrong label), and TP (true positive).
Modality
mAP@IoU (%)
Avg
0.1
0.15
0.2
0.25
0.3
mAP
Single Modality
A (Audio)
18.98
15.47
12.96
9.75
7.09
12.85
V (Visual)
19.27
16.14
12.06
9.14
6.45
12.61
F (Local Face)
22.11
15.97
11.33
8.40
6.62
12.88
Bimodal Combinations
Table 9: Performance comparison of different modality combinations on the TSL task. A: Audio, V: Visual, F: Local Face. Metrics are reported as mAP@IoU (%) across various thresholds and average mAP.
Method
mAP@IoU (%)
Avg
0.1
0.15
0.2
0.25
0.3
mAP
L2
28.06
24.40
20.98
16.49
11.55
20.29
L1
26.53
23.66
20.70
16.36
12.97
20.05
Dot
27.23
24.90
21.21
17.42
14.22
21.00
Cosine(Ours)
29.31
25.47
22.49
16.76
13.24
21.45
Table 10: Ablation of feature distance metrics for Point-aware Sentiment Semantics Contrast modeling.