SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization
Authors: Hanyu Xuan, Haoran Xu, Min Yin, Shuo Wang, Cheng Zhao, Kun Li, Fei Wang, Zhiliang Wu, Guanghui Yue, Hehe Fan
Organizations: the School of Big Data and Statistics, Anhui University, Hefei 230039, China. · the School of Information Science and Technology, University of Science and Technology of China, Hefei, China. · the Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China · the College of Information Technology, United Arab Emirates University, Abu Dhabi, 15551, United Arab Emirates · the Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230026, China. · the College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Singapore · the School of Artificial Intelligence, Zhejiang University, Hangzhou 310007, China.
Open-vocabulary audio-visual event localization aims to infer explicit segment-level categories for both seen and unseen audio-visual events. Existing solutions rely heavily on hand-crafted prompt templates, whose predictions are highly sensitive to subtle wording variations. Learnable prompts remove this manual tuning burden, but when trained solely on seen events, they tend to overfit to seen categories and distort the inter-event semantic topology required for generalization to unseen categories, a failure we term semantic topology drift. In this paper, we propose SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams. Specifically, our SIMPLE combines Learnable Continuous Prompt Tuning (LCPT) with Semantic Topology Consistency Regularization (STCR). The LCPT automates prompt engineering through data-driven optimization, while the STCR module aligns the category-wise similarity distributions of learnable and hand-crafted prompts under random category subset sampling to alleviate semantic topology drift. Comprehensive experiments and analyses on the OV-AVEBench demonstrate that our SIMPLE framework achieves state-of-the-art performance on the seen and unseen splits, while using roughly half the trainable parameters and roughly one-third the training time.
Figures & tables
Fig. 1 : Conceptual comparison of closed-set, open-set, zero-shot, and open-vocabulary settings for Audio-Visual Event Localization (AVEL) task.
Fig. 2 : Performance comparison across various hand-crafted templates on the open-vocabulary AVEL benchmark [ 38 ] .
Fig. 3 : Illustration of semantic topology drift induced by learnable prompts. The cross‑modal mapping learned from seen‑event samples mis-aligns the embeddings of unseen events, leading to distorted inter‑event semantic topology.
Fig. 4 : Overview of the proposed S emantic Structure-al I gned Pro MP t LE arning (SIMPLE) framework for open-vocabulary AVEL.
Fig. 5 : Overview of Semantic Topology Consistency Regularization (STCR) within Learnable Continuous Prompt Tuning (LCPT). Our STCR operates on both seen and synthetic novel categories, aligning the pairwise semantic relationships among text embeddings generated by learnable prompts with those derived from an ensemble of hand-crafted templates.
Method
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Video-LLaMA2 [ 74 ]
50.1
40.6
32.0
40.9
48.5
38.5
29.0
38.6
48.9
39.1
29.8
39.3
CLIP [ 61 ] & CLAP [ 75 ]
51.4
41.4
31.9
41.6
51.6
42.2
31.6
41.8
51.5
41.9
31.7
41.7
OVAVE-FT [ 38 ]
57.5
45.0
34.0
45.5
59.8
47.3
34.0
47.0
59.2
46.7
34.0
46.6
CMRA [ 5 ]
65.2
58.8
54.3
59.4
36.0
31.0
26.3
31.1
44.3
38.9
34.3
39.2
AVE [ 3 ]
76.6
63.6
56.0
65.4
44.6
33.2
24.0
34.0
53.8
41.9
33.2
42.9
TABLE I : Performance comparison with state-of-the-art (SOTA) methods on OV-AVEBench. Bold and underlined values denote the best and second-best results, respectively.
Method
Params . ↓
Time ↓
Optimization
OVAVE-FT [ 38 ]
8.4M
≈ 6.0h
Direct training
Cue2Rule [ 59 ]
N/R
9.8 h/session
3×20 search
Ours
4.2M
≈ 2.0h
Direct training
TABLE II : Optimization efficiency comparison on OV-AVEBench. “Params.” indicates the number of updated parameters, while “Time” denotes the training time until obtaining the optimal checkpoint.
Prompt
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
T1 [ 50 ]
78.9
70.8
65.6
71.8
69.2
58.7
49.2
59.0
67.7
57.4
49.5
58.2
+ Ours
0.1 ↑
0.5 ↓
0.8 ↓
0.4 ↓
1.5 ↑
2.0 ↑
2.2 ↑
1.9 ↑
1.8 ↑
1.8 ↑
1.8 ↑
1.8 ↑
T2 [ 51 ]
78.0
68.9
63.3
70.1
68.2
57.6
48.1
58.0
65.9
55.0
46.8
55.9
+ Ours
0.9 ↑
1.1 ↑
1.5 ↑
1.2 ↑
1.4 ↑
2.2 ↑
2.6 ↑
2.1 ↑
2.5 ↑
2.8 ↑
2.9 ↑
2.9 ↑
T3 [ 51 ]
78.4
70.4
65.0
71.3
68.1
58.5
49.4
58.7
67.3
57.3
49.6
58.1
TABLE III : Generalization performance of the proposed framework across seven diverse hand‑designed prompts ( T1∼T7 ). Upward arrows ( ↑ ) denote performance improvements, and downward arrows ( ↓ ) denote performance degradation compared to the baseline model with vanilla prompts. The detailed definitions of all prompts are listed in Fig. 5 .
Training
Method
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
25%
OVAVE-FT [ 38 ]
73.0
62.6
55.7
63.8
63.8
54.4
47.5
55.2
66.4
56.8
49.8
57.7
Ours
76.1
66.4
60.4
67.6
71.7
60.7
51.3
61.2
68.9
57.3
49.1
58.4
50%
OVAVE-FT [ 38 ]
75.3
66.0
59.9
67.1
63.5
53.5
45.6
54.2
66.7
57.1
49.7
57.8
Ours
77.4
68.2
62.5
69.4
71.9
61.3
52.0
61.7
69.6
58.5
50.5
59.5
75%
OVAVE-FT [ 38 ]
75.1
65.4
59.3
66.6
62.5
53.5
46.1
54.0
66.1
56.9
49.9
57.6
TABLE IV : Validity verification about different ratios of training samples compared with OVAVE-FT [ 38 ] .
Cs
Cu
Ratio
Method
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
15
21
0.71
OVAVE-FT [ 38 ]
71.0
59.1
52.4
60.8
53.5
39.7
30.7
41.3
57.1
43.7
35.1
45.3
Ours
80.0
72.1
67.1
73.1
58.9
48.0
38.8
48.6
65.2
53.6
44.8
54.5
52
0.29
OVAVE-FT [ 38 ]
71.0
59.1
52.4
60.8
60.2
45.0
35.9
47.0
61.3
46.4
37.5
48.4
Ours
80.0
72.1
67.1
73.1
62.6
49.3
40.0
50.6
62.5
49.6
40.7
50.9
25
21
1.19
OVAVE-FT [ 38 ]
73.7
62.9
56.4
64.3
63.6
51.5
42.9
52.7
66.3
54.6
46.6
55.8
TABLE V : Comparison with OVAVE-FT [ 38 ] under different seen-to-unseen event-category ratios, where Cs and Cu denote the numbers of seen and unseen event categories, respectively.
Fig. 6 : Semantic topology structure under different prompt strategies. We derive t-SNE layouts from the inter-class affinity matrices of all 67 event categories from OV‑AVEBench [ 38 ] . Marker shapes denote five semantic groups and colors distinguish individual categories. Within each group, red rings mark an anchor, a semantically related category (long-dashed edge), and a less related category (short-dashed edge).
Fig. 7 : Pairwise semantic relation under different prompt strategies, where each row corresponds to a target event and each column to a candidate neighbor.
Fig. 8 : Qualitative examples for seen and unseen audio-visual event categories.
hand.
LCPT
STCR
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
✓
✗
✗
78.4
70.4
65.0
71.3
68.1
58.5
49.4
58.7
67.3
57.3
49.6
58.1
✗
✓
✗
78.9
70.2
64.8
71.3
69.4
59.5
49.8
59.6
67.5
57.0
48.5
57.7
✗
✓
✓
79.1
70.0
64.5
71.2
71.3
61.5
52.3
61.7
69.8
59.3
51.4
60.2
✗
✓
✓ w/o novel
78.7
70.0
64.5
71.1
69.3
59.3
49.7
59.4
67.6
57.1
48.7
57.8
✗
✓
✓ w/o random
78.8
69.8
64.3
71.0
70.4
60.2
50.8
60.5
68.8
57.9
49.7
58.8
TABLE VI : Ablation Studies on the core components of our framework.
Fig. 9 : Ablation Studies about (a) the learnable context length P in Eq. 3 and (b) the regularization weight λ in Eq. 8 .
Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China · Peng Cheng Laboratory, Shenzhen 518000, China · Harbin Institute of Technology Suzhou Research Institute, Suzhou 215000, China