SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization
Authors: Hanyu Xuan, Haoran Xu, Min Yin, Shuo Wang, Cheng Zhao, Kun Li, Fei Wang, Zhiliang Wu, Guanghui Yue, Hehe Fan
Organizations: the School of Big Data and Statistics, Anhui University, Hefei 230039, China. · the School of Information Science and Technology, University of Science and Technology of China, Hefei, China. · the Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China · the College of Information Technology, United Arab Emirates University, Abu Dhabi, 15551, United Arab Emirates · the Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230026, China. · the College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Singapore · the School of Artificial Intelligence, Zhejiang University, Hangzhou 310007, China.
Open-vocabulary audio-visual event localization aims to infer explicit segment-level categories for both seen and unseen audio-visual events. Existing solutions rely heavily on hand-crafted prompt templates, whose predictions are highly sensitive to subtle wording variations. Learnable prompts remove this manual tuning burden, but when trained solely on seen events, they tend to overfit to seen categories and distort the inter-event semantic topology required for generalization to unseen categories, a failure we term semantic topology drift. In this paper, we propose SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams. Specifically, our SIMPLE combines Learnable Continuous Prompt Tuning (LCPT) with Semantic Topology Consistency Regularization (STCR). The LCPT automates prompt engineering through data-driven optimization, while the STCR module aligns the category-wise similarity distributions of learnable and hand-crafted prompts under random category subset sampling to alleviate semantic topology drift. Comprehensive experiments and analyses on the OV-AVEBench demonstrate that our SIMPLE framework achieves state-of-the-art performance on the seen and unseen splits, while using roughly half the trainable parameters and roughly one-third the training time.
Figures & tables
Fig. 1 : Conceptual comparison of closed-set, open-set, zero-shot, and open-vocabulary settings for Audio-Visual Event Localization (AVEL) task.
Fig. 2 : Performance comparison across various hand-crafted templates on the open-vocabulary AVEL benchmark [ 38 ] .
Fig. 3 : Illustration of semantic topology drift induced by learnable prompts. The cross‑modal mapping learned from seen‑event samples mis-aligns the embeddings of unseen events, leading to distorted inter‑event semantic topology.
Fig. 4 : Overview of the proposed S emantic Structure-al I gned Pro MP t LE arning (SIMPLE) framework for open-vocabulary AVEL.
Fig. 5 : Overview of Semantic Topology Consistency Regularization (STCR) within Learnable Continuous Prompt Tuning (LCPT). Our STCR operates on both seen and synthetic novel categories, aligning the pairwise semantic relationships among text embeddings generated by learnable prompts with those derived from an ensemble of hand-crafted templates.
Method
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Video-LLaMA2 [ 74 ]
50.1
40.6
32.0
40.9
48.5
38.5
29.0
38.6
48.9
39.1
29.8
39.3
CLIP [ 61 ] & CLAP [ 75 ]
51.4
41.4
31.9
41.6
51.6
42.2
31.6
41.8
51.5
41.9
31.7
41.7
OVAVE-FT [ 38 ]
57.5
45.0
34.0
45.5
59.8
47.3
34.0
47.0
59.2
46.7
34.0
46.6
CMRA [ 5 ]
65.2
58.8
54.3
59.4
36.0
31.0
26.3
31.1
44.3
38.9
34.3
39.2
AVE [ 3 ]
76.6
63.6
56.0
65.4
44.6
33.2
24.0
34.0
53.8
41.9
33.2
42.9
TABLE I : Performance comparison with state-of-the-art (SOTA) methods on OV-AVEBench. Bold and underlined values denote the best and second-best results, respectively.
Method
Params . ↓
Time ↓
Optimization
OVAVE-FT [ 38 ]
8.4M
≈ 6.0h
Direct training
Cue2Rule [ 59 ]
N/R
9.8 h/session
3×20 search
Ours
4.2M
≈ 2.0h
Direct training
TABLE II : Optimization efficiency comparison on OV-AVEBench. “Params.” indicates the number of updated parameters, while “Time” denotes the training time until obtaining the optimal checkpoint.
Prompt
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
T1 [ 50 ]
78.9
70.8
65.6
71.8
69.2
58.7
49.2
59.0
67.7
57.4
49.5
58.2
+ Ours
0.1 ↑
0.5 ↓
0.8 ↓
0.4 ↓
1.5 ↑
2.0 ↑
2.2 ↑
1.9 ↑
1.8 ↑
1.8 ↑
1.8 ↑
1.8 ↑
T2 [ 51 ]
78.0
68.9
63.3
70.1
68.2
57.6
48.1
58.0
65.9
55.0
46.8
55.9
+ Ours
0.9 ↑
1.1 ↑
1.5 ↑
1.2 ↑
1.4 ↑
2.2 ↑
2.6 ↑
2.1 ↑
2.5 ↑
2.8 ↑
2.9 ↑
2.9 ↑
T3 [ 51 ]
78.4
70.4
65.0
71.3
68.1
58.5
49.4
58.7
67.3
57.3
49.6
58.1
TABLE III : Generalization performance of the proposed framework across seven diverse hand‑designed prompts ( T1∼T7 ). Upward arrows ( ↑ ) denote performance improvements, and downward arrows ( ↓ ) denote performance degradation compared to the baseline model with vanilla prompts. The detailed definitions of all prompts are listed in Fig. 5 .
Training
Method
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
25%
OVAVE-FT [ 38 ]
73.0
62.6
55.7
63.8
63.8
54.4
47.5
55.2
66.4
56.8
49.8
57.7
Ours
76.1
66.4
60.4
67.6
71.7
60.7
51.3
61.2
68.9
57.3
49.1
58.4
50%
OVAVE-FT [ 38 ]
75.3
66.0
59.9
67.1
63.5
53.5
45.6
54.2
66.7
57.1
49.7
57.8
Ours
77.4
68.2
62.5
69.4
71.9
61.3
52.0
61.7
69.6
58.5
50.5
59.5
75%
OVAVE-FT [ 38 ]
75.1
65.4
59.3
66.6
62.5
53.5
46.1
54.0
66.1
56.9
49.9
57.6
TABLE IV : Validity verification about different ratios of training samples compared with OVAVE-FT [ 38 ] .
Cs
Cu
Ratio
Method
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
15
21
0.71
OVAVE-FT [ 38 ]
71.0
59.1
52.4
60.8
53.5
39.7
30.7
41.3
57.1
43.7
35.1
45.3
Ours
80.0
72.1
67.1
73.1
58.9
48.0
38.8
48.6
65.2
53.6
44.8
54.5
52
0.29
OVAVE-FT [ 38 ]
71.0
59.1
52.4
60.8
60.2
45.0
35.9
47.0
61.3
46.4
37.5
48.4
Ours
80.0
72.1
67.1
73.1
62.6
49.3
40.0
50.6
62.5
49.6
40.7
50.9
25
21
1.19
OVAVE-FT [ 38 ]
73.7
62.9
56.4
64.3
63.6
51.5
42.9
52.7
66.3
54.6
46.6
55.8
TABLE V : Comparison with OVAVE-FT [ 38 ] under different seen-to-unseen event-category ratios, where Cs and Cu denote the numbers of seen and unseen event categories, respectively.
Fig. 6 : Semantic topology structure under different prompt strategies. We derive t-SNE layouts from the inter-class affinity matrices of all 67 event categories from OV‑AVEBench [ 38 ] . Marker shapes denote five semantic groups and colors distinguish individual categories. Within each group, red rings mark an anchor, a semantically related category (long-dashed edge), and a less related category (short-dashed edge).
Fig. 7 : Pairwise semantic relation under different prompt strategies, where each row corresponds to a target event and each column to a candidate neighbor.
Fig. 8 : Qualitative examples for seen and unseen audio-visual event categories.
hand.
LCPT
STCR
Seen
Unseen
Total
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
Acc .
Seg .
Eve .
Avg .
✓
✗
✗
78.4
70.4
65.0
71.3
68.1
58.5
49.4
58.7
67.3
57.3
49.6
58.1
✗
✓
✗
78.9
70.2
64.8
71.3
69.4
59.5
49.8
59.6
67.5
57.0
48.5
57.7
✗
✓
✓
79.1
70.0
64.5
71.2
71.3
61.5
52.3
61.7
69.8
59.3
51.4
60.2
✗
✓
✓ w/o novel
78.7
70.0
64.5
71.1
69.3
59.3
49.7
59.4
67.6
57.1
48.7
57.8
✗
✓
✓ w/o random
78.8
69.8
64.3
71.0
70.4
60.2
50.8
60.5
68.8
57.9
49.7
58.8
TABLE VI : Ablation Studies on the core components of our framework.
Fig. 9 : Ablation Studies about (a) the learnable context length P in Eq. 3 and (b) the regularization weight λ in Eq. 8 .
Open-vocabulary audio-visual event localization (OV-AVEL) grounds a text-queried event in time from video, audio, and language. The supervision sources available to this task can differ in temporal-boundary reliability: on OV-AVEBench, our configured visual teacher gives more reliable boundary cues than the configured audio teacher, although the latter is a strong pretrained audio model and remains semantically informative. This is a setting-specific diagnostic rather than a universal ranking of vision and audio. We formulate the resulting challenge as supervision placement: which teacher signals may shape the localization decision, and which should remain auxiliary. Based on this view, we propose OV-OrthKD, a reliability-aware asymmetric distillation framework. Visual feature transfer shapes a decision-aligned representation, audio feature transfer enriches a complementary auxiliary subspace, a text prototype anchors seen/unseen category semantics, and an orthogonality loss limits directional overlap between the two teacher-specific projections. The student continues to use both modalities through query-aware fusion at inference, while the default training recipe keeps audio-teacher supervision off the segment-logit path. On OV-AVEBench, OV-OrthKD achieves 0.816 segment AP and improves [email protected] over the official fine-tuning baseline by 2.7 points overall and 3.4 points on unseen categories. Path-assignment, role-swap, corruption, and transfer analyses consistently support supervision placement as a task-specific design axis for OV-AVEL.
Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex architecture sets a new state of the art on both OV-AVEL benchmarks. On the open (unseen-class) split of OV-AVEBench we reach 66.5/59.1/54.1% Acc/Seg-F1/Event-F1 (+1.6/+4.1/+6.6 over the previously reported fine-tuned baseline), with consistent gains for seen classes as well. We also modify AVE dataset for this task and observe that our architecture reaches 60.7/51.9/50.4% Acc/Seg-F1/Event-F1, achieving state-of-the-art OV-AVEL results on it as well. We also propose a two-stream alternative, which also sees great improvements over the baseline.
Open-vocabulary audio-visual event localization (OV-AVEL) jointly models audio-visual cues to recognize and temporally localize events, including categories unseen during training. Existing methods primarily learn joint audio-visual representations in Euclidean space, but still face two significant challenges. First, the lack of supervision signals for unseen categories makes it difficult to maintain audio-visual consistency across multiple temporal scales. Second, the lack of hierarchical constraints between segment- and video-level semantics prevents the model from establishing semantic consistency across different levels. To address these challenges, we propose a hierarchical semantic constrained heterogeneous graph (HSCHG) for audio-visual event localization framework. We first construct a heterogeneous hierarchical graph in Euclidean space, which includes audio and visual segment nodes and their corresponding video-level nodes. We use multi-directional temporal edges to capture complete temporal information within each modality. Simultaneously, we employ a dual-threshold filtering gated fusion strategy, introducing cross-modal information only when the alignment confidence is high. Furthermore, we introduce bidirectional semantic constraints between segment- and video-level representations to achieve semantic consistency across different levels. Based on this, we map the multi-level audio-visual representations and text prototypes uniformly into hyperbolic space. We use a hierarchical entailment regularization loss to characterize the hierarchical relationships between videos and segments. Extensive experimental results show that our method outperforms existing methods on the OV-AVEL benchmark. Ablation studies further validate the effectiveness of our method.
Zhe Yang, Ruyi Zhang, Hongtao Chen +4
Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China · Peng Cheng Laboratory, Shenzhen 518000, China · Harbin Institute of Technology Suzhou Research Institute, Suzhou 215000, China