If You Hear It, Help Find It: Orthogonal Knowledge Distillation for Open-Vocabulary Audio-Visual Event Localization

Yi Xu; Cheng Chen; Wenzhuo Lei

cs.MMSep 20, 2026

If You Hear It, Help Find It: Orthogonal Knowledge Distillation for Open-Vocabulary Audio-Visual Event Localization

Authors: Yi Xu, Cheng Chen, Wenzhuo Lei

Organizations: Tongji University Shanghai, China

Abstract

Open-vocabulary audio-visual event localization (OV-AVEL) grounds a text-queried event in time from video, audio, and language. The supervision sources available to this task can differ in temporal-boundary reliability: on OV-AVEBench, our configured visual teacher gives more reliable boundary cues than the configured audio teacher, although the latter is a strong pretrained audio model and remains semantically informative. This is a setting-specific diagnostic rather than a universal ranking of vision and audio. We formulate the resulting challenge as supervision placement: which teacher signals may shape the localization decision, and which should remain auxiliary. Based on this view, we propose OV-OrthKD, a reliability-aware asymmetric distillation framework. Visual feature transfer shapes a decision-aligned representation, audio feature transfer enriches a complementary auxiliary subspace, a text prototype anchors seen/unseen category semantics, and an orthogonality loss limits directional overlap between the two teacher-specific projections. The student continues to use both modalities through query-aware fusion at inference, while the default training recipe keeps audio-teacher supervision off the segment-logit path. On OV-AVEBench, OV-OrthKD achieves 0.816 segment AP and improves [email protected] over the official fine-tuning baseline by 2.7 points overall and 3.4 points on unseen categories. Path-assignment, role-swap, corruption, and transfer analyses consistently support supervision placement as a task-specific design axis for OV-AVEL.

Figures & tables

Figure 1. Overview of OV-OrthKD . AV encoders, query-aware fusion, and temporal modeling produce a shared student representation, which is read by (A) a visual-aligned path supporting boundary prediction, (B) an audio-aligned auxiliary path for audio feature transfer, and (C) a query-aligned branch tied to the text prototype. Query-aware gating controls inference-time fusion; reliability-aware path separation controls training-time supervision routing. The default recipe excludes audio-teacher supervision from segment logits, and the gray visual-logit KD branch is analysis-only. Teachers are discarded at inference. The example video frames are drawn from OV-AVEBench ( Zhou et al., 2025 ) . Pipeline overview of OV-OrthKD with grouped regions for inputs, offline teachers, and student training. The student builds a shared representation with audio-visual encoders, query-aware fusion, and temporal modeling, then routes it into visual-aligned, audio-aligned auxiliary, and query-aligned branches. The visual teacher exports features and an optional analysis-only logit branch; the audio teacher exports auxiliary features and does not supervise segment logits in the default recipe. The objective contains supervised BCE, visual and audio feature transfer, text alignment, and orthogonality. Inference uses only the student.

Figure 2. Geometric intuition for the teacher-specific projection heads. The orthogonality loss reduces overlap between the visual- and audio-aligned projected student features; it does not force the raw inference-time modalities to be orthogonal. A conceptual geometry diagram with three axes for visual-aligned, audio-aligned, and query-aligned student projections, showing separated subspaces and an orthogonality indicator between the visual and audio directions.

Figure 3. Qualitative comparison on four representative OV-AVEBench queries. All displayed video frames are drawn from OV-AVEBench ( Zhou et al., 2025 ) . Each panel shows eight representative frames, temporal prediction curves, and the ground-truth window. The train-horning and dog-barking cases highlight gains over student controls, the electric-shaver case shows broad agreement, and the basketball-bounce example documents a shared hard case. A four-panel qualitative figure with examples for train horning, dog barking, electric shaver, and basketball bounce. Each panel contains a grid of representative video frames, temporal prediction curves for multiple methods, and a ground-truth segment bar.

Item	Setting
Inputs	Video at $224\times 224$ , 16 fps, 16 temporal segments; audio at 16 kHz on the same timeline
Student	ConvNeXtV2-Tiny + EfficientNetV2-B2 ; 4-layer temporal transformer; $D=384$
Teachers	InternVideo2 for visual supervision, BEATs for audio supervision, CLAP for text prototypes
Optimization	AdamW, base learning rate $2\times 10^{-4}$ , step400 , early stopping
Loss Weights	$\lambda_{\mathrm{sup}}=1.0$ , $\lambda_{\mathrm{text}}=0.8$ , $\lambda_{v\text{-feat}}=0.4$ , $\lambda_{a\text{-feat}}=0.1$ , $\lambda_{\mathrm{orth}}=0.5$ , $\lambda_{v\text{-kd}}=0.0$
Deployment	45.8M parameters; student-only inference; runtime excludes offline teacher export and training

Table 1. Compact implementation summary shared by all experiments. Teacher features are exported offline; only the student is active at inference time.

Signal	Val AP	Test AP	Val AUROC	Test AUROC
Visual direct logits	0.767	0.776	0.707	0.716
Visual feature probe	0.764	0.765	0.669	0.670
Audio feature probe	0.716	0.718	0.644	0.645

Table 2. Setting-specific teacher diagnostics on OV-AVEBench. “Direct logits” evaluates exported visual-teacher logits, while “feature probe” fits a light linear probe on frozen teacher features.

ID	Variant	Audio	Orth.	Text	U-AP	AP
1	Visual only	–	no	yes	0.492	0.778
2	Visual + Orth.	–	yes	yes	0.488	0.777
3	Decision KD	Decision	no	yes	0.501	0.745
4	Symmetric transfer	Shared	no	yes	0.505	0.737
5	w/o text	Auxiliary	yes	no	0.086	0.485
6	OV-OrthKD	Auxiliary	yes	yes	0.584	0.816

Table 3. Supervision-placement ablation on OV-AVEBench. “Audio route” records where audio-teacher supervision enters the student. Only completed experimental combinations are reported.

Figure 4. Five-seed sensitivity to

\lambda_{\mathrm{orth}}

. Markers show mean AP and error bars show standard deviation (

\pm 0.003

). Performance peaks near

0.5

; this focused sweep does not cover every loss weight. A line plot showing validation and test AP as a function of the orthogonality weight lambda, with error bars denoting standard deviation across five seeds. Both curves peak at lambda 0.5 and remain above their values at lambda 0 over the tested range.

Model	AP	AUROC	[email protected]	Calib. F1
ImageBind (zero-shot)	0.592	0.534	0.467	0.467
Video-LLaMA2 (LLM)	0.489	0.455	0.391	0.391
Zhou et al. ( Zhou et al., 2025 )	0.745	0.650	0.569	-
Student-only (no teacher)	0.714	0.612	0.523	0.719
Visual feature only	0.778	0.701	0.568	0.774
OV-OrthKD (Ours)	0.816	0.750	0.596	0.781

Table 4. Main OV-AVEBench test results. We compare against the official fine-tuning baseline ( Zhou et al., 2025 ) , a training-free ImageBind baseline ( Girdhar et al., 2023 ) , and the Video-LLaMA2 baseline used in OV-AVEBench. [email protected] follows the standard

0.5

threshold used by Zhou et al.; the AP/AUROC numbers for Zhou et al. are obtained by re-evaluating that official baseline under our ranking protocol.

Split	Model	Acc.	Seg. F1	$\Delta$
Unseen	Zhou et al. ( Zhou et al., 2025 )	64.9	55.0	-
Unseen	OV-OrthKD (Ours)	67.2	58.4	+3.4
Seen	Zhou et al. ( Zhou et al., 2025 )	72.5	61.8	-
Seen	OV-OrthKD (Ours)	76.5	62.5	+0.7
Total	Zhou et al. ( Zhou et al., 2025 )	67.1	56.9	-
Total	OV-OrthKD (Ours)	70.5	59.6	+2.7

Table 5. Protocol-aligned comparison with the official fine-tuning baseline on OV-AVEBench test set. All models use the standard

0.5

threshold.

Variant	Role map	Vision / Audio	AP
Normal (Ours)	Visual-Dec / Audio-Aux	IntV2 / BEATs	0.816
Role Swap	Audio-Dec / Visual-Aux	IntV2 / BEATs	0.737
Alt. Teacher	Visual-Dec / Audio-Aux	CLIP-L / BEATs	0.782
Alt. Teacher	Visual-Dec / Audio-Aux	IntV2 / CLAP	0.798

Table 6. Role assignment and inference efficiency. Role swap tests whether the gain comes from supervision placement rather than adding more branches.

Model	F1 ( $\uparrow$ )	AP ( $\uparrow$ )	AUROC ( $\uparrow$ )
Student-only	0.521	0.707	0.582
Visual feature only	0.615	0.712	0.587
OV-OrthKD (Ours)	0.675	0.728	0.612

Table 7. Cross-dataset transfer to UnAV-100 without fine-tuning.

Panel of Figure 5 — Figure 5. Robustness under corrupted audio cues. Left: temporal jitter, compared with Visual-only and Symmetric KD controls. Right: Gaussian audio-feature noise, compared with Decision-KD. OV-OrthKD maintains a clear lead and degrades substantially less than the corresponding baselines. Two side-by-side line plots comparing robustness under corrupted audio. Panel (a) shows temporal jitter in seconds on the horizontal axis and test AP on the vertical axis; OV-OrthKD stays above the Visual-only and Symmetric KD controls as jitter increases. Panel (b) shows Gaussian noise standard deviation on the horizontal axis and test AP on the vertical axis; OV-OrthKD remains nearly flat while Decision-KD drops much more steeply.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion
Oct 8, 2026Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi +2Audio-Visual Understanding Multimodal Fusion
IIT Kanpur Kanpur, India · Dolby Laboratories Bengaluru, India
Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization
Jun 5, 2026Zhe Yang, Ruyi Zhang, Hongtao Chen +4Audio-Visual Understanding Heterogeneous GNNs
Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China · Peng Cheng Laboratory, Shenzhen 518000, China · Harbin Institute of Technology Suzhou Research Institute, Suzhou 215000, China
SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization
Oct 11, 2026Hanyu Xuan, Haoran Xu, Min Yin +7Audio-Visual Understanding Prompt Learning
the School of Big Data and Statistics, Anhui University, Hefei 230039, China. · the School of Information Science and Technology, University of Science and Technology of China, Hefei, China. · the Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China +4

Oct 8, 2026J/K move · Enter open · S save

Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal+1

IIT Kanpur Kanpur, India · Dolby Laboratories Bengaluru, India

Audio-Visual Understanding Multimodal Fusion

Open paper AskPDF arXiv

Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex architecture sets a new state of the art on both OV-AVEL benchmarks. On the open (unseen-class) split of OV-AVEBench we reach 66.5/59.1/54.1% Acc/Seg-F1/Event-F1 (+1.6/+4.1/+6.6 over the previously reported fine-tuned baseline), with consistent gains for seen classes as well. We also modify AVE dataset for this task and observe that our architecture reaches 60.7/51.9/50.4% Acc/Seg-F1/Event-F1, achieving state-of-the-art OV-AVEL results on it as well. We also propose a two-stream alternative, which also sees great improvements over the baseline.

Abstract

Audio-Visual Understanding

Explore similar work

Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization

SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization