Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning
Authors: Linge Wang, Yingying Chen, Bingke Zhu, Lu Zhou, Jinqiao Wang
Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Objecteye Inc., Beijing, China
Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2% to 37.4% for video-to-audio retrieval and from 27.9% to 37.1% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.
Figures & tables
Figure 1: In existing contrastive masked autoencoder pretraining frameworks, random masking may expose patches with limited cross-modal relevance, such as silent spectrogram regions or visually uninformative background areas. Aggregating such partially observed tokens into global representations can introduce semantic noise into the contrastive branch and weaken cross-modal alignment. In contrast, our framework decouples the masking regimes of reconstruction and alignment, allowing the contrastive branch to use a lower masking ratio better suited to cross-modal alignment, with teacher guidance providing additional structure over the visible tokens.
Figure 2: Overall pipeline of our proposed framework. The model consists of two objective-specific forward passes: (1) the reconstruction branch (left) learns modality-specific generative representations by randomly masking both audio and visual patches and reconstructing them through separate decoders. (2) the contrastive branch (right) adopts a teacher-student scheme for cross-modal alignment under a lower masking ratio. The teacher model observes full, unmasked inputs and produces attention-based token priority cues to guide the student’s masking strategy. The student encodes the masked tokens, appends modality-specific global tokens (gv,ga) , and aligns them via the InfoNCE-based contrastive loss Lcontra . An additional distillation loss Ldis encourages consistency between teacher and student global embeddings. The encoders in the reconstruction branch share weights with the student encoders in the contrastive branch. The final training objective combines all terms as Lall=λ1Lrec+λ2Ldis+λ3Lcontra .
Vision → Audio
Audio → Vision
AudioSet Eval Subset
VGGSound Eval Subset
AudioSet Eval Subset
VGGSound Eval Subset
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
VAB-Encodec [ 33 ]
39.5
65.4
74.6
33.5
63.3
74.3
37.5
64.0
73.7
34.9
62.7
73.1
CAV-MAE [ 16 ]
16.1
38.6
49.3
14.7
35.3
45.9
9.5
22.6
32.4
8.3
23.8
32.4
CAV-MAE Scale+++ [ 16 ]
18.8
39.5
50.1
14.8
34.2
44.0
15.1
34.0
43.0
12.8
30.4
40.3
LanguageBind [ 37 ]
6.4
20.2
28.3
10.3
30.1
39.7
4.4
15.0
22.5
6.5
22.7
33.5
Table 1: Zero-shot retrieval results on AudioSet and VGGSound for Vision → Audio (V → A) and Audio → Vision (A → V) tasks. Our model achieves state-of-the-art zero-shot performance across all retrieval metrics (R@1, R@5, R@10) on both datasets, outperforming existing training-free baselines such as ImageBind, AVSiam, and CAV-MAE variants without any fine-tuning. For reference, we also report the result of VAB-Encodec, which is gray-highlighted because it relies on task-specific fine-tuning on the target retrieval dataset, unlike the other methods in this table that are evaluated in a training-free manner using frozen embeddings.
Method
Pre.
AS20K(mAP)
VGG(Acc)
VAB-Encodec [ 33 ]
AS2M+VGG
33.3
57.6
CAV-MAE [ 16 ]
AS2M
27.3
–
CAV-MAE Scale+++ [ 16 ]
AS2M
25.3
51.6
MaViL [ 22 ]
AS2M
30.0
–
CAV-MAE Sync
AS2M
30.5
52.7
Ours
AS2M
32.0
52.7
Table 2: Audio-visual classification results on AS20K (mAP) and VGGSound (Top-1 accuracy) using attention probing. All models use frozen encoders.
Method
AS20K (Audio-only) mAP
AS20K (Vision-only) mAP
CAV-MAE Sync
29.3
14.3
Ours (TG-DP)
31.2
17.8
Table 3: Unimodal classification on AS20K under the same frozen-encoder attention probing protocol. TG-DP improves both audio-only and vision-only performance over CAV-MAE Sync, indicating that the learned representations are stronger not only for cross-modal alignment but also for single-modality transfer.
Model
Per-epoch (s)
Total
baseline
730
7.1 h
Ours
1045
10.2 h
Table 4: Pretraining runtime comparison. Our method introduces moderate additional training cost due to the extra forward pass and EMA teacher, but incurs no extra computation at inference time.
Dual Pass
VGGSound (V → A)
VGGSound (A → V)
Classification
R@1
R@5
R@10
R@1
R@5
R@10
AS20K (mAP)
w/o
27.9
51.7
61.8
23.2
46.2
58.1
30.5
w/
27.1
51.8
61.2
27.4
50.7
60.1
30.4
Table 5: Ablation on the dual-pass training scheme.
Mask Ratio
VGGSound (V → A)
VGGSound (A → V)
Classification
R@1
R@5
R@10
R@1
R@5
R@10
AS20K (mAP)
0.00
30.4
56.0
64.2
29.8
55.6
64.5
29.6
0.20
32.5
55.7
64.1
31.7
54.5
64.8
30.4
0.50
31.3
54.4
63.9
30.3
53.1
64.3
32.0
0.65
26.4
48.1
57.0
25.1
47.0
58.4
30.5
0.75
27.1
50.4
60.1
26.2
49.4
60.8
30.7
Table 6: Ablation on the contrastive-branch masking ratio . All experiments keep the reconstruction-branch masking unchanged.
Distillation
VGGSound (V → A)
VGGSound (A → V)
Classification
R@1
R@5
R@10
R@1
R@5
R@10
AS20K (mAP)
w/o
29.5
54.7
63.9
29.1
53.1
63.5
30.5
w/
31.3
54.4
63.9
30.3
53.1
64.3
32.0
Table 7: Ablation on the teacher–student distillation mechanism. All experiments use a 0.50 masking ratio in the contrastive branch.
Masking Strategy
VGGSound (V → A)
VGGSound (A → V)
Classification
R@1
R@5
R@10
R@1
R@5
R@10
AS20K (mAP)
Random Mask
31.2
54.3
64.8
30.3
54.5
65.7
30.2
Distinct Guided Mask
31.3
54.4
63.9
30.3
53.1
64.3
32.0
Probabilistic Guided Mask
29.6
56.1
65.4
30.5
54.9
65.0
29.8
Table 8: Ablation on the teacher-guided masking strategy. All experiments are conducted using a 0.50 masking ratio in the contrastive branch.
Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.
Learning aligned multimodal embeddings from weakly paired, label-free corpora is challenging: pipelines often provide only pre-extracted features, clips contain multiple events, and spurious co-occurrences. We propose HSC-MAE (Hierarchical Semantic Correlation-Aware Masked Autoencoder), a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation - from coarse to fine: (i) global-level canonical-geometry correlation via DCCA, which aligns audio and visual embeddings within a shared modality-invariant subspace; (ii) local-level neighborhood-semantics correlation via teacher-mined soft top-k affinities, which preserves multi-positive relational structure among semantically similar instances; and (iii) sample-level conditional-sufficiency correlation via masked autoencoding, which ensures individual embeddings retain discriminative semantic content under partial observation. Concretely, a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE; an EMA teacher operating on unmasked inputs via the CCA path supplies stable canonical geometry and soft positives. Learnable multi-task weights reconcile competing objectives, and an optional distillation loss transfers teacher geometry into the student. Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines, validating that HSC-MAE yields robust and well-structured audio-visual representations.
Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.
Visual object removal can eliminate a target from video frames, yet its acoustic trace persists in the soundtrack, causing obvious audio-visual inconsistency. Existing video inpainting models operate solely on pixels, while audio editing models, especially for the sound removal task, are typically driven by text and therefore rely on limited single-modal control, which is less effective than multimodal guidance that provides stronger semantic grounding and temporal synchronization cues. In this paper, we present Text-Visual Guided Sound Removal (TV-AudioRemover), a target sound removal framework that leverages the visually edited video together with a natural-language instruction to suppress the sound associated with the removed visual object from the original audio mixture. To acquire high-quality training data, we devise a pipeline to construct a million-scale dataset of single-object audio-visual aligned samples, from which we synthesize mixture-target pairs customized for model training. To effectively leverage visual context and follow instruction intent, we augment the model architecture with task tokens, generalizable instruction modeling, and modality-specific global guidance. We further adopt multi-task training to strengthen task-role comprehension, and employ a hard-mixture curriculum that leverages semantically similar acoustic mixtures during fine-tuning to enhance fine-grained source discrimination. To support evaluation, we present AV-Remove-Bench, a comprehensive audio-visual object removal benchmark, along with dedicated objective metrics and an MLLM-based evaluation protocol. Experiments demonstrate that our method achieves state-of-the-art performance on both subjective and objective metrics. Project page: https://yjx-research.github.io/TV-AudioRemover/.