Attribution-Guided Shared-Private Decoupling for Noise-Reduced Audio-Visual Representation Learning
Authors: Linge Wang, Yingying Chen, Bingke Zhu, Lu Zhou, Haixin Wang, Zhipiao Liu, Jinqiao Wang
Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Huawei Technologies Ltd., Beijing, China · Objecteye Inc., Beijing, China
Audio-visual representation learning commonly aligns paired audio and visual signals with sample-level contrastive objectives. However, real-world audio-visual pairs usually contain heterogeneous patch tokens: only a subset carries reliable cross-modal shared semantics, while many others mainly encode modality-private residuals. Aggregating all tokens into a single global representation can therefore introduce semantic noise into cross-modal alignment. Motivated by the minimal-sufficiency principle, we propose an attribution-guided shared-private decoupling framework to mitigate this issue. Our key idea is to identify shared-cover subviews that preserve dominant cross-modal information while suppressing modality-private residuals, and use them to supervise decoupled shared and private representations. To identify such subviews efficiently, we draw on transformer attribution method and introduce Layer-Truncated Attribution (LTA), which provides a lightweight estimate of token-level relevance to cross-modal shared information. We further adopt a teacher-student framework, where the teacher provides attribution-induced supervision and the student internalizes shared-private decoupling into its representations, without introducing extra inference cost. Experiments on retrieval, classification, and sound-prompted semantic segmentation show consistent gains over strong baselines. Compared with baseline, our method improves the average zero-shot R@1 by 4.0 across AudioSet and VGGSound retrieval, and improves AS20K classification mAP by 3.1, demonstrating the effectiveness of fine grained attribution guidance for reducing semantic noise.reducing semantic noise. Code will be released at https://github.com/wanglg20/LTA
Figures & tables
Figure 1: Overall pipeline of our proposed framework. (a) Attribution-guided teacher-student training. A pretrained teacher generates high-relevance shared-cover subviews and low-relevance residual complements, which serve as stop-gradient anchors for training the student’s shared and private tokens. For clarity, we omit the MAE branch inherited from the baseline Wang et al. (2026) . (b) Layer-Truncated Attribution (LTA). Given the cross-modal matching score, LTA computes CLS-to-token gradient-attention relevance from the last Transformer block.
Vision → Audio
Audio → Vision
AudioSet Eval Subset
VGGSound Eval Subset
AudioSet Eval Subset
VGGSound Eval Subset
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
CAV-MAE Gong et al. (2023)
16.1
38.6
49.3
14.7
35.3
45.9
9.5
22.6
32.4
8.3
23.8
32.4
CAV-MAE Scale+++ Araujo et al. (2025)
18.8
39.5
50.1
14.8
34.2
44.0
15.1
34.0
43.0
12.8
30.4
40.3
LanguageBind Zhu et al. (2023)
6.4
20.2
28.3
10.3
30.1
39.7
4.4
15.0
22.5
6.5
22.7
33.5
AVSiam Lin and Bertasius (2024)
19.7
–
–
19.0
–
–
17.6
–
–
20.4
–
–
Table 1: Zero-shot cross-modal retrieval on the AudioSet-20K and VGGSound evaluation subsets. We report Recall@1, Recall@5, and Recall@10 for both Vision → Audio (V → A) and Audio → Vision (A → V) retrieval using frozen representations.
Table 3
V → A
A → V
Subview Score
R@1
R@5
R@10
R@1
R@5
R@10
Full view
31.7
56.0
65.9
30.5
54.0
65.3
Similarity
21.6
46.2
56.4
23.2
44.8
54.0
Attention
28.5
54.9
64.2
28.7
52.7
63.7
CheferCAM Chefer et al. (2021b)
37.2
61.6
70.6
37.5
60.9
68.3
LTA
44.1
68.5
77.3
41.2
67.2
75.8
Table 4: Empirical validation of LTA-induced subviews on VGGSound. We compare teacher retrieval performance using subviews constructed by different token scoring functions.
Similarity
Attention
LTA
Pair condition
mIoU
mAP
mIoU
mAP
mIoU
mAP
Matched
22.3
25.2
34.2
42.8
44.2
60.1
Mismatched
22.2
25.4
34.4
43.2
34.7
45.7
Table 5: Sound-source localization under matched and mismatched audio-video pairing on AVSBench. For mismatched pairs, the audio is replaced while the visual input and its original sound-source annotation are retained. Both mIoU and mAP are reported in percentage.
Token
Direction
R@1
R@5
R@10
Private
V → A
29.5
53.9
63.9
Private
A → V
28.2
52.9
63.8
Shared
V → A
35.4
59.6
70.5
Shared
A → V
34.3
57.9
69.8
Table 6: VGGSound zero-shot retrieval using different tokens as the global representation.
Setting
Anchor
Cons.
Dec.
Avg. R@1
AS20K mAP
Seg. mAP
Backbone
30.8
32.0
24.3
+ Anchor
✓
33.0
32.2
28.3
+ Consistency
✓
✓
34.3
32.0
26.3
+ Decoupled tokens
✓
✓
✓
34.9
35.1
38.2
Table 7: Incremental ablation of the main training components. Anchor, Cons., and Dec. denote attribution-induced anchor supervision, consistency regularization, and shared-private token decoupling, respectively. Avg. R@1 is averaged over the two retrieval directions on VGGSound. Seg. mAP denotes sound-prompted semantic segmentation mAP on AVSBench.
Retention
Avg. R@1
AS20K mAP
Seg. mAP
0.25
32.6
35.3
38.3
0.35
33.9
35.2
37.3
0.50
34.9
35.1
38.2
0.75
34.8
35.5
31.5
Table 8: Effect of the retention ratio.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Teacher Pretrain
Student Pretrain
AS20K Attn. Probe
VGGSound Attn. Probe
Dataset
AS-2M
AS-2M
AS-20K
VGGSound
Optimizer
Adam
Adam
Adam
Adam
Learning Rate
2×10−4
2×10−4
5×10−3
1×10−3
LR Scheduler
Cosine
Cosine
Cosine
Cosine
Epochs
18
25
10
10
Warmup Epochs
3.5
2.5
1.5
1
Appendix
Table 9: Hyperparameters for pretraining and attention probing. The MAE and CL masking ratios apply to the reconstruction and contrastive branches, respectively, in both modalities. During student pretraining, LTA uses the frozen teacher’s last unified block with retention ratio ρ=0.50 for both modalities. The encoder-freezing setting refers to the encoder being optimized or probed in each column; the supervising teacher remains frozen during student training. A dash denotes not applicable.
Figure 2: Dynamics of teacher-induced subview quality during teacher pretraining. For each teacher checkpoint, we construct LTA-induced subviews with retention ratio ρ=0.50 and evaluate zero-shot cross-modal retrieval using the resulting subview representations. Avg. R@1 and Avg. R@5 denote the mean retrieval performance over the two retrieval directions. The subview quality improves rapidly in the early stage and becomes saturated around epoch 18, after which later checkpoints provide only marginal gains. We therefore stop teacher pretraining at epoch 18 rather than using the full 35-epoch training schedule.
Teacher stop
Retrieval
Localization
epoch
V → A R@1
A → V R@1
Avg. R@1
mIoU
mAP
15
35.38
33.62
34.50
29.6
37.1
18
35.44
34.27
34.86
30.6
38.2
21
35.19
33.81
34.50
29.5
36.5
Appendix
Table 10: Sensitivity of final student performance to the teacher stopping epoch. The default stopping epoch is shaded. Avg. R@1 is averaged over both retrieval directions.
AVSBench localization
AS20K classification
Method
mIoU
mAP
mAP
Ours
29.88±0.44
37.26±0.57
35.20±0.13
Appendix
Table 11: Repeated-run results, reported as mean ± standard deviation. Retrieval and localization results use five independent pretraining runs with different random seeds. Classification results use five attention-probing runs with the same pretrained checkpoint. The corresponding means in Tables 1 – 3 are rounded to one decimal place.
Training component
Time (hours)
TG-DP baseline ( 35 epochs)
19.00
Teacher pretraining ( 18 epochs)
9.77
Student base computation ( 25 epochs)
13.57
Additional consistency regularization
2.40
Teacher-anchor construction and LTA scoring
1.39
Our total pretraining time
27.13
Appendix
Table 12: Breakdown of pretraining time under the same hardware setting of 8 NVIDIA A100 GPUs. Times are reported in wall-clock hours. The four components below the baseline sum to our total pretraining time; the last three comprise the student stage.
Component
GFLOPs
Memory (MB)
Relative FLOPs
Teacher forward
71.78
746.83
1.00 ×
Teacher forward + LTA
76.47
779.10
1.07 ×
LTA overhead
+4.69
+32.27
+6.54%
Appendix
Table 13: Computation cost of LTA scoring per audio-visual pair, excluding data loading and audio/image preprocessing. The full scoring process includes one teacher forward pass and the additional attribution computation. Relative FLOPs are computed with respect to the teacher forward.
AVSBench
ADE20K-Sound
Method
mIoU
mAP
mIoU
mAP
Original heatmap
22.3
25.2
21.9
25.4
CheferCAM
44.5
61.2
32.3
41.9
LTA
44.2
60.1
31.1
39.6
Appendix
Table 14: Sound-prompted semantic segmentation results of different heatmaps on AVSBench and ADE20K-Sound. We report mean IoU (mIoU) and mean average precision (mAP) in percentage.
Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.
Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2% to 37.4% for video-to-audio retrieval and from 27.9% to 37.1% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.
Linge Wang, Yingying Chen, Bingke Zhu +2
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Objecteye Inc., Beijing, China
Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.
Learning aligned multimodal embeddings from weakly paired, label-free corpora is challenging: pipelines often provide only pre-extracted features, clips contain multiple events, and spurious co-occurrences. We propose HSC-MAE (Hierarchical Semantic Correlation-Aware Masked Autoencoder), a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation - from coarse to fine: (i) global-level canonical-geometry correlation via DCCA, which aligns audio and visual embeddings within a shared modality-invariant subspace; (ii) local-level neighborhood-semantics correlation via teacher-mined soft top-k affinities, which preserves multi-positive relational structure among semantically similar instances; and (iii) sample-level conditional-sufficiency correlation via masked autoencoding, which ensures individual embeddings retain discriminative semantic content under partial observation. Concretely, a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE; an EMA teacher operating on unmasked inputs via the CCA path supplies stable canonical geometry and soft positives. Learnable multi-task weights reconcile competing objectives, and an optional distillation loss transfers teacher geometry into the student. Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines, validating that HSC-MAE yields robust and well-structured audio-visual representations.