Attribution-Guided Shared-Private Decoupling for Noise-Reduced Audio-Visual Representation Learning
Authors: Linge Wang, Yingying Chen, Bingke Zhu, Lu Zhou, Haixin Wang, Zhipiao Liu, Jinqiao Wang
Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Huawei Technologies Ltd., Beijing, China · Objecteye Inc., Beijing, China
Audio-visual representation learning commonly aligns paired audio and visual signals with sample-level contrastive objectives. However, real-world audio-visual pairs usually contain heterogeneous patch tokens: only a subset carries reliable cross-modal shared semantics, while many others mainly encode modality-private residuals. Aggregating all tokens into a single global representation can therefore introduce semantic noise into cross-modal alignment. Motivated by the minimal-sufficiency principle, we propose an attribution-guided shared-private decoupling framework to mitigate this issue. Our key idea is to identify shared-cover subviews that preserve dominant cross-modal information while suppressing modality-private residuals, and use them to supervise decoupled shared and private representations. To identify such subviews efficiently, we draw on transformer attribution method and introduce Layer-Truncated Attribution (LTA), which provides a lightweight estimate of token-level relevance to cross-modal shared information. We further adopt a teacher-student framework, where the teacher provides attribution-induced supervision and the student internalizes shared-private decoupling into its representations, without introducing extra inference cost. Experiments on retrieval, classification, and sound-prompted semantic segmentation show consistent gains over strong baselines. Compared with baseline, our method improves the average zero-shot R@1 by 4.0 across AudioSet and VGGSound retrieval, and improves AS20K classification mAP by 3.1, demonstrating the effectiveness of fine grained attribution guidance for reducing semantic noise.reducing semantic noise. Code will be released at https://github.com/wanglg20/LTA
Figures & tables
Figure 1: Overall pipeline of our proposed framework. (a) Attribution-guided teacher-student training. A pretrained teacher generates high-relevance shared-cover subviews and low-relevance residual complements, which serve as stop-gradient anchors for training the student’s shared and private tokens. For clarity, we omit the MAE branch inherited from the baseline Wang et al. (2026) . (b) Layer-Truncated Attribution (LTA). Given the cross-modal matching score, LTA computes CLS-to-token gradient-attention relevance from the last Transformer block.
Vision → Audio
Audio → Vision
AudioSet Eval Subset
VGGSound Eval Subset
AudioSet Eval Subset
VGGSound Eval Subset
Method
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
CAV-MAE Gong et al. (2023)
16.1
38.6
49.3
14.7
35.3
45.9
9.5
22.6
32.4
8.3
23.8
32.4
CAV-MAE Scale+++ Araujo et al. (2025)
18.8
39.5
50.1
14.8
34.2
44.0
15.1
34.0
43.0
12.8
30.4
40.3
LanguageBind Zhu et al. (2023)
6.4
20.2
28.3
10.3
30.1
39.7
4.4
15.0
22.5
6.5
22.7
33.5
AVSiam Lin and Bertasius (2024)
19.7
–
–
19.0
–
–
17.6
–
–
20.4
–
–
Table 1: Zero-shot cross-modal retrieval on the AudioSet-20K and VGGSound evaluation subsets. We report Recall@1, Recall@5, and Recall@10 for both Vision → Audio (V → A) and Audio → Vision (A → V) retrieval using frozen representations.
Table 3
V → A
A → V
Subview Score
R@1
R@5
R@10
R@1
R@5
R@10
Full view
31.7
56.0
65.9
30.5
54.0
65.3
Similarity
21.6
46.2
56.4
23.2
44.8
54.0
Attention
28.5
54.9
64.2
28.7
52.7
63.7
CheferCAM Chefer et al. (2021b)
37.2
61.6
70.6
37.5
60.9
68.3
LTA
44.1
68.5
77.3
41.2
67.2
75.8
Table 4: Empirical validation of LTA-induced subviews on VGGSound. We compare teacher retrieval performance using subviews constructed by different token scoring functions.
Similarity
Attention
LTA
Pair condition
mIoU
mAP
mIoU
mAP
mIoU
mAP
Matched
22.3
25.2
34.2
42.8
44.2
60.1
Mismatched
22.2
25.4
34.4
43.2
34.7
45.7
Table 5: Sound-source localization under matched and mismatched audio-video pairing on AVSBench. For mismatched pairs, the audio is replaced while the visual input and its original sound-source annotation are retained. Both mIoU and mAP are reported in percentage.
Token
Direction
R@1
R@5
R@10
Private
V → A
29.5
53.9
63.9
Private
A → V
28.2
52.9
63.8
Shared
V → A
35.4
59.6
70.5
Shared
A → V
34.3
57.9
69.8
Table 6: VGGSound zero-shot retrieval using different tokens as the global representation.
Setting
Anchor
Cons.
Dec.
Avg. R@1
AS20K mAP
Seg. mAP
Backbone
30.8
32.0
24.3
+ Anchor
✓
33.0
32.2
28.3
+ Consistency
✓
✓
34.3
32.0
26.3
+ Decoupled tokens
✓
✓
✓
34.9
35.1
38.2
Table 7: Incremental ablation of the main training components. Anchor, Cons., and Dec. denote attribution-induced anchor supervision, consistency regularization, and shared-private token decoupling, respectively. Avg. R@1 is averaged over the two retrieval directions on VGGSound. Seg. mAP denotes sound-prompted semantic segmentation mAP on AVSBench.
Retention
Avg. R@1
AS20K mAP
Seg. mAP
0.25
32.6
35.3
38.3
0.35
33.9
35.2
37.3
0.50
34.9
35.1
38.2
0.75
34.8
35.5
31.5
Table 8: Effect of the retention ratio.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Teacher Pretrain
Student Pretrain
AS20K Attn. Probe
VGGSound Attn. Probe
Dataset
AS-2M
AS-2M
AS-20K
VGGSound
Optimizer
Adam
Adam
Adam
Adam
Learning Rate
2×10−4
2×10−4
5×10−3
1×10−3
LR Scheduler
Cosine
Cosine
Cosine
Cosine
Epochs
18
25
10
10
Warmup Epochs
3.5
2.5
1.5
1
Appendix
Table 9: Hyperparameters for pretraining and attention probing. The MAE and CL masking ratios apply to the reconstruction and contrastive branches, respectively, in both modalities. During student pretraining, LTA uses the frozen teacher’s last unified block with retention ratio ρ=0.50 for both modalities. The encoder-freezing setting refers to the encoder being optimized or probed in each column; the supervising teacher remains frozen during student training. A dash denotes not applicable.
Figure 2: Dynamics of teacher-induced subview quality during teacher pretraining. For each teacher checkpoint, we construct LTA-induced subviews with retention ratio ρ=0.50 and evaluate zero-shot cross-modal retrieval using the resulting subview representations. Avg. R@1 and Avg. R@5 denote the mean retrieval performance over the two retrieval directions. The subview quality improves rapidly in the early stage and becomes saturated around epoch 18, after which later checkpoints provide only marginal gains. We therefore stop teacher pretraining at epoch 18 rather than using the full 35-epoch training schedule.
Teacher stop
Retrieval
Localization
epoch
V → A R@1
A → V R@1
Avg. R@1
mIoU
mAP
15
35.38
33.62
34.50
29.6
37.1
18
35.44
34.27
34.86
30.6
38.2
21
35.19
33.81
34.50
29.5
36.5
Appendix
Table 10: Sensitivity of final student performance to the teacher stopping epoch. The default stopping epoch is shaded. Avg. R@1 is averaged over both retrieval directions.
AVSBench localization
AS20K classification
Method
mIoU
mAP
mAP
Ours
29.88±0.44
37.26±0.57
35.20±0.13
Appendix
Table 11: Repeated-run results, reported as mean ± standard deviation. Retrieval and localization results use five independent pretraining runs with different random seeds. Classification results use five attention-probing runs with the same pretrained checkpoint. The corresponding means in Tables 1 – 3 are rounded to one decimal place.
Training component
Time (hours)
TG-DP baseline ( 35 epochs)
19.00
Teacher pretraining ( 18 epochs)
9.77
Student base computation ( 25 epochs)
13.57
Additional consistency regularization
2.40
Teacher-anchor construction and LTA scoring
1.39
Our total pretraining time
27.13
Appendix
Table 12: Breakdown of pretraining time under the same hardware setting of 8 NVIDIA A100 GPUs. Times are reported in wall-clock hours. The four components below the baseline sum to our total pretraining time; the last three comprise the student stage.
Component
GFLOPs
Memory (MB)
Relative FLOPs
Teacher forward
71.78
746.83
1.00 ×
Teacher forward + LTA
76.47
779.10
1.07 ×
LTA overhead
+4.69
+32.27
+6.54%
Appendix
Table 13: Computation cost of LTA scoring per audio-visual pair, excluding data loading and audio/image preprocessing. The full scoring process includes one teacher forward pass and the additional attribution computation. Relative FLOPs are computed with respect to the teacher forward.
AVSBench
ADE20K-Sound
Method
mIoU
mAP
mIoU
mAP
Original heatmap
22.3
25.2
21.9
25.4
CheferCAM
44.5
61.2
32.3
41.9
LTA
44.2
60.1
31.1
39.6
Appendix
Table 14: Sound-prompted semantic segmentation results of different heatmaps on AVSBench and ADE20K-Sound. We report mean IoU (mIoU) and mean average precision (mAP) in percentage.
Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Objecteye Inc., Beijing, China