cs.MMApr 5, 2026

Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning

Authors: Donghuo Zeng, Hao Niu, Masato Taya

Organizations: KDDI Research, Inc., Saitama, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learning aligned multimodal embeddings from weakly paired, label-free corpora is challenging: pipelines often provide only pre-extracted features, clips contain multiple events, and spurious co-occurrences. We propose HSC-MAE (Hierarchical Semantic Correlation-Aware Masked Autoencoder), a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation - from coarse to fine: (i) global-level canonical-geometry correlation via DCCA, which aligns audio and visual embeddings within a shared modality-invariant subspace; (ii) local-level neighborhood-semantics correlation via teacher-mined soft top-k affinities, which preserves multi-positive relational structure among semantically similar instances; and (iii) sample-level conditional-sufficiency correlation via masked autoencoding, which ensures individual embeddings retain discriminative semantic content under partial observation. Concretely, a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE; an EMA teacher operating on unmasked inputs via the CCA path supplies stable canonical geometry and soft positives. Learnable multi-task weights reconcile competing objectives, and an optional distillation loss transfers teacher geometry into the student. Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines, validating that HSC-MAE yields robust and well-structured audio-visual representations.

Figures & tables

Fig. 1: Overview of the HSC-MAE architecture. Pre-extracted audio and visual features are processed by shared encoders and a cross-attention fusion block under two coordinated training modes. The student MAE path applies sample-level value masking and is optimized by reconstruction, a teacher-guided soft top- kk InfoNCE, producing robust embeddings (Za,Zv)(Z_{a},Z_{v}) . In parallel, the EMA teacher CCA path preserves input values and enforces global cross-modal alignment via DCCA, yielding clean embeddings (Za′,Zv′)(Z^{\prime}_{a},Z^{\prime}_{v}) . The teacher provides stable semantic affinities and geometric targets for neighborhood mining and distillation, while gradients are blocked from flowing into the teacher.
\Block 2-1 Methods\Block 1-3 AVE dataset  \Block 1-3 VEGAS dataset  
 A2VV2AAvg.A2VV2AAvg.
Random0.1270.1240.1260.1100.1090.109
CCA0.1900.1890.1900.3320.3270.330
KCCA0.30620.18180.24400.2880.2730.281
DCCA0.33820.34380.34100.75410.75740.7558
InfoNCE0.46460.46430.46440.73590.70970.7228
TABLE I: mAP comparison across state-of-the-art methods. The best results are shown in bold, and the second-best results are underlined.
\Block 2-1 CCA\Block 2-1 Rec.\Block 2-1 Soft InfoNCE\Block 2-1 Self-distillation\Block 1-3 AVE dataset  \Block 1-3 VEGAS dataset  
    A2VV2AAvg.A2VV2AAvg.
✓✓✓✓0.77470.77280.77370.80840.82280.8156
−-✓✓✓0.62290.62390.62340.77500.78820.7816
✓✓✓−-0.74860.74270.74570.80600.81020.8081
✓✓−-✓0.67350.67460.67410.78050.77490.7777
✓✓−-−-0.57280.57010.57150.79030.78940.7899
TABLE II: Component-wise impact on the final objective loss for different methods on the AVE and VEGAS datasets.
Fig. 2: (Left) Decomposition of training losses; (Right) test mAP over epochs (1-100) on AVE under component-wise ablations.
Fig. 3: Effect of mask ratio on UCMR task on both AVE and VEGAS datasets. Solid curves show the average mAP, while shaded regions indicate the absolute gap between the two retrieval directions.
Fig. 4: Qualitative audio–visual cross-modal retrieval results on AVE. For each query (audio or visual), the top-10 retrieved results are shown.
More in this topic

Canonical Correlation Analysis

See all 10 papers

Explore similar work

CardsList
Apr 9, 2026

Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2% to 37.4% for video-to-audio retrieval and from 27.9% to 37.1% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.

Linge Wang, Yingying Chen, Bingke Zhu +2

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Objecteye Inc., Beijing, China

Audio-Visual Representation LearningMultimodal PretrainingAudio-Visual AlignmentMultimodal Contrastive Learning
AskPDF
Oct 5, 2026

LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.

Benjamin Robson, Santeri Mentu, Wenshuai Zhao +1

ELLIS Institute Finland Aalto University

Audio-Visual Representation LearningJoint-Embedding Predictive ArchitectureAudio-Visual AlignmentMultimodal SSL
AskPDF
Jun 23, 2026

MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning. Since audio and visual streams naturally co-occur in video data, extending this success to jointly learn from both modalities is a natural next step, yet it remains challenging. Existing audio-visual self-supervised methods rely on modality-specific encoders and complex combinations of contrastive or reconstruction objectives, limiting cross-modal synergy and scalability. Joint Embedding Predictive Architectures (JEPAs) offer a simple, modality-agnostic alternative, but have to date been applied primarily to individual modalities. We introduce MJEPA, a joint-embedding predictive architecture for audio-visual learning that uses a single, unified encoder for both modalities. Our approach uses only a single predictive objective, applied both within and across modalities. We show that cross-modal prediction is critical: without it, a shared encoder degrades below unimodal baselines; with it, each modality's representation benefits from the other. Our frozen ViT-g model outperforms the best prior frozen baseline by over 6.8 mAP on AudioSet-20K, surpasses fully finetuned models on ESC-50 and FSD50K, and is competitive on video benchmarks despite using 10x less video data.

Revant Teotia, Adrien Bardes, Michael Rabbat +3

1FAIR at Meta · 2New York University

Audio-Visual Representation LearningJoint-Embedding Predictive ArchitectureMultimodal Pretraining
AskPDF