cs.CVNov 18, 2026

Why Multi-Source Audio Fails to Compose: A Geometric Diagnosis and Its Remedy

Authors: Anurag Singh, Prajwal Singh, Shanmuganathan Raman

Organizations: CVIG Lab, IIT Gandhinagar Gandhinagar, Gujarat, India

Abstract

Multi-source audio-to-image generation exhibits consistent compositional failures when audio tokens occupy geometrically displaced, semantically unanchored positions in the CLIP text-token space, where the diffusion U-Net lacks the prior to reliably attend to them or ground them in distinct spatial regions. We formalize this as the geometric displacement phenomenon and expose it through a two-level clustering analysis comparing BEATs and CLAP embeddings at the raw encoder and projected CLIP token levels. Three geometric properties emerge as important: intra-class compactness, angular alignment with the CLIP text manifold, and inter-token spatial separation. We show that CLAP, whose contrastive language-audio pretraining aligns embeddings with natural language semantics, substantially outperforms BEATs on all three properties, improving the silhouette score by more than 3x and the Audio-Image Content score as well. Building on this analysis, we introduce a multi-token extension that requires no diffusion-backbone retraining and assigns each audio source to a distinct placeholder token in the CLIP tokenizer, complemented by a mixed inference-time objective that combines Attend-and-Excite with an A* spatial separation loss to resolve residual cross-token interference. For evaluation, we construct multi-audio compositional benchmarks from VGGSound (VGG2, VGG3, and VGGN) and extend six standard audio-image metrics to the multi-source setting. Across five benchmarks and four strong baselines, our method achieves strong results on the metrics most directly tied to compositional faithfulness, with AIC improvements of up to 2.25x over the strongest baseline and FID reduced by nearly 2x on two-source generation. We propose geometric displacement as a diagnostic framework for cross-modal generation systems that project learned embeddings into a pretrained contrastive space.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation

    Sep 30, 2026Tiernon Riesenmy, You Zhang, Gautam Bhattacharya +1Text-to-Audio GenerationAudio-Video Generation

  2. COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings

    May 28, 2026Yonggang Zhu, Liting Gao, Aidong Men +1Audio UnderstandingAudio-Language Models

  3. Do Audio Representations Compose Additively?

    Sep 23, 2026Chenhao Xue, Zhijin Guo, Joyraj Chakraborty +2Audio Representation LearningCompositional Representation Learning