cs.SDNov 9, 2026

Direct Conditioning of Audio Diffusion Transformers on fMRI Reveals Cortical Contributions to Sound Reconstruction

Authors: Matteo Ciferri, Tonio Weidler, Matteo Ferrante, Nicola Toschi, Elia Formisano

Organizations: Department of Biomedicine and Prevention, University of Rome Tor Vergata, Italy · Maastricht Centre for Systems Biology (MaCSBio), Faculty of Science and Engineering, Maastricht University, The Netherlands · Tether Evo, El Salvador · Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, USA · 

Abstract

Reconstructing audio from brain activity remains challenging. Existing brain-to-audio pipelines typically map neural responses to predefined stimulus representations before synthesis. While recent work in the visual domain has explored direct conditioning on brain activity, this approach has not yet been extended to audio diffusion models. We introduce MAAS (Multi-Area Auditory Synthesis), an end-to-end framework that conditions a pretrained audio diffusion transformer on ROI-wise fMRI responses without first supervising a decoder to predict an external stimulus representation. MAAS uses region-specific projections and learned brain tokens to condition Stable Audio through a dedicated cross-attention pathway. On a 7T fMRI dataset, we demonstrate consistent improvements over baselines, including two-stage fMRI-to-latent conditioning, across both acoustic and semantic evaluation metrics. In addition, we introduce a fidelity metric grounded in a computational model of auditory neural processing, which allows us to quantify the contribution of individual auditory cortical regions to reconstructed audio representations.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction

    Aug 10, 2026Ambuj Mehrish, Sebastiano VasconNeural DecodingMusic Generation

  2. Stage-adaptive audio diffusion modeling

    May 6, 2026Xuanhao Zhang, Chang LiText-to-Audio GenerationDiffusion Model Guidance

  3. Audio-Anchored Fusion of Multi-Ratio DiT Reconstruction Residuals for Cross-Domain Audio Deepfake Detection

    Jul 29, 2026Haotian Mo, Jie Liu, Siqi Shen +8Audio Deepfake DetectionReconstruction-Based Anomaly Detection