Direct Conditioning of Audio Diffusion Transformers on fMRI Reveals Cortical Contributions to Sound Reconstruction
Organizations: Department of Biomedicine and Prevention, University of Rome Tor Vergata, Italy · Maastricht Centre for Systems Biology (MaCSBio), Faculty of Science and Engineering, Maastricht University, The Netherlands · Tether Evo, El Salvador · Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, USA ·
Abstract
Reconstructing audio from brain activity remains challenging. Existing brain-to-audio pipelines typically map neural responses to predefined stimulus representations before synthesis. While recent work in the visual domain has explored direct conditioning on brain activity, this approach has not yet been extended to audio diffusion models. We introduce MAAS (Multi-Area Auditory Synthesis), an end-to-end framework that conditions a pretrained audio diffusion transformer on ROI-wise fMRI responses without first supervising a decoder to predict an external stimulus representation. MAAS uses region-specific projections and learned brain tokens to condition Stable Audio through a dedicated cross-attention pathway. On a 7T fMRI dataset, we demonstrate consistent improvements over baselines, including two-stage fMRI-to-latent conditioning, across both acoustic and semantic evaluation metrics. In addition, we introduce a fidelity metric grounded in a computational model of auditory neural processing, which allows us to quantify the contribution of individual auditory cortical regions to reconstructed audio representations.
Figures & tables
| Condition | CLAP sim. | ID acc. | Mel-SSIM | Log-mel dist. |
|---|---|---|---|---|
| Brain-conditioning baselines | ||||
| MAAS (our adapter) | ||||
| MAAS (no backbone fine-tuning) | ||||
| Brain-plus-empty | ||||
| Brain-only | ||||
| Two-stage fMRI-to-latent | ||||
| Training pairing | Correct | Within-category shuffle | Unrestricted shuffle | Chance |
|---|---|---|---|---|
| Pairwise ID |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Selected | Search range |
|---|---|---|
| Number of brain queries | 16 | {6, 16, 64, 128} |
| Conditioning strategy | adapter | {brain-only, residual, adapter} |
| BrainTokenGen. layers | 1 | {1, 2, 4} |
| Transformer dimension | 1024 | {512, 1024, 2048} |
| Learning rate | ||
| Weight decay |
Audio Generation
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
See all 140 papers