Organizations: Department of Biomedicine and Prevention, University of Rome Tor Vergata, Italy · Maastricht Centre for Systems Biology (MaCSBio), Faculty of Science and Engineering, Maastricht University, The Netherlands · Tether Evo, El Salvador · Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, USA · Maastricht Brain Imaging Center, Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, The Netherlands
Reconstructing audio from brain activity remains challenging. Existing brain-to-audio pipelines typically map neural responses to predefined stimulus representations before synthesis. While recent work in the visual domain has explored direct conditioning on brain activity, this approach has not yet been extended to audio diffusion models. We introduce MAAS (Multi-Area Auditory Synthesis), an end-to-end framework that conditions a pretrained audio diffusion transformer on ROI-wise fMRI responses without first supervising a decoder to predict an external stimulus representation. MAAS uses region-specific projections and learned brain tokens to condition Stable Audio through a dedicated cross-attention pathway. On a 7T fMRI dataset, we demonstrate consistent improvements over baselines, including two-stage fMRI-to-latent conditioning, across both acoustic and semantic evaluation metrics. In addition, we introduce a fidelity metric grounded in a computational model of auditory neural processing, which allows us to quantify the contribution of individual auditory cortical regions to reconstructed audio representations.
Figures & tables
Figure 1: General overview of our approach. fMRI responses from auditory ROIs are projected and transformed into brain tokens, which are injected into the cross-attention layers of a Diffusion Transformer via a dedicated adapter. The model is conditioned solely on brain activity (empty text prompt) and generates audio through a frozen latent diffusion backbone.
Condition
CLAP sim. ↑
ID acc. ↑
Mel-SSIM ↑
Log-mel dist. ↓
Brain-conditioning baselines
MAAS (our adapter)
0.614±0.13
85.4±4.2%
0.297±0.09
0.152±0.04
MAAS (no backbone fine-tuning)
0.548±0.17
73.1±4.8%
0.283±0.08
0.171±0.04
Brain-plus-empty
0.436±0.14
63.1±5.7%
0.273±0.10
0.185±0.05
Brain-only
0.403±0.15
58.3±5.9%
0.226±0.08
0.197±0.06
Two-stage fMRI-to-latent
0.509±0.16
70.1±5.9%
0.270±0.10
0.177±0.08
Table 1: Reconstruction performance. Higher is better for CLAP similarity, pairwise identification (ID) accuracy, and Mel-SSIM; lower is better for log-mel distance. Caption-based conditions use ground-truth test captions.
Training pairing
Correct
Within-category shuffle
Unrestricted shuffle
Chance
Pairwise ID
85.4±4.2
73.6±4.0
53.4±2.7
50.0
Table 2: Neural-information controls (mean ± SD, %). Top: pairwise identification after training with correct or shuffled brain–stimulus pairings (within-category or unrestricted stimulus identity shuffled). Bottom: retrieval from fMRI without generation (72 test stimuli).
Figure 2: Analysis of cortical contributions in MAAS. Evolution of per-ROI transported attention αˉt(r) across 100 denoising steps for two test samples. Each row shows the reconstructed spectrogram at a given denoising step alongside the ground truth and the corresponding ROI attention profile.
Figure 3: (a) Relative change in NSL-based reconstruction fidelity, Δi , by ROI and semantic category, computed from Fisher- z averaged correlations across the four cross-validation folds; brackets show the range [min, max] of fold-wise Δi (%). Dots mark cells in which all four folds agree in sign. Left: leave-one-ROI-out analysis. Right: single-ROI analysis. Each partial-ROI condition uses a separately trained model with the same fixed mask during training and testing; negative values indicate lower fidelity. (b) NSL-derived spectrotemporal representations for two individual stimuli, woman speech and monkey call. Columns compare the ground truth, all-ROI model, and models trained without HG or mSTG. Rows show scale–rate, frequency–rate, and frequency–scale projections, respectively, obtained by averaging over the remaining dimension of the time-averaged representation.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Selected
Search range
Number of brain queries Q
16
{6, 16, 64, 128}
Conditioning strategy
adapter
{brain-only, residual, adapter}
BrainTokenGen. layers
1
{1, 2, 4}
Transformer dimension
1024
{512, 1024, 2048}
Learning rate
1×10−4
{1⋅10−5,5⋅10−4,1⋅10−4}
Weight decay
1×10−4
{0,1⋅10−5,1⋅10−4,1⋅10−3}
Appendix
Table A1: Hyperparameter selection. We report the final configuration and the grid explored during validation.
Figure A1: Category-averaged frequency–scale projections of NSL-derived spectrotemporal representations, obtained by averaging over temporal modulation rate. Rows correspond to semantic categories; columns show the ground-truth audio, full-model reconstructions, and reconstructions from models trained without HG or mSTG, respectively. The horizontal axis denotes spectral modulation scale (cycles/octave), and the vertical axis denotes characteristic frequency (Hz).
Figure A2: Category-averaged rate–frequency projections of NSL-derived spectrotemporal representations, obtained by averaging over spectral modulation scale. Rown and columns follow the same structure as frequency × scale. The horizontal axis denotes temporal modulation rate (Hz), and the vertical axis denotes characteristic frequency (Hz).
Figure A3: Category-averaged rate–scale projections of NSL-derived spectrotemporal representations, obtained by averaging over characteristic frequency. The horizontal axis denotes temporal modulation rate (Hz), and the vertical axis denotes spectral modulation scale (cycles/octave).
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
Brain-to-audio reconstruction is limited by prior domination: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator's sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from 0.14--0.18 for direct generation, near the 0.10 chance level, to 0.40--0.43, comparable to retrieval. It also reduces Fréchet Audio Distance by roughly an order of magnitude, from 13.49 to 1.25 for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.
Recent progress in diffusion-based audio generation and restoration has substantially improved performance across heterogeneous conditioning regimes, including text-conditioned audio generation and audio-conditioned super-resolution. However, training audio diffusion models remains computationally expensive, and most existing pipelines still rely on static optimization recipes that treat the relative importance of training signals as fixed throughout learning. In this work, we argue that a major source of inefficiency lies in the evolving balance between semantic acquisition and generation-oriented refinement. Early training places stronger emphasis on acquiring condition-aligned semantic structure and coarse global organization, whereas later training increasingly emphasizes temporal consistency, perceptual fidelity, and fine-detail refinement. To characterize this evolving balance, we introduce a progress-based regime variable derived from the training-time slope of an SSL-space discrepancy, which measures semantic progress during training. Based on this signal, we develop three complementary stage-aware mechanisms: decayed SSL guidance for early semantic bootstrapping, self-adaptive timestep sampling driven by the regime variable, and structure-aware regularization activated from convergent grouped organization in parameter space. We evaluate these mechanisms on text-conditioned audio generation and audio-conditioned super-resolution. Across both settings, the proposed stage-aware strategies improve convergence behavior and yield gains on the primary generation and spectral reconstruction metrics over standard static baselines. These results support the view that efficient audio diffusion training can benefit from treating external guidance, internal organization, and optimization emphasis as stage-dependent components rather than fixed ingredients.
Xuanhao Zhang, Chang Li
China Pharmaceutical University · University of Science and Technology of China
Audio deepfake detectors often degrade when generators, corpora, or recording conditions change. We use a Diffusion Transformer (DiT), trained only on bona fide speech, as a frozen reconstruction probe. Reconstructions at masking ratios 0.5, 0.75, and 0.9 yield explicit multi-ratio residual maps. Because these residuals are domain sensitive, our audio-anchored detector passes the projected frozen-WavLM auditory representation into the fusion sum without gate-based attenuation and uses residuals only as a scalar-gated additive correction. The pre-specified seed-42 run obtains 6.5442% EER / 0.18456 min-DCF on ASVspoof 5 Eval and 13.8372% / 0.36921 on ITW Full; three-seed means are 6.8885 (0.3308)% and 15.3328 (2.0719)%. The latter is below a separately optimized WavLM-ResNet18 reference under both supervision settings. Auxiliary supervision raises dynamic competitive fusion from 18.4007% to 25.2968% mean ITW EER, worsening all three seeds. The results support reconstruction residuals as complementary evidence and motivate a non-competitive auditory path for ASVspoof 5-to-ITW transfer, without claiming a componentwise causal ablation of anchoring alone.
Haotian Mo, Jie Liu, Siqi Shen +8
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · School of Informatics, Xiamen University, Xiamen, China · Hunan Zhongke Youxin Technology Co., Ltd., Changsha, China