Generating audio that is acoustically consistent with a scene is essential for immersive virtual environments. Recent neural acoustic field methods enable spatially continuous sound rendering but remain scene-specific, requiring dense audio measurements and costly training for each environment. Few-shot approaches improve scalability across rooms but still rely on multiple recordings and, being deterministic, fail to capture the inherent uncertainty of scene acoustics under sparse context. We introduce flow-matching acoustic generation (FLAC), a probabilistic method for few-shot acoustic synthesis that models the distribution of plausible room impulse responses (RIRs) given minimal scene context. FLAC leverages a diffusion transformer trained with a flow-matching objective to generate RIRs at arbitrary positions in novel scenes, conditioned on spatial, geometric, and acoustic cues. FLAC outperforms state-of-the-art eight-shot baselines with one-shot on both the AcousticRooms and Hearing Anything Anywhere datasets. To complement standard perceptual metrics, we further introduce AGREE, a joint acoustic-geometry embedding, enabling geometry-consistent evaluation of generated RIRs through retrieval and distributional metrics. This work is the first to apply generative flow matching to explicit RIR synthesis, establishing a new direction for robust and data-efficient acoustic synthesis.
Figures & tables
Figure 2: Training and inference pipelines of FLAC: During training, a pre-trained VAE encodes ground-truth RIRs into latents z0 . Latents are linearly interpolated with noise to form zt . A DiT is trained to predict the velocity vt that transports zt toward the original data distribution. At inference, RIRs are generated from random noise, guided by the few-shot spatial, geometric and acoustic context.
Figure 3: FLAC diffusion transformer: The noise timestep t and the target RIR pose are injected via AdaLN. Acoustic, spatial and geometric context are provided through cross-attention.
Figure 4: AGREE contrastive framework: Audio and geometry inputs are encoded into a shared latent space, where a contrastive objective maximizes similarity for matching pairs (diagonal entries) and minimizes it for mismatched ones.
Method
K
G
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@1 (%) ↑
R@5 (%) ↑
R@10 (%) ↑
FDG↓
Random Across Rooms
✗
✗
44.73
7.676
306.29
0.02
0.06
0.32
0.111
Random Same Room
✗
✗
17.36
5.490
168.17
0.25
1.09
2.16
0.001
FLAC
✗
✓
23.41 ±0.02
2.554 ±0.002
109.75 ±0.09
5.12 ±0.12
16.47 ±0.14
23.12 ±0.14
0.337
Nearest Neighbor
1
✗
15.22
5.212
157.94
0.00
2.26
4.56
0.001
Fast-RIR
1
✓
18.97
3.257
121.21
0.17
0.66
1.64
0.456
xRIR
1
✓
14.47
1.961
74.45
0.28
1.36
2.59
0.263
Table 1: Performance on unseen AcousticRooms scenes: Results are shown for K∈{8,1, ✗ } reference RIRs. For FLAC, we report mean and standard deviation over 5 generations. FLAC outperforms all baselines even in the one-shot setting. denotes ablations with either geometry (G) or audio conditioning removed.
Method
A2G
G2A
R@1 ↑
R@5 ↑
R@10 ↑
R@1 ↑
R@5 ↑
R@10 ↑
AGREE
59.78
83.53
89.35
59.10
85.56
91.04
AGREE †
85.37
99.70
99.98
84.38
99.53
99.97
Table 2: Zero-shot cross-modal retrieval on the unseen AcousticRooms set: We report acoustic-to-geometry (A2G) and geometry-to-acoustic (G2A) recall at 1, 5 and 10. † indicates training on the full dataset for benchmarking few-shot methods.
Figure 5: Impact of classifier-free guidance and inference steps: Evolution of T60 and FD G as a function of the guidance scale ω and the number of timesteps.
Figure 6: Robustness to limited context RIRs in novel scenes: Performance as the number of reference RIRs ( K ) decreases for KNN, xRIR, and FLAC. FLAC remains the most stable and outperforms state-of-the-art methods with K=8 even in one-shot.
Figure 7: Octave-band analysis of 4 RIRs in unseen rooms: 100 samples per instance are generated with FLAC. The mean and ±3 standard deviation (covering 99.7% of the distribution) are shown. Std increases at low frequencies.
Method
K
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@5 (%) ↑
FD G ↓
Random Across Rooms
✗
17.40
10.283
533.99
1.49
0.460
Random Same Room
✗
8.00
4.805
180.15
1.86
0.169
Nearest Neighbor
1
8.19
5.000
187.55
1.20
0.177
xRIR
1
8.63
4.862
183.27
14.85
0.363
FLAC
1
3.45 ±0.02
2.170 ±0.014
90.02 ±0.24
17.94 ±0.62
0.564
Linear Interpolation
8
4.12
2.695
88.19
3.62
0.904
Table 3: Sim-to-real transfer to the Hearing-Anything-Anywhere dataset: Few-shot methods are compared against Diff-RIR and INRAS, which require per-scene training ( † ). For FLAC, we report mean and standard deviation over 5 generations. With K=8 , FLAC matches or exceeds xRIR and Diff-RIR on perceptual metrics, and with one-shot, it outperforms KNN and xRIR.
Figure 8: Uncertainty persistence time and band-wise energy decay, averaged over 100 unseen samples. Uncertainty lasts longer at low frequencies and decays faster at high frequencies.
ϕG
ϕA
K
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@5 (%) ↑
FD G ↓
ViT
WDINO
[ 52 ]
✗
ResNet
1
10.91
1.166
42.41
11.17
0.328
S/16 [ 73 ]
✗
ResNet
1
10.81
1.090
42.11
15.40
0.318
S/16 [ 73 ]
❄
ResNet
1
10.42
1.427
51.79
5.29
0.373
S/16 [ 73 ]
✓
ResNet
1
9.95
1.046
40.04
18.9
0.303
S/16 [ 73 ]
✓
VAE
1
9.40
1.057
39.31
17.11
0.310
Table 4: Impact of geometry and acoustic encoders: Performance on unseen AcousticRooms scenes using different configurations of the geometry ϕG and acoustic ϕA encoders. We compare xRIR’s ViT and DINOv3 ViT-S/16 with three initialization strategies: trained from scratch, frozen, or fine-tuned ( WDINO ). For ϕA , we evaluate the ResNet-18 and our frozen VAE encoder.
Method
K
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@5 (%) ↑
FD G ↓
In-Context
1
69.68
11.199
1236.98
0.06
1.270
CA
1
15.68
1.750
85.98
6.10
0.424
AdaLN+CA
1
9.95
1.046
40.04
18.92
0.303
In-Context
8
8.12
1.081
41.97
0.194
0.316
CA
8
9.31
1.234
45.81
11.93
0.342
AdaLN+CA
8
8.60
0.970
37.13
19.38
0.305
Table 5: Impact of DiT variants: Performance on unseen AcousticRooms scenes with In-Context, Cross-Attention (CA), and hybrid AdaLN+CA conditioning.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Geometry module pipeline: A panoramic depth map captured at the receiver position is unprojected into a 3D point cloud using the equirectangular projection P , then reprojected so that each pixel encodes its corresponding 3D coordinates. We subtract the source pose associated with the RIR (previously projected in the receiver frame). The resulting representation is processed by a ViT followed by a linear layer to produce geometry-aware features. In HAA, the source and receiver are interchanged.
Figure A.2 : DiT architecture variants: In-Context and Cross-Attention –only architectures. The In-Context variant concatenates all conditioning information with the input before self-attention. In the Cross-Attention variant, conditioning is applied solely via cross-attention layers.
ϕG
A2G
G2A
ViT
WDINO
R@1 ↑
R@5 ↑
R@10 ↑
R@1 ↑
R@5 ↑
R@10 ↑
[ 52 ]
✗
6.26
17.82
25.69
6.63
19.20
27.30
S/16 [ 37 ]
✗
26.53
54.96
67.19
27.47
55.78
67.68
S/16 [ 73 ]
✗
42.26
72.08
81.11
42.32
71.28
80.29
S/16 [ 73 ]
❄
2.00
7.42
12.77
1.94
7.40
13.08
S/16 [ 73 ]
✓
59.78
83.53
89.35
59.10
85.56
91.04
Appendix
Table A.1 : Zero-shot retrieval on the unseen split of the AcousticRooms dataset using different geometry encoders: We report both acoustic-to-geometry (A2G) and geometry-to-acoustic (G2A) results for several ViT variants. WDINO denotes DINOv3 pre-trained weights. Models marked with † are trained on the full AcousticRooms dataset.
Method
K
G
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@1 (%) ↑
R@5 (%) ↑
R@10 (%) ↑
FDG↓
Random Across Rooms
✗
✗
44.02
6.415
274.43
0.02
0.10
0.21
0.061
Random Same Room
✗
✗
13.47
3.582
126.28
1.03
3.12
4.49
0.006
FLAC
✗
✓
24.56 ±0.03
2.464 ±0.003
114.84 ±0.11
4.97 ±0.14
15.63 ±0.17
22.52 ±0.13
0.326
Nearest Neighbor
1
✗
11.91
3.406
120.72
0.00
6.79
9.96
0.001
FastRIR
1
✓
15.88
2.152
92.44
0.23
0.89
1.64
0.383
xRIR
1
✓
10.66
1.476
58.67
0.39
1.72
3.15
0.270
Appendix
Table A.2 : Performance on seen AcousticRooms scenes: Results are shown for K∈{8,1, ✗ } reference RIRs. For FLAC, we report mean and standard deviation over 5 generations. FLAC outperforms all baselines even in the one-shot setting. denotes ablations with either geometry (G) or audio conditioning removed.
ϕG
ϕA
K
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@5 (%) ↑
FD G ↓
ViT
WDINO
[ 52 ]
✗
ResNet
1
6.77
0.741
30.15
12.9
0.319
S/16 [ 73 ]
✗
ResNet
1
6.83
0.740
30.23
17.65
0.310
S/16 [ 73 ]
❄
ResNet
1
7.61
0.956
36.98
6.40
0.365
S/16 [ 73 ]
✓
ResNet
1
6.46
0.692
28.11
21.54
0.296
S/16 [ 73 ]
✓
VAE
1
6.81
0.692
27.97
20.28
0.309
Appendix
Table A.3 : Effect of the geometry and acoustic conditioning encoders (seen): We evaluate different configurations of the geometry encoder ϕG and acoustic encoder ϕA on the seen set of the AcousticRooms dataset. For ϕG , we compare the ViT from xRIR and DINOv3 ViT-S/16 under various initialization strategies ( WDINO ): trained from scratch, frozen, or initialized with DINO weights. For ϕA , we experiment with the ResNet-18 used in xRIR and our frozen VAE encoder.
Method
K
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@5 (%) ↑
FD G ↓
In-Context
1
72.37
11.299
1318.71
0.05
1.302
CA
1
14.44
1.411
75.13
7.43
0.406
AdaLN+CA
1
6.46
0.692
28.11
21.54
0.296
In-Context
8
5.31
0.651
26.58
21.10
0.307
CA
8
6.01
0.782
30.16
14.40
0.328
AdaLN+CA
8
5.32
0.643
25.69
21.89
0.299
Appendix
Table A.4 : Effect of DiT architecture variants (seen): Performance comparison of different conditioning strategies on the seen set of the AcousticRooms dataset. We report results for In-Context, Cross-Attention (CA), and our hybrid design combining AdaLN for target information and CA for contextual cues.
Method
K
G
Unseen
Seen
R@1 ↑
R@5 ↑
R@10 ↑
R@1 ↑
R@5 ↑
R@10 ↑
Random Across Rooms
✗
✗
0.03
0.13
0.21
0.00
0.05
0.16
Random Same Room
✗
✗
0.22
0.98
1.78
0.88
2.45
3.78
FLAC
✗
✓
5.12 ±0.15
16.10 ±0.19
24.01 ±0.12
5.58 ±0.10
16.18 ±0.12
23.09 ±0.20
Nearest Neighbor
1
✗
0.05
2.04
4.09
0.21
5.65
8.41
Fast-RIR
1
✓
0.16
0.79
1.52
0.15
0.82
1.61
Appendix
Table A.5: Acoustic-to-geometry retrieval on the AcousticRooms dataset: Results are shown for K∈{8,1, ✗ } reference RIRs. For FLAC, we report mean and standard deviation over 5 generations. denotes ablations with either geometry (G) or audio conditioning removed. FLAC achieves higher acoustic-to-geometry recall than the baselines, demonstrating higher scene consistency.
Method
K
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@5 (%) ↑
FD G ↓
Random Across Rooms
✗
17.40
10.283
533.99
2.02
0.418
Random Same Room
✗
8.00
4.805
180.15
1.51
0.253
Nearest Neighbor
1
8.19
5.000
187.55
1.32
0.268
xRIR
1
8.63
4.862
183.27
8.43
0.372
FLAC
1
3.02 ±0.04
1.676 ±0.023
75.14 ±0.98
10.31 ±1.10
0.899
Linear Interpolation
8
4.12
2.695
88.19
8.24
0.824
Appendix
Table A.6 : Sim-to-real transfer on the Hearing-Anything-Anywhere dataset with inverted receiver pose: Few-shot methods are compared to Diff-RIR and INRAS, which require per-scene training ( † ). For FLAC, we report the mean and standard deviation over 5 generations. Inverting the receiver pose improves T60/C50/EDT but reduces geometry-consistency metrics.
Method
K
MAG ↓
ENV ↓
Random Across Rooms
✗
6.97
1.581
Random Same Room
✗
3.22
0.470
Nearest Neighbor
1
3.24
0.475
xRIR
1
3.97
0.577
FLAC
1
3.16
0.393
Linear Interpolation
8
3.63
0.835
Appendix
Table A.7: Additional metrics on the Hearing-Anything-Anywhere dataset: Few-shot methods are compared against Diff-RIR, which requires per-scene training ( † ) using MAG and ENV error.
Timestep Sampler
Unseen
Seen
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
Uniform
8.97
0.943
38.18
5.22
0.636
25.83
Ones
8.17
1.044
37.48
5.29
0.636
25.60
Logit-Normal
9.77
1.068
42.65
5.80
0.681
27.98
LogSNR
8.60
0.970
37.13
5.32
0.643
25.69
Appendix
Table A.8 : Comparison of timestep sampling strategies during training on the generation performance: LogSNR emphasizes high t values, Logit-Normal concentrates on intermediate values, Ones fixes t=1 ( i.e . , full noise), and Uniform samples t uniformly. LogSNR provides the best overall trade-off between seen and unseen performance.
Figure A.3 : t-SNE visualization of generated RIRs across unseen rooms: Each color represents a different room category. Samples cluster tightly within the same conditioning and remains well separated across conditionings and rooms, confirming consistent yet diverse RIR generation.
Figure A.4 : User study interface. We created 2 synthetic audio using FLAC one-shot and xRIR one-shot and ask people which one matches closely with the ground truth.
Method
K
T60 (%) ↓
C50 (dB) ↓
EDT (ms) ↓
R@5 (%) ↑
FDG↓
xRIR
1
14.47
1.961
74.45
1.36
0.263
FLAC-S
1
9.71 ±0.04
1.010 ±0.002
39.58 ±0.18
17.80 ±0.11
0.264
FLAC
1
9.95 ±0.05
1.046 ±0.002
40.04 ±0.22
18.92 ±0.10
0.303
xRIR
8
9.98
1.354
49.40
2.00
0.307
FLAC-S
8
8.69 ±0.02
0.942 ±0.001
37.14 ±0.08
17.88 ±0.17
0.267
FLAC
8
8.60 ±0.01
0.970 ±0.002
37.13 ±0.02
19.38 ±0.15
0.305
Appendix
Table A.10 : Comparison between FLAC-S, FLAC, and xRIR on the unseen AcousticRooms set : FLAC-S reduces the parameter count by 13M compared to FLAC. Despite having a similar number of parameters to xRIR, it achieves performance comparable to FLAC, outperforming xRIR.
Figure A.5 : Additional octave band analysis on the AcousticRooms dataset: We generate 100 samples per instance in the unseen set with FLAC under identical conditioning, and plot the mean along with a ±3 standard deviation interval (covering 99.7% of the distribution). Ground truth and xRIR predictions are shown for comparison.
Figure A.6 : Octave band analysis on the HAA dataset.
Figure A.7 : Qualitative comparison of predicted RIR waveforms on the unseen set of the AcousticRooms dataset: We compare xRIR (orange), FLAC (red), and FLAC with K=1 (purple) against the ground truth (green).
Figure A.8 : Qualitative comparison of predicted RIR waveforms on the HAA dataset: We compare xRIR (orange), FLAC (red), and FLAC with K=1 (purple) against the ground truth (green).
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.
Existing video-to-audio (V2A) generation methods predominantly rely on text prompts alongside visual information to synthesize audio. However, two critical bottlenecks persist: semantic granularity gaps in training data, such as conflating acoustically distinct sounds under coarse labels, and textual ambiguity in describing micro-acoustic features. These bottlenecks make it difficult to perform fine-grained sound synthesis using text-controlled modes. To address these limitations, we propose AC-Foley, an audio-conditioned V2A model that directly leverages reference audio to achieve precise and fine-grained control over generated sounds. This approach enables fine-grained sound synthesis, timbre transfer, zero-shot sound generation, and improved audio quality. By directly conditioning on audio signals, our approach bypasses the semantic ambiguities of text descriptions while enabling precise manipulation of acoustic attributes. Empirically, AC-Foley achieves state-of-the-art performance for Foley generation when conditioned on reference audio, while remaining competitive with state-of-the-art video-to-audio methods even without audio conditioning. Code and demo are available at: https://ff2416.github.io/AC-Foley-Page
Pengjun Fang, Yingqing He, Yazhou Xing +3
The Hong Kong University of Science and Technology · University of Central Florida
In recent years, flow-matching models have produced significant improvements in zero-shot text-to-speech synthesis. Conditioned on an audio prompt and text, these models learn a velocity field and generate speech by iteratively solving an ODE. During inference, the solver evolves a single state spanning both the prompt and the region to be generated, although only the generated region is ultimately retained. The discarded prompt state, however, still matters; its intermediate values influence generation through the velocity field that couples the two regions. As sampling continues, this state can drift away from the prescribed conditional path, introducing a discrepancy into subsequent generation updates. Unlike the unknown generated trajectory, the prompt path is available in closed form from the reference audio and the initial noise. We exploit this observation with Prompt-Consistency Inference (PCI), a training-free rule that restores the prompt block to its analytic value before each velocity evaluation, while leaving the generated block unchanged. PCI improves speaker similarity and intelligibility across the evaluated flow-matching TTS backbones without additional network evaluations. Our ablation studies further show that PCI keeps post-step prompt discrepancies smaller and that corrections covering the later sampling stages recover much of the observed similarity gain.
Vasily Zadorozhnyy, Can Goksen, Kazuhito Koishida +1
Applied Sciences Group, Microsoft Corporation, Redmond, USA
Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.