Few-shot Acoustic Synthesis with Multimodal Flow Matching

Amandine Brunetto

cs.SDMar 19, 2026

Few-shot Acoustic Synthesis with Multimodal Flow Matching

Authors: Amandine Brunetto

Organizations: Center for Robotics, Mines Paris - PSL University

Abstract

Generating audio that is acoustically consistent with a scene is essential for immersive virtual environments. Recent neural acoustic field methods enable spatially continuous sound rendering but remain scene-specific, requiring dense audio measurements and costly training for each environment. Few-shot approaches improve scalability across rooms but still rely on multiple recordings and, being deterministic, fail to capture the inherent uncertainty of scene acoustics under sparse context. We introduce flow-matching acoustic generation (FLAC), a probabilistic method for few-shot acoustic synthesis that models the distribution of plausible room impulse responses (RIRs) given minimal scene context. FLAC leverages a diffusion transformer trained with a flow-matching objective to generate RIRs at arbitrary positions in novel scenes, conditioned on spatial, geometric, and acoustic cues. FLAC outperforms state-of-the-art eight-shot baselines with one-shot on both the AcousticRooms and Hearing Anything Anywhere datasets. To complement standard perceptual metrics, we further introduce AGREE, a joint acoustic-geometry embedding, enabling geometry-consistent evaluation of generated RIRs through retrieval and distributional metrics. This work is the first to apply generative flow matching to explicit RIR synthesis, establishing a new direction for robust and data-efficient acoustic synthesis.

Figures & tables

Figure 2: Training and inference pipelines of FLAC: During training, a pre-trained VAE encodes ground-truth RIRs into latents $\mathbf{z}_{0}$ . Latents are linearly interpolated with noise to form $\mathbf{z}_{t}$ . A DiT is trained to predict the velocity $\widehat{\mathbf{v}}_{t}$ that transports $\mathbf{z}_{t}$ toward the original data distribution. At inference, RIRs are generated from random noise, guided by the few-shot spatial, geometric and acoustic context.

Figure 3: FLAC diffusion transformer: The noise timestep $t$ and the target RIR pose are injected via AdaLN. Acoustic, spatial and geometric context are provided through cross-attention.

Figure 4: AGREE contrastive framework: Audio and geometry inputs are encoded into a shared latent space, where a contrastive objective maximizes similarity for matching pairs (diagonal entries) and minimizes it for mismatched ones.

Method	K	G	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@1 (%) $\uparrow$	R@5 (%) $\uparrow$	R@10 (%) $\uparrow$	$\text{FD}_{G}$ $\downarrow$
Random Across Rooms	✗	✗	44.73	7.676	306.29	0.02	0.06	0.32	0.111
Random Same Room	✗	✗	17.36	5.490	168.17	0.25	1.09	2.16	0.001
FLAC	✗	✓	23.41 ±0.02	2.554 ±0.002	109.75 ±0.09	5.12 ±0.12	16.47 ±0.14	23.12 ±0.14	0.337
Nearest Neighbor	1	✗	15.22	5.212	157.94	0.00	2.26	4.56	0.001
Fast-RIR	1	✓	18.97	3.257	121.21	0.17	0.66	1.64	0.456
xRIR	1	✓	14.47	1.961	74.45	0.28	1.36	2.59	0.263

Table 1: Performance on unseen AcousticRooms scenes: Results are shown for

K\in\{8,1,

✗

\}

reference RIRs. For FLAC, we report mean and standard deviation over 5 generations. FLAC outperforms all baselines even in the one-shot setting. denotes ablations with either geometry (G) or audio conditioning removed.

Method	A2G			G2A
Method	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$
AGREE	59.78	83.53	89.35	59.10	85.56	91.04
AGREE †	85.37	99.70	99.98	84.38	99.53	99.97

Table 2: Zero-shot cross-modal retrieval on the unseen AcousticRooms set: We report acoustic-to-geometry (A2G) and geometry-to-acoustic (G2A) recall at 1, 5 and 10. † indicates training on the full dataset for benchmarking few-shot methods.

Figure 5: Impact of classifier-free guidance and inference steps: Evolution of T60 and FD G as a function of the guidance scale $\omega$ and the number of timesteps.

Figure 6: Robustness to limited context RIRs in novel scenes: Performance as the number of reference RIRs ( $K$ ) decreases for KNN, xRIR, and FLAC. FLAC remains the most stable and outperforms state-of-the-art methods with $K{=}8$ even in one-shot.

Figure 7: Octave-band analysis of 4 RIRs in unseen rooms: 100 samples per instance are generated with FLAC. The mean and $\pm 3$ standard deviation (covering 99.7% of the distribution) are shown. Std increases at low frequencies.

Method	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
Random Across Rooms	✗	17.40	10.283	533.99	1.49	0.460
Random Same Room	✗	8.00	4.805	180.15	1.86	0.169
Nearest Neighbor	1	8.19	5.000	187.55	1.20	0.177
xRIR	1	8.63	4.862	183.27	14.85	0.363
FLAC	1	3.45 ±0.02	2.170 ±0.014	90.02 ±0.24	17.94 ±0.62	0.564
Linear Interpolation	8	4.12	2.695	88.19	3.62	0.904

Table 3: Sim-to-real transfer to the Hearing-Anything-Anywhere dataset: Few-shot methods are compared against Diff-RIR and INRAS, which require per-scene training ( † ). For FLAC, we report mean and standard deviation over 5 generations. With

K{=}8

, FLAC matches or exceeds xRIR and Diff-RIR on perceptual metrics, and with one-shot, it outperforms KNN and xRIR.

Figure 8: Uncertainty persistence time and band-wise energy decay, averaged over 100 unseen samples. Uncertainty lasts longer at low frequencies and decays faster at high frequencies.

$\mathbf{\phi}_{\text{G}}$		$\mathbf{\phi}_{\text{A}}$	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
ViT	$\mathcal{W}_{\text{DINO}}$	$\mathbf{\phi}_{\text{A}}$	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
[ 52 ]	✗	ResNet	1	10.91	1.166	42.41	11.17	0.328
S/16 [ 73 ]	✗	ResNet	1	10.81	1.090	42.11	15.40	0.318
S/16 [ 73 ]	❄	ResNet	1	10.42	1.427	51.79	5.29	0.373
S/16 [ 73 ]	✓	ResNet	1	9.95	1.046	40.04	18.9	0.303
S/16 [ 73 ]	✓	VAE	1	9.40	1.057	39.31	17.11	0.310

Table 4: Impact of geometry and acoustic encoders: Performance on unseen AcousticRooms scenes using different configurations of the geometry

\phi_{G}

and acoustic

\phi_{A}

encoders. We compare xRIR’s ViT and DINOv3 ViT-S/16 with three initialization strategies: trained from scratch, frozen, or fine-tuned (

\mathcal{W}_{\text{DINO}}

). For

\phi_{A}

, we evaluate the ResNet-18 and our frozen VAE encoder.

Method	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
In-Context	1	69.68	11.199	1236.98	0.06	1.270
CA	1	15.68	1.750	85.98	6.10	0.424
AdaLN+CA	1	9.95	1.046	40.04	18.92	0.303
In-Context	8	8.12	1.081	41.97	0.194	0.316
CA	8	9.31	1.234	45.81	11.93	0.342
AdaLN+CA	8	8.60	0.970	37.13	19.38	0.305

Table 5: Impact of DiT variants: Performance on unseen AcousticRooms scenes with In-Context, Cross-Attention (CA), and hybrid AdaLN+CA conditioning.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Geometry module pipeline: A panoramic depth map captured at the receiver position is unprojected into a 3D point cloud using the equirectangular projection $\mathcal{P}$ , then reprojected so that each pixel encodes its corresponding 3D coordinates. We subtract the source pose associated with the RIR (previously projected in the receiver frame). The resulting representation is processed by a ViT followed by a linear layer to produce geometry-aware features. In HAA, the source and receiver are interchanged.

Figure A.2 : DiT architecture variants: In-Context and Cross-Attention –only architectures. The In-Context variant concatenates all conditioning information with the input before self-attention. In the Cross-Attention variant, conditioning is applied solely via cross-attention layers.

$\mathbf{\phi}_{\text{G}}$		A2G			G2A
ViT	$\mathcal{W}_{\text{DINO}}$	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$
[ 52 ]	✗	6.26	17.82	25.69	6.63	19.20	27.30
S/16 [ 37 ]	✗	26.53	54.96	67.19	27.47	55.78	67.68
S/16 [ 73 ]	✗	42.26	72.08	81.11	42.32	71.28	80.29
S/16 [ 73 ]	❄	2.00	7.42	12.77	1.94	7.40	13.08
S/16 [ 73 ]	✓	59.78	83.53	89.35	59.10	85.56	91.04

Appendix

Table A.1 : Zero-shot retrieval on the unseen split of the AcousticRooms dataset using different geometry encoders: We report both acoustic-to-geometry (A2G) and geometry-to-acoustic (G2A) results for several ViT variants.

\mathcal{W}_{\text{DINO}}

denotes DINOv3 pre-trained weights. Models marked with † are trained on the full AcousticRooms dataset.

Method	K	G	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@1 (%) $\uparrow$	R@5 (%) $\uparrow$	R@10 (%) $\uparrow$	$\text{FD}_{G}$ $\downarrow$
Random Across Rooms	✗	✗	44.02	6.415	274.43	0.02	0.10	0.21	0.061
Random Same Room	✗	✗	13.47	3.582	126.28	1.03	3.12	4.49	0.006
FLAC	✗	✓	24.56 ±0.03	2.464 ±0.003	114.84 ±0.11	4.97 ±0.14	15.63 ±0.17	22.52 ±0.13	0.326
Nearest Neighbor	1	✗	11.91	3.406	120.72	0.00	6.79	9.96	0.001
FastRIR	1	✓	15.88	2.152	92.44	0.23	0.89	1.64	0.383
xRIR	1	✓	10.66	1.476	58.67	0.39	1.72	3.15	0.270

Appendix

Table A.2 : Performance on seen AcousticRooms scenes: Results are shown for

K\in\{8,1,

✗

\}

$\mathbf{\phi}_{\text{G}}$		$\mathbf{\phi}_{\text{A}}$	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
ViT	$\mathcal{W}_{\text{DINO}}$	$\mathbf{\phi}_{\text{A}}$	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
[ 52 ]	✗	ResNet	1	6.77	0.741	30.15	12.9	0.319
S/16 [ 73 ]	✗	ResNet	1	6.83	0.740	30.23	17.65	0.310
S/16 [ 73 ]	❄	ResNet	1	7.61	0.956	36.98	6.40	0.365
S/16 [ 73 ]	✓	ResNet	1	6.46	0.692	28.11	21.54	0.296
S/16 [ 73 ]	✓	VAE	1	6.81	0.692	27.97	20.28	0.309

Appendix

Table A.3 : Effect of the geometry and acoustic conditioning encoders (seen): We evaluate different configurations of the geometry encoder

\phi_{G}

and acoustic encoder

\phi_{A}

on the seen set of the AcousticRooms dataset. For

\phi_{G}

, we compare the ViT from xRIR and DINOv3 ViT-S/16 under various initialization strategies (

\mathcal{W}_{\text{DINO}}

): trained from scratch, frozen, or initialized with DINO weights. For

\phi_{A}

, we experiment with the ResNet-18 used in xRIR and our frozen VAE encoder.

Method	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
In-Context	1	72.37	11.299	1318.71	0.05	1.302
CA	1	14.44	1.411	75.13	7.43	0.406
AdaLN+CA	1	6.46	0.692	28.11	21.54	0.296
In-Context	8	5.31	0.651	26.58	21.10	0.307
CA	8	6.01	0.782	30.16	14.40	0.328
AdaLN+CA	8	5.32	0.643	25.69	21.89	0.299

Appendix

Table A.4 : Effect of DiT architecture variants (seen): Performance comparison of different conditioning strategies on the seen set of the AcousticRooms dataset. We report results for In-Context, Cross-Attention (CA), and our hybrid design combining AdaLN for target information and CA for contextual cues.

Method	K	G	Unseen			Seen
Method	K	G	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$	R@1 $\uparrow$	R@5 $\uparrow$	R@10 $\uparrow$
Random Across Rooms	✗	✗	0.03	0.13	0.21	0.00	0.05	0.16
Random Same Room	✗	✗	0.22	0.98	1.78	0.88	2.45	3.78
FLAC	✗	✓	5.12 ±0.15	16.10 ±0.19	24.01 ±0.12	5.58 ±0.10	16.18 ±0.12	23.09 ±0.20
Nearest Neighbor	1	✗	0.05	2.04	4.09	0.21	5.65	8.41
Fast-RIR	1	✓	0.16	0.79	1.52	0.15	0.82	1.61

Appendix

Table A.5: Acoustic-to-geometry retrieval on the AcousticRooms dataset: Results are shown for

K\in\{8,1,

✗

\}

reference RIRs. For FLAC, we report mean and standard deviation over 5 generations. denotes ablations with either geometry (G) or audio conditioning removed. FLAC achieves higher acoustic-to-geometry recall than the baselines, demonstrating higher scene consistency.

Method	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	FD G $\downarrow$
Random Across Rooms	✗	17.40	10.283	533.99	2.02	0.418
Random Same Room	✗	8.00	4.805	180.15	1.51	0.253
Nearest Neighbor	1	8.19	5.000	187.55	1.32	0.268
xRIR	1	8.63	4.862	183.27	8.43	0.372
FLAC	1	3.02 ±0.04	1.676 ±0.023	75.14 ±0.98	10.31 ±1.10	0.899
Linear Interpolation	8	4.12	2.695	88.19	8.24	0.824

Appendix

Table A.6 : Sim-to-real transfer on the Hearing-Anything-Anywhere dataset with inverted receiver pose: Few-shot methods are compared to Diff-RIR and INRAS, which require per-scene training ( † ). For FLAC, we report the mean and standard deviation over 5 generations. Inverting the receiver pose improves T60/C50/EDT but reduces geometry-consistency metrics.

Method	K	MAG $\downarrow$	ENV $\downarrow$
Random Across Rooms	✗	6.97	1.581
Random Same Room	✗	3.22	0.470
Nearest Neighbor	1	3.24	0.475
xRIR	1	3.97	0.577
FLAC	1	3.16	0.393
Linear Interpolation	8	3.63	0.835

Appendix

Table A.7: Additional metrics on the Hearing-Anything-Anywhere dataset: Few-shot methods are compared against Diff-RIR, which requires per-scene training ( † ) using MAG and ENV error.

Timestep Sampler	Unseen			Seen
Timestep Sampler	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$
Uniform	8.97	0.943	38.18	5.22	0.636	25.83
Ones	8.17	1.044	37.48	5.29	0.636	25.60
Logit-Normal	9.77	1.068	42.65	5.80	0.681	27.98
LogSNR	8.60	0.970	37.13	5.32	0.643	25.69

Appendix

Table A.8 : Comparison of timestep sampling strategies during training on the generation performance: LogSNR emphasizes high

t

values, Logit-Normal concentrates on intermediate values, Ones fixes

t=1

( i.e . , full noise), and Uniform samples

t

uniformly. LogSNR provides the best overall trade-off between seen and unseen performance.

Figure A.3 : t-SNE visualization of generated RIRs across unseen rooms: Each color represents a different room category. Samples cluster tightly within the same conditioning and remains well separated across conditionings and rooms, confirming consistent yet diverse RIR generation.

Figure A.4 : User study interface. We created 2 synthetic audio using FLAC one-shot and xRIR one-shot and ask people which one matches closely with the ground truth.

Method	K	T60 (%) $\downarrow$	C50 (dB) $\downarrow$	EDT (ms) $\downarrow$	R@5 (%) $\uparrow$	$\text{FD}_{G}$ $\downarrow$
xRIR	1	14.47	1.961	74.45	1.36	0.263
FLAC-S	1	9.71 ±0.04	1.010 ±0.002	39.58 ±0.18	17.80 ±0.11	0.264
FLAC	1	9.95 ±0.05	1.046 ±0.002	40.04 ±0.22	18.92 ±0.10	0.303
xRIR	8	9.98	1.354	49.40	2.00	0.307
FLAC-S	8	8.69 ±0.02	0.942 ±0.001	37.14 ±0.08	17.88 ±0.17	0.267
FLAC	8	8.60 ±0.01	0.970 ±0.002	37.13 ±0.02	19.38 ±0.15	0.305

Appendix

Table A.10 : Comparison between FLAC-S, FLAC, and xRIR on the unseen AcousticRooms set : FLAC-S reduces the parameter count by 13M compared to FLAC. Despite having a similar number of parameters to xRIR, it achieves performance comparable to FLAC, outperforming xRIR.

Figure A.5 : Additional octave band analysis on the AcousticRooms dataset: We generate 100 samples per instance in the unseen set with FLAC under identical conditioning, and plot the mean along with a $\pm 3$ standard deviation interval (covering 99.7% of the distribution). Ground truth and xRIR predictions are shown for comparison.

Figure A.6 : Octave band analysis on the HAA dataset.

Figure A.7 : Qualitative comparison of predicted RIR waveforms on the unseen set of the AcousticRooms dataset: We compare xRIR (orange), FLAC (red), and FLAC with $K=1$ (purple) against the ground truth (green).

Figure A.8 : Qualitative comparison of predicted RIR waveforms on the HAA dataset: We compare xRIR (orange), FLAC (red), and FLAC with $K=1$ (purple) against the ground truth (green).

Audio Generation

Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.

See all 133 papers

AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer
Mar 16, 2026Pengjun Fang, Yingqing He, Yazhou Xing +3Audio-Video Generation Audio Generation
The Hong Kong University of Science and Technology · University of Central Florida
Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models
Oct 3, 2026Vasily Zadorozhnyy, Can Goksen, Kazuhito Koishida +1TTS Synthesis Zero-Shot TTS
Applied Sciences Group, Microsoft Corporation, Redmond, USA
WavFlow: Audio Generation in Waveform Space
May 18, 2026Feiyan Zhou, Luyuan Wang, Shoufa Chen +6Text-to-Audio Generation Audio-Video Generation
Meta AI · Northeastern University

Mar 16, 2026J/K move · Enter open · S save

AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer

Pengjun Fang, Yingqing He, Yazhou Xing, Qifeng Chen+2

The Hong Kong University of Science and Technology · University of Central Florida

Audio-Video Generation Audio Generation Sound Effects Generation

Open paper AskPDF arXiv

Existing video-to-audio (V2A) generation methods predominantly rely on text prompts alongside visual information to synthesize audio. However, two critical bottlenecks persist: semantic granularity gaps in training data, such as conflating acoustically distinct sounds under coarse labels, and textual ambiguity in describing micro-acoustic features. These bottlenecks make it difficult to perform fine-grained sound synthesis using text-controlled modes. To address these limitations, we propose AC-Foley, an audio-conditioned V2A model that directly leverages reference audio to achieve precise and fine-grained control over generated sounds. This approach enables fine-grained sound synthesis, timbre transfer, zero-shot sound generation, and improved audio quality. By directly conditioning on audio signals, our approach bypasses the semantic ambiguities of text descriptions while enabling precise manipulation of acoustic attributes. Empirically, AC-Foley achieves state-of-the-art performance for Foley generation when conditioned on reference audio, while remaining competitive with state-of-the-art video-to-audio methods even without audio conditioning. Code and demo are available at: https://ff2416.github.io/AC-Foley-Page

Abstract

Audio Generation

Explore similar work

AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer

Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models

WavFlow: Audio Generation in Waveform Space