FOAdapter: A Plug-and-Play Spatial Tokenizer for Low-Bitrate FOA Speech Coding

Kirak Kim; Yoonjeong Park; Juhan Nam; Sungyoung Kim; Minje Kim

eess.ASOct 12, 2026

FOAdapter: A Plug-and-Play Spatial Tokenizer for Low-Bitrate FOA Speech Coding

Authors: Kirak Kim, Yoonjeong Park, Juhan Nam, Sungyoung Kim, Minje Kim

Organizations: Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea · Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, USA

Abstract

Spatially encoded speech conveys not only linguistic information but also spatial cues. This enables listeners to distinguish and localize speakers, contributing to more natural and realistic auditory experiences in applications such as remote conferencing and immersive media. Neural speech codecs (NSCs) enable high-quality reconstruction of monaural speech at increasingly low bitrates while preserving linguistic information, but their ability to effectively preserve spatial cues remains limited. We propose FOAdapter, a plug-and-play spatial tokenizer that leverages a pretrained monaural speech codec to efficiently represent both linguistic and spatial information in first-order ambisonic (FOA) speech at low bitrates. Comprehensive evaluation demonstrates that FOAdapter consistently preserves spatial information across all evaluated codecs. Source code and audio examples are available on our demo website.

Figures & tables

Figure 1: DOA error of reconstructed FOA signal versus total bitrate on the Spatial LibriSpeech test set. FOAdapter achieves approximately

7^{\circ}

error across the evaluated monaural codecs using a 275-bit/s spatial stream.

Figure 2: Overview of the proposed FOAdapter framework.

		Total bitrate	Spatial fidelity				Speech quality				Intelligibility
Codec	Coding	(kbps)	Az. $\downarrow$	Ele. $\downarrow$	DOA $\downarrow$	RTF $\downarrow$	UTMOS $\uparrow$	PESQ $\uparrow$	SNR $\uparrow$	STOI $\uparrow$	WER $\downarrow$
Ground Truth		–	–	–	–	–	2.127	4.644	$\infty$	1.000	4.47
DAC-1	CI	3.000	12.95	6.65	15.69	1.5271	1.277	1.207	1.31	0.716	51.95
	+ FOAdapter	1.025	4.45	4.26	6.86	1.4893	1.269	1.146	$-0.19$	0.656	76.10
DAC-2	CI	6.000	4.70	3.39	6.42	1.521	1.369	1.560	3.34	0.802	23.22
	+ FOAdapter	1.775	4.43	4.18	6.78	1.4879	1.319	1.338	1.23	0.732	42.03

Table 1: Reconstruction on the Spatial LibriSpeech test set. FOAdapter framework bitrates include a 0.275-kbps spatial stream.

Figure 3: Subjective listening test results. Error bars indicate 95% confidence intervals.

Spatial Audio

Spatial audio studies how sound is captured, represented, generated, and perceived with direction, distance, motion, and room acoustics.

See all 100 papers

RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding
Oct 21, 2026Chihiro Arata, Shohei Taniguchi, Gouki Minegishi +1Neural Audio Codecs Audio Token Compression
Third Intelligence, Inc.
Exploring Token-Space Manipulation in Latent Audio Tokenizers
May 11, 2026Francesco Paissan, Luca Della Libera, Mirco Ravanelli +1Audio Representation Learning Neural Audio Codecs
1Mila – Québec AI Institute · 2Université Laval · 3Concordia University
Probing Spatial Structure in Pretrained Audio Representations
Jun 4, 2026Chuyang Chen, Sivan Ding, Adrian S. Roman +1Audio Representation Learning Representation Probing
Music and Audio Research Laboratory, New York University, USA

Oct 21, 2026J/K move · Enter open · S save

RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

Chihiro Arata, Shohei Taniguchi, Gouki Minegishi, Kan Hatakeyama-Sato

Third Intelligence, Inc.

Neural Audio Codecs Audio Token Compression Streaming TTS Synthesis

Open paper AskPDF arXiv

Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information. Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Its Adaptive Firing mechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor. Discrete codes and an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates. A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.

Abstract

Spatial Audio

Explore similar work

RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

Exploring Token-Space Manipulation in Latent Audio Tokenizers

Probing Spatial Structure in Pretrained Audio Representations