Geometric Iterative Retrieval for Neural Audio Codec Resynthesis

Leo Schmidt-Traub; Frédéric Berdoz; Luca A. Lanzendörfer; Roger Wattenhofer

cs.SDAug 19, 2026

Geometric Iterative Retrieval for Neural Audio Codec Resynthesis

Authors: Leo Schmidt-Traub, Frédéric Berdoz, Luca A. Lanzendörfer, Roger Wattenhofer

Organizations: ETH Zurich

Abstract

Neural audio codecs based on Residual Vector Quantization (RVQ) have become the dominant discrete representation for token-based general audio generation, yet resynthesizing high-quality audio from coarse codec tokens remains an open problem and bounds the fidelity of every system that generates them. Prior work has framed resynthesis as a choice between discrete token prediction and continuous regression. We argue that this dichotomy is incomplete and introduce geometric iterative retrieval, a paradigm that uses the RVQ layer hierarchy itself as a natural iterative decomposition in continuous codebook space. Rather than classifying over discrete vocabularies or regressing to a single target vector, our method performs contrastive retrieval in the codebook's geometric space. We evaluate our method on codec restoration tasks across speech and music, and show improvements over both single-pass token prediction and one-step regression baselines.

Figures & tables

	Single-step	Iterative
Discrete	(degenerate)	Token prediction
Continuous	Regression	Diffusion / Retrieval (ours)

Table 1: Codec resynthesis methods organized along two axes, the prediction space (discrete vs. continuous) and the refinement strategy (single-step vs. iterative). Prior work occupies three of the four cells. Our method shares the continuous-iterative cell with diffusion, but iterates along the codec’s own RVQ hierarchy rather than along an externally imposed noise schedule, giving each step a meaning grounded in the codec rather than in the sampler.

Figure 1: Combined view of geometric iterative retrieval. (a) During training, the model predicts layers

2,\dots,D

from the masked input stack in a single forward pass, supervised by a CLIP-style contrastive loss per predicted layer. (b) At inference, layer

k+1

is predicted from layers

1,\dots,k

, quantized by nearest-neighbor lookup against the layer-

k+1

codebook, and concatenated for the next step. (c) Architecture: per-position codebook embeddings are aggregated by self-attention into one hidden state, processed by the DeBERTa-v3 encoder, and projected by

W_{\text{out}}

into the codebook space. Orange = discrete tokens, blue = continuous latents.

Method	LSD $\downarrow$	SI-SDR $\uparrow$	FAD $\downarrow$
Naive	$\underline{10.82\!\pm\!0.08}$	$-1.88\!\pm\!0.24$	0.99
CE	$12.42\!\pm\!0.20$	$-6.93\!\pm\!0.30$	2.46
OSR	$11.03\!\pm\!0.07$	$\mathbf{-0.22\!\pm\!0.27}$	0.61
MSE	$12.86\!\pm\!0.22$	$-7.83\!\pm\!0.30$	3.08
Ours	$\mathbf{10.61\!\pm\!0.07}$	$\underline{-0.63\!\pm\!0.28}$	0.94

Table 2: Codec restoration results on the combined val split (1500 clips). Lower is better for LSD and FAD, higher is better for SI-SDR. Best per column in bold , second-best underlined . LSD and SI-SDR are reported as mean

\pm

half-width of the 95 % normal CI over the 1500 clips. FAD is a single set-level number per cell and admits no per-sample CI. OSR is one-step, layer-wise regression under cosine-similarity loss. MSE is the same backbone trained with MSE against the residual codebook latent.

$K$	LSD $\downarrow$	SI-SDR $\uparrow$	FAD $\downarrow$
1 (input only)	10.81	$-1.88$	0.99
2	10.26 ( $-5.1\%$ )	$-0.34$ ( $+82\%$ )	0.43 ( $-57\%$ )
3	$\mathbf{10.20}$ ( $\mathbf{-5.6\%}$ )	$\mathbf{-0.22}$ ( $\mathbf{+88\%}$ )	$\mathbf{0.43}$ ( $\mathbf{-57\%}$ )
4	10.23 ( $-5.4\%$ )	$-0.30$ ( $+84\%$ )	0.56 ( $-43\%$ )
5	10.34 ( $-4.3\%$ )	$-0.41$ ( $+78\%$ )	0.69 ( $-30\%$ )
6	10.41 ( $-3.7\%$ )	$-0.50$ ( $+73\%$ )	0.77 ( $-22\%$ )

Table 3: Reconstruction quality as a function of the number of predicted residual layers

K

(layer

1

is the ground-truth input, layers

2,\dots,K

are our predictions). All three objective metrics improve from

K{=}1

through

K{=}3

and then degrade monotonically, because each additional residual layer brings progressively less spectral signal while still injecting prediction noise of roughly the same magnitude. Beyond

K{=}3

, the noise outweighs the signal. Parenthesized values report the percentage change relative to the

K{=}1

input-only baseline. Best per column in bold .

Condition	Mean (95% CI)	$\Delta$	W/L
Naive	23.8 [21.0, 26.7]	$+31.4$	19 / 148
CE	0 9.1 [7.5, 10.7]	$+45.5$	0 6 / 164
MSE	0 7.6 [5.7, 9.6]	$+46.8$	0 7 / 164
OSR	43.4 [39.7, 47.1]	$+11.2$	48 / 117
Ours (K=3)	49.3 [45.7, 52.8]	$+5.3$	64 / 0 99
Ours	54.0 [50.1, 57.9]	—	—

Table 4: Double-blind human evaluation (19 raters

\times

9 clips). Higher is better.

\Delta

is the mean per-trial difference (Ours

-

baseline). W ins/ L osses count the trials in which the baseline scored above or below Ours (ties omitted). All five pairwise differences favor our method and are significant under a Wilcoxon signed-rank test (Naive, CE, MSE with

|z|>9.7

p<10^{-21}

; OSR with

|z|=5.5

p=3.4{\times}10^{-8}

; Ours (K=3) with

|z|=3.9

p=1.1{\times}10^{-4}

Variant	LSD $\downarrow$	SI-SDR $\uparrow$	FAD $\downarrow$
Ours	$\mathbf{10.61\pm 0.07}$	$\mathbf{-0.63\pm 0.28}$	0.94
Full-residual	$12.29\pm 0.22$	$-4.22\pm 0.32$	1.14
No-contrastive	$\underline{10.75\pm 0.07}$	$\underline{-1.17\pm 0.31}$	0.99
Additive	$11.74\pm 0.09$	$-1.24\pm 0.30$	0.96

Table 5: Ablation results on the combined validation split. Each variant perturbs one component, the prediction target (per-layer

\mathbf{e}_{k}

vs. cumulative residual

\sum_{j=k+1}^{D}\mathbf{e}_{j}

), the loss (contrastive retrieval vs. cosine regression to the codebook vector), or the layer-aggregation mechanism (self-attention vs. additive summation). Best per column in bold , second-best underlined . LSD and SI-SDR are reported as mean

\pm

half-width of the 95 % normal CI over the 1500 clips. FAD is a single set-level number per cell.

Neural Audio Codecs

Neural audio codecs learn compact representations for compressing and reconstructing speech, music, and other sounds.

See all 87 papers

HARP: Harmonic-Aware Residual Partitioning for Neural Audio Codecs
Jul 18, 2026Qiaoyu Yang, Lixing He, Binyue Deng +1Audio Representation Learning Neural Audio Codecs
Georgia Institute of Technology, Atlanta, United States · The Chinese University of Hong Kong, Hong Kong, China · Tencent Music Entertainment, Shenzhen, China
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
Apr 20, 2026Ho-Lam Chung, Yiming Chen, Hung-yi LeeNeural Audio Codecs Multi-Token Prediction
Graduate Institute of Communication Engineering, National Taiwan University · ASUS Intelligent Cloud Services · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding
Oct 3, 2026Ron Aluf, Alon Canfi, Eliya NachmaniNeural Audio Codecs
School of Electrical and Computer Engineering Ben-Gurion University of the Negev

Jul 18, 2026J/K move · Enter open · S save

HARP: Harmonic-Aware Residual Partitioning for Neural Audio Codecs

Qiaoyu Yang, Lixing He, Binyue Deng, Weifeng Zhao

Georgia Institute of Technology, Atlanta, United States · The Chinese University of Hong Kong, Hong Kong, China · Tencent Music Entertainment, Shenzhen, China

Audio Representation Learning Neural Audio Codecs Residual VQ Vector Quantization

Open paper AskPDF arXiv

Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.

Abstract

Neural Audio Codecs

Explore similar work

HARP: Harmonic-Aware Residual Partitioning for Neural Audio Codecs

LLM-Codec: Neural Audio Codec Meets Language Model Objectives

GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding