GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding

Ron Aluf; Alon Canfi; Eliya Nachmani

cs.SDOct 3, 2026

GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding

Authors: Ron Aluf, Alon Canfi, Eliya Nachmani

Organizations: School of Electrical and Computer Engineering Ben-Gurion University of the Negev

Abstract

Neural audio codecs compress waveforms into compact discrete tokens that underpin speech language models, real-time communication, and large-scale audio storage. Almost every dominant design, including residual vector quantization, finite scalar quantization, and single-codebook variants, follows the VQ-VAE template by partitioning the encoder latent through a learned codebook or a fixed scalar grid. We ask whether this partition is necessary. We introduce GS-Codec, a neural speech codec whose bottleneck is a parametric signal decomposition rather than a quantizer. We adapt Gaussian splatting from 3D scene reconstruction to one-dimensional latents. An inner optimization loop fits each encoder segment as a weighted sum of 1D Gaussian primitives. The decoder then reconstructs the waveform from the rendered sum. To avoid the cost of this iterative inner loop at inference time, we additionally train a lightweight GS Predictor Net that regresses the primitive parameters in a single forward pass. The encoder and decoder are trained end-to-end through the inner loop, with no quantizer anywhere in the training pipeline: the bottleneck is the decomposition itself, and scalar quantization is applied only post-training to the fitted parameters. Rather than relying on discrete codebook stages for bitrate control, our representation exposes a fine-grained rate-quality tradeoff: a single trained checkpoint supports post-training bitrate control by varying the number of primitives and the per-parameter bit depth, with no retraining required. GS-Codec matches or exceeds well-established open-source codecs such as EnCodec and DAC on speaker similarity (SIM), intelligibility (STOI), and perceptual quality (UTMOS) at comparable bitrates, while achieving comparable semantic performance (WER). Code and audio samples are available at https://ronaluf.github.io/gs-codec/

Figures & tables

Figure 1: GS-Codec pipeline. Training (top): The encoder latent $Z$ is approximated by $N_{G}$ Gaussian primitives via an inner optimization loop. Inference (bottom): A GS Predictor Net regresses primitive parameters in a single pass, which are then quantized, transmitted, and decoded.

Figure 2: Visualization of the GS inner loop on a single latent channel across optimization steps 0, 50, and 100. Ground-truth $Z_{c}$ is shown in blue, rendered approximation $\hat{Z}_{c}$ in red, and individual weighted Gaussian primitives are shown underneath. The horizontal axis denotes the temporal latent frame ( $t$ ), while the vertical axis represents the latent value.

Dataset	Band	Model	BW $\downarrow$	tok/s $\downarrow$	UTMOS $\uparrow$	PESQ $\uparrow$	STOI $\uparrow$	V/UV F1 $\uparrow$
LibriTTS test-clean		DAC (ref)	9.0 kbps	900	3.910	3.908	0.970	0.978
	$\geq$ 5 kbps high	EnCodec	6.0 kbps	600	3.040	2.720	0.939	0.953
		Vocos	6.0 kbps	600	3.695	2.807	0.943	0.944
		SpeechTokenizer	6.0 kbps	600	3.879	2.612	0.917	0.950
		GS-Codec ( $N_{G}{=}90$ , 6b, Pred, frozen)	5.94 kbps	990	4.076	3.289	0.970	0.956
		GS-Codec ( $N_{G}{=}90$ , 6b, frozen)	5.94 kbps	990	4.094	3.348	0.972	0.960

Table 1: Objective reconstruction results on LibriTTS test-clean (clean), LibriTTS test-other (noisy), and LJSpeech (out-of-domain), following the protocol of Ji et al. [31] Table 1. PESQ is wideband (WB). Best results in each band are bold .

	Model	BR	Param	WER $\downarrow$	PESQ $\uparrow$	UTMOS $\uparrow$	SIM $\uparrow$
		(kbps)		(%)
$\geq$ 5 kbps high	DAC-75Hz	6.0	74M	2.27	3.77	3.62	0.90
	EnCodec-75Hz	6.0	15M	2.24	3.12	3.01	0.89
	GS-Codec (130g, 4b, frozen)	5.72	55M	2.25	3.67	3.93	0.93
	GS-Codec (120g, 4b, frozen)	5.28	55M	2.22	3.65	3.94	0.93
	GS-Codec (110g, 5b, frozen)	6.05	55M	2.19	3.71	4.01	0.94

Table 2: Comparison with open-source neural audio codecs on LibriSpeech test-clean . WER evaluated with facebook/hubert-large-ls960-ft on 4-10 s subset. Acoustic metrics on RVQ1:8 for stacked codecs; for GS-Codec this is the full 33-row token stream (see Eq. 4 ). Baseline results from published results.

	RVQ	FSQ	GS Predictor
Payload (kbps)	6.000	6.000	5.950
PESQ-NB $\uparrow$	$3.8\pm.011$	$3.8\pm.013$	$3.8\pm.009$
UTMOS $\uparrow$	$3.94\pm.020$	$3.94\pm.019$	$4.05\pm.017$
STOI $\uparrow$	$.96\pm.0008$	$.96\pm.0008$	$.96\pm.0011$
SIM $\uparrow$	$.93\pm.0013$	$.92\pm.0014$	$.94\pm.0012$
WER (%) $\downarrow$	$2.1\pm.25$	$2.2\pm.26$	$2.1\pm.25$

Table 3: Same-backbone comparison at approximately 6 kbps. Values are means

\pm

95% utterance-bootstrap confidence interval half-widths (

n=1{,}237

). PESQ is narrowband. GS uses predicted centers.

Figure 3: $N_{G}$ sweep at fixed 5-bit PTQ on LibriSpeech test-clean (4-10 s subset, $n{=}1237$ ; WER: hubert-large-ls960-ft ).

Figure 4: Bit-depth sweep for $N_{G}{=}60$ (orange squares) and $N_{G}{=}80$ (blue circles) on LibriSpeech test-clean (4-10 s subset, $n{=}1237$ ). Metrics saturate past 6 bits; $N_{G}{=}80$ outperforms $N_{G}{=}60$ at every bit-depth.

Figure 5: Effect of unfreezing primitive positions at fixed 6-bit PTQ. Left column: absolute UTMOS and PESQ versus $N_{G}$ for frozen (solid blue) and learned (dashed orange) positions. Right column: metric delta $\Delta=\text{learned}-\text{frozen}$ .

Model	Mean	Median	p95	RTF
GS-Codec iterative (300-step Adam)	171.2 ms	152.6 ms	208.4 ms	0.057
GS-Pred	7.9 ms	7.9 ms	7.9 ms	0.0026
EnCodec 24 kHz @ 6 kbps	5.4 ms	5.3 ms	5.4 ms	0.0018
DAC 24 kHz @ 6 kbps	5.0 ms	5.0 ms	5.1 ms	0.0017

Table 4: Encoder GPU execution time on NVIDIA L40S. RTF

=t_{\text{encode}}/t_{\text{audio}}

; lower is better. Measured over 200 segments

\times

3 s with 10 warmup segments. torch.cuda.Event timing; single run.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Min	Max	Levels ( $2^{B}$ )
Scale $\sigma_{i}$	0.5	56.25	64
Amplitude $w_{i,c}$	$-3.0$	$3.0$	64

Appendix

Table 5: Per-parameter

k

-means PTQ codebooks (6-bit,

2^{6}{=}64

centroids). Ranges estimated on held-out calibration set; loaded at dequantisation.

System	kbps	PESQ-NB $\uparrow$	UTMOS $\uparrow$	STOI $\uparrow$	SIM $\uparrow$	WER $\downarrow$	Steps/s	Tok./step	Tok./s
GS iterative, 102g/5b	5.95	$3.8\pm.009$	$4.07\pm.017$	$.96\pm.0010$	$.94\pm.0012$	$2.1\pm.26$	34	34	1,156
GS Predictor, 102g/5b	5.95	$3.8\pm.009$	$4.05\pm.017$	$.96\pm.0011$	$.94\pm.0012$	$2.1\pm.25$	34	34	1,156
FSQ, 32-dim	6.00	$3.8\pm.013$	$3.94\pm.019$	$.96\pm.0008$	$.92\pm.0014$	$2.2\pm.26$	75	32	2,400
RVQ, 8 codebooks	6.00	$3.8\pm.011$	$3.94\pm.020$	$.96\pm.0008$	$.93\pm.0013$	$2.1\pm.25$	75	8	600
DAC-75Hz	6.00	$3.7\pm.010$	$3.62\pm.023$	$.95\pm.0007$	$.88\pm.0017$	$2.3\pm.26$	75	8	600
EnCodec-75Hz	6.00	$3.1\pm.018$	$3.09\pm.027$	$.93\pm.0010$	$.89\pm.0015$	$2.2\pm.26$	75	8	600

Appendix

Table 6: Unified evaluation on the pinned 1,237 utterances. DAC and EnCodec are public checkpoints rerun with the same protocol.

Subset	GS Predictor	FSQ	RVQ	$\Delta$ vs FSQ [95% CI]	Holm $p$
High F0	3.97	3.84	3.85	$+0.125$ [ $+0.109$ , $+0.140$ ]	$3\times 10^{-43}$
High frication proxy	4.01	3.88	3.90	$+0.122$ [ $+0.108$ , $+0.137$ ]	$6\times 10^{-47}$
High spectral flux	4.04	3.95	3.95	$+0.096$ [ $+0.083$ , $+0.109$ ]	$8\times 10^{-40}$
Low energy	4.05	3.94	3.93	$+0.114$ [ $+0.100$ , $+0.129$ ]	$7\times 10^{-46}$
Lowest peak-to-floor ratio	3.98	3.86	3.86	$+0.119$ [ $+0.106$ , $+0.132$ ]	$2\times 10^{-50}$
All	4.05	3.94	3.94	$+0.109$ [ $+0.101$ , $+0.116$ ]	$2\times 10^{-133}$

Appendix

Table 7: UTMOS by subgroup.

Representation	Values	Bits / 3 s	NMSE $\downarrow$
Gaussian, free centers, 102	3,468	17,850	.0259
Triangular, 102	3,468	17,850	.0275
Laplacian, 102	3,468	17,850	.0295
Gabor, 97	3,492	17,945	.0303
Gaussian, grid centers, 109	3,597	17,985	.0317
No decomposition: decimate to 102 frames	3,264	17,850	.0496

Appendix

Table 8: Latent NMSE at matched payload.

GS $N_{G}$ /bits	kbps	UTMOS	AE frames/bits	kbps	UTMOS	$\Delta$
102/5	5.950	4.051	102/5 (trained)	5.440	3.985	$+0.066$
91/5	5.005	3.699	94/5	5.013	2.823	$+0.876$
131/4	5.764	3.969	136/4	5.803	3.118	$+0.851$
127/5	6.985	4.025	131/5	6.987	3.145	$+0.880$

Appendix

Table 9: GS versus a continuous autoencoder with post-hoc

k

-means. UTMOS on the 1,237 utterances.

Bottleneck	Time/update	Updates/s	400k updates
FSQ	1.160 s	0.862	$\approx$ 5.3 days
GS, 100-step inner loop	1.209 s	0.827	$\approx$ 5.6 days
RVQ, 8 codebooks	1.319 s	0.758	$\approx$ 6.1 days

Appendix

Table 10: Training cost per update.

System	kbps	MUSHRA $\uparrow$	MOS gap $\downarrow$
Hidden reference / natural speech	n/a	$78.62\pm 5.41$	0.00 [0.00, 0.00]
GS iterative, unfrozen, 100g/5b	5.833	$77.67\pm 5.01$	0.14 [ $-0.28$ , 0.56]
GS Predictor, 109g/5b	5.995	$75.36\pm 5.30$	0.61 [0.19, 1.03]
GS iterative, frozen, 110g/5b	6.050	$75.13\pm 5.14$	0.50 [0.00, 0.97]
RVQ	6.000	$72.53\pm 5.66$	0.69 [0.25, 1.14]
FSQ	3.000	$70.78\pm 5.78$	0.78 [0.22, 1.31]

Appendix

Table 11: Listening scores. MUSHRA: mean

\pm

95% CI half-width. MOS gap: mean [95% CI].

kbps	$N_{G}$	Bits	PESQ-NB $\uparrow$	UTMOS $\uparrow$	WER $\downarrow$	SIM $\uparrow$
3.30	100	3	$2.926\pm.055$	$3.425\pm.098$	$2.48\pm.74$	$.844\pm.006$
3.30	60	5	$2.484\pm.042$	$2.845\pm.066$	$2.38\pm.72$	$.797\pm.009$
3.96	120	3	$3.155\pm.051$	$3.583\pm.103$	$2.33\pm.73$	$.867\pm.007$
3.96	60	6	$2.515\pm.044$	$2.880\pm.066$	$2.48\pm.74$	$.804\pm.009$
3.96	40	9	$1.898\pm.031$	$1.905\pm.065$	$5.48\pm 1.35$	$.612\pm.013$
4.40	100	4	$3.494\pm.044$	$3.889\pm.087$	$2.02\pm.68$	$.912\pm.004$

Appendix

Table 12: Fixed-budget allocation between primitive count and precision. WER in percent.

Neural Audio Codecs

Neural audio codecs learn compact representations for compressing and reconstructing speech, music, and other sounds.

See all 85 papers

Sep 10, 2026

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

Luca Della Libera, Cem Subakan, Mirco Ravanelli

Concordia University · Mila-Quebec AI Institute · Universit´e Laval

Audio Token Compression Neural Audio Codecs Speech Processing

Ask PDF

Sep 15, 2025

CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents

Text-guided sound separation enables flexible audio editing, assistive listening, and open-domain source extraction, but systems such as AudioSep remain too expensive for low-latency edge or codec-mediated deployment. Existing neural audio codec separators are efficient, yet largely restricted to fixed stems or closed taxonomies. We introduce CodecSep, a prompt-driven universal sound separation framework that extracts sources directly in neural audio codec latent space. CodecSep combines a frozen DAC backbone with a lightweight FiLM-conditioned Transformer masker driven by CLAP text embeddings, enabling open-vocabulary separation while preserving codec-native efficiency. Across dnr-v2 and five open-domain benchmarks, CodecSep consistently improves over AudioSep in SI-SDR, remains competitive in ViSQOL, and achieves clear gains in human MOS-LQS. Controlled analyses show that fine-grained prompts outperform coarse labels, and that explicit latent masking is substantially more effective than decoder-style latent generation in codec space. Qualitative diagnostics show that neural audio codec latents retain source-dependent structure, which CodecSep exploits mainly through channel-wise source-conditioned modulation. CodecSep also provides a practical code-stream deployment path. When audio is transmitted as neural audio codec codes, CodecSep maps codes to embeddings, separates directly in codec space, and outputs waveforms or re-quantized codes, avoiding the decode-separate-re-encode loop. In this regime, CodecSep requires only 1.35 GMACs end-to-end: about 54 times less compute than AudioSep in the same pipeline and 25 times lower separator-only compute, with much lower latency and memory. More broadly, CodecSep offers a blueprint for codec-native downstream audio processing.

Adhiraj Banerjee, Vipul Arora

Department of Electrical Engineering · Indian Institute of Technology, Kanpur

Audio Source Separation

Ask PDF

Aug 6, 2026

LILAC: An Idempotent Neural Speech Codec

Neural Audio Codecs are widely adopted in speech generation and editing. However, existing neural audio codecs are not idempotent: across the paper's twelve baseline systems, every configuration tested rewrites, on average, at least 15% of its tokens in a single decode-re-encode pass. This poses a problem for utilizing Neural Audio Codecs as token interfaces in pipelines where re-encoding decoded outputs can occur. We present LILAC, a fully convolutional 24 kHz speech codec at 9.375 Hz and 0.75 kbit/s that is codec idempotent by construction; re-encoding the decoded audio of any valid token stream returns the identical stream. LILAC achieves idempotency while maintaining competitive quality, reaching UTMOS 4.14 and 4.24 on LibriSpeech and LibriTTS-R test sets, comparable to SOTA sub-1 kbit/s Neural Audio Codecs.

June Young Yi, Dongwook Lee, Jiheum Yeom +1

Department of Computer Science and Engineering, Seoul National University · 2Interdisciplinary Program in Artificial Intelligence, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University +1

Neural Audio Codecs

Ask PDF

Abstract

Neural Audio Codecs

Explore similar work

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents

LILAC: An Idempotent Neural Speech Codec