GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding

Ron Aluf; Alon Canfi; Eliya Nachmani

cs.SDOct 3, 2026

GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding

Authors: Ron Aluf, Alon Canfi, Eliya Nachmani

Organizations: School of Electrical and Computer Engineering Ben-Gurion University of the Negev

Abstract

Neural audio codecs compress waveforms into compact discrete tokens that underpin speech language models, real-time communication, and large-scale audio storage. Almost every dominant design, including residual vector quantization, finite scalar quantization, and single-codebook variants, follows the VQ-VAE template by partitioning the encoder latent through a learned codebook or a fixed scalar grid. We ask whether this partition is necessary. We introduce GS-Codec, a neural speech codec whose bottleneck is a parametric signal decomposition rather than a quantizer. We adapt Gaussian splatting from 3D scene reconstruction to one-dimensional latents. An inner optimization loop fits each encoder segment as a weighted sum of 1D Gaussian primitives. The decoder then reconstructs the waveform from the rendered sum. To avoid the cost of this iterative inner loop at inference time, we additionally train a lightweight GS Predictor Net that regresses the primitive parameters in a single forward pass. The encoder and decoder are trained end-to-end through the inner loop, with no quantizer anywhere in the training pipeline: the bottleneck is the decomposition itself, and scalar quantization is applied only post-training to the fitted parameters. Rather than relying on discrete codebook stages for bitrate control, our representation exposes a fine-grained rate-quality tradeoff: a single trained checkpoint supports post-training bitrate control by varying the number of primitives and the per-parameter bit depth, with no retraining required. GS-Codec matches or exceeds well-established open-source codecs such as EnCodec and DAC on speaker similarity (SIM), intelligibility (STOI), and perceptual quality (UTMOS) at comparable bitrates, while achieving comparable semantic performance (WER). Code and audio samples are available at https://ronaluf.github.io/gs-codec/

Figures & tables

Figure 1: GS-Codec pipeline. Training (top): The encoder latent $Z$ is approximated by $N_{G}$ Gaussian primitives via an inner optimization loop. Inference (bottom): A GS Predictor Net regresses primitive parameters in a single pass, which are then quantized, transmitted, and decoded.

Figure 2: Visualization of the GS inner loop on a single latent channel across optimization steps 0, 50, and 100. Ground-truth $Z_{c}$ is shown in blue, rendered approximation $\hat{Z}_{c}$ in red, and individual weighted Gaussian primitives are shown underneath. The horizontal axis denotes the temporal latent frame ( $t$ ), while the vertical axis represents the latent value.

Dataset	Band	Model	BW $\downarrow$	tok/s $\downarrow$	UTMOS $\uparrow$	PESQ $\uparrow$	STOI $\uparrow$	V/UV F1 $\uparrow$
LibriTTS test-clean		DAC (ref)	9.0 kbps	900	3.910	3.908	0.970	0.978
	$\geq$ 5 kbps high	EnCodec	6.0 kbps	600	3.040	2.720	0.939	0.953
		Vocos	6.0 kbps	600	3.695	2.807	0.943	0.944
		SpeechTokenizer	6.0 kbps	600	3.879	2.612	0.917	0.950
		GS-Codec ( $N_{G}{=}90$ , 6b, Pred, frozen)	5.94 kbps	990	4.076	3.289	0.970	0.956
		GS-Codec ( $N_{G}{=}90$ , 6b, frozen)	5.94 kbps	990	4.094	3.348	0.972	0.960

Table 1: Objective reconstruction results on LibriTTS test-clean (clean), LibriTTS test-other (noisy), and LJSpeech (out-of-domain), following the protocol of Ji et al. [31] Table 1. PESQ is wideband (WB). Best results in each band are bold .

	Model	BR	Param	WER $\downarrow$	PESQ $\uparrow$	UTMOS $\uparrow$	SIM $\uparrow$
		(kbps)		(%)
$\geq$ 5 kbps high	DAC-75Hz	6.0	74M	2.27	3.77	3.62	0.90
	EnCodec-75Hz	6.0	15M	2.24	3.12	3.01	0.89
	GS-Codec (130g, 4b, frozen)	5.72	55M	2.25	3.67	3.93	0.93
	GS-Codec (120g, 4b, frozen)	5.28	55M	2.22	3.65	3.94	0.93
	GS-Codec (110g, 5b, frozen)	6.05	55M	2.19	3.71	4.01	0.94

Table 2: Comparison with open-source neural audio codecs on LibriSpeech test-clean . WER evaluated with facebook/hubert-large-ls960-ft on 4-10 s subset. Acoustic metrics on RVQ1:8 for stacked codecs; for GS-Codec this is the full 33-row token stream (see Eq. 4 ). Baseline results from published results.

	RVQ	FSQ	GS Predictor
Payload (kbps)	6.000	6.000	5.950
PESQ-NB $\uparrow$	$3.8\pm.011$	$3.8\pm.013$	$3.8\pm.009$
UTMOS $\uparrow$	$3.94\pm.020$	$3.94\pm.019$	$4.05\pm.017$
STOI $\uparrow$	$.96\pm.0008$	$.96\pm.0008$	$.96\pm.0011$
SIM $\uparrow$	$.93\pm.0013$	$.92\pm.0014$	$.94\pm.0012$
WER (%) $\downarrow$	$2.1\pm.25$	$2.2\pm.26$	$2.1\pm.25$

Table 3: Same-backbone comparison at approximately 6 kbps. Values are means

\pm

95% utterance-bootstrap confidence interval half-widths (

n=1{,}237

). PESQ is narrowband. GS uses predicted centers.

Figure 3: $N_{G}$ sweep at fixed 5-bit PTQ on LibriSpeech test-clean (4-10 s subset, $n{=}1237$ ; WER: hubert-large-ls960-ft ).

Figure 4: Bit-depth sweep for $N_{G}{=}60$ (orange squares) and $N_{G}{=}80$ (blue circles) on LibriSpeech test-clean (4-10 s subset, $n{=}1237$ ). Metrics saturate past 6 bits; $N_{G}{=}80$ outperforms $N_{G}{=}60$ at every bit-depth.

Figure 5: Effect of unfreezing primitive positions at fixed 6-bit PTQ. Left column: absolute UTMOS and PESQ versus $N_{G}$ for frozen (solid blue) and learned (dashed orange) positions. Right column: metric delta $\Delta=\text{learned}-\text{frozen}$ .

Model	Mean	Median	p95	RTF
GS-Codec iterative (300-step Adam)	171.2 ms	152.6 ms	208.4 ms	0.057
GS-Pred	7.9 ms	7.9 ms	7.9 ms	0.0026
EnCodec 24 kHz @ 6 kbps	5.4 ms	5.3 ms	5.4 ms	0.0018
DAC 24 kHz @ 6 kbps	5.0 ms	5.0 ms	5.1 ms	0.0017

Table 4: Encoder GPU execution time on NVIDIA L40S. RTF

=t_{\text{encode}}/t_{\text{audio}}

; lower is better. Measured over 200 segments

\times

3 s with 10 warmup segments. torch.cuda.Event timing; single run.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Min	Max	Levels ( $2^{B}$ )
Scale $\sigma_{i}$	0.5	56.25	64
Amplitude $w_{i,c}$	$-3.0$	$3.0$	64

Appendix

Table 5: Per-parameter

k

-means PTQ codebooks (6-bit,

2^{6}{=}64

centroids). Ranges estimated on held-out calibration set; loaded at dequantisation.

System	kbps	PESQ-NB $\uparrow$	UTMOS $\uparrow$	STOI $\uparrow$	SIM $\uparrow$	WER $\downarrow$	Steps/s	Tok./step	Tok./s
GS iterative, 102g/5b	5.95	$3.8\pm.009$	$4.07\pm.017$	$.96\pm.0010$	$.94\pm.0012$	$2.1\pm.26$	34	34	1,156
GS Predictor, 102g/5b	5.95	$3.8\pm.009$	$4.05\pm.017$	$.96\pm.0011$	$.94\pm.0012$	$2.1\pm.25$	34	34	1,156
FSQ, 32-dim	6.00	$3.8\pm.013$	$3.94\pm.019$	$.96\pm.0008$	$.92\pm.0014$	$2.2\pm.26$	75	32	2,400
RVQ, 8 codebooks	6.00	$3.8\pm.011$	$3.94\pm.020$	$.96\pm.0008$	$.93\pm.0013$	$2.1\pm.25$	75	8	600
DAC-75Hz	6.00	$3.7\pm.010$	$3.62\pm.023$	$.95\pm.0007$	$.88\pm.0017$	$2.3\pm.26$	75	8	600
EnCodec-75Hz	6.00	$3.1\pm.018$	$3.09\pm.027$	$.93\pm.0010$	$.89\pm.0015$	$2.2\pm.26$	75	8	600

Appendix

Table 6: Unified evaluation on the pinned 1,237 utterances. DAC and EnCodec are public checkpoints rerun with the same protocol.

Subset	GS Predictor	FSQ	RVQ	$\Delta$ vs FSQ [95% CI]	Holm $p$
High F0	3.97	3.84	3.85	$+0.125$ [ $+0.109$ , $+0.140$ ]	$3\times 10^{-43}$
High frication proxy	4.01	3.88	3.90	$+0.122$ [ $+0.108$ , $+0.137$ ]	$6\times 10^{-47}$
High spectral flux	4.04	3.95	3.95	$+0.096$ [ $+0.083$ , $+0.109$ ]	$8\times 10^{-40}$
Low energy	4.05	3.94	3.93	$+0.114$ [ $+0.100$ , $+0.129$ ]	$7\times 10^{-46}$
Lowest peak-to-floor ratio	3.98	3.86	3.86	$+0.119$ [ $+0.106$ , $+0.132$ ]	$2\times 10^{-50}$
All	4.05	3.94	3.94	$+0.109$ [ $+0.101$ , $+0.116$ ]	$2\times 10^{-133}$

Appendix

Table 7: UTMOS by subgroup.

Representation	Values	Bits / 3 s	NMSE $\downarrow$
Gaussian, free centers, 102	3,468	17,850	.0259
Triangular, 102	3,468	17,850	.0275
Laplacian, 102	3,468	17,850	.0295
Gabor, 97	3,492	17,945	.0303
Gaussian, grid centers, 109	3,597	17,985	.0317
No decomposition: decimate to 102 frames	3,264	17,850	.0496

Appendix

Table 8: Latent NMSE at matched payload.

GS $N_{G}$ /bits	kbps	UTMOS	AE frames/bits	kbps	UTMOS	$\Delta$
102/5	5.950	4.051	102/5 (trained)	5.440	3.985	$+0.066$
91/5	5.005	3.699	94/5	5.013	2.823	$+0.876$
131/4	5.764	3.969	136/4	5.803	3.118	$+0.851$
127/5	6.985	4.025	131/5	6.987	3.145	$+0.880$

Appendix

Table 9: GS versus a continuous autoencoder with post-hoc

k

-means. UTMOS on the 1,237 utterances.

Bottleneck	Time/update	Updates/s	400k updates
FSQ	1.160 s	0.862	$\approx$ 5.3 days
GS, 100-step inner loop	1.209 s	0.827	$\approx$ 5.6 days
RVQ, 8 codebooks	1.319 s	0.758	$\approx$ 6.1 days

Appendix

Table 10: Training cost per update.

System	kbps	MUSHRA $\uparrow$	MOS gap $\downarrow$
Hidden reference / natural speech	n/a	$78.62\pm 5.41$	0.00 [0.00, 0.00]
GS iterative, unfrozen, 100g/5b	5.833	$77.67\pm 5.01$	0.14 [ $-0.28$ , 0.56]
GS Predictor, 109g/5b	5.995	$75.36\pm 5.30$	0.61 [0.19, 1.03]
GS iterative, frozen, 110g/5b	6.050	$75.13\pm 5.14$	0.50 [0.00, 0.97]
RVQ	6.000	$72.53\pm 5.66$	0.69 [0.25, 1.14]
FSQ	3.000	$70.78\pm 5.78$	0.78 [0.22, 1.31]

Appendix

Table 11: Listening scores. MUSHRA: mean

\pm

95% CI half-width. MOS gap: mean [95% CI].

kbps	$N_{G}$	Bits	PESQ-NB $\uparrow$	UTMOS $\uparrow$	WER $\downarrow$	SIM $\uparrow$
3.30	100	3	$2.926\pm.055$	$3.425\pm.098$	$2.48\pm.74$	$.844\pm.006$
3.30	60	5	$2.484\pm.042$	$2.845\pm.066$	$2.38\pm.72$	$.797\pm.009$
3.96	120	3	$3.155\pm.051$	$3.583\pm.103$	$2.33\pm.73$	$.867\pm.007$
3.96	60	6	$2.515\pm.044$	$2.880\pm.066$	$2.48\pm.74$	$.804\pm.009$
3.96	40	9	$1.898\pm.031$	$1.905\pm.065$	$5.48\pm 1.35$	$.612\pm.013$
4.40	100	4	$3.494\pm.044$	$3.889\pm.087$	$2.02\pm.68$	$.912\pm.004$

Appendix

Table 12: Fixed-budget allocation between primitive count and precision. WER in percent.

Neural Audio Codecs

Neural audio codecs learn compact representations for compressing and reconstructing speech, music, and other sounds.

See all 85 papers

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding
Sep 10, 2026Luca Della Libera, Cem Subakan, Mirco RavanelliAudio Token Compression Neural Audio Codecs
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents
Sep 15, 2025Adhiraj Banerjee, Vipul AroraAudio Source Separation
Department of Electrical Engineering · Indian Institute of Technology, Kanpur
LILAC: An Idempotent Neural Speech Codec
Aug 6, 2026June Young Yi, Dongwook Lee, Jiheum Yeom +1Neural Audio Codecs
Department of Computer Science and Engineering, Seoul National University · 2Interdisciplinary Program in Artificial Intelligence, Seoul National University · Department of Electrical and Computer Engineering, Seoul National University +1

Sep 10, 2026J/K move · Enter open · S save

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Luca Della Libera, Cem Subakan, Mirco Ravanelli

Concordia University · Mila-Quebec AI Institute · Universit´e Laval

Audio Token Compression Neural Audio Codecs Speech Processing

Open paper AskPDF arXiv

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

Abstract

Neural Audio Codecs

Explore similar work

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents

LILAC: An Idempotent Neural Speech Codec