RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

Chihiro Arata; Shohei Taniguchi; Gouki Minegishi; Kan Hatakeyama-Sato

cs.SDOct 21, 2026

RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

Authors: Chihiro Arata, Shohei Taniguchi, Gouki Minegishi, Kan Hatakeyama-Sato

Organizations: Third Intelligence, Inc.

Abstract

Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information. Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Its Adaptive Firing mechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor. Discrete codes and an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates. A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.

Figures & tables

System	Temporal allocation	Event-count control	Timing representation	Training procedure
TFC [ 11 ]	Resolution selection on waveform entropy	Exact	Hierarchical mask	Single-stage
CodecSlime [ 12 ]	DP segmentation on codec-feature distortion	Exact	Explicit lengths	Multi-stage (3 stages)
VARSTok [ 13 ]	Contiguous clustering on codec latents	Content-dependent	In token ID	Adaptation (pretrained WavTokenizer)
FlexiCodec [ 14 ]	Similarity merging on SenseVoice features	Content-dependent	Explicit lengths	Single-stage (frozen SenseVoice)
DTM-Codec [ 15 ]	Threshold crossing on accumulated codec-latent change	Content-dependent	Binary mask	Single-stage
DyCAST [ 22 ]	MMS character alignment or boundary prediction on WavLM features	Content-dependent	Explicit or predicted lengths	Multi-stage (4 stages; frozen WavLM)

Table 1: Temporal allocation, event-count control, timing representations, and training procedures in VFR speech codecs. Training procedures describe the recipes reported in the respective papers.

Figure 1: RubatoCodec architecture. The codec builds on the X-Codec-2 encoder and decoder backbone [ 7 ] and is shown at the default retention ratio of 0.25. Latent change and waveform spectral flux guide budgeted event allocation. The interval aggregator combines each event’s onset feature with a learned correction based on that feature, the mean of its frame features, and its duration. Quantized event representations and durations restore the 50-Hz decoding grid; the Inworld-TTS-inspired upsampler increases the synthesis feature rate [ 20 ] .

Stage	Temporal rate	Shape / specification
Waveform input	16 kHz	Mono waveform
Acoustic convolutional encoder	50 Hz	$T\times 1024$ ; strides $2,\allowbreak 2,\allowbreak 4,\allowbreak 4,\allowbreak 5$
Frozen w2v-BERT 2.0 [ 28 ] + semantic projection	50 Hz	Layer-16 features projected to $T\times 1024$
Concatenation and prior projection	50 Hz	$T\times 2048$
Adaptive segmentation and aggregation	Approximately 12.5 events/s	$K\times 2048$
Hybrid residual quantizer	Same event sequence	16 indices/event; 4,096 values/layer

Table 2: Main architecture in the default configuration. Tensor shapes omit the batch dimension.

Figure 2: Cumulative allocation for a synthetic firing signal. The example uses

T=20

R=0.25

, and

K=5

. (a) Firing signal. (b) Cumulative bounded increments cross thresholds 0.5, 1.5, 2.5, and 3.5 at frames 3, 7, 12, and 17. (c) Together with the fixed start at frame 0, these crossings define five events spanning 3, 4, 5, 5, and 3 grid frames, respectively. All positions are computed from the displayed signal using equations ( 8 – 9 ); no position correction is needed in this example.

Architecture	Target event rate	Retention ratio	Encoder hop at 16 kHz	Latent grid	Decoder upsampling factors	ISTFT FFT / hop
Adaptive	6.25 / 12.5 / 25 Hz	0.125 / 0.25 / 0.5	320	50 Hz	3	1,176 / 294
Fixed	6.25 Hz	—	2,560	6.25 Hz	4 $\times$ 3	2,352 / 588
Fixed	12.5 Hz	—	1,280	12.5 Hz	4 $\times$ 3	1,176 / 294
Fixed	25 Hz	—	640	25 Hz	3 $\times$ 2	1,176 / 294

Table 3: Temporal configurations of the main architecture comparisons. Encoder hop sizes, FFT sizes, and ISTFT hop sizes are given in samples. Each adaptive rate uses a separately trained model.

Dataset	Language	Utterances	Evaluation subset
LibriSpeech [ 42 ]	English	2,620	Full test-clean split
VCTK 0.92 [ 43 ]	English	1,308	Filtered mic1 recordings; 12 utterances per speaker
JVS [ 44 ]	Japanese	3,000	nonpara30: 30 utterances from each of 100 speakers
JVNV [ 45 ]	Japanese	1,615	Full corpus: four speakers, six emotions, including nonverbal vocalizations
seed-tts-eval [ 46 ]	Mandarin Chinese	2,020	test-zh recordings

Table 4: Speech reconstruction evaluation datasets.

Codec	Training data	Semantic features or supervision	Event rate / allocation	Output (kHz)	Causal enc. / dec.	Codec params. (M)	Extra enc. params. (M) a
EnCodec [ 2 ]	Speech, music, and general audio	None	75 Hz	24	Yes / Yes	14.9	—
DAC (24 kHz) [ 3 ]	Speech, music, and environmental audio, including VCTK	None	75 Hz	24	No / No	74.7	—
BigCodec [ 17 ]	LibriSpeech, 960 h	None	80 Hz	16	No / No	159.4	—
WavTokenizer [ 54 ]	Speech, music, and general audio	None	75 Hz	24	No / No	80.6	—
X-Codec-2 [ 7 ]	Emilia and Multilingual LibriSpeech, approximately 150k h	Frozen w2v-BERT 2.0 features	50 Hz	16	No / No	242.2	580
Mimi [ 6 ]	Moshi speech collection, 7M h b	WavLM distillation	12.5 Hz	24	Yes / Yes	79.3	—

Table 5: Specifications of the evaluated codecs.

Codec	Depth	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
Fixed-rate	8	1.20	3.365	3.293	2.322	0.915	0.84	0.776
Adaptive	8	1.23	3.448	3.314	2.379	0.917	0.80	0.757
Fixed-rate	16	2.40	3.363	3.288	2.442	0.923	0.72	0.810
Adaptive	16	2.43	3.444	3.307	2.556	0.929	0.62	0.807

Table 6: Adaptive versus fixed-rate coding at 12.5 Hz on LibriSpeech test-clean. Depth denotes the number of active quantizer layers. Bold marks the best score within each quantizer depth.

Figure 3: Adaptive and fixed-rate reconstruction at 12.5 Hz on LibriSpeech test-clean. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.

Event rate (Hz)	Codec	Depth	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
6.25	Fixed-rate	8	0.60	3.290	3.292	1.715	0.870	1.78	0.582
6.25	Adaptive	8	0.62	3.342	3.314	1.784	0.872	1.75	0.567
6.25	Fixed-rate	16	1.20	3.300	3.285	1.812	0.881	1.24	0.627
6.25	Adaptive	16	1.22	3.381	3.314	1.932	0.889	1.08	0.621
25	Fixed-rate	8	2.40	3.380	3.302	2.828	0.945	0.52	0.885
25	Adaptive	8	2.44	3.433	3.309	2.815	0.944	0.53	0.879

Table 7: Adaptive versus fixed-rate coding at 6.25 and 25 Hz on LibriSpeech test-clean. Each codec is trained separately at its target event rate using one training seed. Bold marks the best score within each event rate and quantizer depth.

Figure 4: Adaptive and fixed-rate reconstruction at 6.25 Hz on LibriSpeech test-clean. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.

Figure 5: Adaptive and fixed-rate reconstruction at 25 Hz on LibriSpeech test-clean. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.

Event placement	Depth	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
Uniform stride	8	1.21	3.442	3.321	2.655	0.932	0.70	0.849
Adaptive	8	1.23	3.451	3.331	2.681	0.933	0.76	0.859
Uniform stride	16	2.42	3.461	3.309	2.957	0.949	0.52	0.913
Adaptive	16	2.43	3.457	3.317	2.989	0.949	0.44	0.914

Table 8: Reconstruction with uniform-stride and adaptive event placement on LibriSpeech test-clean. Bold marks the best score within each quantizer depth. Uniform-stride bitrates slightly exceed 1.2 and 2.4 kbit/s because a final interval shorter than four frames still requires a complete set of code indices.

Figure 6: Uniform-stride and adaptive event placement across quantizer depths on LibriSpeech test-clean. Both models use a nominal event rate of 12.5 Hz. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.

Event rate (Hz)	Depth	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
6.25	16	1.22	3.381	3.314	1.932	0.889	1.08	0.621
12.5	8	1.23	3.448	3.314	2.379	0.917	0.80	0.757
25	4	1.24	3.387	3.324	2.387	0.921	0.84	0.771
12.5	16	2.43	3.444	3.307	2.556	0.929	0.62	0.807
25	8	2.44	3.433	3.309	2.815	0.944	0.53	0.879

Table 9: Adaptive coding at similar bitrates on LibriSpeech test-clean. Bold marks the best score within each bitrate group.

Codec / configuration	Allocation	Event rate (Hz)	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
GT	—	—	—	3.233	3.272	4.644	1.000	0.00	1.000
X-Codec-2	Fixed	50.0	0.80	3.233	3.299	2.434	0.919	1.23	0.821
FlexiCodec, $\tau$ =0.91, $D$ =8	Adaptive	8.3	0.84	3.174	3.307	2.463	0.919	1.04	0.777
WavTokenizer, 75 Hz	Fixed	75.0	0.90	2.772	3.342	2.382	0.912	3.16	0.682
DyCAST, CA	Adaptive	14.2	0.91	2.868	3.316	1.277	0.213	2.01	0.769
BigCodec	Fixed	80.0	1.04	3.362	3.333	2.681	0.935	1.30	0.844

Table 10: Reconstruction on LibriSpeech test-clean. Event rate is measured in coded events per second. Thick rules separate the bitrate bands described above. Bold and underline indicate the best and second-best scores in each of the two comparison bands, excluding GT; ties at displayed precision receive the same formatting.

D

denotes the number of active quantizer layers; DTM-Codec labels are nominal targets, and CA/BP denote DyCAST’s character-aligned/boundary-prediction settings.

Codec / configuration	Allocation	Event rate (Hz)	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
GT	—	—	—	3.566	3.202	4.644	1.000	0.00	1.000
FlexiCodec, $\tau$ =0.91, $D$ =8	Adaptive	7.0	0.71	3.354	3.191	2.384	0.846	1.13	0.738
DyCAST, CA	Adaptive	11.0	0.71	2.594	3.179	1.245	0.202	10.47	0.625
X-Codec-2	Fixed	50.0	0.80	3.443	3.190	2.430	0.858	1.48	0.758
DyCAST, BP minimum gap 1	Adaptive	13.6	0.88	2.805	3.226	1.324	0.242	8.24	0.692
WavTokenizer, 75 Hz	Fixed	75.0	0.90	3.198	3.208	2.397	0.853	4.45	0.662

Table 11: Reconstruction on VCTK. Allocation labels, bitrate bands, and rank formatting follow Table 10 . Configurations are grouped using their bitrate on this dataset.

Figure 7: English reconstruction quality versus estimated bitrate. Stars denote RubatoCodec at depths 8 and 16; circles denote adaptive allocation and triangles fixed allocation for the other codecs. Colors identify codec families. The bitrate and dWER axes are logarithmic. Operating points match Tables 10 – 11 .

Codec / configuration	Allocation	Event rate (Hz)	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dCER (%) $\downarrow$	SIM $\uparrow$
GT	—	—	—	3.621	3.334	4.644	0.999	0.00	1.000
X-Codec-2	Fixed	50.0	0.80	3.234	3.312	2.585	0.924	2.01	0.829
FlexiCodec, $\tau$ =0.91, $D$ =8	Adaptive	8.5	0.85	2.772	3.292	2.562	0.925	2.44	0.736
WavTokenizer, 75 Hz	Fixed	75.0	0.90	2.656	3.336	2.448	0.913	5.51	0.711
DTM-Codec, 80-Hz target	Adaptive	63.3	1.02	3.275	3.354	2.929	0.944	3.35	0.838
BigCodec	Fixed	80.0	1.04	3.244	3.348	2.816	0.938	2.47	0.803

Table 12: Reconstruction on JVS nonpara30. Event rates, bitrate grouping, and score highlighting follow Table 10 .

Codec / configuration	Allocation	Event rate (Hz)	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dCER (%) $\downarrow$	SIM $\uparrow$
GT	—	—	—	2.526	3.098	4.644	1.000	0.00	1.000
FlexiCodec, $\tau$ =0.91, $D$ =8	Adaptive	7.5	0.75	1.742	3.133	1.718	0.817	9.06	0.714
DTM-Codec, 80-Hz target	Adaptive	46.1	0.77	2.523	3.154	2.081	0.862	14.82	0.783
X-Codec-2	Fixed	50.0	0.80	2.369	3.148	1.944	0.858	6.15	0.807
WavTokenizer, 75 Hz	Fixed	75.0	0.90	1.769	3.150	1.846	0.842	12.48	0.672
DyCAST, BP minimum gap 1	Adaptive	15.3	0.98	1.797	3.071	1.178	0.156	11.67	0.677

Table 13: Reconstruction on JVNV. Event rates, bitrate grouping, and score highlighting follow Table 10 .

Codec / configuration	Allocation	Event rate (Hz)	Bitrate (kbit/s)	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dCER (%) $\downarrow$	SIM $\uparrow$
GT	—	—	—	3.003	3.203	4.644	1.000	0.00	1.000
X-Codec-2	Fixed	50.0	0.80	2.783	3.217	2.237	0.902	1.09	0.853
WavTokenizer, 75 Hz	Fixed	75.0	0.90	2.406	3.258	2.175	0.892	2.96	0.747
BigCodec	Fixed	80.0	1.04	2.850	3.237	2.264	0.909	1.55	0.800
Mimi, $D$ =8	Fixed	12.5	1.10	2.397	3.035	2.052	0.882	2.16	0.732
CodecSlime, FSQ18K, 80 Hz	Fixed	80.0	1.13	2.840	3.236	2.144	0.897	2.52	0.692

Table 14: Reconstruction on seed-tts-eval test-zh. Event rates, bitrate grouping, and score highlighting follow Table 10 .

Figure 8: Japanese and Mandarin reconstruction quality versus estimated bitrate. Marker and color conventions follow Figure 7 . The bitrate and dCER axes are logarithmic. Operating points match Tables 12 – 13 and 14 .

System / setting	LibriSpeech test-clean	VCTK	JVS nonpara30	JVNV	seed-tts-eval test-zh
RubatoCodec	12.5	12.5	12.5	12.5	12.5
CodecSlime, FSQ18K, 40 Hz	40.0	40.0	40.0	40.0	40.0
DTM-Codec, 80-Hz target	81.5	80.2	63.3	46.1	80.3
FlexiCodec, $\tau$ =0.91	8.3	7.0	8.5	7.5	7.2
DyCAST, character-aligned	14.2	11.0	4.4	3.9	3.7
DyCAST, boundary predictor, min-gap 1	17.4	13.6	18.0	15.3	18.1

Table 15: Mean event rates (Hz) across datasets at fixed encoding settings. Rates follow Section 5.1.2 . DTM-Codec’s 80-Hz label denotes a nominal target.

Figure 9: Event durations in reconstructed-speech evaluation. Left: event-weighted distribution across all five datasets, with durations above 300 ms grouped in the final bin. Right: the percentage of events longer than each duration, shown separately for each dataset; both axes are logarithmic.

Dataset	Median (ms)	95th percentile (ms)	99th percentile (ms)	Events >300 ms (%)
LibriSpeech test-clean	80	120	160	0.098
VCTK	80	140	180	0.004
JVS nonpara30	60	200	280	0.363
JVNV	60	180	360	1.932
seed-tts-eval test-zh	80	120	140	0.006

Table 16: Event-duration statistics by dataset. Percentages are computed over encoded events.

Figure 10: Adaptive event placement on English, Japanese, and Mandarin speech. The upper trace in each example is the input waveform, normalized by its peak amplitude; the lower strip partitions the same timeline into encoded events. Each rectangle begins at a firing position and has width proportional to its duration. Alternating colors distinguish neighboring events. Events lasting at least 300 ms are labeled in milliseconds and shaded on the waveform.

Codec / configuration	Encoding RTFx $\uparrow$	Decoding RTFx $\uparrow$
X-Codec-2 [ 7 ]	72.7	271.9
Mimi, $D$ =16 [ 6 ]	219.7	335.4
MOSS-Audio-Tokenizer, $D$ =20 [ 9 ]	161.9	199.2
Qwen3-TTS-Tokenizer-12Hz [ 8 ]	416.3	279.8
DTM-Codec, 80-Hz target [ 15 ]	471.7	518.6
CodecSlime, FSQ18K, 40 Hz [ 12 ]	25.7	190.4

Table 17: Full-utterance encoding and decoding throughput for selected codecs.

Figure 11: MOSS-TTS adapted to RubatoCodec, shown with eight codebooks. The temporal Transformer predicts each event from the conditioning prefix and the completed preceding events; the local Transformer then predicts continuation or stop, the codebook tokens, and finally the duration. Completed code–duration pairs can be passed to the causal RubatoCodec decoder for streaming synthesis.

Codec	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	UTMOSv2 $\uparrow$
Fixed-rate codec	3.20	0.392	4.080	3.239
Adaptive codec	3.10	0.392	4.162	3.295

Table 18: TTS results on the English seed-tts-eval benchmark. Both systems use eight codebooks at a nominal event rate of 12.5 Hz. Bold marks the better score; ties share the formatting.

Figure 12: Generated event rates on English seed-tts-eval. Each observation is one utterance. The histogram uses 0.05-events/s bins centered on multiples of 0.05, and the dashed line marks the codec’s nominal rate of 12.5 events/s. All 1,088 utterances are included.

Figure 13: Generated event durations on English seed-tts-eval. Bars show the percentage of events at each 20-ms duration increment through 300 ms. The final, hatched bin groups all durations above 300 ms. All events, including the long tail, contribute to the reported mean, median, and range.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Firing signal	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
Adjacent latent dissimilarity	1.262	0.787	3.30	0.397
Multi-scale latent dissimilarity	1.307	0.804	2.81	0.458
Waveform flux	1.327	0.810	2.20	0.447
Adjacent latent dissimilarity + flux	1.324	0.808	2.53	0.426
Multi-scale latent dissimilarity + flux	1.345	0.810	2.31	0.454
Learned difficulty	1.335	0.811	2.31	0.412

Appendix

Table A1: Firing-signal ablations with a single FSQ codebook. Bold marks the best score in each column.

Decoder	Depth	UTMOSv2 $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
Non-causal	8	3.465	2.478	0.923	0.76	0.792
Causal	8	3.448	2.379	0.917	0.80	0.757
Non-causal	16	3.471	2.651	0.933	0.60	0.836
Causal	16	3.444	2.556	0.929	0.62	0.807

Appendix

Table A2: Reconstruction quality with fully non-causal and fully causal decoders. Bold marks the best score within each quantizer depth.

Grid composition	Depth	UTMOSv2 $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
Repeated event vector	8	3.463	2.324	0.915	0.84	0.759
Sparse grid	8	3.448	2.379	0.917	0.80	0.757
Repeated event vector	16	3.475	2.507	0.926	0.65	0.809
Sparse grid	16	3.444	2.556	0.929	0.62	0.807

Appendix

Table A3: Sparse grid composition versus repeating the event representation across its interval. Bold marks the best score within each quantizer depth.

Firing rule	Depth	UTMOSv2 $\uparrow$	DNSMOS $\uparrow$	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
Level thresholding	8	3.299	3.311	1.657	0.859	5.62	0.613
Cumulative allocation	8	3.448	3.314	2.379	0.917	0.80	0.757
Level thresholding	16	3.319	3.306	1.751	0.871	3.33	0.649
Cumulative allocation	16	3.444	3.307	2.556	0.929	0.62	0.807

Appendix

Table A4: Firing-rule comparison with the residual quantizer and causal decoder. Bold marks the best score within each quantizer depth. dWER follows Section 5.1.2 .

Figure A1: Inference-time event-rate variation without retraining. All curves use the same model trained at 12.5 Hz, marked by the dashed vertical line. The horizontal axis shows the nominal inference event rate; colors and markers identify quantizer depth. The dWER axis is logarithmic.

Inference ratio	Depth	Event rate (Hz)	Bitrate (kbit/s)	PESQ-WB $\uparrow$	STOI $\uparrow$	dWER (%) $\downarrow$	SIM $\uparrow$
0.10	8	5.0	0.50	1.214	0.708	45.66	0.383
0.15	8	7.5	0.74	1.799	0.872	3.29	0.714
0.20	8	10.0	0.98	2.421	0.918	1.11	0.830
0.25 (trained)	8	12.5	1.23	2.684	0.931	0.75	0.871
0.30	8	15.0	1.47	2.811	0.938	0.58	0.891
0.35	8	17.5	1.71	2.895	0.942	0.58	0.903

Appendix

Table A5: Reconstruction at inference ratios different from the training ratio. The training ratio is 0.25. Event rates and bitrates are measured utterance averages; bitrate includes duration information. Depth-4 results are shown in Figure A1 .

Duration prediction	Local prediction order	Duration CE (nats) $\downarrow$	WER (%) $\downarrow$
Independent head	Parallel duration prediction; autoregressive codes	0.9698	4.71
Before all codes	$d_{i}\rightarrow c_{i,\allowbreak 0}\rightarrow\cdots\rightarrow c_{i,\allowbreak 7}$	0.9737	4.43
After the semantic code	$c_{i,\allowbreak 0}\rightarrow d_{i}\rightarrow c_{i,\allowbreak 1}\rightarrow\cdots\rightarrow c_{i,\allowbreak 7}$	0.8538	4.40
After all codes	$c_{i,\allowbreak 0}\rightarrow\cdots\rightarrow c_{i,\allowbreak 7}\rightarrow d_{i}$	0.4265	4.00

Appendix

Table B1: Effect of duration prediction structure and order. The codebook indices follow Equation ( 18 ). Lower values are better for both metrics; bold marks the best score in each column.

Setting	Adaptive-versus-fixed comparisons	Uniform-stride-versus-adaptive placement	Multilingual model
Training data	HiFiTTS-2, 44-kHz subset; approximately 5,000 h	HiFiTTS-2, full English 44-kHz training partition	Public and internal multilingual speech; approximately 375,000 h
Input / target sample rate	16 / 44.1 kHz	16 / 44.1 kHz	16 / 44.1 kHz
Crop duration	6 s	6 s	6 s
Training length	100,000 steps	1,000,000 steps	1,500,000 steps
Distributed training	2 GPUs; 32 examples/GPU; global batch 64; no gradient accumulation	8 GPUs; 8 examples/GPU; global batch 64; no gradient accumulation	8 GPUs; 20 examples/GPU; global batch 160; no gradient accumulation
Precision	bfloat16 mixed precision; residual quantization in float32	bfloat16 mixed precision; residual quantization in float32	bfloat16 mixed precision; residual quantization in float32

Appendix

Table C1: Training settings for the architecture and placement comparisons and multilingual model.

Setting	Configuration
Training data	HiFiTTS-2, 44-kHz English partition; approximately 31,300 h [ 34 ]
Codec	12.5-Hz adaptive and fixed-rate codecs (Section 4.2 )
Temporal backbone	Qwen3-0.6B [ 57 ] ; 28 layers; hidden size 1,024; 16 query / 8 key-value heads
Local Transformer	4 layers; hidden size 1,024; 16 attention heads
Codec representation	8 codebooks; 4,096 entries per codebook
Duration prediction	Adaptive codec: final local position, after all 8 codes; durations 1–64 frames on the 50-Hz grid. Fixed-rate codec: known frame interval, without a duration head

Appendix

Table C2: TTS training settings shared by the adaptive and fixed-rate comparison.

Resource	Role in this study	Source links
RubatoCodec	Proposed codec	Code Model
EnCodec [ 2 ]	Codec baseline	Code Model: 24 kHz
DAC [ 3 ]	Codec baseline	Code Model: 24 kHz
BigCodec [ 17 ]	Codec baseline	Code Model
WavTokenizer [ 54 ]	Codec baseline	Code Model: 75 Hz
X-Codec-2 [ 7 ]	Codec baseline	Code Model

Appendix

Table D1: Codec implementations and checkpoints.

Resource	Role in this study	Source links
HiFiTTS-2 [ 34 ]	Codec and TTS training / validation	Dataset
Emilia-YODAS [ 35 ]	Multilingual codec training	Dataset
ReazonSpeech [ 36 ]	Multilingual codec training	Dataset
The People’s Speech [ 37 ]	Multilingual codec training	Dataset
VoxPopuli [ 38 ]	Multilingual codec training	Dataset
Multilingual LibriSpeech [ 39 ]	Multilingual codec training	Dataset

Appendix

Table D2: Datasets.

Resource	Role in this study	Source links
w2v-BERT 2.0 [ 28 ]	Semantic feature extraction and supervision	Model
wav2vec2-XLS-R-300M [ 33 ]	Perceptual-loss feature extraction	Model
Qwen3-0.6B [ 57 ]	TTS temporal backbone	Model
Whisper-large-v3 [ 47 ]	English and Japanese ASR	Model
Paraformer-zh [ 48 ]	Mandarin ASR	Model
WavLM-Large speaker verification [ 53 ]	Speaker similarity evaluation	UniSpeech implementation SV checkpoint Evaluation instructions

Appendix

Table D3: Pretrained components and evaluation tools.

Neural Audio Codecs

Neural audio codecs learn compact representations for compressing and reconstructing speech, music, and other sounds.

See all 87 papers

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding
Sep 10, 2026Luca Della Libera, Cem Subakan, Mirco RavanelliAudio Token Compression Neural Audio Codecs
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
FOAdapter: A Plug-and-Play Spatial Tokenizer for Low-Bitrate FOA Speech Coding
Oct 12, 2026Kirak Kim, Yoonjeong Park, Juhan Nam +2Spatial Audio Neural Audio Codecs
Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea · Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, USA
CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
Jun 3, 2026Eugene Kwek, Feng Liu, Rui Zhang +1Audio Token Compression Audio Representation Learning
Pennsylvania State University · Drexel University

Sep 10, 2026J/K move · Enter open · S save

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

Luca Della Libera, Cem Subakan, Mirco Ravanelli

Concordia University · Mila-Quebec AI Institute · Universit´e Laval

Audio Token Compression Neural Audio Codecs Speech Processing

Open paper AskPDF arXiv

Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.

Abstract

Neural Audio Codecs

Explore similar work

ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding

FOAdapter: A Plug-and-Play Spatial Tokenizer for Low-Bitrate FOA Speech Coding

CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding