Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information. Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Its Adaptive Firing mechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor. Discrete codes and an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates. A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.
Figures & tables
System
Temporal allocation
Event-count control
Timing representation
Training procedure
TFC [ 11 ]
Resolution selection on waveform entropy
Exact
Hierarchical mask
Single-stage
CodecSlime [ 12 ]
DP segmentation on codec-feature distortion
Exact
Explicit lengths
Multi-stage (3 stages)
VARSTok [ 13 ]
Contiguous clustering on codec latents
Content-dependent
In token ID
Adaptation (pretrained WavTokenizer)
FlexiCodec [ 14 ]
Similarity merging on SenseVoice features
Content-dependent
Explicit lengths
Single-stage (frozen SenseVoice)
DTM-Codec [ 15 ]
Threshold crossing on accumulated codec-latent change
Content-dependent
Binary mask
Single-stage
DyCAST [ 22 ]
MMS character alignment or boundary prediction on WavLM features
Content-dependent
Explicit or predicted lengths
Multi-stage (4 stages; frozen WavLM)
Table 1: Temporal allocation, event-count control, timing representations, and training procedures in VFR speech codecs. Training procedures describe the recipes reported in the respective papers.
Figure 1: RubatoCodec architecture. The codec builds on the X-Codec-2 encoder and decoder backbone [ 7 ] and is shown at the default retention ratio of 0.25. Latent change and waveform spectral flux guide budgeted event allocation. The interval aggregator combines each event’s onset feature with a learned correction based on that feature, the mean of its frame features, and its duration. Quantized event representations and durations restore the 50-Hz decoding grid; the Inworld-TTS-inspired upsampler increases the synthesis feature rate [ 20 ] .
Stage
Temporal rate
Shape / specification
Waveform input
16 kHz
Mono waveform
Acoustic convolutional encoder
50 Hz
T×1024 ; strides 2,2,4,4,5
Frozen w2v-BERT 2.0 [ 28 ] + semantic projection
50 Hz
Layer-16 features projected to T×1024
Concatenation and prior projection
50 Hz
T×2048
Adaptive segmentation and aggregation
Approximately 12.5 events/s
K×2048
Hybrid residual quantizer
Same event sequence
16 indices/event; 4,096 values/layer
Table 2: Main architecture in the default configuration. Tensor shapes omit the batch dimension.
Figure 2: Cumulative allocation for a synthetic firing signal. The example uses T=20 , R=0.25 , and K=5 . (a) Firing signal. (b) Cumulative bounded increments cross thresholds 0.5, 1.5, 2.5, and 3.5 at frames 3, 7, 12, and 17. (c) Together with the fixed start at frame 0, these crossings define five events spanning 3, 4, 5, 5, and 3 grid frames, respectively. All positions are computed from the displayed signal using equations ( 8 – 9 ); no position correction is needed in this example.
Architecture
Target event rate
Retention ratio
Encoder hop at 16 kHz
Latent grid
Decoder upsampling factors
ISTFT FFT / hop
Adaptive
6.25 / 12.5 / 25 Hz
0.125 / 0.25 / 0.5
320
50 Hz
3
1,176 / 294
Fixed
6.25 Hz
—
2,560
6.25 Hz
4 × 3
2,352 / 588
Fixed
12.5 Hz
—
1,280
12.5 Hz
4 × 3
1,176 / 294
Fixed
25 Hz
—
640
25 Hz
3 × 2
1,176 / 294
Table 3: Temporal configurations of the main architecture comparisons. Encoder hop sizes, FFT sizes, and ISTFT hop sizes are given in samples. Each adaptive rate uses a separately trained model.
Dataset
Language
Utterances
Evaluation subset
LibriSpeech [ 42 ]
English
2,620
Full test-clean split
VCTK 0.92 [ 43 ]
English
1,308
Filtered mic1 recordings; 12 utterances per speaker
JVS [ 44 ]
Japanese
3,000
nonpara30: 30 utterances from each of 100 speakers
JVNV [ 45 ]
Japanese
1,615
Full corpus: four speakers, six emotions, including nonverbal vocalizations
Speech, music, and environmental audio, including VCTK
None
75 Hz
24
No / No
74.7
—
BigCodec [ 17 ]
LibriSpeech, 960 h
None
80 Hz
16
No / No
159.4
—
WavTokenizer [ 54 ]
Speech, music, and general audio
None
75 Hz
24
No / No
80.6
—
X-Codec-2 [ 7 ]
Emilia and Multilingual LibriSpeech, approximately 150k h
Frozen w2v-BERT 2.0 features
50 Hz
16
No / No
242.2
580
Mimi [ 6 ]
Moshi speech collection, 7M h b
WavLM distillation
12.5 Hz
24
Yes / Yes
79.3
—
Table 5: Specifications of the evaluated codecs.
Codec
Depth
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
Fixed-rate
8
1.20
3.365
3.293
2.322
0.915
0.84
0.776
Adaptive
8
1.23
3.448
3.314
2.379
0.917
0.80
0.757
Fixed-rate
16
2.40
3.363
3.288
2.442
0.923
0.72
0.810
Adaptive
16
2.43
3.444
3.307
2.556
0.929
0.62
0.807
Table 6: Adaptive versus fixed-rate coding at 12.5 Hz on LibriSpeech test-clean. Depth denotes the number of active quantizer layers. Bold marks the best score within each quantizer depth.
Figure 3: Adaptive and fixed-rate reconstruction at 12.5 Hz on LibriSpeech test-clean. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.
Event rate (Hz)
Codec
Depth
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
6.25
Fixed-rate
8
0.60
3.290
3.292
1.715
0.870
1.78
0.582
6.25
Adaptive
8
0.62
3.342
3.314
1.784
0.872
1.75
0.567
6.25
Fixed-rate
16
1.20
3.300
3.285
1.812
0.881
1.24
0.627
6.25
Adaptive
16
1.22
3.381
3.314
1.932
0.889
1.08
0.621
25
Fixed-rate
8
2.40
3.380
3.302
2.828
0.945
0.52
0.885
25
Adaptive
8
2.44
3.433
3.309
2.815
0.944
0.53
0.879
Table 7: Adaptive versus fixed-rate coding at 6.25 and 25 Hz on LibriSpeech test-clean. Each codec is trained separately at its target event rate using one training seed. Bold marks the best score within each event rate and quantizer depth.
Figure 4: Adaptive and fixed-rate reconstruction at 6.25 Hz on LibriSpeech test-clean. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.
Figure 5: Adaptive and fixed-rate reconstruction at 25 Hz on LibriSpeech test-clean. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.
Event placement
Depth
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
Uniform stride
8
1.21
3.442
3.321
2.655
0.932
0.70
0.849
Adaptive
8
1.23
3.451
3.331
2.681
0.933
0.76
0.859
Uniform stride
16
2.42
3.461
3.309
2.957
0.949
0.52
0.913
Adaptive
16
2.43
3.457
3.317
2.989
0.949
0.44
0.914
Table 8: Reconstruction with uniform-stride and adaptive event placement on LibriSpeech test-clean. Bold marks the best score within each quantizer depth. Uniform-stride bitrates slightly exceed 1.2 and 2.4 kbit/s because a final interval shorter than four frames still requires a complete set of code indices.
Figure 6: Uniform-stride and adaptive event placement across quantizer depths on LibriSpeech test-clean. Both models use a nominal event rate of 12.5 Hz. Each curve evaluates one trained model at quantizer depths 1, 2, 4, 8, and 16; the dWER axis is logarithmic.
Event rate (Hz)
Depth
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
6.25
16
1.22
3.381
3.314
1.932
0.889
1.08
0.621
12.5
8
1.23
3.448
3.314
2.379
0.917
0.80
0.757
25
4
1.24
3.387
3.324
2.387
0.921
0.84
0.771
12.5
16
2.43
3.444
3.307
2.556
0.929
0.62
0.807
25
8
2.44
3.433
3.309
2.815
0.944
0.53
0.879
Table 9: Adaptive coding at similar bitrates on LibriSpeech test-clean. Bold marks the best score within each bitrate group.
Codec / configuration
Allocation
Event rate (Hz)
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
GT
—
—
—
3.233
3.272
4.644
1.000
0.00
1.000
X-Codec-2
Fixed
50.0
0.80
3.233
3.299
2.434
0.919
1.23
0.821
FlexiCodec, τ =0.91, D =8
Adaptive
8.3
0.84
3.174
3.307
2.463
0.919
1.04
0.777
WavTokenizer, 75 Hz
Fixed
75.0
0.90
2.772
3.342
2.382
0.912
3.16
0.682
DyCAST, CA
Adaptive
14.2
0.91
2.868
3.316
1.277
0.213
2.01
0.769
BigCodec
Fixed
80.0
1.04
3.362
3.333
2.681
0.935
1.30
0.844
Table 10: Reconstruction on LibriSpeech test-clean. Event rate is measured in coded events per second. Thick rules separate the bitrate bands described above. Bold and underline indicate the best and second-best scores in each of the two comparison bands, excluding GT; ties at displayed precision receive the same formatting. D denotes the number of active quantizer layers; DTM-Codec labels are nominal targets, and CA/BP denote DyCAST’s character-aligned/boundary-prediction settings.
Codec / configuration
Allocation
Event rate (Hz)
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
GT
—
—
—
3.566
3.202
4.644
1.000
0.00
1.000
FlexiCodec, τ =0.91, D =8
Adaptive
7.0
0.71
3.354
3.191
2.384
0.846
1.13
0.738
DyCAST, CA
Adaptive
11.0
0.71
2.594
3.179
1.245
0.202
10.47
0.625
X-Codec-2
Fixed
50.0
0.80
3.443
3.190
2.430
0.858
1.48
0.758
DyCAST, BP minimum gap 1
Adaptive
13.6
0.88
2.805
3.226
1.324
0.242
8.24
0.692
WavTokenizer, 75 Hz
Fixed
75.0
0.90
3.198
3.208
2.397
0.853
4.45
0.662
Table 11: Reconstruction on VCTK. Allocation labels, bitrate bands, and rank formatting follow Table 10 . Configurations are grouped using their bitrate on this dataset.
Figure 7: English reconstruction quality versus estimated bitrate. Stars denote RubatoCodec at depths 8 and 16; circles denote adaptive allocation and triangles fixed allocation for the other codecs. Colors identify codec families. The bitrate and dWER axes are logarithmic. Operating points match Tables 10 – 11 .
Codec / configuration
Allocation
Event rate (Hz)
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dCER (%) ↓
SIM ↑
GT
—
—
—
3.621
3.334
4.644
0.999
0.00
1.000
X-Codec-2
Fixed
50.0
0.80
3.234
3.312
2.585
0.924
2.01
0.829
FlexiCodec, τ =0.91, D =8
Adaptive
8.5
0.85
2.772
3.292
2.562
0.925
2.44
0.736
WavTokenizer, 75 Hz
Fixed
75.0
0.90
2.656
3.336
2.448
0.913
5.51
0.711
DTM-Codec, 80-Hz target
Adaptive
63.3
1.02
3.275
3.354
2.929
0.944
3.35
0.838
BigCodec
Fixed
80.0
1.04
3.244
3.348
2.816
0.938
2.47
0.803
Table 12: Reconstruction on JVS nonpara30. Event rates, bitrate grouping, and score highlighting follow Table 10 .
Codec / configuration
Allocation
Event rate (Hz)
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dCER (%) ↓
SIM ↑
GT
—
—
—
2.526
3.098
4.644
1.000
0.00
1.000
FlexiCodec, τ =0.91, D =8
Adaptive
7.5
0.75
1.742
3.133
1.718
0.817
9.06
0.714
DTM-Codec, 80-Hz target
Adaptive
46.1
0.77
2.523
3.154
2.081
0.862
14.82
0.783
X-Codec-2
Fixed
50.0
0.80
2.369
3.148
1.944
0.858
6.15
0.807
WavTokenizer, 75 Hz
Fixed
75.0
0.90
1.769
3.150
1.846
0.842
12.48
0.672
DyCAST, BP minimum gap 1
Adaptive
15.3
0.98
1.797
3.071
1.178
0.156
11.67
0.677
Table 13: Reconstruction on JVNV. Event rates, bitrate grouping, and score highlighting follow Table 10 .
Codec / configuration
Allocation
Event rate (Hz)
Bitrate (kbit/s)
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dCER (%) ↓
SIM ↑
GT
—
—
—
3.003
3.203
4.644
1.000
0.00
1.000
X-Codec-2
Fixed
50.0
0.80
2.783
3.217
2.237
0.902
1.09
0.853
WavTokenizer, 75 Hz
Fixed
75.0
0.90
2.406
3.258
2.175
0.892
2.96
0.747
BigCodec
Fixed
80.0
1.04
2.850
3.237
2.264
0.909
1.55
0.800
Mimi, D =8
Fixed
12.5
1.10
2.397
3.035
2.052
0.882
2.16
0.732
CodecSlime, FSQ18K, 80 Hz
Fixed
80.0
1.13
2.840
3.236
2.144
0.897
2.52
0.692
Table 14: Reconstruction on seed-tts-eval test-zh. Event rates, bitrate grouping, and score highlighting follow Table 10 .
Figure 8: Japanese and Mandarin reconstruction quality versus estimated bitrate. Marker and color conventions follow Figure 7 . The bitrate and dCER axes are logarithmic. Operating points match Tables 12 – 13 and 14 .
System / setting
LibriSpeech test-clean
VCTK
JVS nonpara30
JVNV
seed-tts-eval test-zh
RubatoCodec
12.5
12.5
12.5
12.5
12.5
CodecSlime, FSQ18K, 40 Hz
40.0
40.0
40.0
40.0
40.0
DTM-Codec, 80-Hz target
81.5
80.2
63.3
46.1
80.3
FlexiCodec, τ =0.91
8.3
7.0
8.5
7.5
7.2
DyCAST, character-aligned
14.2
11.0
4.4
3.9
3.7
DyCAST, boundary predictor, min-gap 1
17.4
13.6
18.0
15.3
18.1
Table 15: Mean event rates (Hz) across datasets at fixed encoding settings. Rates follow Section 5.1.2 . DTM-Codec’s 80-Hz label denotes a nominal target.
Figure 9: Event durations in reconstructed-speech evaluation. Left: event-weighted distribution across all five datasets, with durations above 300 ms grouped in the final bin. Right: the percentage of events longer than each duration, shown separately for each dataset; both axes are logarithmic.
Dataset
Median (ms)
95th percentile (ms)
99th percentile (ms)
Events >300 ms (%)
LibriSpeech test-clean
80
120
160
0.098
VCTK
80
140
180
0.004
JVS nonpara30
60
200
280
0.363
JVNV
60
180
360
1.932
seed-tts-eval test-zh
80
120
140
0.006
Table 16: Event-duration statistics by dataset. Percentages are computed over encoded events.
Figure 10: Adaptive event placement on English, Japanese, and Mandarin speech. The upper trace in each example is the input waveform, normalized by its peak amplitude; the lower strip partitions the same timeline into encoded events. Each rectangle begins at a firing position and has width proportional to its duration. Alternating colors distinguish neighboring events. Events lasting at least 300 ms are labeled in milliseconds and shaded on the waveform.
Codec / configuration
Encoding RTFx ↑
Decoding RTFx ↑
X-Codec-2 [ 7 ]
72.7
271.9
Mimi, D =16 [ 6 ]
219.7
335.4
MOSS-Audio-Tokenizer, D =20 [ 9 ]
161.9
199.2
Qwen3-TTS-Tokenizer-12Hz [ 8 ]
416.3
279.8
DTM-Codec, 80-Hz target [ 15 ]
471.7
518.6
CodecSlime, FSQ18K, 40 Hz [ 12 ]
25.7
190.4
Table 17: Full-utterance encoding and decoding throughput for selected codecs.
Figure 11: MOSS-TTS adapted to RubatoCodec, shown with eight codebooks. The temporal Transformer predicts each event from the conditioning prefix and the completed preceding events; the local Transformer then predicts continuation or stop, the codebook tokens, and finally the duration. Completed code–duration pairs can be passed to the causal RubatoCodec decoder for streaming synthesis.
Codec
WER (%) ↓
SIM ↑
UTMOS ↑
UTMOSv2 ↑
Fixed-rate codec
3.20
0.392
4.080
3.239
Adaptive codec
3.10
0.392
4.162
3.295
Table 18: TTS results on the English seed-tts-eval benchmark. Both systems use eight codebooks at a nominal event rate of 12.5 Hz. Bold marks the better score; ties share the formatting.
Figure 12: Generated event rates on English seed-tts-eval. Each observation is one utterance. The histogram uses 0.05-events/s bins centered on multiples of 0.05, and the dashed line marks the codec’s nominal rate of 12.5 events/s. All 1,088 utterances are included.
Figure 13: Generated event durations on English seed-tts-eval. Bars show the percentage of events at each 20-ms duration increment through 300 ms. The final, hatched bin groups all durations above 300 ms. All events, including the long tail, contribute to the reported mean, median, and range.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Firing signal
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
Adjacent latent dissimilarity
1.262
0.787
3.30
0.397
Multi-scale latent dissimilarity
1.307
0.804
2.81
0.458
Waveform flux
1.327
0.810
2.20
0.447
Adjacent latent dissimilarity + flux
1.324
0.808
2.53
0.426
Multi-scale latent dissimilarity + flux
1.345
0.810
2.31
0.454
Learned difficulty
1.335
0.811
2.31
0.412
Appendix
Table A1: Firing-signal ablations with a single FSQ codebook. Bold marks the best score in each column.
Decoder
Depth
UTMOSv2 ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
Non-causal
8
3.465
2.478
0.923
0.76
0.792
Causal
8
3.448
2.379
0.917
0.80
0.757
Non-causal
16
3.471
2.651
0.933
0.60
0.836
Causal
16
3.444
2.556
0.929
0.62
0.807
Appendix
Table A2: Reconstruction quality with fully non-causal and fully causal decoders. Bold marks the best score within each quantizer depth.
Grid composition
Depth
UTMOSv2 ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
Repeated event vector
8
3.463
2.324
0.915
0.84
0.759
Sparse grid
8
3.448
2.379
0.917
0.80
0.757
Repeated event vector
16
3.475
2.507
0.926
0.65
0.809
Sparse grid
16
3.444
2.556
0.929
0.62
0.807
Appendix
Table A3: Sparse grid composition versus repeating the event representation across its interval. Bold marks the best score within each quantizer depth.
Firing rule
Depth
UTMOSv2 ↑
DNSMOS ↑
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
Level thresholding
8
3.299
3.311
1.657
0.859
5.62
0.613
Cumulative allocation
8
3.448
3.314
2.379
0.917
0.80
0.757
Level thresholding
16
3.319
3.306
1.751
0.871
3.33
0.649
Cumulative allocation
16
3.444
3.307
2.556
0.929
0.62
0.807
Appendix
Table A4: Firing-rule comparison with the residual quantizer and causal decoder. Bold marks the best score within each quantizer depth. dWER follows Section 5.1.2 .
Figure A1: Inference-time event-rate variation without retraining. All curves use the same model trained at 12.5 Hz, marked by the dashed vertical line. The horizontal axis shows the nominal inference event rate; colors and markers identify quantizer depth. The dWER axis is logarithmic.
Inference ratio
Depth
Event rate (Hz)
Bitrate (kbit/s)
PESQ-WB ↑
STOI ↑
dWER (%) ↓
SIM ↑
0.10
8
5.0
0.50
1.214
0.708
45.66
0.383
0.15
8
7.5
0.74
1.799
0.872
3.29
0.714
0.20
8
10.0
0.98
2.421
0.918
1.11
0.830
0.25 (trained)
8
12.5
1.23
2.684
0.931
0.75
0.871
0.30
8
15.0
1.47
2.811
0.938
0.58
0.891
0.35
8
17.5
1.71
2.895
0.942
0.58
0.903
Appendix
Table A5: Reconstruction at inference ratios different from the training ratio. The training ratio is 0.25. Event rates and bitrates are measured utterance averages; bitrate includes duration information. Depth-4 results are shown in Figure A1 .
Table B1: Effect of duration prediction structure and order. The codebook indices follow Equation ( 18 ). Lower values are better for both metrics; bold marks the best score in each column.
Setting
Adaptive-versus-fixed comparisons
Uniform-stride-versus-adaptive placement
Multilingual model
Training data
HiFiTTS-2, 44-kHz subset; approximately 5,000 h
HiFiTTS-2, full English 44-kHz training partition
Public and internal multilingual speech; approximately 375,000 h
Input / target sample rate
16 / 44.1 kHz
16 / 44.1 kHz
16 / 44.1 kHz
Crop duration
6 s
6 s
6 s
Training length
100,000 steps
1,000,000 steps
1,500,000 steps
Distributed training
2 GPUs; 32 examples/GPU; global batch 64; no gradient accumulation
8 GPUs; 8 examples/GPU; global batch 64; no gradient accumulation
8 GPUs; 20 examples/GPU; global batch 160; no gradient accumulation
Precision
bfloat16 mixed precision; residual quantization in float32
bfloat16 mixed precision; residual quantization in float32
bfloat16 mixed precision; residual quantization in float32
Appendix
Table C1: Training settings for the architecture and placement comparisons and multilingual model.
Setting
Configuration
Training data
HiFiTTS-2, 44-kHz English partition; approximately 31,300 h [ 34 ]
Codec
12.5-Hz adaptive and fixed-rate codecs (Section 4.2 )
Adaptive codec: final local position, after all 8 codes; durations 1–64 frames on the 50-Hz grid. Fixed-rate codec: known frame interval, without a duration head
Appendix
Table C2: TTS training settings shared by the adaptive and fixed-rate comparison.
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Luca Della Libera, Cem Subakan, Mirco Ravanelli
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
Spatially encoded speech conveys not only linguistic information but also spatial cues. This enables listeners to distinguish and localize speakers, contributing to more natural and realistic auditory experiences in applications such as remote conferencing and immersive media. Neural speech codecs (NSCs) enable high-quality reconstruction of monaural speech at increasingly low bitrates while preserving linguistic information, but their ability to effectively preserve spatial cues remains limited. We propose FOAdapter, a plug-and-play spatial tokenizer that leverages a pretrained monaural speech codec to efficiently represent both linguistic and spatial information in first-order ambisonic (FOA) speech at low bitrates. Comprehensive evaluation demonstrates that FOAdapter consistently preserves spatial information across all evaluated codecs. Source code and audio examples are available on our demo website.
Kirak Kim, Yoonjeong Park, Juhan Nam +2
Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea · Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, USA
Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content. We reframe audio tokenization as a selective information bottleneck problem and propose CleanCodec, a denoising audio codec which learns to encode only perceptually important features and discard imperceptible information. At just 12.5 tokens per second, CleanCodec achieves state-of-the-art tokenization efficiency, substantially outperforming existing codecs in speaker similarity and speech intelligibility. Evaluations on downstream text-to-speech and voice conversion tasks further demonstrate improved performance and up to 17x faster inference, highlighting significant efficiency gains.