Neural audio codecs are increasingly used to encode speech for integration with large language models. This paper investigates the phonetic representations in the first codebook (CB0) of the Mimi codec. ABX style evaluation is poorly suited to Mimi fixed 80 ms tokenization window, which often spans multiple phonemes; we instead use forced alignment, entropy analysis, and Phone-Normalized Mutual Information (PNMI). Using TIMIT speech (58/55 speakers, SA1/SA2), individual tokens are strongly associated with specific phonemes (PNMI = 0.84, above random), yet phonemes are realized by multiple competing tokens rather than a one-to-one mapping.
Figures & tables
Figure 1: A schematic representation of the Mimi encoding-decoding pipeline
Figure 2: Distribution of Mimi token frames (80 ms each) in TIMIT SA1/SA2 across speakers. The modal counts (42 for SA1, 35 for SA2) are indicated by the tallest bar in each histogram.
Figure 3: Token variability in Mimi’s first codebook across TIMIT SA1 and SA2
Phon.
Obs.
Null
95%
p
G
0.693
0.166
0.214
.001
IY0
0.693
0.089
0.130
.001
IH0
0.693
0.213
0.282
.001
S
0.693
0.108
0.137
.001
R
0.689
0.057
0.077
.001
K
0.668
0.114
0.150
.001
Table 1: Permutation test results.
Model
Layer
PNMI
Baseline
HuBERT
KM500
0.43
—
EnCodec
RVQ-1
0.28
—
SpeechTokenizer
RVQ-1
0.71
—
Mimi (SA1)
CB0
0.84
0.22±0.004
Mimi (SA2)
CB0
0.84
0.29±0.004
Table 2: PNMI of discrete speech representations. Values for HuBERT, EnCodec, and SpeechTokenizer are from Zhang et al. (2024) , Table 8, computed on the full TIMIT training set. Mimi values are computed on the modal-length SA1/SA2 subsets; Mimi baseline: token shuffling, 100 iterations.
Codebook
SA1 PNMI
SA2 PNMI
CB0
0.8407
0.8394
CB1
0.6203
0.5566
CB2
0.6415
0.6613
CB3
0.6260
0.6583
CB4
0.6218
0.6648
CB5
0.6296
0.6541
Table 3: Phoneme Normalized Mutual Information (PNMI) measured independently for each Mimi codebook. CB0 exhibits the strongest phonological correspondence, while all remaining codebooks retain substantial phoneme-predictive information.
Condition
WER
Original audio
0.194
Mimi reconstruction
0.493
CB 0 single token
0.493
CB 0 all tokens
0.701
Table 4: WER under four token manipulation conditions (SI/SX, speaker FCJF0, n=8 ).
Phoneme
SA1 Entropy
SA2 Entropy
Distinct
(bits)
(bits)
Tokens
L
1.70
3.93
3–18
K
1.28
3.36
4–17
T
2.12
3.60
6–17
R
2.99
3.00
9–21
AE1
1.78
4.68
5–35
Table 5: Representative phoneme entropy values for the conditional distribution P(token∣phoneme) in SA1 and SA2. Low entropy indicates that a phoneme is realized by a small set of tokens; higher entropy reflects greater token variability.