cs.CLAug 16, 2026
SaveUsing the Mimi codec for metalinguistic representations
Organizations: ALTAE, Université Paris Cité, F-75013 Paris, France · SCIAM, 10 rue de Penthièvre, F-75008 Paris, France
Abstract
In this paper, we focus on the dictionary of 2048 tokens used in Mimi semantic token codebook, the neural codec of the Moshi language model. We show that the ABX experiment carried out with Mimi fails to capture the mapping of the semantic tokens to phone realisations. By realigning Mimi representations to the TIMIT corpus transcriptions, we show that the 2048 tokens IDs of the semantic codebook map to quadphone, triphone, biphone, phone and subphone realisations.
Figures & tables
Figure 1: Levels of granularity in the Tyneside Linguistic Survey, from Jones-Sargent (1985)
| token | start (in ms) | end (in ms) | phone | word |
| 888 | 640 | 720 | epi|n|aw | now |
| 495 | 720 | 800 | aw | now |
| 1586 | 800 | 880 | aw|ix | now/ adjourned |
Table 1: Example of token to phone to word alignment
| (majority) TIMIT correspondence | n |
| allophones/subphones | 570 |
| diphones | 980 |
| triphones | 232 |
| quadphones | 11 |
Table 2: Distribution of TIMIT phone (sequences) matching Mimi semantic tokens
Figure 2: Jensen–Shannon divergence between token distributions for shared sentences (SA1 vs SA2) across phonemes. High divergence values indicate substantial variation in token assignments across different phonetic contexts.
Figure 3: Allophone network for the word water Cohen et al. (1987)
Figure 4: Comparison of the TIMIT phone transcriptions and Mimi semantic token representations of the allophone network for the pronunciations of the word greasy in the TIMIT training data
| Language Family | Number |
| Germanic | 85 |
| Romance | 223 |
| Slavic | 187 |
| Finno-Ugric | 313 |
| Baltic | 368 |
| Hellenic | 520 |
Table 4: Distribution of VoxPopuli languages by family.
Figure 5: Token distribution quality across language families and codebooks. Left: Shannon entropy (higher indicates more uniform distributions). Middle: number of missing tokens (lower indicates better coverage). Right: maximum token probability (lower indicates less dominance).
Figure 6: Token probability density across codebooks (CB0–CB7) and language families. Each subplot shows the distribution over token indices, highlighting differences in sparsity and dominance across codebooks.
More in this topic
Speech Processing
Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.
See all 412 papers