Variable-Length Semantic IDs for Recommender Systems

Kirill Khrylchenko

cs.IRFeb 18, 2026

Variable-Length Semantic IDs for Recommender Systems

Authors: Kirill Khrylchenko

Organizations: HSE University, Moscow, Russia

Abstract

Generative models are increasingly used in recommender systems, both for modeling user behavior as event sequences and for integrating large language models into recommendation pipelines. A key challenge in this setting is the extremely large cardinality of item spaces, which makes training generative models difficult and introduces a vocabulary gap between natural language and item identifiers. Semantic identifiers (semantic IDs), which represent items as sequences of low-cardinality tokens, have recently emerged as an effective solution to this problem. However, existing approaches generate semantic identifiers of fixed length, assigning the same description length to all items. This is inefficient, misaligned with natural language, and ignores the highly skewed frequency structure of real-world catalogs, where popular items and rare long-tail items exhibit fundamentally different information requirements. In parallel, the emergent communication literature studies how agents develop discrete communication protocols, often producing variable-length messages in which frequent concepts receive shorter descriptions. Despite the conceptual similarity, these ideas have not been systematically adopted in recommender systems. In this work, we bridge recommender systems and emergent communication by introducing variable-length semantic identifiers for recommendation. We propose a discrete variational autoencoder with Gumbel-Softmax reparameterization that learns item representations of adaptive length under a principled probabilistic framework, avoiding the instability of REINFORCE-based training and the fixed-length constraints of prior semantic ID methods.

Figures & tables

Dataset	#Items	#Events	Emb. dim	Head share
Yambda	268K	82M	128	0.747
VK-LSVD	900K	95M	64	0.276
Amazon T&G	305K	12M	128	0.597

Table 1. Dataset statistics after preprocessing. Head share denotes the fraction of interactions attributed to the top-30k most popular items.

Method	$\mathcal{L}_{\text{recon}}$	$\mathcal{L}_{\text{recon}}^{\text{cold}}$	$\mathbb{E}_{p_{\text{data}}}[L]$	PPL	$\mathcal{L}_{\text{recon}}$	$\mathcal{L}_{\text{recon}}^{\text{cold}}$	$\mathbb{E}_{p_{\text{data}}}[L]$	PPL	$\mathcal{L}_{\text{recon}}$	$\mathcal{L}_{\text{recon}}^{\text{cold}}$	$\mathbb{E}_{p_{\text{data}}}[L]$	PPL
	Yambda				VK-LSVD				Amazon Toys&Games
R-KMeans	0.038	0.063	5	3721	0.102	0.149	5	4010	0.094	0.138	5	3288
dVAE (fixed-length)	0.025	0.054	5	3557	0.123	0.147	5	3959	0.076	0.170	5	3797
dVAE (varlen, $\lambda{=}\lambda_{1}$ )	0.025	0.057	3.62	3574	0.146	0.160	4.06	3913	0.068	0.167	4.63	3800
dVAE (varlen, $\lambda{=}\lambda_{2}$ )	0.026	0.061	2.96	3609	0.155	0.176	3.45	3879	0.071	0.167	3.01	3623
dVAE (varlen, $\lambda{=}\lambda_{3}$ )	0.029	0.076	2.29	3564	0.173	0.193	2.94	3839	0.076	0.182	2.38	3339

Table 2. Semantic ID reconstruction quality and efficiency.

\mathbb{E}_{p_{\text{data}}}[L]

denotes average semantic ID length under

p_{\text{data}}

distribution.

Semantic ID length	Mean Pop.	Max Pop.	# Items	Mean Pop.	Max Pop.	# Items	Mean Pop.	Max Pop.	# Items
	Yambda			VK-LSVD			Amazon Toys&Games
1	2248.9	82097	7562	165.2	12319	18572	392.0	18275	5020
2	1154.1	23034	12485	151.3	6222	59305	99.8	2705	31384
3	292.2	10082	28421	115.0	3600	94966	39.2	724	31412
4	214.1	10037	80505	104.8	3079	159327	28.7	632	54192
5	125.4	7027	112276	89.4	2695	477542	18.6	575	152429

Table 3. Item popularity statistics by semantic ID length on the training data

	Train		Cold
Dataset	$\mathbb{E}_{p_{\text{catalog}}}[L]$	$\mathbb{E}_{p_{\text{data}}}[L]$	$\mathbb{E}_{p_{\text{catalog}}}[L]$	$\mathbb{E}_{p_{\text{data}}}[L]$
Yambda	4.15	2.96	4.53	4.40
VK-LSVD	3.66	3.45	4.16	3.94
Amazon T&G	4.16	3.01	4.49	4.49

Table 4. Average semantic ID length under catalog and data distributions

Method	Yambda					VK-LSVD
	Recall@100	Tokens	Events	$L_{\text{cand}}$	Cov@100	Recall@100	Tokens	Events	$L_{\text{cand}}$	Cov@100
R-KMeans	0.1637 (+0.0%)	364.8	61.49	6.00	18.2 (+0.0%)	0.0363 (+0.0%)	326.7	55.87	6.00	5.1 (+0.0%)
dVAE (fixed-length)	0.1645 (+0.5%)	364.8	61.49	6.00	20.8 (+14.3%)	0.0410 (+12.9%)	326.7	55.87	6.00	6.6 (+29.4%)
dVAE (varlen, $\lambda{=}1$ )	0.1676 (+2.4%)	331.2	71.72	4.47	20.9 (+14.8%)	0.0415 (+14.3%)	308.5	66.60	4.25	8.2 (+60.8%)
dVAE (varlen, $\lambda{=}2$ )	0.1794 (+9.6%)	287.7	88.93	3.05	25.1 (+37.9%)	0.0418 (+15.2%)	302.6	70.49	3.96	9.0 (+76.5%)
dVAE (varlen, $\lambda{=}3$ )	0.1820 (+11.2%)	277.8	93.13	2.86	25.8 (+41.8%)	0.0459 (+26.5%)	291.7	78.24	3.50	10.3 (+102.0%)

Table 5. Sequential recommendation performance under a fixed token budget. Tokens and Events denote the number of historical tokens and interaction events for the test users.

L_{\text{cand}}

denotes the average length of semantic IDs generated for candidate items during evaluation

Method	$\mathcal{L}_{\text{recon}}$	$\mathcal{L}_{\text{recon}}^{\text{cold}}$	$\mathbb{E}_{p_{\text{data}}}[L]$	PPL
dVAE (fixed-length)	0.0246	0.0536	5.00	3558
dVAE (varlen)	0.0294	0.0763	2.29	3564
REINFORCE (fixed-length)	0.0259	0.0607	5.00	2720
REINFORCE (varlen)	0.0528	0.1499	2.65	316

Table 6. Comparison of dVAE and REINFORCE-based training on Yambda

$\boldsymbol{T}$	$\boldsymbol{\|\mathcal{V}\|}$	$\boldsymbol{\mathcal{L}_{\text{recon}}}$	$\boldsymbol{\mathcal{L}_{\text{recon}}^{\text{cold}}}$	$\boldsymbol{\mathbb{E}_{p_{\text{data}}}[L]}$	PPL
5	4096	0.0254	0.0567	3.62	3575
20	4096	0.0177	0.0300	8.33	2733
5	32768	0.0150	0.0544	2.79	21350

Table 7. Scaling behavior of variable-length dVAE on Yambda.

T

denotes the maximum semantic ID length, and

|\mathcal{V}|

denotes the vocabulary size.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$\mathcal{L}_{\text{recon}}$ @1	$\mathcal{L}_{\text{recon}}$ @2	$\mathcal{L}_{\text{recon}}$ @3	$\mathcal{L}_{\text{recon}}$ @4	$\mathcal{L}_{\text{recon}}$ @5
R-KMeans	0.1644	0.0959	0.0665	0.0494	0.0384
dVAE (fixed-length)	0.1400	0.0678	0.0512	0.0355	0.0246
dVAE (varlen)	0.1344	0.0606	0.0404	0.0313	0.0270

Appendix

Table 8. Reconstruction loss from semantic ID prefixes on Yambda

Method	PPL@1	PPL@2	PPL@3	PPL@4	PPL@5
R-KMeans	3895	3674	3489	3414	3393
dVAE (fixed-length)	1959	3366	3315	3284	3135
dVAE (varlen)	2168	3383	3215	3167	3175
REINFORCE (fixed-length)	1340	2146	2689	3035	3232

Appendix

Table 9. Position-wise token perplexity on Yambda (higher means more diverse token usage)

Figure 1. Position-wise token perplexity for varlen dVAE with

\texttt{maxlen}=20

on Yambda

Semantic ID Recommendation

Semantic identifier recommendation uses discrete, content-aware item codes to retrieve and rank items, especially in generative systems.

See all 38 papers

Learning Variable-Length Tokenization for Generative Recommendation
May 18, 2026Minhao Wang, Bowen Wu, Wei ZhangResidual VQ Generative Recommendation
East China Normal University · Shanghai, China · & Shanghai Innovation Institute
VaLiDRec: Variable-Length LLM-Aligned Semantic IDs for Generative Recommendation
Jul 28, 2026Shutong Qiao, Wei Yuan, Tong Chen +3Semantic ID Recommendation Graph Prompt Learning
University of Queensland Brisbane, Australia · Computer Network Information Center, Chinese Academy of Sciences Beijing, China · Griffith University Gold Coast, Australia
JevRec: Linguistic Encoding with Jev-like Decision Modeling for LLM-based Generative Recommendation
Oct 27, 2026Qiyong Zhong, Jiajie Su, Ming Yang +1Generative Recommendation Large Language Model-Based Recommendation
Zhejiang University, Hangzhou, China

May 18, 2026J/K move · Enter open · S save

Learning Variable-Length Tokenization for Generative Recommendation

Minhao Wang, Bowen Wu, Wei Zhang

East China Normal University · Shanghai, China · & Shanghai Innovation Institute

Residual VQ Generative Recommendation

Open paper AskPDF arXiv

Generative recommendation reformulates recommendation as next-token prediction over discrete semantic identifiers (IDs). A fundamental yet unexplored design choice is that existing methods employ fixed-length tokenization for all items, implicitly assuming uniform encoding capacity regardless of item characteristics. Through systematic experiments across four datasets, we discover the Popularity-Length Paradox: popular items achieve optimal performance with short IDs, while tail items require substantially longer codes to capture discriminative semantics. This reveals a critical mismatch where popular items benefit from abundant collaborative signals and require minimal semantic detail, whereas tail items must rely on fine-grained content features due to sparse interaction data. To address this, we propose VarLenRec, a framework for learning variable-length tokenization. We develop Popularity-Weighted Information Budget Allocation (PIBA), an information-theoretic framework proving that optimal ID length should scale as a negative power of popularity. Directly implementing variable-length allocation faces two technical challenges: standard Euclidean residual quantization lacks geometric capacity to support diverse code lengths without distortion, and discrete length decisions are non-differentiable. We address these through Hyperbolic Residual Quantization, which leverages the exponential volume growth of the Poincaré ball to naturally stratify encoding capacity, and a Soft Length Controller, which enables differentiable length prediction via continuous layer retention probabilities regularized by PIBA-derived priors. Extensive experiments demonstrate that VarLenRec achieves significant improvements over state-of-the-art methods in recommendation accuracy and training/inference efficiency, revealing the importance of adaptive encoding capacity in generative recommendation.

Abstract

Semantic ID Recommendation

Explore similar work

Learning Variable-Length Tokenization for Generative Recommendation

VaLiDRec: Variable-Length LLM-Aligned Semantic IDs for Generative Recommendation

JevRec: Linguistic Encoding with Jev-like Decision Modeling for LLM-based Generative Recommendation