Knowledge Distillation across Item and Code Spaces for Parallel Generative Recommendation

Yixuan Xu; Hailin Liu; Linxiang Peng; Can Yang

cs.IROct 23, 2026

Knowledge Distillation across Item and Code Spaces for Parallel Generative Recommendation

Authors: Yixuan Xu, Hailin Liu, Linxiang Peng, Can Yang

Organizations: The Hong Kong University of Science and Technology, Hong Kong, China · University of Science and Technology of China, Hefei, China

Abstract

Generative recommendation with semantic IDs represents each item as a sequence of discrete codes derived from its semantic representation and predicts these codes to generate recommendations. Existing semantic-ID recommenders are typically trained with hard labels for the observed next item. Although such supervision identifies the target, it provides no explicit distributional information about alternative items conditioned on the same user history. Sequential recommenders can supply this item-level predictive knowledge, but transferring it to semantic-ID generation is challenging because the teacher and student operate in item and code spaces, respectively. We propose two complementary knowledge-distillation objectives for parallel semantic-ID recommenders, which predict all code positions simultaneously without autoregressive prefix conditioning. Item Distribution Matching (IDM) reconstructs a catalogue-level item distribution from the student's parallel code predictions and aligns it with the teacher's item distribution. Code Marginal Matching (CMM) projects the teacher's item probabilities onto each code position using fixed item-to-code assignments, then matches the resulting code marginals to the corresponding student heads. Both objectives transfer the teacher's soft distributional knowledge while preserving the student's parallel prediction structure. Experiments on benchmark datasets show that both methods consistently outperform existing state-of-the-art approaches. On average, IDM improves Recall@10 by 15.39%, while CMM improves NDCG@10 by 8.28%. These results demonstrate that distillation in either reconstructed item space or code-marginal space can substantially strengthen hard-label supervision for parallel generative recommendation.

Figures & tables

Figure 1. Overview of item- and code-space knowledge distillation for parallel semantic-ID recommendation. (a) A parallel SID student learns from hard labels and a frozen teacher using either IDM or CMM. (b) Fixed assignments

M_{h}

connect teacher item probabilities with student code predictions. IDM matches distributions over valid catalogue items; CMM matches teacher-derived code marginals with student head distributions. (c) At inference, the student predicts SID codes in parallel and ranks valid items without the teacher. Three panels show the training, cross-space distillation, and inference. A frozen teacher produces item probabilities and a trainable parallel SID student produces code logits. Fixed item-to-code assignments gather student logits into catalogue scores for IDM and aggregate teacher probabilities into code marginals for CMM. Each objective is trained separately with hard-label supervision. At inference, the student produces catalogue scores and selects the top-ranked items.

Statistic	Beauty	Toys	Sports
Users	22,363	19,412	35,598
Items	12,101	11,924	18,357
Interactions	198,502	167,597	296,337
Training targets	131,413	109,361	189,543

Table 1. Dataset statistics. Each user contributes one Validation target and one Test target.

Item-ID-based
Method	Beauty				Toys				Sports
	R@5	R@10	N@5	N@10	R@5	R@10	N@5	N@10	R@5	R@10	N@5	N@10
Caser	0.0205	0.0347	0.0131	0.0176	0.0166	0.0270	0.0107	0.0141	0.0116	0.0194	0.0072	0.0097
GRU4Rec	0.0164	0.0283	0.0099	0.0137	0.0097	0.0176	0.0059	0.0084	0.0129	0.0204	0.0086	0.0110
HGN	0.0325	0.0512	0.0206	0.0266	0.0321	0.0497	0.0221	0.0277	0.0189	0.0313	0.0120	0.0159
BERT4Rec	0.0203	0.0347	0.0124	0.0170	0.0116	0.0203	0.0071	0.0099	0.0115	0.0191	0.0075	0.0099

Table 2. Overall Test results. R and N denote Recall and NDCG. Results for RPG, AsymRec, IDM, and CMM are averaged over three runs; all other baseline results are taken from RPG ( Hou et al., 2025a ) , Table 2. Bold and underline mark the best and second-best values in each metric column. Gain is the relative improvement of the best IDM/CMM mean over hard-label AsymRec. Significance testing:

*

indicates statistically significant improvements of IDM/CMM over AsymRec via a two-sided paired

t

-test over users (

p<0.05

), while

-

denotes no significant difference.

Variant	Beauty		Toys		Sports
	R@10	N@10	R@10	N@10	R@10	N@10
IDM (full)	$\mathbf{0.1094}$	$\mathbf{0.0590}$	$\mathbf{0.1176}$	$\mathbf{0.0638}$	$\mathbf{0.0644}$	$\mathbf{0.0333}$
Top-1 item	$0.1041$	$0.0548$	$0.1141$	$0.0608$	$0.0606$	$0.0294$
Uniform-smoothed item	$0.1038$	$0.0548$	$0.1144$	$0.0612$	$0.0602$	$0.0293$
CMM (full)	$\mathbf{0.1069}$	$\mathbf{0.0592}$	$\mathbf{0.1144}$	$\mathbf{0.0647}$	$\mathbf{0.0639}$	$0.0341$
Hard code	$0.1010$	$0.0545$	$0.1119$	$0.0616$	$0.0595$	$0.0292$

Table 3. Ablation results (mean performance over three random seeds). Variants change only the auxiliary targets or supervised code heads; hard-label supervision covers all 48 heads. Bold denotes the highest mean within each objective group and metric.

System	Data	Beauty		Toys		Sports
		R@10	N@10	R@10	N@10	R@10	N@10
IDM	25%	$0.0934$	$0.0479$	$0.1065$	$0.0549$	$0.0577$	$0.0284$
	50%	$0.1037$	$0.0534$	$0.1097$	$0.0572$	$0.0610$	$0.0302$
	100%	$\mathbf{0.1094}$	$\mathbf{0.0590}$	$\mathbf{0.1176}$	$\mathbf{0.0638}$	$\mathbf{0.0644}$	$\mathbf{0.0333}$
CMM	25%	$0.0962$	$0.0511$	$0.1076$	$0.0589$	$0.0585$	$0.0297$
	50%	$0.1025$	$0.0547$	$0.1100$	$0.0607$	$0.0618$	$0.0321$

Table 4. Effect of teacher training data (mean performance over three random seeds). Bold marks the highest performance across data fractions within each system and metric. Learning-rate-matched hard-label baselines (

r=0

) have R@10 and N@10 of 0.0940 and 0.0547 (Beauty, IDM), 0.0912 and 0.0537 (Beauty, CMM), 0.1000 and 0.0583 (Toys), and 0.0573 and 0.0322 (Sports).

System	$\tau_{T}$	Beauty		Toys		Sports
		R@10	N@10	R@10	N@10	R@10	N@10
IDM	0.5	$0.1077$	$0.0559$	$0.1171$	$0.0620$	$0.0633$	$0.0299$
	1	$\mathbf{0.1094}$	$\mathbf{0.0590}$	$\mathbf{0.1176}$	$\mathbf{0.0638}$	$\mathbf{0.0644}$	$\mathbf{0.0333}$
	2	$0.0980$	$0.0570$	$0.1060$	$0.0619$	$0.0583$	$0.0326$
	4	$0.0954$	$0.0556$	$0.1017$	$0.0590$	$0.0576$	$0.0324$
CMM	0.5	$0.1027$	$0.0551$	$0.1130$	$0.0618$	$0.0602$	$0.0294$

Table 5. Teacher-temperature sensitivity (mean performance over three random seeds). Bold marks the best temperature per objective and metric.

Figure 2. Distillation-strength sensitivity. Bands denote

\pm

one sample standard deviation over three runs, and rings indicate the validation-selected configurations. The

r=0

controls match the learning rate of each curve and coincide on Toys and Sports. Rows correspond to Beauty, Toys, and Sports; columns show Recall at 10 and NDCG at 10 across eight strengths. Recall benefits persist over positive strengths, while NDCG can decline at large strengths, particularly on Sports.

Beauty
System	R@5	R@10	N@5	N@10
IDM	$0.0719$	$\mathbf{0.1094}$	$0.0468$	$0.0590$
CMM	$0.0714$	$0.1069$	$0.0478$	$0.0592$
Joint	$0.0711$	$0.1069$	$0.0470$	$0.0586$
Fusion	$\mathbf{0.0728}$	$0.1093$	$\mathbf{0.0479}$	$\mathbf{0.0597}$
Toys

Table 6. Standalone IDM/CMM, joint training, and same-seed score fusion (mean performance over three random seeds). Bold denotes the highest unrounded mean for each dataset and metric.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	LR	$r_{m}^{\star}$	$\lambda_{m}^{\star}$	Ramp steps
Beauty	AsymRec	0.005	—	—	—
Beauty	IDM	0.005	0.8	1.716	1048
Beauty	CMM	0.02	0.6	0.802	1048
Toys	AsymRec	0.02	—	—	—
Toys	IDM	0.01	0.6	1.530	903
Toys	CMM	0.01	0.4	0.493	903

Appendix

Table 7. Selected single-model hyperparameters. All three runs use the same dataset–objective coefficient.

Variant	Beauty				Toys				Sports
	R@5	R@10	N@5	N@10	R@5	R@10	N@5	N@10	R@5	R@10	N@5	N@10
IDM (full)	$0.0719$	$0.1094$	$0.0468$	$0.0590$	$0.0804$	$0.1176$	$0.0518$	$0.0638$	$0.0423$	$0.0644$	$0.0262$	$0.0333$
Top-1 item	$0.0687$	$0.1041$	$0.0434$	$0.0548$	$0.0773$	$0.1141$	$0.0489$	$0.0608$	$0.0387$	$0.0606$	$0.0223$	$0.0294$
Uniform-smoothed item	$0.0688$	$0.1038$	$0.0435$	$0.0548$	$0.0782$	$0.1144$	$0.0495$	$0.0612$	$0.0389$	$0.0602$	$0.0224$	$0.0293$
CMM (full)	$0.0714$	$0.1069$	$0.0478$	$0.0592$	$0.0784$	$0.1144$	$0.0532$	$0.0647$	$0.0422$	$0.0639$	$0.0272$	$0.0341$
Hard code	$0.0677$	$0.1010$	$0.0438$	$0.0545$	$0.0767$	$0.1119$	$0.0503$	$0.0616$	$0.0376$	$0.0595$	$0.0221$	$0.0292$

Appendix

Table 8. Four-metric ablation results (mean performance over three random seeds). Variant names and objective groups follow Table 3 .

Variant	Beauty		Toys		Sports
	$\lambda$	$\epsilon$	$\lambda$	$\epsilon$	$\lambda$	$\epsilon$
Top-1 item	0.867649	—	0.779020	—	0.611078	—
Uniform-smoothed item	1.462578	0.406484	1.240714	0.371723	1.690206	0.638363
Hard code	0.542772	—	0.379317	—	0.528670	—
Uniform-smoothed code	0.853681	0.487760	0.518572	0.465291	1.146013	0.752142
16 heads (level 1)	0.444682	—	0.292024	—	0.548509	—

Appendix

Table 9. Gradient-calibrated coefficients and entropy-matched smoothing parameters used by the ablation variants. Values are fixed across runs.

Dataset	$\alpha$	$r_{J}$	$\lambda_{\mathrm{CMM}}$	$\lambda_{\mathrm{IDM}}$
Beauty	0.75	0.8	0.859846	0.460198
Toys	0.25	0.6	0.205164	1.274258
Sports	0.50	0.4	0.373973	0.748308

Appendix

Table 10. Selected joint-training configurations. The final columns give the effective post-ramp weights for CMM and IDM, respectively.

Semantic ID Recommendation

Semantic identifier recommendation uses discrete, content-aware item codes to retrieve and rank items, especially in generative systems.

See all 38 papers

REGER: Modeling inter-item Relations with Semantic IDs for Generative Recommendation
Oct 27, 2026Yuxi Huang, Silin Zhou, Chenhao Wang +5Generative Recommendation Semantic ID Recommendation
University of Electronic Science and Technology of China
Understanding Semantic IDs: From Item Representation to Item Selection in Generative Recommendation
Jul 27, 2026Junting Wang, Xinrui He, Yunzhe Li +1Semantic ID Recommendation Language Model Decoding
University of Illinois Urbana-Champaign Urbana, Illinois, USA
EPIC: Explicit Posterior Item Conditioning for Semantic ID Diffusion Recommendation
Sep 3, 2026Tuan-Binh Tran, Thanh Tam Nguyen, Quoc Viet Hung Nguyen +3Semantic ID Recommendation Sequential Recommendation
College of Engineering and Computer Science VinUniversity Hanoi, Vietnam · School of Information and Communication Technology Griffith University Gold Coast, Queensland, Australia · Department of Computer Science Aalborg University Copenhagen, Denmark

Oct 27, 2026J/K move · Enter open · S save

REGER: Modeling inter-item Relations with Semantic IDs for Generative Recommendation

Yuxi Huang, Silin Zhou, Chenhao Wang, Zihan Li+4

University of Electronic Science and Technology of China

Generative Recommendation Semantic ID Recommendation Preference Modeling Sequential Recommendation

Open paper AskPDF arXiv

Personalized recommendation depends on learning semantic and collaborative relations among items from online content and user interactions. Generative recommendation (GR) offers a promising paradigm by autoregressively generating discrete semantic IDs (SIDs) of items. However, randomly initialized SID token embeddings and GR models trained from scratch do not directly inherit the semantic relations inherent in items, while inter-item collaborative relations are learned largely implicitly through token-level attention and next-token supervision. To address this limitation, we propose REGER, a unified framework that explicitly incorporates inter-item relations into semantic pretraining, history encoding, and SID generation. We pretrain the GR model through generative semantic retrieval, enabling it to inherit inter-item semantic similarity by learning to generate semantic neighbors' SIDs. We combine global attention with local attention aligned with SID boundaries to capture historical context and directed transitions between adjacent items. We further develop a theoretical framework for hierarchical preference decomposition over the SID prefix tree, proving an exact factorization of collaborative preference distributions and a weighted decomposition of item-level KL divergence into prefix-conditional KL divergences. Building on this analysis, we distill a discriminative teacher's full conditional distributions over all legal next tokens at ground-truth prefixes along the target SID path, jointly optimizing this objective with cross-entropy. Extensive experiments on real-world datasets demonstrate consistent improvements over existing GR baselines and support the effectiveness of the proposed preference distillation.

Abstract

Semantic ID Recommendation

Explore similar work

REGER: Modeling inter-item Relations with Semantic IDs for Generative Recommendation

Understanding Semantic IDs: From Item Representation to Item Selection in Generative Recommendation

EPIC: Explicit Posterior Item Conditioning for Semantic ID Diffusion Recommendation