Off-Policy Evaluation for Semantic ID Recommenders: Does the Model's Own Code Hierarchy Help?

Artem Betlei

cs.LGAug 28, 2026

Off-Policy Evaluation for Semantic ID Recommenders: Does the Model's Own Code Hierarchy Help?

Authors: Artem Betlei

Organizations: Criteo AI Lab Grenoble, France

Abstract

Generative recommenders increasingly emit semantic IDs (SIDs): each item is a short sequence of hierarchical discrete codes from a residual quantizer, decoded autoregressively. Before spending scarce A/B-test, a team may decide offline which decoder or reranking variants are worth testing - a job for off-policy evaluation (OPE). We ask a simple question: can the model's own SID tree serve as the action abstraction for that OPE? Our answer has three parts. (i) Under the near-argmax logging real recommenders use, per-item OPE is hopeless - as item-level effective sample size is usually small on production logs - but marginalizing items to code-prefix clusters restores estimable support and cuts error. (ii) This gain is thanks to coarsening, not to the hierarchy specifically; but the SID tree is what makes coarsening feasible in a generative system - each cluster's mass is exactly and cheaply returned by the decoder, whereas flat clustering requires enumerating item/leaf masses that a code-only decoder does not directly expose. (iii) Resolution depth is the operative knob - coarser under scarce support - and a conditional bias bound links the coarsening bias to the quantizer's worst-case reconstruction residual and the target-logging divergence.

Figures & tables

Table 1. RQ2, KuaiRand, leakage-free (

|\mathcal{A}|{=}7339

M{=}5000

;

\mathrm{hier}

at the oracle-selected level). (a) Estimator comparison. (b) SID tree vs. flat

k

-means at matched cluster counts (metadata,

\mathrm{OffCEM}^{\mathrm{act}}_{\mathrm{hier}}

), each row refitted under

10

shared clustering seeds;

\Delta

is the paired tree

{-}

flat difference.

$M$	estimator	Kendall $\tau$	top-3 hit
$1000$	$\mathrm{SNIPS}_{\mathrm{item}}$	$0.08\pm 0.03$	$10\%$
$1000$	$\mathrm{SNIPS}_{\mathrm{hier}}$	$\mathbf{0.13}\pm 0.03$	$12\%$
$5000$	$\mathrm{SNIPS}_{\mathrm{item}}$	$0.15\pm 0.03$	$6\%$
$5000$	$\mathrm{SNIPS}_{\mathrm{hier}}$	$\mathbf{0.18}\pm 0.02$	$\mathbf{14\%}$

Table 2. RQ2 (selection). Ranking

16

candidate variants off one log against the oracle ranking (level 3, pre-specified). For realistic cross-cluster variants,

\mathrm{SNIPS}_{\mathrm{hier}}

ranks better than

\mathrm{SNIPS}_{\mathrm{item}}

, more so under scarce support.

Figure 1. RQ3. KuaiRand RMSE by code level (coarse

\to

fine), metadata source, two support regimes (

200

seeds,

95\%

bootstrap bands). Under scarce support (left) an intermediate level is best; with more data (right) the optimum moves finer.

$M$	$\mathrm{SNIPS}_{\mathrm{item}}$	SLOPE (deployable)	oracle-selected
1000	0.251	0.133	0.105
5000	0.155	0.109	0.086
10000	0.160	0.102	0.084

Table 3. RQ3. KuaiRand support sweep, leakage-free, content-metadata codes (

|\mathcal{A}|{=}7339

200

seeds; RMSE). oracle is

\mathrm{SNIPS}_{\mathrm{hier}}

at the oracle-selected level (an upper bound); SLOPE selects that level from logged data alone - applied to

\mathrm{MIPS}_{\mathrm{hier}}

, whose per-level standard errors the rule needs - and is therefore deployable. SLOPE beats

\mathrm{SNIPS}_{\mathrm{item}}

at every

M

, capturing

67

81\%

of the available gain. The

M{=}5000

row is Table 1(a) ’s configuration;

0.086

vs.

0.087

is the independent run’s seed draw.

source	Residual K-Means	RQ-VAE	$\Delta$ (paired)
metadata	$0.087$ [.075,.098]	$0.084$ [.078,.089]	$-0.003$ [ $-$ .016, $+$ .010]
semantic	$0.097$ [.096,.098]	$0.096$ [.094,.097]	$-0.001$ [ $-$ .003, $+$ .001]
collaborative	$0.131$ [.127,.134]	$0.131$ [.130,.131]	$+0.000$ [ $-$ .003, $+$ .004]
fused	$0.119$ [.116,.125]	$0.115$ [.114,.116]	$\mathbf{-0.004}$ [ $-$ .010, $-$ .001]

Table 4. RQ4. Quantizer ablation (KuaiRand, leakage-free,

M{=}5000

100

simulation seeds per fit), oracle-selected-level

\mathrm{OffCEM}^{\mathrm{act}}_{\mathrm{hier}}

RMSE. Each cell is refitted under

10

independent quantizer seeds; the interval is over those seeds , so it covers variability in the fitted tree and not only in the evaluation.

\Delta

is the paired RQ-VAE

{-}k

-means difference (both fitted under the same seed). Source separates the sources by up to

0.047

; every tokenizer difference except fused contains zero.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

$\alpha$	med. $\pi_{0}(a)$	${>}0.9$	level	med. $p_{\text{code}}$	ESS/ $n$
3	0.036	5.7%	1 (coarse)	0.365	0.300
4	0.096	9.8%	2	0.125	0.191
6	0.145	16.4%	3 (fine)	0.089	0.167

Appendix

Table 5. RQ1/RQ2 production diagnostics. Left: served-item propensity concentration by logging exponent

\alpha

(item ESS/

n{\approx}0.003

\alpha{=}4

). Right: coarsening lifts greedy-target ESS (

n{=}29

k).

$M$	$\|\mathcal{A}\|/M$	$\mathrm{SNIPS}_{\mathrm{item}}$	$\mathrm{OffCEM}^{\mathrm{act}}_{\mathrm{hier}}$
1000	21.0	0.114	0.110
2000	10.5	0.089	0.080
5000	4.2	0.055	0.052
10000	2.1	0.036	0.036

Appendix

Table 6. RQ6. Amazon Reviews (Musical Instruments, metadata codes,

80

seeds;

\mathrm{OffCEM}^{\mathrm{act}}_{\mathrm{hier}}

at the oracle-selected level). The support-dependent pattern replicates.

Off-Policy Evaluation

Off-policy evaluation estimates a policy’s value from data collected by another policy, using weighting, models, and uncertainty-aware methods.

See all 74 papers

Understanding Semantic IDs: From Item Representation to Item Selection in Generative Recommendation
Jul 27, 2026Junting Wang, Xinrui He, Yunzhe Li +1Semantic ID Recommendation Language Model Decoding
University of Illinois Urbana-Champaign Urbana, Illinois, USA
From Interests to Semantic IDs: Retrieval-Grounded Credit Assignment for Generative Recommendation
Sep 24, 2026Mengdan Zhu, Yufan Zhao, Yao Zhao +5Semantic ID Recommendation Credit Assignment
Emory University Atlanta, GA, USA · Microsoft Redmond, WA, USA · Cornell University Ithaca, NY, USA
Difficulty-Aware Semantic-ID Optimization for Generative Recommendation
Aug 20, 2026Xin Yu, Stephen Li, Sina Aghaei +6Semantic ID Recommendation Generative Recommendation
Meta · The Pennsylvania State University

Jul 27, 2026J/K move · Enter open · S save

Understanding Semantic IDs: From Item Representation to Item Selection in Generative Recommendation

Junting Wang, Xinrui He, Yunzhe Li, Hari Sundaram

University of Illinois Urbana-Champaign Urbana, Illinois, USA

Semantic ID Recommendation Language Model Decoding Generative Recommendation

Open paper AskPDF arXiv

Semantic IDs (SIDs) are now a central component of generative recommendation. Current SID-based systems assign three roles to the same token sequence. Shared prefixes are intended to organize related items, the complete SID identifies an individual item, and each generated token narrows the items that can still be returned. We systematically investigate SIDs from item encoding and SID construction to autoregressive generation and final recommendation. We examine how SID construction changes item representations and how those changes affect generation. Across three Amazon domains and eight SID constructions, SID neighborhoods recover only 32.2% of the encoder's ten nearest neighbors on average. Alternative item descriptions still retrieve the corresponding item first in 99.57% of controlled cases, yet change 38.4% of exact SIDs. These results show that SIDs retain broad organization but lose much of the encoder's fine local structure, while their exact tokens are not determined by item meaning alone. This loss becomes consequential during generation. After the final semantic token, TIGER retains only 29.9% of held-out targets that were plausible recommendations before SID filtering. Motivated by these findings, we propose Item-Supported Decoding (ISD), a lightweight inference-time method that allows a user-specific item ranking to support corresponding SID prefixes before beam search discards them. The same ranking then orders the generated items. ISD requires no additional parameters or retraining of the SID constructor or decoder. We empirically show that ISD improves NDCG@10 over the corresponding SID backbone in every evaluated setting, with relative gains of up to 31.2%. Our results show that SIDs provide useful coarse item organization, but their fine boundaries should not alone determine which items remain available during generation.

Abstract

Off-Policy Evaluation

Explore similar work

Understanding Semantic IDs: From Item Representation to Item Selection in Generative Recommendation

From Interests to Semantic IDs: Retrieval-Grounded Credit Assignment for Generative Recommendation

Difficulty-Aware Semantic-ID Optimization for Generative Recommendation