Next-item prediction is inherently ambiguous: a single interaction history often admits multiple plausible continuations across distinct catalogue regions. Standard sequential recommenders compress these alternatives into a single prediction, conflating user intents. We introduce COllapse-resistant Multiple hyPOThesis lEarning (COMPOTE), a multi-hypothesis transformer trained with an annealed objective that transitions from uniform supervision to a winner-takes-all assignment. This mechanism forces hypotheses to specialize into distinct catalogue regions, while inference relies on an efficient, normalizer-free max-similarity rule. Theoretically, we show that in the population case, the optimal solutions partition the conditional next-item distribution according to a max-entropy quantization principle, while pooled scores preserve the Bayes-optimal ranking. We also prove a generalization bound scaling logarithmically with the number of hypotheses. Across three Amazon Reviews 2023 benchmarks, COMPOTE outperforms strong ID-based, generative, and multi-interest baselines using item IDs alone. Matched-capacity ablations confirm that performance gains stem directly from the annealed objective, preventing the head collapse observed in conventional multi-head architectures, rather than added capacity, ensembling, or routing.
Figures & tables
Figure 1: Multiple-choice transformer. An item-id history x is embedded, summed with a learned positional embedding, and encoded by a stack of L pre-norm causal transformer blocks shared by all heads; the representation z at the last (most recent) position is read off. Two modules consume z : the M hypothesis heads h1,…,hM , each producing hj=hj(z)∈Rd , and the confidence score head γ^ , whose j -th output is the predicted probability that head j is the best expert for x . Each hypothesis is turned into a distribution over the catalogue by fj,i(x)=\softmaxi(cos(hj,ei)/β) , yielding one score vector fj,⋅(x)∈ΔN\mhyphen1 per head. Everything left of z is a single shared trunk; right of it the M branches are independent, which is what makes head specialisation possible.
Instrument
Scientific
Game
Method
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
Caser ( Tang and Wang, 2018 )
2.41
3.86
1.51
1.97
1.59
2.57
1.01
1.32
3.30
5.53
2.09
2.81
HGN ( Ma et al., 2019 )
3.21
5.17
2.02
2.65
2.12
3.51
1.31
1.76
4.24
6.87
2.71
3.56
GRU4Rec ( Hidasi et al., 2016 )
3.24
5.01
2.09
2.66
2.02
3.38
1.29
1.73
4.99
7.99
3.20
4.16
BERT4Rec ( Sun et al., 2019 )
3.07
4.85
1.95
2.52
1.86
2.96
1.19
1.55
4.60
7.35
2.98
3.86
SASRec ( Kang and McAuley, 2018 )
3.33
5.23
2.13
2.74
2.59
4.12
1.50
1.99
5.35
8.47
3.31
4.38
Table 1: Performance (%) on three Amazon 2023 subsets (H@ K : Hit Rate@ K , N@ K : NDCG@ K ). First-block numbers are from Zheng et al. (2025) ; multi-interest models are trained by us under the same protocol ( Appendix F ). COMPOTE uses max-similarity aggregation ( Equation 10 ), equivalent to max-softmax pooling ( Section 4.1 ). Bold : best, underline : second best.
Aggregator
Normalizer
H@5
H@10
N@5
N@10
g – ”max softmax”
required
6.04
9.27
4.01
5.05
g~ – ”max cosine similarity”
not required
6.06
9.26
4.01
5.04
Table 2: Effect of the inference aggregator on COMPOTE performance (%) on Game, for the same trained model with M=3 heads and the same checkpoint. “Normalizer” indicates whether ranking requires the per-head catalogue partition function Zm . Bold : best.
Instrument
Scientific
Game
Method
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
H@5
H@10
N@5
N@10
MCT-SASRec
3.63
5.70
2.34
2.97
2.55
3.87
1.71
2.13
5.73
8.84
3.78
4.78
MCT-EnsCE
3.54
5.53
2.28
2.96
2.42
3.80
1.60
2.03
5.74
8.81
3.78
4.77
MCT-Router
3.57
5.48
2.28
2.89
2.40
3.60
1.64
2.00
5.74
8.81
3.78
4.77
COMPOTE (ours)
3.75
5.83
2.45
3.10
2.84
4.20
1.92
2.35
6.06
9.26
4.01
5.04
Table 3: Ablation over the prediction head and the training objective (%). Variants share the same encoder backbone and parameter budget, and rank items with max-similarity aggregation ( Equation 10 ): MCT-SASRec is a single-hypothesis iso-parameter head, MCT-EnsCE uses an ensemble cross-entropy objective, and MCT-Router routes queries to a single expert. Bold : best.
M
In-cell mass
Win frequency (%)
Sector share (%)
L^\textscwta
L^\textscwta+logM
Sector agreement
3
0.929
[32.0,34.4]
[27.9,36.3]
7.32
8.41
98.5%
4
0.915
[24.5,25.6]
[23.4,27.1]
7.02
8.41
98.7%
5
0.937
[18.9,21.1]
[15.5,25.3]
6.71
8.32
99.4%
Table 4: Signatures of Theorem 7 on Game as the number of hypotheses M varies. All columns are computed on the full test set of 94,762 query–target pairs. Win frequencies and sector shares are reported as the range over heads.
Figure 2: Catalogue tiling on Game ( M=3 ). Normalized items (dots) and head outputs (stars) are projected onto the plane spanned by the static head directions hˉm , the average normalized output of each head over the test set. Items are coloured by the direction they are most aligned with, and the green cross marks the target. Each panel is a test query won by a different head, and in each the target lies in the sector of its winning head.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Distribution of the per-head log-normalizer logZm(x) . Histogram of logZm(x)=log∑i≤Nexp(sm,i(x)) over the 94,762 test queries of Game , one colour per head, for the COMPOTE checkpoint with M=3 of Table 2 ( β=0.07 , N=25,612 ). The three heads have almost superimposed distributions, concentrated between 12.2 and 15.0 on a scale where logZm could a priori take any value in [logN−1/β,logN+1/β]=[−4.1,24.4] . For a fixed query, the standard deviation of logZm(x) across the three heads is 0.13 on average, against a mean value of 13.7 .
Dataset
#Users
#Items
#Interactions
Sparsity
Avg. len
Instrument
57,439
24,587
511,836
99.964%
8.91
Scientific
50,985
25,848
412,947
99.969%
8.10
Game
94,762
25,612
814,586
99.966%
8.60
Appendix
Table 5: Statistics of the preprocessed datasets [ Zheng et al., 2025 ] . Avg. len is the average length of the user interaction sequences.
Figure 4: Effect of the score-loss weight α . N@5 (%) on Scientific with M=3 under max-similarity aggregation, as a function of the weight α of the score loss in Equation 33 . Ticks are evenly spaced and do not follow the scale of α . Without the score loss ( α=0 ), COMPOTE already reaches 1.82 , above every single-hypothesis variant of Table 3 ; the best weight α=0.1 adds 0.09 points.
Figure 5: Effect of the number of hypotheses. N@5 on Scientific under max-similarity aggregation, as a function of M . Performance improves sharply from M=1 to M=3 , then plateaus: increasing M from 3 to 12 does not degrade ranking quality, in line with the sublinear dependence on M in Theorem 4 .
Method
Instrument
Scientific
Game
MCT-EnsCE
1.00
1.00
1.00
MCT-Router
1.00
1.00
1.00
COMPOTE (ours)
0.64
0.61
0.58
Appendix
Table 6: Mean pairwise cosine similarity between hypothesis heads on the test set. A value of 1 means the heads are indistinguishable, lower is more specialized.
Instrument
Scientific
Game
Method
Aggregated
Oracle
Aggregated
Oracle
Aggregated
Oracle
MCT-EnsCE
2.28
2.28
1.60
1.60
3.78
3.78
MCT-Router
2.28
2.28
1.64
1.64
3.78
3.78
COMPOTE (ours)
2.45
3.75
1.92
3.52
4.01
7.78
Appendix
Table 7: NDCG@5 (%) under max-similarity aggregation ( Equation 10 ) and under oracle head selection ( Equation 11 ). Equal columns indicate that all heads induce the same ranking, i.e. complete collapse.
M
Mean off-diag. cos(h)
Cross-head top- 10 Jaccard
Region loyalty
3
−0.09
0.008
98.9%
4
−0.17
0.003
99.4%
5
−0.04
0.001
99.8%
Appendix
Table 8: Latent-space tiling on Game as the number of hypotheses M varies. Mean off-diagonal cosine similarity is computed between the static head directions hm (the dataset-wide average of normalized head outputs). The top- 10 diagnostics use 4,000 test queries sampled uniformly without replacement. Cross-head Jaccard is the mean pairwise overlap of these lists across heads. Region loyalty is the fraction of a head’s top- 10 items that fall in its catalogue sector.
Figure 6: Latent-space tiling on Game . Cosine similarity between the static head directions hˉm (the dataset-wide average of normalized head outputs) for M∈{3,4,5} . Off-diagonal values remain near zero or negative, indicating that the heads occupy separated directions rather than collapsing onto one another.
Figure 7: Catalogue partition on Scientific ( M=3 ). PCA projections on three test sequences (queries 4934, 23025, and 26314) under COMPOTE. Each panel shows the L2-normalized item catalogue in two dimensions (grey), the target next item (green cross), and the three head query embeddings (stars). The heads occupy distinct regions of the catalogue rather than collapsing onto a single direction.
Sequential recommender systems typically infer user preferences through single-pass encoding of interaction histories without iterative refinement, relying on increasingly deep architectures to capture complex patterns. In this work, we revisit sequential recommendation from a recursive inference perspective: can user preferences be modeled as a persistent latent state that is recursively refined? We propose RecRec (Recursive Recommendation), a lightweight model that maintains a compact latent state and updates it through a shared recursive module conditioned on interaction evidence. Unlike prior recursive models, RecRec introduces an evidence-anchored correction mechanism that stabilizes refinement by grounding each update in the original interaction context, preventing semantic drift during deep recursive reasoning. Experiments on three benchmark datasets under standard evaluation protocols show that RecRec matches or outperforms state-of-the-art sequential, graph-based, and reasoning-enhanced recommenders while using only 3.9M to 14M parameters. Ablation studies demonstrate that both recursive refinement and the evidence-anchored correction gate contribute significantly to performance, highlighting the effectiveness of recursive latent inference as a scalable alternative to deeper or language-based architectures. Code is available at https://anonymous.4open.science/r/RecRec-6B67/README.md.
Existing sequential recommenders usually allocate a fixed computational budget to every item in a user's interaction history, regardless of the item's contribution to the final prediction. We challenge this de facto design with two diagnostic findings. (1) Non-uniform computational need: items within the same sequence vary drastically in how much computation they require. Most converge after shallow processing while a critical minority only stabilizes at full depth. (2) Suboptimality of fixed allocation: given oracle access to the target item, a non-uniform depth assignment consistently outranks the fixed one, indicating that existing uniform computation both wastes capacity on already-converged items and sacrifices overall recommendation quality. These observations motivate a shift from fixed-depth computation to item-level adaptive reasoning. To this end, we propose LoopRec. LoopRec recursively applies a single weight-shared block and introduces a lightweight gate at each recursion boundary that makes a binary exit-or-continue decision for every item independently. Items whose representations have sufficiently matured exit early, while those requiring deeper reasoning continue to subsequent recursions. This design converts the fixed global depth into a learned, per-item property without auxiliary depth labels, and is readily adaptable to diverse backbone architectures. Extensive experiments on public benchmarks with four backbone families demonstrate that LoopRec consistently outperforms matched fixed-depth counterparts. Further analysis reveals that LoopRec's learned allocation exhibits interpretable patterns aligned with our post-hoc oracle analysis: lower-frequency items and more recent interactions naturally receive deeper computation. Our code is available at https://github.com/Elpmis117/LoopRec.
Zhanshuo Cao, Jiakai Tang, Yuqi Zhou +2
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Generative recommendation autoregressively generates the semantic IDs of the target item, unifying preference modeling and index retrieval within the shared token space. Recent attempts have introduced Multi-Token Prediction (MTP) into this field, yet they primarily inherit its efficiency merit, leaving its potential as dense supervision unexplored. Unlocking this potential hinges on whether future behaviors qualify as informative supervision. Our analysis reveals that future behaviors carry a semantic echo of the current one far above that of random pairs, which nevertheless decays along horizons under intent transitions, making them informative yet order-dependent signals. Motivated by this, we propose EchoRec, which empowers MTP with cycle-consistent holistic preference alignment across multi-horizon for generative recommendation. It comprises two synergistic modules. Horizon-aware Preference Generation (HPG) sequentially chains lightweight auxiliary branches upon the base recommender, where each branch conditions on its predecessor to respect preference evolution. Verifiable Holistic-Preference Alignment (VHA) further consolidates them into the holistic preference and echoes it back through cycle-consistent projectors to suppress spurious alignment, with theoretical guarantees that exclude the rank-collapse form of spurious alignment under an invertible transport, enabling the holistic preference to be retained in the decoding representation. All auxiliary components serve as disposable scaffolding discarded at inference, introducing negligible online serving overhead. Extensive experiments on three datasets demonstrate the superiority of our EchoRec, together with its naturally acquired multi-item generation ability. Our code and datasets will be available upon acceptance.
Haokai Ma, Aoqi Hu, Yueao Xing +5
National University of Singapore Singapore, Singapore · Beijing University of Posts and Telecommunications Beijing, China · Tencent Beijing, China +1