Transformer-based sequential recommenders with causal self-attention often rely heavily on the most recent interaction at inference time, but how this behavior is structurally expressed in the representation used for prediction remains unclear. We combine prediction-time diagnostics with norm-based analysis of the full attention block. First, we show that SASRec-style models exhibit highly localized last-item reliance. We then find that, although self-attention aggregates contextual information, residual addition sharply shifts the full-block representation toward same-position contributions, which we term residual dominance. To probe this interpretation, we use inference-time residual scaling as a controlled diagnostic intervention. Changing the residual strength induces a monotonic trade-off between structural mixing and last-item reliance, while reducing residual strength recovers a subset of final-position misses for which representations at non-final positions already rank the ground-truth item correctly. Our results provide a structural account linking extreme last-item reliance to residual dominance at inference time. The code is publicly available.
Figures & tables
Figure 1. Impact of sequence ordering at inference time. Empty bars: chronologically ordered sequences; light diagonal hatch patterns: partial shuffle where only the final position vt is kept fixed and the remaining items are permuted; heavy diagonal hatch patterns: full shuffle where all positions, including vt , are permuted. Key takeaway: on the sparse and dense datasets, preserving only vt causes a substantially smaller change in SASRec’s performance than full shuffling, indicating a discontinuous last-position privilege; this contrast is weak on the non-sequential dataset. The figure consists of a sequence-shuffling diagram on the left and nine bar charts on the right. In the partial-shuffle diagram, the final item remains fixed at the final position while all preceding items are randomly permuted. In the full-shuffle diagram, all items, including the original final item, are permuted. The bar charts are arranged in three rows for GRU4Rec, BERT4Rec, and SASRec, and three columns for Toys, ML-1M, and Steam. Each chart compares NDCG at 10 for the original chronological sequence, partial shuffle, and full shuffle. For SASRec on Toys and ML-1M, partial shuffling produces a substantially smaller performance change than full shuffling. On Steam, both perturbations produce relatively small changes. GRU4Rec and BERT4Rec generally exhibit weaker contrasts between partial and full shuffling than SASRec.
Dataset
#Users
#Items
#Interact.
Avg. Len.
Density
#Days
Beauty
22,363
12,101
198,502
8.9
0.07%
4,424
Sports
35,598
18,357
296,337
8.3
0.05%
4,521
Video
24,303
10,672
231,780
9.5
0.09%
5,395
Diginetica
61,279
25,593
485,903
7.9
0.03%
152
Toys
19,412
11,924
167,597
8.6
0.07%
5,108
Steam
281,349
11,961
3,550,272
12.6
0.11%
2,639
Table 1. Statistics of the processed datasets after p -core filtering ( p=5 ) and consecutive-repeat removal.
Figure 2. HRLI@1 and HRL2I@1 across models and datasets (computed without filter-seen). HRLI@1 measures whether the last interacted item vt appears in the top-1 list; HRL2I@1 is defined analogously for vt−1 . A grouped bar chart compares HRLI at 1 and HRL2I at 1 for five models across nine datasets: Beauty, Sports, Video, Diginetica, Toys, Steam, BeerAdvocate, ML-1M, and Zvuk. For each model and dataset, a solid bar represents the frequency with which the last interacted item is ranked first, and a hatched bar represents the corresponding frequency for the second-to-last item. SASRec, DuoRec, and BSARec generally show tall solid bars and very small hatched bars, producing a large gap between HRLI and HRL2I. The largest solid bars occur for several sparse datasets, including Beauty, Sports, and Toys, as well as Zvuk. BERT4Rec and GRU4Rec generally show lower HRLI values and smaller differences between the two positional metrics. Diginetica and Steam exhibit comparatively smaller last-item localization than several other datasets, although the solid bars for the causal self-attention models remain higher than their hatched counterparts.
Figure 3. Input-wise contribution patterns in SASRec for the representative sparse and dense datasets, Toys and ML-1M. From left to right: Toys (Attn), Toys (AttnResLN), ML-1M (Attn), and ML-1M (AttnResLN). Compared with the attention-only contributions (Attn), the complete attention block outputs after residual addition and layer normalization (AttnResLN) exhibit markedly stronger diagonal components, indicating stronger preservation of same-position information. Four heatmaps visualizing position-wise contribution patterns for SASRec on the Toys and ML-1M datasets. From left to right, the figure shows attention-only outputs (Attn) and full attention-block representations including residual connections and layer normalization (AttnResLN) for Toys and ML-1M. Compared to attention-only outputs, the full attention block exhibits stronger diagonal components, indicating increased preservation of self-information.
SASRec
DuoRec
Dataset
Attn
+Res
+LN
Attn
+Res
+LN
Beauty
0.776
0.119
0.119
0.776
0.116
0.115
Sports
0.775
0.126
0.122
0.790
0.119
0.119
Toys
0.772
0.123
0.123
0.780
0.121
0.121
Video
0.798
0.169
0.169
0.773
0.170
0.170
Diginetica
0.721
0.150
0.149
0.724
0.152
0.151
Table 2. Mixing ratios of SASRec and DuoRec. Attn denotes the attention-only output, +Res denotes the output after residual addition, and +LN denotes the subsequent output after layer normalization.
Dataset
Layer
L
L−1
L−2
Toys
1
0.1278
0.1283
0.1251
2
0.1182
0.1178
0.1135
ML-1M
1
0.3984
0.3819
0.3798
2
0.3660
0.3617
0.3606
Table 3. Position- and layer-specific AttnResLN mixing ratios of SASRec on Toys and ML-1M, representing sparse and dense settings, respectively. Values are averaged over test sequences.
Figure 4. Effect of inference-time residual scaling ( α ) applied to the residual branch at all sequence positions in SASRec. Left: structural mixing averaged over positions and layers after the full attention block (AttnResLN; mixing ratio) and recency-localized reliance (HRLI@1) as functions of α ; the red dashed line indicates the standard setting α=1 . Right: accuracy–reliance trade-off, plotting HR@10 and NDCG@10 against HRLI@1; each colored point corresponds to a value of α (colorbar), and baseline markers indicate BERT4Rec and GRU4Rec. Effect of inference-time residual scaling on structural mixing, recency bias, and accuracy.
Dataset
Metric
Beauty
Sports
Toys
Video
Diginetica
ML-1M
Steam
BeerAdvocate
Zvuk
HR@10 at L (standard inference)
0.0381
0.0342
0.0574
0.0403
0.2900
0.1588
0.0639
0.0389
0.2416
HR@10 at L−1 and miss at L
0.0116
0.0157
0.0185
0.0171
0.1153
0.0347
0.0235
0.0150
0.0583
HR@10 at L−2 and miss at L
0.0124
0.0125
0.0134
0.0145
0.0782
0.0422
0.0241
0.0141
0.0415
HR@10 at L−3 and miss at L
0.0112
0.0115
0.0150
0.0124
0.0914
0.0298
0.0236
0.0139
0.0431
Table 4. Correct non-final predictions among final-position misses. The first row reports the standard HR@10 obtained from the final-position representation. For each preceding position L−k , the remaining rows report the joint proportion of all test sequences for which the final-position prediction misses the ground-truth item, while the representation at position L−k ranks it within the top 10. These values indicate the prevalence of potentially useful predictive signals at non-final positions.
Figure 5. Recovery of final-position misses under inference-time residual scaling. For each dataset, we consider sequences for which the standard final-position inference ( α=1 ) fails to rank the ground-truth item within the top 10, while at least one representation at L−1 , L−2 , or L−3 ranks it within the top 10. The curve reports the fraction of such sequences for which the final-position representation ranks the ground-truth item within the top 10 after applying residual scaling with factor α at all sequence positions. The recovery rate is zero at α=1 by definition. A multi-line plot shows the recovery rate of final-position misses as a function of the residual scaling factor alpha for nine datasets. The horizontal axis ranges from complete removal of the residual contribution at alpha equal to zero to the standard setting at alpha equal to one. The vertical axis shows the fraction of conditioned sequences that are recovered at the final position. All curves reach zero at alpha equal to one by construction. For most datasets, recovery increases as alpha is reduced from one, reaches its maximum at an intermediate value around 0.1 to 0.3, and then remains stable or decreases slightly as alpha approaches zero. Diginetica shows the highest peak recovery, exceeding 0.3, while the remaining datasets exhibit lower but consistently positive peaks. The common non-monotonic pattern indicates that moderate residual reduction generally recovers more cases than either the standard residual strength or complete residual removal.
Figure 6. Validation-based reproduction of the accuracy–recovery trade-off. (Top) Comparison between validation-selected and oracle-selected α across different values of ε . (Bottom) Corresponding test performance, showing the trade-off between Δ HR@10 relative to the α=1 baseline and recovery under validation-based selection. On Toys, validation-based selection reproduces the same accuracy–recovery trade-off observed under oracle tuning. Two vertically stacked plots summarize validation-based residual-scaling selection on Toys. The top plot shows the selected residual scaling factor alpha as a function of the allowed accuracy-loss parameter epsilon. The validation-selected curve and the oracle-selected curve follow similar downward trajectories: selected alpha is high under a strict accuracy constraint, decreases as epsilon increases, and stabilizes near a small value under looser constraints. The bottom plot shows the resulting test-set trade-off. The left vertical axis reports the change in HR at 10 relative to standard inference, and the right vertical axis reports recovery. As epsilon increases, the HR at 10 change becomes more negative, while recovery increases. Both quantities stabilize once the selected alpha becomes approximately constant. The two panels together show that validation-based selection reproduces a trade-off similar to oracle selection without selecting alpha directly on the test set.
Sequential self-attention models usually rely on additive positional embeddings, which inject positional information into item representations at the input. In the absence of positional signals, the attention block is permutation-equivariant over sequence positions and thus has no intrinsic notion of temporal order beyond causal masking. We argue that additive positional embeddings make the attention mechanism only superficially sensitive to sequence order: positional information is entangled with item embedding semantics, propagates weakly in deep architectures, and limits the ability to capture rich sequential patterns. To address these limitations, we introduce a kernelized self-attention mechanism, where a learnable positional kernel operates purely in the position space, disentangled from semantic similarity, and directly modulates attention weights. When applied per attention block, this kernel enables adaptive multi-scale sequential modeling. Experiments on standard next-item prediction benchmarks show that our positional kernel attention consistently improves over strong competing baselines.
Timur Nabiev, Evgeny Frolov
Skolkovo Institute of Science and Technology · AIRI
Candidate-dependent routing lets a sequential recommender weight a user's past behaviors differently for each item it scores. We test whether this flexibility helps by adding a small residual routing module to a self-attentive baseline, in two versions that differ only in what sets the routing weights: the user's history representation or the candidate item's embedding. Across five paired seeds on the REES46 cosmetics-shop dataset, with matched training for all three models, candidate-dependent routing leaves the primary metric, normalized discounted cumulative gain at cutoff ten (NDCG@10), essentially unchanged (mean relative change +0.003%). Secondary and exploratory analyses show that this flat average hides two effects that cancel. The candidate-dependent model places more relevant items in its top ten (recall +1.81%) but ranks the first one lower (mean reciprocal rank -1.09%), and most of its new hits land at positions six to ten. Its gains come from queries with at least one target among the user's last 50 events, while queries whose targets all lie outside that window get worse. In this form, candidate-dependent routing trades accuracy at the top of the list for recall further down.
Han Chen, Yingrui Li
Department of Computer Science, The University of Texas at Austin, Austin, Texas, United States · Unaffiliated, Fairfax, Virginia, United States
Transformer architectures have achieved remarkable success across diverse domains; however, directly applying their standard self-attention mechanism to recommendation often yields suboptimal performance, sometimes even trailing behind well-designed simple recommendation models. In this paper, we reveal that this performance bottleneck stems from severe embedding and attention collapse unique to recommendation scenarios. The heterogeneity and long-tail nature of recommendation data lead to a severe spectral collapse dominated by a few principal singular values. We further theoretically demonstrate that this triggers a vicious cycle in recommendation model's forward and backward propagation, which accelerates embedding and attention collapse and limits the model's scaling capability with increased depth. To address these issues, we propose SpecFormer, a novel Spectral-Aware Transformer designed for mitigating embedding and attention collapse in recommendation. Specifically, SpecFormer introduces 1) a Learnable Spectral Softening module to dynamically smooth the singular values distribution of the input token embeddings; 2) a Spectrum-softened Attention mechanism to model feature interaction under a more uniform spectral distribution space; 3) a Spectral Residual Position Encoding via Taylor expansion of singular values, explicitly providing a spectral inductive bias for feature interactions. Extensive experiments on one industrial and two public datasets demonstrate that SpecFormer significantly outperforms state-of-the-art baselines. Notably, SpecFormer has been successfully deployed in a real-world commercial recommender system and exhibits exceptional scaling capabilities: stacking SpecFormer layers actively improves the attention effective rank and recommendation performance.
Yu Cui, Yi Xu, Jiahao Wang +6
Zhejiang University Hangzhou, China · Alibaba Group Beijing, China