Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders

Keito Kozaki; Keigo Sakurai; Ren Togo; Takahiro Ogawa; Miki Haseyama

cs.AIAug 14, 2026

Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders

Authors: Keito Kozaki, Keigo Sakurai, Ren Togo, Takahiro Ogawa, Miki Haseyama

Organizations: Hokkaido University Sapporo, Hokkaido, Japan

Abstract

Transformer-based sequential recommenders with causal self-attention often rely heavily on the most recent interaction at inference time, but how this behavior is structurally expressed in the representation used for prediction remains unclear. We combine prediction-time diagnostics with norm-based analysis of the full attention block. First, we show that SASRec-style models exhibit highly localized last-item reliance. We then find that, although self-attention aggregates contextual information, residual addition sharply shifts the full-block representation toward same-position contributions, which we term residual dominance. To probe this interpretation, we use inference-time residual scaling as a controlled diagnostic intervention. Changing the residual strength induces a monotonic trade-off between structural mixing and last-item reliance, while reducing residual strength recovers a subset of final-position misses for which representations at non-final positions already rank the ground-truth item correctly. Our results provide a structural account linking extreme last-item reliance to residual dominance at inference time. The code is publicly available.

Figures & tables

Figure 1. Impact of sequence ordering at inference time. Empty bars: chronologically ordered sequences; light diagonal hatch patterns: partial shuffle where only the final position $v_{t}$ is kept fixed and the remaining items are permuted; heavy diagonal hatch patterns: full shuffle where all positions, including $v_{t}$ , are permuted. Key takeaway: on the sparse and dense datasets, preserving only $v_{t}$ causes a substantially smaller change in SASRec’s performance than full shuffling, indicating a discontinuous last-position privilege; this contrast is weak on the non-sequential dataset. The figure consists of a sequence-shuffling diagram on the left and nine bar charts on the right. In the partial-shuffle diagram, the final item remains fixed at the final position while all preceding items are randomly permuted. In the full-shuffle diagram, all items, including the original final item, are permuted. The bar charts are arranged in three rows for GRU4Rec, BERT4Rec, and SASRec, and three columns for Toys, ML-1M, and Steam. Each chart compares NDCG at 10 for the original chronological sequence, partial shuffle, and full shuffle. For SASRec on Toys and ML-1M, partial shuffling produces a substantially smaller performance change than full shuffling. On Steam, both perturbations produce relatively small changes. GRU4Rec and BERT4Rec generally exhibit weaker contrasts between partial and full shuffling than SASRec.

Dataset	#Users	#Items	#Interact.	Avg. Len.	Density	#Days
Beauty	22,363	12,101	198,502	8.9	0.07%	4,424
Sports	35,598	18,357	296,337	8.3	0.05%	4,521
Video	24,303	10,672	231,780	9.5	0.09%	5,395
Diginetica	61,279	25,593	485,903	7.9	0.03%	152
Toys	19,412	11,924	167,597	8.6	0.07%	5,108
Steam	281,349	11,961	3,550,272	12.6	0.11%	2,639

Table 1. Statistics of the processed datasets after

p

-core filtering (

p=5

) and consecutive-repeat removal.

Figure 2. HRLI@1 and HRL2I@1 across models and datasets (computed without filter-seen). HRLI@1 measures whether the last interacted item

v_{t}

appears in the top-1 list; HRL2I@1 is defined analogously for

v_{t-1}

. A grouped bar chart compares HRLI at 1 and HRL2I at 1 for five models across nine datasets: Beauty, Sports, Video, Diginetica, Toys, Steam, BeerAdvocate, ML-1M, and Zvuk. For each model and dataset, a solid bar represents the frequency with which the last interacted item is ranked first, and a hatched bar represents the corresponding frequency for the second-to-last item. SASRec, DuoRec, and BSARec generally show tall solid bars and very small hatched bars, producing a large gap between HRLI and HRL2I. The largest solid bars occur for several sparse datasets, including Beauty, Sports, and Toys, as well as Zvuk. BERT4Rec and GRU4Rec generally show lower HRLI values and smaller differences between the two positional metrics. Diginetica and Steam exhibit comparatively smaller last-item localization than several other datasets, although the solid bars for the causal self-attention models remain higher than their hatched counterparts.

Figure 3. Input-wise contribution patterns in SASRec for the representative sparse and dense datasets, Toys and ML-1M. From left to right: Toys (Attn), Toys (AttnResLN), ML-1M (Attn), and ML-1M (AttnResLN). Compared with the attention-only contributions (Attn), the complete attention block outputs after residual addition and layer normalization (AttnResLN) exhibit markedly stronger diagonal components, indicating stronger preservation of same-position information. Four heatmaps visualizing position-wise contribution patterns for SASRec on the Toys and ML-1M datasets. From left to right, the figure shows attention-only outputs (Attn) and full attention-block representations including residual connections and layer normalization (AttnResLN) for Toys and ML-1M. Compared to attention-only outputs, the full attention block exhibits stronger diagonal components, indicating increased preservation of self-information.

	SASRec			DuoRec
Dataset	Attn	+Res	+LN	Attn	+Res	+LN
Beauty	0.776	0.119	0.119	0.776	0.116	0.115
Sports	0.775	0.126	0.122	0.790	0.119	0.119
Toys	0.772	0.123	0.123	0.780	0.121	0.121
Video	0.798	0.169	0.169	0.773	0.170	0.170
Diginetica	0.721	0.150	0.149	0.724	0.152	0.151

Table 2. Mixing ratios of SASRec and DuoRec. Attn denotes the attention-only output, +Res denotes the output after residual addition, and +LN denotes the subsequent output after layer normalization.

Dataset	Layer	$L$	$L-1$	$L-2$
Toys	1	0.1278	0.1283	0.1251
	2	0.1182	0.1178	0.1135
ML-1M	1	0.3984	0.3819	0.3798
	2	0.3660	0.3617	0.3606

Table 3. Position- and layer-specific AttnResLN mixing ratios of SASRec on Toys and ML-1M, representing sparse and dense settings, respectively. Values are averaged over test sequences.

Figure 4. Effect of inference-time residual scaling ( $\alpha$ ) applied to the residual branch at all sequence positions in SASRec. Left: structural mixing averaged over positions and layers after the full attention block (AttnResLN; mixing ratio) and recency-localized reliance (HRLI@1) as functions of $\alpha$ ; the red dashed line indicates the standard setting $\alpha=1$ . Right: accuracy–reliance trade-off, plotting HR@10 and NDCG@10 against HRLI@1; each colored point corresponds to a value of $\alpha$ (colorbar), and baseline markers indicate BERT4Rec and GRU4Rec. Effect of inference-time residual scaling on structural mixing, recency bias, and accuracy.

	Dataset
Metric	Beauty	Sports	Toys	Video	Diginetica	ML-1M	Steam	BeerAdvocate	Zvuk
HR@10 at $L$ (standard inference)	0.0381	0.0342	0.0574	0.0403	0.2900	0.1588	0.0639	0.0389	0.2416
HR@10 at $L-1$ and miss at $L$	0.0116	0.0157	0.0185	0.0171	0.1153	0.0347	0.0235	0.0150	0.0583
HR@10 at $L-2$ and miss at $L$	0.0124	0.0125	0.0134	0.0145	0.0782	0.0422	0.0241	0.0141	0.0415
HR@10 at $L-3$ and miss at $L$	0.0112	0.0115	0.0150	0.0124	0.0914	0.0298	0.0236	0.0139	0.0431

Table 4. Correct non-final predictions among final-position misses. The first row reports the standard HR@10 obtained from the final-position representation. For each preceding position

L-k

, the remaining rows report the joint proportion of all test sequences for which the final-position prediction misses the ground-truth item, while the representation at position

L-k

ranks it within the top 10. These values indicate the prevalence of potentially useful predictive signals at non-final positions.

Figure 5. Recovery of final-position misses under inference-time residual scaling. For each dataset, we consider sequences for which the standard final-position inference (

\alpha=1

) fails to rank the ground-truth item within the top 10, while at least one representation at

L-1

L-2

, or

L-3

ranks it within the top 10. The curve reports the fraction of such sequences for which the final-position representation ranks the ground-truth item within the top 10 after applying residual scaling with factor

\alpha

at all sequence positions. The recovery rate is zero at

\alpha=1

by definition. A multi-line plot shows the recovery rate of final-position misses as a function of the residual scaling factor alpha for nine datasets. The horizontal axis ranges from complete removal of the residual contribution at alpha equal to zero to the standard setting at alpha equal to one. The vertical axis shows the fraction of conditioned sequences that are recovered at the final position. All curves reach zero at alpha equal to one by construction. For most datasets, recovery increases as alpha is reduced from one, reaches its maximum at an intermediate value around 0.1 to 0.3, and then remains stable or decreases slightly as alpha approaches zero. Diginetica shows the highest peak recovery, exceeding 0.3, while the remaining datasets exhibit lower but consistently positive peaks. The common non-monotonic pattern indicates that moderate residual reduction generally recovers more cases than either the standard residual strength or complete residual removal.

Figure 6. Validation-based reproduction of the accuracy–recovery trade-off. (Top) Comparison between validation-selected and oracle-selected

\alpha

across different values of

\varepsilon

. (Bottom) Corresponding test performance, showing the trade-off between

\Delta

HR@10 relative to the

\alpha=1

baseline and recovery under validation-based selection. On Toys, validation-based selection reproduces the same accuracy–recovery trade-off observed under oracle tuning. Two vertically stacked plots summarize validation-based residual-scaling selection on Toys. The top plot shows the selected residual scaling factor alpha as a function of the allowed accuracy-loss parameter epsilon. The validation-selected curve and the oracle-selected curve follow similar downward trajectories: selected alpha is high under a strict accuracy constraint, decreases as epsilon increases, and stabilizes near a small value under looser constraints. The bottom plot shows the resulting test-set trade-off. The left vertical axis reports the change in HR at 10 relative to standard inference, and the right vertical axis reports recovery. As epsilon increases, the HR at 10 change becomes more negative, while recovery increases. Both quantities stabilize once the selected alpha becomes approximately constant. The two panels together show that validation-based selection reproduces a trade-off similar to oracle selection without selecting alpha directly on the test set.

Sequential Recommendation

Sequential recommendation predicts what a user will choose next from ordered interactions, adapting to changing interests and context.

See all 132 papers

Position-Aware Sequential Attention for Accurate Next Item Recommendations
Feb 24, 2026Timur Nabiev, Evgeny FrolovAttention Mechanisms Sequential Recommendation
Skolkovo Institute of Science and Technology · AIRI
Candidate-dependent residual routing trades top-of-list accuracy for recall in sequential recommendation
Oct 16, 2026Han Chen, Yingrui LiSequential Recommendation Attention Routing
Department of Computer Science, The University of Texas at Austin, Austin, Texas, United States · Unaffiliated, Fairfax, Virginia, United States
SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
Jul 27, 2026Yu Cui, Yi Xu, Jiahao Wang +6Representation Collapse Transformer
Zhejiang University Hangzhou, China · Alibaba Group Beijing, China

Feb 24, 2026J/K move · Enter open · S save

Position-Aware Sequential Attention for Accurate Next Item Recommendations

Timur Nabiev, Evgeny Frolov

Skolkovo Institute of Science and Technology · AIRI

Attention Mechanisms Sequential Recommendation Self-Attention Multi-Scale Attention

Open paper AskPDF arXiv

Sequential self-attention models usually rely on additive positional embeddings, which inject positional information into item representations at the input. In the absence of positional signals, the attention block is permutation-equivariant over sequence positions and thus has no intrinsic notion of temporal order beyond causal masking. We argue that additive positional embeddings make the attention mechanism only superficially sensitive to sequence order: positional information is entangled with item embedding semantics, propagates weakly in deep architectures, and limits the ability to capture rich sequential patterns. To address these limitations, we introduce a kernelized self-attention mechanism, where a learnable positional kernel operates purely in the position space, disentangled from semantic similarity, and directly modulates attention weights. When applied per attention block, this kernel enables adaptive multi-scale sequential modeling. Experiments on standard next-item prediction benchmarks show that our positional kernel attention consistently improves over strong competing baselines.

Abstract

Sequential Recommendation

Explore similar work

Position-Aware Sequential Attention for Accurate Next Item Recommendations

Candidate-dependent residual routing trades top-of-list accuracy for recall in sequential recommendation

SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation