HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation

Lei Xin; Yuhao Zheng; Ke Cheng; Changjiang Jiang; Zifan Zhang; Fanhu Zeng

cs.IRFeb 20, 2026

HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation

Authors: Lei Xin, Yuhao Zheng, Ke Cheng, Changjiang Jiang, Zifan Zhang, Fanhu Zeng

Organizations: Shanghai Dewu Information Group · Wuhan University · USTC · Beihang University

Abstract

Modeling long sequences of user behaviors has emerged as a critical frontier in generative recommendation. However, existing solutions face a dilemma: linear attention mechanisms achieve efficiency at the cost of retrieval precision due to limited state capacity, while softmax attention suffers from prohibitive computational overhead. To address this challenge, we propose HyTRec, a model featuring a Hybrid Attention architecture that explicitly decouples long-term stable preferences from short-term intent spikes. By assigning massive historical sequences to a linear attention branch and reserving a specialized softmax attention branch for recent interactions, our approach restores precise retrieval capabilities within industrial-scale contexts involving ten thousand interactions. To mitigate the lag in capturing rapid interest drifts within the linear layers, we furthermore design Temporal-Aware Delta Network (TADN) to dynamically upweight fresh behavioral signals while effectively suppressing historical noise. Empirical results on industrial-scale datasets confirm the superiority that our model maintains linear inference speed and outperforms strong baselines, notably delivering over 8% improvement in Hit Rate for users with ultra-long sequences with great efficiency.

Figures & tables

Figure 1: The Evolution of Attention Mechanisms.

Model	Recall@10	GAUC	AUC
SASRec	0.0235	0.4224	0.4533
Ours	0.0317	0.8758	0.9327

Table 1: Cross-domain transfer experiments based on the 2022 Huawei Advertising Challenge Dataset. The table presents the performance on Recall@10, GAUC and AUC metrics.

Model	Beauty			Electronics			Movies&TV
Model	H@500	NDCG@500	AUC	H@500	NDCG@500	AUC	H@500	NDCG@500	AUC
GRU4Rec	0.5263	0.1648	0.8397	0.2854	0.0752	0.8353	0.5380	0.1261	0.9307
SASRec	0.5776	0.3280	0.8497	0.3003	0.0945	0.8625	0.6979	0.6226	0.9370
HSTU	0.5838	0.4480	0.8602	0.3156	0.1244	0.8550	0.7042	0.6278	0.9372
DIN	0.5748	0.4297	0.8556	0.2700	0.1063	0.8560	0.7155	0.6236	0.9263
GLA	0.5732	0.4157	0.8548	0.3176	0.1042	0.8657	0.7053	0.6153	0.9124

Table 2: We compare with various methods on three public Amazon datasets in terms of H@500, NDCG@500 and AUC. The best and second-best methods are marked in bold and underlined, respectively.

Figure 3: We compare the training throughput of models with the same parameter scale on a single V100 GPU under different behavior sequence lengths.

Model	TADN	Short-Term Attention	H@500	NDCG@500	AUC
HyTRec	$\times$	$\times$	0.6043	0.3130	0.8355
	$\times$	$\checkmark$	0.6343	0.3300	0.8505
	$\checkmark$	$\times$	0.6493	0.3380	0.8575
	$\checkmark$	$\checkmark$	0.6643	0.3480	0.8655

Table 3: Ablation experiments of the model on the Amazon Beauty dataset.

Figure 4: Performance Comparison Under Different Hybrid Attention Ratios.

Badcase Type	H@500	NDCG@500	AUC
Cold-start for new users	0.6622	0.3473	0.8616
Silent old users	0.6340	0.2808	0.8661

Table 4: Model performance on several typical business bad cases.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Performance Comparison Under Different Number of Experts

Figure 6: Performance Comparison Under Different Number of Attention Heads

Model	Efficiency	Temporal Info	Short-term Intent	Rec Capability
Softmax Attention	$\times$	$\times$	$\times$	$\times$
Linear Attention	$\checkmark$	$\times$	$\times$	$\times$
Qwen-next	$\checkmark$	$\times$	$\times$	$\times$
Ours	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Appendix

Table 5: Comparison of different models on key evaluation metrics. HyTRec is the first work in the recommendation field to achieve a balance between efficiency and accuracy based on hybrid attention, enabling effective modeling of long user behavior sequences.

nums_head	H@500	NDCG@500	AUC	Latency
1	0.6514	0.3467	0.8717	1.1981
2	0.6623	0.3459	0.8634	1.1308
4	0.6643	0.3480	0.8655	1.1092
8	0.6579	0.3493	0.8690	1.1806

Appendix

Table 6: Comparison of different nums_head on key recommendation evaluation metrics. The table shows the performance (H@500, NDCG@500, AUC) and efficiency (Latency) under different nums_head settings.

Expert_nums	H@500	NDCG@500	AUC	Latency
4	0.6643	0.3480	0.8655	1.1092
6	0.6624	0.3449	0.8677	1.6221
8	0.6579	0.3434	0.8729	2.0128

Appendix

Table 7: Comparison of different expert_nums on key recommendation evaluation metrics. The table presents the performance and latency under various expert number settings.

Ratio	H@500	NDCG@500	AUC	Latency
2:1	0.6559	0.3452	0.8649	1.0802
3:1	0.6643	0.3480	0.8655	1.1092
4:1	0.6527	0.3478	0.8672	1.4397
5:1	0.6650	0.3474	0.8619	1.8070
6:1	0.6637	0.3507	0.8675	2.5296

Appendix

Table 8: Comparison of different Ratio on key recommendation evaluation metrics. The table presents the performance and latency under various ratio settings.

Sequential Recommendation

Sequential recommendation predicts what a user will choose next from ordered interactions, adapting to changing interests and context.

See all 136 papers

SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching
Sep 8, 2026Lin Guan, Jia-Qi Yang, Zhishan Zhao +15Sequential Recommendation Long-Context Modeling
ByteDance
DP-Rec: Towards Dynamic Patching for Efficient Long-Sequence Recommendation
Sep 26, 2026Dwipam Katariya, Thomas Caputo, Akshat Shreemali +6Sequential Recommendation Sequence Modeling
Capital One, AI Foundations McLean, VA, USA
BehaveRec: When Does Long-Term Behavioral History Help Sequential Recommendation? A Study with Session, History, and Community Signals
Oct 26, 2026Aysenur Inan, Reza Yousefi Maragheh, Jayesh Uddhav Kudase +4Sequential Recommendation Recommender Systems
Walmart Global Tech, Sunnyvale, CA, USA

Sep 8, 2026J/K move · Enter open · S save

SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching

Lin Guan, Jia-Qi Yang, Zhishan Zhao, Jiaqi Huang+14

ByteDance

Sequential Recommendation Long-Context Modeling User Preference Modeling

Open paper AskPDF arXiv

Modeling long-term user behavior is central to sequential recommendation and billion-scale industrial recommender systems, yet production ranking models operate under strict latency, memory, communication, and training-throughput constraints. At the 100K scale, the challenge extends beyond attention complexity: raw sequence features must be stored, transferred, and repeatedly processed during training and online serving. Existing approaches based on history truncation, multi-stage behavior retrieval, compressed lifelong histories, or train-short/infer-long extrapolation either weaken end-to-end optimization or retain substantial length-dependent cost. We present SequenceO1, an end-to-end framework for ultra-long user behavior sequence modeling, deployed at full traffic on Douyin with histories of up to 100K interactions. SequenceO1 follows a compress-then-reason design. Its Sketch Attention (SA) uses learnable prototypes and prototype-wise normalization to compress the raw history into a fixed-size, target-agnostic user representation. Target-conditioned Stacked Target-to-History Cross Attention (STCA) then models complementary time scales: a recent 10K suffix for short-term interests and the compact sketch for long-term preferences. To make training and inference practical, SequenceO1 combines low-rank user representation caching, multi-request user-level batching, pipeline lift, and a fused FlashSA kernel to amortize feature storage, communication, and computation across targets, training instances, and consecutive requests. Production experiments show consistent offline and online gains, while the compact cached sketch retains most of the benefit of directly scaling end-to-end sequence ranking to 100K. These results provide a practical model-system approach to efficient attention, sequence compression, and scalable long-sequence and long-context recommendation systems.

Abstract

Sequential Recommendation

Explore similar work

SequenceO1: End-to-End Ultra-Long (100K) Sequence Modeling in Recommendation with Low-Rank Caching

DP-Rec: Towards Dynamic Patching for Efficient Long-Sequence Recommendation

BehaveRec: When Does Long-Term Behavioral History Help Sequential Recommendation? A Study with Session, History, and Community Signals