TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds

Yifeng Zhou; Yuehong Hu; Zhixiang Feng; Junwei Pan; Kaihui Wu; Hanyong Li; Shangyu Zhang; Shudong Huang

cs.IRApr 15, 2026

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds

Authors: Yifeng Zhou, Yuehong Hu, Zhixiang Feng, Junwei Pan, Kaihui Wu, Hanyong Li, Shangyu Zhang, Shudong Huang,

Organizations: Tencent Inc., Shenzhen, China

Abstract

Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling.

Figures & tables

Figure 1. The central tradeoff in unified recommendation modeling. Left: representation discriminability measured by mutual information (MI). Right: dimensional robustness measured by the singular value spectrum and effective rank. Compared with a sequence-only Transformer ( green ), naive unification of sequential and non-sequential features with a vanilla Transformer ( blue ) improves discriminability, but also induces much steeper spectral decay and lower effective rank in sequential representations, revealing sequential collapse propagation . The proposed BFTS and NLIR mitigate this collapse, restore dimensional robustness, and further improve discriminability.

Figure 2. Overview of TokenFormer . TokenFormer represents multi-field features $\mathcal{F}$ , sequential behavior tokens $\mathcal{T}$ , and target features $\mathcal{V}$ as a unified token stream, which is processed by stacked Unified Interaction Blocks (UIBs). Each UIB combines the proposed Bottom-Full-Top-Sliding (BFTS) attention design, which applies full causal attention in shallow layers and shrinking SWA in deeper layers, with the Non-Linear Interaction Representation (NLIR) for multiplicative feature interaction.

Paradigm	Model	Macro AUC	$\Delta$ (‰)	Params	GFLOPs
User-Centric	Transformer	0.85467	-	3.41M	3.7G
	HSTU	0.85694	+2.27	0.50M	0.3G
	HSTU-Ultra	0.85762	+2.95	0.50M	0.2G
	TokenFormer-T	0.85967	+5.00	0.48M	0.8G
	TokenFormer-S	0.86043	+5.76	3.77M	3.5G
	TokenFormer-M	0.86116	+6.49	5.64M	5.3G

Table 1. Main results on KuaiRand-27k.

\Delta

denotes the AUC improvement relative to the Transformer baseline in each category, measured in per mille (‰).

Figure 3. Discriminability analysis across varying cluster numbers $K$ . To clearly illustrate the performance gains, we report the scaled MI. The proposed BFTS and NLIR can effectively improve discriminability for output embeddings.

Figure 4. Block-wise effective-rank trajectory of sequential behavioral tokens ( $\mathcal{T}$ ) for the Vanilla Transformer , $+$ BFTS and $+$ NLIR . We track the output across each block include: Attention output (Eq. 11 ), the successive residual additions (Eq. 18 ), and the FFN output (Eq. 19 ).

Figure 5. Effective rank comparison of sequential behavioral tokens ( $\mathcal{T}$ ) across layers. To demonstrate the protective effect of the non-linear gating mechanism, we compare the layer-wise outputs given the input $\mathbf{X}^{(l)}$ . While the linear attention output (Eq. 11 ) suffers from sharply rank degradation, the non-linear gated output of TokenFormer (Eq. 17 ) significantly mitigates this collapse in the early stages.

Figure 6. Evolution of attention patterns. Top: Vanilla Transformer suffers from redundant revisiting of static fields in last layers; once the initial cross-interactions is completed in shallow layers, such back-attend introduces noise. Bottom: TokenFormer decouple this by BFTS. It establishes cross-interactions in shallow layers and switches to shrinking window SWA in deeper layers to refine sequential representations.

Figure 7. Left: Attention receptive field distributions . Histograms left to right correspond to first and last layer, respectively. In shallow layers, TokenFormer exhibits a wider receptive field to facilitate comprehensive cross-integration. In deeper layers, while the Transformer maintains a broad distribution due to redundant back-attention to static tokens. Right: AUC for ablation study.

Figure 8. Efficiency and effectiveness trade-offs of various BFTS configurations. Explicit attention constraints of BFTS improve AUC, while linear-complexity SWA simultaneously reduces GFLOPs

Model	Depth	Dim	Head Num
TokenFormer-T	4	64	4
TokenFormer-S	4	256	4
TokenFormer-M	6	256	4
TokenFormer-L	8	256	8

Table 2. Architectural configurations of TokenFormer across different model scales.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9. Block-wise effective-rank trajectory of sequential behavioral tokens ( $\mathcal{T}$ ) for the Vanilla Transformer , Transformer with only sequence tokens, TokenFormer , $+$ BFTS and $+$ NLIR . We track the output across each block include: Attention output (Eq. 11 ), the successive residual additions (Eq. 18 ), and the FFN output (Eq. 19 ).

Figure 10. Per-layer normalized singular value spectra. Compared with the non-gated baseline, TokenFormer exhibits less top-heavy spectra, indicating a more balanced allocation of variance across latent dimensions.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 504 papers

From Feature Interaction to Feature Transport - A Unified Block for Scalable Recommendation Models
Aug 31, 2026Zichen Luo, Jiachen Guo, Keming Gu +1Recommender Systems Feature Interaction Modeling
Tianjin University Jinnan Qu, Tianjin Shi, China
UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation
Aug 17, 2026Rongcheng Lin, Yan Sun, Jamey Zhang +4Recommender Systems Feature Interaction Modeling
Meta Menlo Park, USA
SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
Jul 27, 2026Yu Cui, Yi Xu, Jiahao Wang +6Representation Collapse Transformer
Zhejiang University Hangzhou, China · Alibaba Group Beijing, China

Aug 31, 2026J/K move · Enter open · S save

From Feature Interaction to Feature Transport - A Unified Block for Scalable Recommendation Models

Zichen Luo, Jiachen Guo, Keming Gu, Jie Zhang

Tianjin University Jinnan Qu, Tianjin Shi, China

Recommender Systems Feature Interaction Modeling Sequential Recommendation

Open paper AskPDF arXiv

Unified recommendation models aim to jointly model non-sequential multi-field features and sequential user behaviors, but existing interaction-centric designs mainly focus on mixing heterogeneous tokens within each layer. We argue that scalable unified recommendation also requires controlling how intent information is carried, filtered, and preserved across stacked blocks. Inspired by flow-based representation dynamics, we introduce feature transport, a view that treats deep unified recommendation as a discrete context-conditioned representation evolution process. We propose CRAFT, a Contextual Residual Adaptive Feature Transport block, which summarizes non-sequential features into a reliability-aware contextual field and uses it to generate residual displacement and memory-preserving signals for intent and sequence representations. In this way, non-sequential context acts as an active controller of representation evolution rather than a passive object of interaction. In the TAAC2026 advertising recommendation competition, CRAFT achieves a test AUC of 0.838090, surpassing the previous leaderboard-best score of 0.83798. Scaling experiments further show that CRAFT benefits from both depth and width expansion: stacking CRAFT to six blocks improves test AUC to 0.838148, while increasing the hidden dimension reaches 0.838106. These results demonstrate the effectiveness, scalability, and generalization potential of the feature transport paradigm. Source code: https://github.com/AshleyLuo001/CRAFT

Abstract

Recommender Systems

Explore similar work

From Feature Interaction to Feature Transport - A Unified Block for Scalable Recommendation Models

UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation

SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation