From Feature Interaction to Feature Transport - A Unified Block for Scalable Recommendation Models

Zichen Luo; Jiachen Guo; Keming Gu; Jie Zhang

cs.IRAug 31, 2026

From Feature Interaction to Feature Transport - A Unified Block for Scalable Recommendation Models

Authors: Zichen Luo, Jiachen Guo, Keming Gu, Jie Zhang

Organizations: Tianjin University Jinnan Qu, Tianjin Shi, China

Abstract

Unified recommendation models aim to jointly model non-sequential multi-field features and sequential user behaviors, but existing interaction-centric designs mainly focus on mixing heterogeneous tokens within each layer. We argue that scalable unified recommendation also requires controlling how intent information is carried, filtered, and preserved across stacked blocks. Inspired by flow-based representation dynamics, we introduce feature transport, a view that treats deep unified recommendation as a discrete context-conditioned representation evolution process. We propose CRAFT, a Contextual Residual Adaptive Feature Transport block, which summarizes non-sequential features into a reliability-aware contextual field and uses it to generate residual displacement and memory-preserving signals for intent and sequence representations. In this way, non-sequential context acts as an active controller of representation evolution rather than a passive object of interaction. In the TAAC2026 advertising recommendation competition, CRAFT achieves a test AUC of 0.838090, surpassing the previous leaderboard-best score of 0.83798. Scaling experiments further show that CRAFT benefits from both depth and width expansion: stacking CRAFT to six blocks improves test AUC to 0.838148, while increasing the hidden dimension reaches 0.838106. These results demonstrate the effectiveness, scalability, and generalization potential of the feature transport paradigm. Source code: https://github.com/AshleyLuo001/CRAFT

Figures & tables

Figure 1. Effect of CRAFT and distillation on predictive performance and representation quality.

Figure 2. Overall architecture of CRAFT. Heterogeneous non-sequential and sequential features are first tokenized into non-sequential tokens, sequence summaries, and intent tokens. Each stacked CRAFT block performs contextual residual feature transport, where non-sequential context generates adaptive residual transformations to guide intent and sequence representations before unified interaction. This design enables layer-wise feature transport across heterogeneous signals while preserving stackability for depth scaling.

Paradigm	Model	Test AUC	$\Delta$	Params	GFLOPs	Intent-Rank	Seq-Rank
Original	Original Baseline	0.832688	–	182.21M	–	–	–
Interaction	Optimized Baseline	0.837165	+4.48	307.27M	6.955	74.02	8.06
Interaction	Optimized Baseline + Distill	0.837983	+5.30	334.87M	9.889	57.68	8.89
Transport	CRAFT	0.837231	+4.54	308.69M	6.970	77.22	9.86
Transport	CRAFT + Distill	0.838090	+5.40	308.69M	6.970	84.33	10.90

Table 1. Main results on TAAC2026.

\Delta

denotes the AUC improvement over the original baseline, measured in per mille (‰). Params are counted at inference time; GFLOPs exclude embedding lookup.

Model	Blocks	$d_{\mathrm{model}}$	Test AUC	GFLOPs
CRAFT-1L	1	420	0.837975	3.960
CRAFT-2L	2	420	0.838090	6.970
CRAFT-4L	4	420	0.838093	12.992
CRAFT-6L	6	420	0.838148	19.013
CRAFT-D600	2	600	0.838106	13.679

Table 2. Depth and width scaling of CRAFT. GFLOPs are measured per sample and exclude embedding lookup.

Figure 3. Block-stacking diagnostic for CRAFT. We compare the effective ranks of query carriers, final outputs, and sequence summaries when increasing the number of CRAFT blocks from 1 to 6. Query carriers show a large rank increase from 70.4 to 158.2, while the final-output rank decreases and the sequence-summary rank remains low. This indicates that depth scaling mainly expands the query-mediated transport space rather than uniformly increasing the dimensionality of all representation spaces.

Group	Raw form	Count	Model-side token type
Basic columns	IDs and timestamps	5	Supervision/context fields
User integer fields	Scalar IDs / ID arrays	54	User non-sequential tokens
User dense fields	Float vectors / arrays	17	Dense and pair-aware tokens
Item integer fields	Scalar IDs / ID arrays	17	Item non-sequential tokens
Item dense fields	Float vectors	4	Item dense tokens
Behavior sequences	Ordered ID sequences	45 in 4 domains	Sequence tokens

Table 3. Feature groups in the TAAC2026 advertising recommendation data.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Raw form	Encoder	Output
User IDs	Sparse IDs	RankMixer	User tokens
Item IDs	Sparse IDs	RankMixer	Item tokens
Multi-value IDs	ID arrays	Masked pool	Field tokens
Dense fields	Vectors	MLP / structured	Dense tokens
ID-dense pairs	Paired arrays	Pair encoder	Pair tokens
Behaviors	Sequences	Seq encoder	Seq tokens

Appendix

Table 4. Tokenization pipeline for heterogeneous recommendation features.

Pattern	Observed structure	Encoder	Output
Plain vector	Continuous vector	BN + MLP	1 token
Segmented vector	Repeated sub-vectors	Masked segment pool	1 token
Positioned segments	Position + segments + tail	Pos + segment + tail fusion	1 token
Embedding + metadata	Vector prefix + suffix	Prefix projection	1 token
Item dense	Item-side vectors	Field pool	1 token
ID-dense pair	Aligned ID/value arrays	Pair encoder	Field token

Appendix

Table 5. Structure-aware dense tokenization. Dense fields are grouped by observed structural pattern rather than anonymized field ID.

Feature type	Dataset pattern	Operation
Scalar ID	Missing code	Reserved embedding
ID array	Padding code	Masked aggregation
Dense vector	Invalid values	Sanitization + norm
Segmented dense	Empty segment	Segment mask
Behavior sequence	Padded events	Sequence mask
ID-dense pair	Invalid pair	Pair-validity mask

Appendix

Table 6. Robust handling of missingness and padding.

Component	Setting	Applied to
Sparse optimizer	Adagrad	Sparse embeddings
Dense optimizer	MuonPlus/AdamW-style update	Dense parameters
EMA	Dense-parameter EMA	Validation/checkpoint selection
Embedding refresh	Early sparse reinitialization	Sparse embeddings
LR schedule	Step-wise cosine decay	Dense optimizer
Model selection	Validation-based early stopping	Whole model

Appendix

Table 7. Optimization and stabilization settings.

Figure 4. Comparison of horizontal and vertical scaling routes. Horizontal scaling increases

d_{\mathrm{model}}

at fixed depth, while vertical scaling increases the number of CRAFT blocks at fixed

d_{\mathrm{model}}=420

Stage	Teacher source	Student initialization	Distillation weight
Supervised anchor	None	Random	0
First-generation distillation	Strong supervised anchor	Random	0.50
Second-generation distillation	First-generation distilled model	Random	0.65

Appendix

Table 8. Progressive distillation protocol.

Model	Trans	Distill	T-AUC	Params
Optim. Base	No	No	0.837165	307.27M
CRAFT	Yes	No	0.837231	308.69M
Optim. Base + Distill	No	Yes	0.837983	334.87M
CRAFT + Distill	Yes	Yes	0.838090	308.69M

Appendix

Table 9. CRAFT ablation results. “Trans” indicates contextual feature transport; “T-AUC” denotes official test AUC.

Model	Test AUC	I-Rank	I-Cos	NS-Rank	NS-Cos	Seq-Rank	Seq-Cos	Final-Rank	Final-Cos
Optimized Baseline	0.837165	74.02	0.171	187.68	0.188	8.06	0.476	35.67	0.263
CRAFT	0.837231	77.22	0.210	180.98	0.127	9.86	0.485	33.29	0.263
Optimized Baseline + Distill	0.837983	57.68	0.225	207.44	0.210	8.89	0.477	47.55	0.239
CRAFT + Distill	0.838090	84.33	0.233	202.70	0.125	10.90	0.514	32.15	0.270

Appendix

Table 10. Representation diagnostics on sampled validation batches. Rank denotes effective rank; Cos denotes average absolute cosine similarity.

Panel of Figure 5 — Figure 5. Qualitative projections of layer-2 hidden representations in CRAFT-2L, CRAFT-4L, and CRAFT-6L. Rows correspond to non-sequential, sequence-summary, and intent-carrier representations, and columns correspond to model depth. Both PCA and t-SNE show that the clearest depth-dependent restructuring occurs in the intent-carrier space.

Model	Blocks	$d_{\mathrm{model}}$	Test AUC	Params	GFLOPs
CRAFT-2L	2	420	0.838090	308.69M	6.970
CRAFT-4L	4	420	0.838093	330.63M	12.992
CRAFT-6L	6	420	0.838148	352.56M	19.013
CRAFT-D600	2	600	0.838106	375.41M	13.679

Appendix

Table 11. Clean CRAFT scaling results.

Figure 6. Width-scaling diagnostic for CRAFT. Increasing

d_{\mathrm{model}}

from 420 to 600 increases the combined effective-rank budget of intent-carrier and non-sequential context representations.

Direction	Controlled change	Observed outcome
Dense tokenization	Flat projection for structured dense vectors	Weaker scaling behavior
Segmented dense fields	Remove segment-aware pooling	Lower robustness
Stable dense embeddings	Use metadata suffix with vector prefix	Worse generalization
Dense tokenizer complexity	Add heavier self-attention tokenizer	Higher cost and instability
Optimization	Use one optimizer for all parameters	Weaker stability
FFN expansion	Increase hidden multiplier	Limited or negative gain

Appendix

Table 12. Negative or limited-gain findings from controlled explorations.

Figure 7. Layer-wise evolution of sequence effective rank. Layer 0 denotes the sequence summary before the first modeling block. The interaction baseline progressively compresses the sequence space, whereas CRAFT enables later blocks to recover task-relevant representation directions. Effective rank is used as a geometric diagnostic rather than a direct measure of representation quality.

Figure 8. KMeans-MI heatmap of label-discriminative information across CRAFT representation spaces. KMeans clustering with $K=128$ is applied to each representation, and mutual information is computed between cluster assignments and labels.

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 504 papers

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds
Apr 15, 2026Yifeng Zhou, Yuehong Hu, Zhixiang Feng +9Recommender Systems Sequential Recommendation
Tencent Inc., Shenzhen, China
UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation
Aug 17, 2026Rongcheng Lin, Yan Sun, Jamey Zhang +4Recommender Systems Feature Interaction Modeling
Meta Menlo Park, USA
WHALE: A Scalable Unified Model for Recommendation with Wukong-HSTU Architecture
Jul 19, 2026Renqin Cai, Dawei Sun, Yuanjun Yao +8Sequential Recommendation Feature Interaction Modeling
Meta Platforms, Inc. USA

Apr 15, 2026J/K move · Enter open · S save

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds

Yifeng Zhou, Yuehong Hu, Zhixiang Feng, Junwei Pan+8

Tencent Inc., Shenzhen, China

Recommender Systems Sequential Recommendation Feature Interaction Modeling

Open paper AskPDF arXiv

Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling.

Abstract

Recommender Systems

Explore similar work

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds

UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation

WHALE: A Scalable Unified Model for Recommendation with Wukong-HSTU Architecture