Industrial recommenders rely on two model families that have evolved largely independently: feature-interaction models over multi-field user/item features, and sequential models over user-behavior histories. Production systems couple them only loosely. To unify the two, we present UniDot, a novel architecture for post-click conversion prediction built from the factorization-machine (FM) point of view: the embedding inner product---which powers collaborative filtering and lets a recommender generalize to unseen user--item pairs---is the same primitive as attention's query dot key scoring, so a single dot-product of tokens can underlie both feature interaction and sequence modeling. UniDot tokenizes non-sequential fields and multi-domain behavioral sequences into one shared token space and stacks a single macro-block in which a token-mixing bus and a sequence-retrieval bus (item tokens cross-attending the histories) run in parallel and exchange state each layer through an MLP-Mixer fusion, while an FM Highway carries explicit per-layer dot-product interactions around the residual stack directly to the classifier. The sequence side is embedded once per forward pass and shared by all consumers, bounding inference latency. Trained with a dual sparse/dense (Adagrad + Muon) optimizer, an auxiliary conversion-delay head, and multi-path mutual learning, UniDot finished as the runner-up on the Industrial track of the TAAC KDD Cup 2026.
Figures & tables
Figure 1. UniDot architecture (§ 4 ): two buses co-evolve through L macro-blocks, exchange state via the per-layer MLP-Mixer fusion, and the FM Highway routes explicit per-layer dot-products past the fuser to the classifier.
Rank
AUC ↑
Gap to #1
1
0.83254
—
2 (UniDot, ours)
0.83217
0.037%
(UniDot, single path) ⋆
0.83184
0.070%
3
0.83145
0.109%
4
0.83080
0.174%
5
0.83073
0.181%
Table 1. Final Industrial-track leaderboard (round 2, top 10). ⋆ Serving a single path of our all-data DML model (1 × inference cost) would still rank second—multi-path training pulls even a single model to a better minimum (§ 4.6 ).
Figure 2. One sequence trunk (§ 4.3 ).
Figure 3. MultiChannelSeqPool (§ 4.4 ): positions are gated into C channels by independent state-conditioned sigmoid weights.
Table 2. Training dynamics by epoch of the d=128 , N=2 multi-path run. Eval / Test (sel.) : the model-selection run ( remote , 10% held-out). Test (final) : the final submission, retrained on all data with EMA. LL = LogLoss (train over both paths, eval over their mean); “—” = not submitted.
#
Change
Test AUC
Δ
0
competition baseline
0.81398
—
1
+ UniDot
0.82500
+1.102
2
+ item-id hash embedding
0.82565
+0.064
3
tune token counts
0.82609
+0.044
4
tune batch size
0.82704
+0.095
5
+ more FM-Highway dot products
0.82722
+0.018
Table 3. Round-2 incremental improvements.
Variant
AUC ↑
LogLoss ↓
Δ AUC
Full UniDot (6 layers)
0.83657
0.2151
—
2 macro-layers
0.83600
0.2157
−0.057%
4 macro-layers
0.83603
0.2154
−0.054%
8 macro-layers
0.83624
0.2152
−0.033%
10 macro-layers
0.83615
0.2152
−0.042%
− FM Highway (all)
0.83530
0.2159
−0.127%
Table 4. Component ablations ( tiny mode, single-path, d=64 ; in-distribution held-out eval). Δ is AUC vs. the full model.
Configuration
Dense par.
GFLOP
Test AUC ↑
Δ
d=64 , sparse fold × 2
18.8M
5.9
—
no win
d=64 (baseline)
18.8M
5.9
0.82993
—
d=96
36.7M
12.4
0.83024
+0.031%
d=128
60.3M
21.2
0.83043
+0.050%
d=64 , 2-path
37.6M
11.7
0.83128
+0.135%
d=128 , 2-path
120.6M
42.5
0.83196
+0.203%
Table 5. Scaling study.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
Industrial track
Train / test samples
35M / 12M (2M, r1)
Columns (categories)
142 (7)
User Int / Dense fids
54 / 17
Item Int / Dense fids
17 / 4
Pre-trained emb fids (u/i)
7 / 4
Aligned per-position fids
10
Appendix
Table 6. Industrial-track dataset and schema (second round; first round in parentheses).
Mode
Data (train / eval)
GPUs
Role
tiny
4M / 1M, random hold-out
1
rapid design A/B
remote
35M / 10% random hold-out
4
full-scale validation
final
35M / no hold-out
4
submission (EMA weights)
Appendix
Table 7. Training/evaluation modes.
Group
Setting
Macro-layers L / mix-blocks-per-layer W
6 / 2
dmodel
128
Token-mix block
Wukong (parallel LCB + FMB, rank 32)
User NS / user emb tokens
8 / 2 ( × 7 fids) ⇒Tu=22
Item NS tokens (mix / retrieval bus)
4 / 16 (+ 4 emb fids × 2) ⇒Ti=12 , Tih=24
Fuse / classifier hidden mult
4 / 8
Appendix
Table 8. Submitted configuration behind the reported numbers.
Figure 6. Field-aware feature embedding (FAFE, § C.3 ): a DIN-style attention pool scores each value of a multi-value field against the ranking candidate, yielding a candidate-dependent field token; all other fields keep the static NCB pooling.
Figure 7. Data scaling ( tiny mode, single-path d=64 ): held-out AUC (left, blue ) and LogLoss (right, red ) vs. training examples ( log2 axis). Both improve log-linearly across the full 4M–32M range.
Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling.
Post-click conversion rate (pCVR) prediction requires jointly modeling feature interactions and sequential user behaviors. The KDD Cup 2026 Tencent UniRec Challenge calls for a unified architecture addressing both. We observe that existing unified architectures often generate query tokens---the central information hub---with projection-based multi-layer perceptrons (MLPs), without explicit token-to-query attention for refining the query side. We propose QueryFormer, centered on a stackable unified field--sequence block that bridges non-sequential multi-field features and behavioral sequences, and provide a latency-aware scaling study over view width H, model width, depth, data, and compute. The block generates queries through cross-attention and packs sequence queries into shared-parameter attention. QueryFormer secured 1st place in the Industrial Track, achieving an official test area under the ROC curve (AUC) of 0.83254; a modest post-competition scale-up reached 0.832713. Within our grid, H-scaling improves validation AUC from 0.84540 to 0.84615 and beats HyFormer at comparable budgets. Ablation identifies query generation as the largest contributor. Packed shared-parameter cross-attention keeps H=8 inference latency to only 1.89x that of H=1, positioning the bridge as an efficient stackable unified block.
Unified recommendation models aim to jointly model non-sequential multi-field features and sequential user behaviors, but existing interaction-centric designs mainly focus on mixing heterogeneous tokens within each layer. We argue that scalable unified recommendation also requires controlling how intent information is carried, filtered, and preserved across stacked blocks. Inspired by flow-based representation dynamics, we introduce feature transport, a view that treats deep unified recommendation as a discrete context-conditioned representation evolution process. We propose CRAFT, a Contextual Residual Adaptive Feature Transport block, which summarizes non-sequential features into a reliability-aware contextual field and uses it to generate residual displacement and memory-preserving signals for intent and sequence representations. In this way, non-sequential context acts as an active controller of representation evolution rather than a passive object of interaction. In the TAAC2026 advertising recommendation competition, CRAFT achieves a test AUC of 0.838090, surpassing the previous leaderboard-best score of 0.83798. Scaling experiments further show that CRAFT benefits from both depth and width expansion: stacking CRAFT to six blocks improves test AUC to 0.838148, while increasing the hidden dimension reaches 0.838106. These results demonstrate the effectiveness, scalability, and generalization potential of the feature transport paradigm. Source code: https://github.com/AshleyLuo001/CRAFT