Industrial recommendation faces a resolution--complexity dilemma: dense interactions make detailed modeling of long behavior histories expensive, while compact history summaries restrict the evidence available to prediction. A unified backbone must preserve rich behavioral details within strict serving budgets. We introduce TokenFormer V2, a unified architecture that explicitly decouples representation resolution from interaction density. Its group-wise token layout jointly encodes behavior segments with non-sequential context, expands interaction ranges via hierarchical group merging, and coordinates distributed groups through NS Fusion and Global Sync modules. Through this local--global decomposition paradigm, computation is concentrated on structured interactions while preserving fine-grained evidence, ensuring that every encoded event remains directly accessible to the Full-Resolution Readout module. On the Tencent Advertising Algorithm Competition 2026 (TAAC 2026) benchmark, TokenFormer V2 achieves top-ranking accuracy among unified recommendation baselines while using \textbf{5.1× fewer FLOPs} than leading baselines, sustaining its advantage across long histories and imbalanced behavior domains. Furthermore, in billion-parameter evaluations on WeChat Channels advertising, it consistently outperforms state-of-the-art industrial baselines, yielding up to a +0.07 pp overall AUC gain. In live online A/B testing against the production model, TokenFormer V2 delivers a statistically significant 2.33% increase in GMV. These results establish TokenFormer V2 as a battle-tested blueprint for scalable, full-resolution behavior modeling in industrial recommendation.
Figures & tables
Figure 1. Overview of TokenFormer V2, which separates representation resolution from interaction density. (a) Full-resolution sequence passes through hierarchical interaction layers and remains directly accessible to the candidate-aware Full-Resolution Readout. (b) Local Group Attention, NS Fusion, and Global Sync coordinate local evidence and shared context; Hierarchical Group Merging expands group scope without compression, preserving individual representations throughout the backbone. The supplied architecture diagram has two panels. Panel a traces the Full-Resolution Sequence through repeated interaction layers, candidate-aware cross-attention, and Full-Resolution Readout to task logits. Panel b illustrates Local Group Attention, Global Sync, NS Fusion, and Hierarchical Group Merging with expanding group scope. Behavior, non-sequential, group relay, and readout tokens are distinguished by color.
Symbol
Token type
Count
S
Behavior tokens
nS
N
Non-sequential tokens
nN
G
Group relay tokens
nG
O
Readout tokens
nO
Table 1. Token types and counts. Each token has width d .
Model
Test AUC ↑
Log loss ↓
GFLOPs
Train (samples/s)
Inference ( μ s/sample)
Dense params
Transformer ( Vaswani et al., 2017 )
0.8168
0.2288
32.59
161
2921
∼ 11M
TokenFormer ( Zhou et al., 2026 )
0.8198
0.2281
28.52
333
1945
OneTrans ( Zhang et al., 2026 )
0.8192
0.2281
1.90
923
157
HyFormer ( Huang et al., 2026b )
0.8204
0.2270
3.13
900
131
EST ( Liu et al., 2026 )
0.8159
0.2290
2.75
779
271
Kunlun ( Hou et al., 2026 )
0.8176
0.2288
2.05
929
171
Table 2. Accuracy and efficiency on TAAC 2026 at approximately 11M dense parameters. Best predictive metrics are bold; tied second-best values are underlined.
Figure 2. Validation AUC over one training epoch. TokenFormer V2 establishes its lead early and remains ahead at every measured checkpoint. Validation AUC against training steps for TokenFormer V2, MixFormer, HyFormer and TokenFormer. TokenFormer V2 has the highest curve at every measured checkpoint.
Figure 3. Training time and peak memory as fully populated histories grow from 384 to 3,072 events, with batch size 256. Two log-log plots of training time and peak memory against history length. TokenFormer V2 follows growth close to HyFormer and below MixFormer. TokenFormer requires substantially more time and exceeds device memory at the largest history.
1,536 events
3,072 events
Model
AUC
GFLOPs
AUC
GFLOPs
TokenFormer
0.8198
28.52
0.8201
68.94
HyFormer
0.8204
3.13
0.8204
6.22
MixFormer
0.8211
19.75
0.8210
39.50
TokenFormer V2
0.8221
3.86
0.8219
9.07
Table 3. Accuracy and forward cost after doubling the history. TokenFormer V2 leads at both input budgets.
Figure 4. AUC gains across dominant-domain and inactive-domain user segments. TokenFormer V2 improves every reported segment, with larger gains away from the dominant-domain majority. Grouped bars of AUC gains over HyFormer and MixFormer. All gains are positive. Gains over MixFormer are about 0.08 percentage points for users dominated by D, 0.17 to 0.22 for users dominated by A to C, and up to 0.30 for users with an inactive domain.
Figure 5. Predictive information in the common 128-dimensional representation. TokenFormer V2 leads in probe AUC and the MI lower bound (a), and in normalized class separation (b). Two dual-axis bar charts compare TokenFormer, HyFormer, MixFormer and TokenFormer V2. TokenFormer V2 has the highest probe AUC, mutual-information lower bound, class-centroid gap, and distance ratio.
Variant
Test AUC
Δ (pp)
TokenFormer V2
0.8221
–
fixed groups (no merging)
0.8219
-0.02
w/o NS in groups
0.8212
-0.09
w/o NS Fusion
0.8217
-0.04
w/o Global Sync
0.8209
-0.12
w/o attention-output gating
0.8218
-0.03
Table 4. Component ablations, each retrained on the full training split. Removing any component reduces test AUC.
Model
Overall
Params
GFLOPs
MixFormer-1B
0.8004
1.01B
15.3
HyFormer-1B
−0.0962
1.07B
15.8
OneTrans-1B
−0.7792
1.04B
121.0
TokenFormer-1B
−0.0830
1.00B
45.5
TokenFormer V2-1B
+0.0572
1.00B
70.2
TokenFormer V2-1B ∗
+0.0712
1.00B
23.1
Table 5. Overall pCVR performance on historical WeChat Channels logs. MixFormer is reported in absolute AUC, while results for all other models are presented as relative changes (in percentage points, pp) against MixFormer.
Model
Backbone configuration
Learning rate (dense / embedding)
Transformer
d=352 , 4 layers, 8 heads, GELU 4×
5⋅10−4 / 0.1
TokenFormer
d=320 , 4 layers, windows 32/16
5⋅10−4 / 0.05
OneTrans
d=128 , 4 layers, pyramid 1/2ℓ
5⋅10−4 / 0.05
HyFormer
d=128 , 4 blocks, 6 queries
5⋅10−4 / 0.02
EST
d=144 , 4 layers, top-32 GSU
5⋅10−4 / 0.05
Kunlun
d=168 , 4 layers, window 32
5⋅10−4 / 0.1
Table 6. Selected backbone configurations and learning rates on TAAC 2026.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Metric
Token- Former
HyFormer
MixFormer
Token- Former V2
Primary measures (higher is better)
Probe AUC
0.8182
0.8181
0.8184
0.8204
MI (bits)
0.0798
0.0799
0.0797
0.0816
Ratio
1.68
1.68
1.50
1.70
Centroid
0.516
0.491
0.486
0.520
Components of Ratio (diagnostic)
Appendix
Table 7. Representation measures on 200k test impressions. Ratio summarizes relative class separation; its three distance components are diagnostic. Rank describes spectral concentration.
Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling.
Industrial recommenders rely on two model families that have evolved largely independently: feature-interaction models over multi-field user/item features, and sequential models over user-behavior histories. Production systems couple them only loosely. To unify the two, we present UniDot, a novel architecture for post-click conversion prediction built from the factorization-machine (FM) point of view: the embedding inner product---which powers collaborative filtering and lets a recommender generalize to unseen user--item pairs---is the same primitive as attention's query dot key scoring, so a single dot-product of tokens can underlie both feature interaction and sequence modeling. UniDot tokenizes non-sequential fields and multi-domain behavioral sequences into one shared token space and stacks a single macro-block in which a token-mixing bus and a sequence-retrieval bus (item tokens cross-attending the histories) run in parallel and exchange state each layer through an MLP-Mixer fusion, while an FM Highway carries explicit per-layer dot-product interactions around the residual stack directly to the classifier. The sequence side is embedded once per forward pass and shared by all consumers, bounding inference latency. Trained with a dual sparse/dense (Adagrad + Muon) optimizer, an auxiliary conversion-delay head, and multi-path mutual learning, UniDot finished as the runner-up on the Industrial track of the TAAC KDD Cup 2026.
Large Recommendation Models (LRMs) have demonstrated promising capabilities in industry-scale recommendation tasks. However, holistically integrating traditional signals into these transformer-based architectures effectively and efficiently remains a major challenge. Conventional approaches that "textualize" these signals directly or create discrete item representations often lead to excessively long prompts, substantial memory footprints, and high computational overhead. To overcome these limitations, we propose "Token Factory", a framework designed to transform traditional signals into "soft tokens" that can be directly processed by LRMs. This approach enables efficient integration and compression of heterogeneous input features, preventing prompt length explosion while enhancing model performance. We detail the architecture of Token Factory and present experimental results validating its effectiveness in a production-scale recommendation environment.