QueryFormer: Winning Solution for KDD Cup 2026 Tencent UniRec Challenge

Yuanzhe Zhou; Zhaoyang Zeng

cs.AISep 15, 2026

QueryFormer: Winning Solution for KDD Cup 2026 Tencent UniRec Challenge

Authors: Yuanzhe Zhou, Zhaoyang Zeng

Organizations: Wuhan University · Sun Yat-sen University

Abstract

Post-click conversion rate (pCVR) prediction requires jointly modeling feature interactions and sequential user behaviors. The KDD Cup 2026 Tencent UniRec Challenge calls for a unified architecture addressing both. We observe that existing unified architectures often generate query tokens---the central information hub---with projection-based multi-layer perceptrons (MLPs), without explicit token-to-query attention for refining the query side. We propose QueryFormer, centered on a stackable unified field--sequence block that bridges non-sequential multi-field features and behavioral sequences, and provide a latency-aware scaling study over view width $H$ , model width, depth, data, and compute. The block generates queries through cross-attention and packs sequence queries into shared-parameter attention. QueryFormer secured 1st place in the Industrial Track, achieving an official test area under the ROC curve (AUC) of 0.83254; a modest post-competition scale-up reached 0.832713. Within our grid, $H$ -scaling improves validation AUC from 0.84540 to 0.84615 and beats HyFormer at comparable budgets. Ablation identifies query generation as the largest contributor. Packed shared-parameter cross-attention keeps H=8 inference latency to only 1.89x that of H=1, positioning the bridge as an efficient stackable unified block.

Figures & tables

Figure 1. QueryFormer architecture. The reusable stackable query-generation bridge takes non-sequential field tokens and behavioral sequence tokens as input, returns generated query probes and mixed representations, and preserves the

H

dimension as parallel tokenization columns with shape-

(B_{s},H,T,d_{\text{model}})

views, where

B_{s}

is batch size and

T

is token count, before per-column outputs are combined. Architecture diagram showing QueryFormer's query-driven bridge from non-sequential tokens to behavioral sequences through CoTransformerBlock, QuerySeqCrossAttn, and SeqQueryCrossAttn, with H independent tokenization columns.

Figure 2. Scaling evidence for the stackable query-generation bridge. (a) Parameter scaling: QueryFormer traces a stronger AUC–dense-parameter frontier than HyFormer ( Huang et al., 2026 ) within the explored budgets, with separate

d_{\text{model}}

- and

H

-scaling paths. (b) Data efficiency: QueryFormer (

P_{\mathrm{dense}}{=}87.1\mathrm{M}

H{=}4

) remains above HyFormer (

P_{\mathrm{dense}}{=}105.8\mathrm{M}

) from about 20% to full training data. Both x-axes are logarithmic;

P_{\mathrm{dense}}

denotes dense parameter count. Two-panel scaling analysis. (a) Parameter scaling: AUC vs dense parameters for QueryFormer d_model-scaling, H-scaling, and HyFormer. (b) Data efficiency: AUC vs training samples for HyFormer (P_dense=105.8M) and QueryFormer (P_dense=87.1M, H=4). Both subplots use log x-axes.

Method	Query	Tok $\to$ Q	Bridge	Views	Packed
HyFormer ( Huang et al., 2026 )	MLP	$\times$	implicit	$\times$	$\times$
HeMix ( Wang et al., 2026a )	Mixed	$\times$	mixed	$\times$	$\times$
LENS ( Wang et al., 2026b )	Gate	$\times$	gated	$\times$	$\times$
GAP-Net ( Ke et al., 2026 )	Cascade	$\times$	cascade	$\times$	$\times$
QueryFormer	Cross-Attn	✓	token I/O	$H$	✓

Table 1. Unified-block positioning. Tok

\to

Q: explicit token-to-query attention; token I/O: token input/output; Packed: shared-parameter packed attention for multi-view queries.

Stage	Mechanism	Type	Interaction
Co-Transformer	QuerySelfAttn	SelfAttn	$\langle N_{u},N_{u}\rangle$ , $\langle N_{i},N_{i}\rangle$
Co-Transformer	QueryCrossAttn	CrossAttn	$\langle N_{u},N_{i}\rangle$
QuerySeqCrossAttn	—	CrossAttn	$\langle N_{i},B\rangle$
SeqQueryCrossAttn	—	CrossAttn	$\langle Q_{B},N\rangle$

Table 2. Attention mechanisms by architectural stage. For SeqQueryCrossAttn,

Q_{B}

denotes behavior-initialized query probes.

Figure 3. Validation AUC and LogLoss vs. training steps for

H\in\{1,2,4,8\}

d_{\text{model}}{=}272

. Training curves showing that wider embedding matrices achieve better final AUC and LogLoss.

Model	$H$	Dense	Val AUC	LogLoss
HyFormer ( Huang et al., 2026 )	–	105.8	0.84477	–
QueryFormer	4	87.1	0.84606	0.210309
QueryFormer	4	220.8	0.84631	0.210241

Table 3. Validation comparison under the optimized training recipe. Dense parameters are in millions; all runs use

L_{\mathrm{seq}}{=}256

Configuration	Best AUC	$\Delta$
Full Model (baseline)	0.84606	—
- Replace SeqQueryCrossAttn with MLP	0.84568	$-$ 0.00038
- Remove QueryCrossAttn	0.84577	$-$ 0.00029
- Remove QuerySeqCrossAttn	0.84583	$-$ 0.00023
- Remove QuerySelfAttn	0.84586	$-$ 0.00020
- Remove all four attention mechanisms	0.84496	$-$ 0.00110

Table 4. Ablation of attention mechanisms (

H{=}4

d_{\text{model}}{=}272

Figure 4. Per-epoch ablation of four attention mechanisms. Validation AUC across training epochs for four ablation configurations: replacing cross-attention query generation with MLP causes the largest drop, followed by removing QueryCrossAttn, QuerySeqCrossAttn, and QuerySelfAttn, confirming each mechanism independently contributes.

Rank	Team	Test AUC	$\Delta$
1st	QueryFormer (日之光面)	0.83254	—
2nd	RClaw	0.83217	$-$ 0.00037
3rd	load_state_dict	0.83145	$-$ 0.00109
–	QF-scaled-post	0.832713	$+$ 0.000173

Table 5. Industrial Track leaderboard (top 3), with post-competition improvement.

$H$	Train Time	Eval Time	Eval Throughput	Latency / sample
1	43.7 min	1.8 min	32.2k samples/s	31.1 $\mu$ s
2	56.0 min	2.0 min	29.0k samples/s	34.5 $\mu$ s
4	80.1 min	2.5 min	23.2k samples/s	43.1 $\mu$ s
8	137.3 min	3.4 min	17.0k samples/s	58.7 $\mu$ s

Table 6. Latency-aware

H

scaling of the packed bridge on 31.35M training samples and 3.48M validation samples.

Setting	Sparse	Dense	Val AUC	LogLoss
$H{=}1$	530	50	0.84540	0.210651
$H{=}2$	563	62	0.84575	0.210432
$H{=}4$	628	87	0.84606	0.210309
$H{=}8$	757	137	0.84615	0.210225
$K{=}1$	628	77	0.84598	0.210302
$K{=}3$	628	97	0.84616	0.210270

Table 7. Multi-axis scaling grid (SwiGLU encoders; params in millions). Unspecified values use

K{=}2

H{=}4

d_{\text{model}}{=}272

, and 4 epochs.

Conversion Rate Prediction

See all 12 papers

Field-Aware RankMixer with Dual-Stream Bilinear Fusion for the Tencent UNI-REC Challenge
Jul 17, 2026Yufeng Zhang, Zhengqi Xu, Jiajun CuiFeature Interaction Modeling Conversion Rate Prediction
Independent Researcher · Meituan
RoleMix: Unifying Sequential and Non-Sequential Features via Semantic Tokenization for Post-Click Conversion Rate Prediction
Jul 18, 2026Wenan Wang, Qin Zhao, Zhixiang LuFeature Interaction Modeling Recommender Systems
Macau Polytechnic University · University of Liverpool
UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation
Aug 17, 2026Rongcheng Lin, Yan Sun, Jamey Zhang +4Recommender Systems Feature Interaction Modeling
Meta Menlo Park, USA

Jul 17, 2026J/K move · Enter open · S save

Field-Aware RankMixer with Dual-Stream Bilinear Fusion for the Tencent UNI-REC Challenge

Yufeng Zhang, Zhengqi Xu, Jiajun Cui

Independent Researcher · Meituan

Feature Interaction Modeling Conversion Rate Prediction Recommender Systems User Preference Modeling Tabular Prediction Sequence Modeling

Open paper AskPDF arXiv

This paper presents our solution to the KDD Cup 2026 Tencent UNIREC Challenge. The task requires joint modeling of multi-domain user behavior sequences and non-sequential multi-field features for target-ad pCVR prediction. We develop a Field-Aware RankMixer (FA-RankMixer) with dual-stream bilinear fusion. The model first applies target-aware DIN modules to extract user interests from multiple behavior domains. It also models recent and earlier interests separately for the longest behavior sequence. The model then forms semantic tokens based on feature fields and behavior domains and uses RankMixer blocks for cross-token interaction. A shallow MLP stream complements the deep RankMixer stream, and a group-wise bilinear module fuses their representations. Our final solution ranks ninth on the official leaderboard. Our code is available at https://github.com/PixelCookie-zyf/TAAC-2026-SeRankMixer.

Abstract

Conversion Rate Prediction

Explore similar work

Field-Aware RankMixer with Dual-Stream Bilinear Fusion for the Tencent UNI-REC Challenge

RoleMix: Unifying Sequential and Non-Sequential Features via Semantic Tokenization for Post-Click Conversion Rate Prediction

UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation