SAGA: Structure-Attended Generative Action Embedding Model that encodes Multi-Surface User Action Sequences

Tsz Fung Pang; Po Jen Chen; Nimish Ronghe; Farhad Farahani; Bo Zhang

cs.LGAug 15, 2026

SAGA: Structure-Attended Generative Action Embedding Model that encodes Multi-Surface User Action Sequences

Authors: Tsz Fung Pang, Po Jen Chen, Nimish Ronghe, Farhad Farahani, Bo Zhang

Organizations: PayPal AI San Jose, CA, USA

Abstract

Prior embedding models for sequential recommendation typically operate within a homogeneous action space, limiting their ability to capture cross-surface behavioral signals spanning distinct behavioral domains. We present SAGA, a generative action embedding model that encodes multi-surface user interaction sequences across a Financial Service organization's ecosystems, from checkout, peer-to-peer (P2P) transactions, in-app engagement, email to account actions, into a unified user representation for downstream recommendation tasks. Central to SAGA is a per-field tokenization schema that decomposes each action event into multiple field-level tokens (e.g. product, interaction, surface), enabling field-level attention and per-field training objectives that fused single-token approaches cannot support. Through an offline ablation study on loss formulation, tokenization granularity and training data scope, we isolate the contribution of each design choice. A downstream model integrated with SAGA-generated user embeddings delivers the strongest overall click and conversion lift across diverse downstream touchpoints, compared to all ablated and alternative architectures.

Figures & tables

Figure 1. Structure-attended action sequences (right,

K{=}7

) enabled per-field attention. Fusing all fields into a single token per event (left,

K{=}1

) limits attention to event-level granularity, losing fine-grained dependencies.

Figure 2. SAGA dual-head architecture. A decoder-only transformer with round-robin per-field tokenization (

K

fields per event),

K

independent LM heads, and an event-boundary contrastive head with stop-gradient compositional targets. Architecture diagram showing decoder-only transformer with round-robin per-field tokenization, (

K

fields per event),

K

independent LM heads, and an event-boundary contrastive head with stop-gradient compositional targets.

Figure 3. Structural cadence of the dual-head firing pattern across two consecutive events. The LM head

\mathcal{L}_{\mathrm{LM}}

computes a next-token prediction loss at every position (

7{\times}

per event), while the contrastive head

\mathcal{L}_{\mathrm{con}}

fires only at the mcc event-boundary token (

1{\times}

per event). The bottom row (

\hat{y}_{i}

) shows the shifted prediction targets for the LM head.

Symbol	Definition
$\mathbf{x}_{1:T}$	Input token sequence of length $T$
$m$	Number of tokens in the training sequence
$K$	Number of fields per event ( ${=}\,7$ )
$s$	Stride for multi-token prediction ( $s{=}K$ in round-robin mode)
$\mathbf{h}_{i}$	Hidden state at position $i$ from the transformer backbone
$\hat{\mathbf{y}}_{i}$	Logit vector at position $i$ , produced by the LM head

Table 1. Notation for training objectives.

Abl.	ID	Variable changed	$K$	Training data
Objective	A1	LM heads only ( $\lambda_{c}{=}0$ )	7	All
	A2	Contrastive only ( $\alpha_{\text{ntl}}{=}\alpha_{\text{mtl}}{=}0$ )	7	All
	A3	SAGA (dual-head)	7	All
Tok (A)	B1 A	Fused-key, 146 evt	1	All
	B2 A	Interleaved, 146 evt	2	All
	B3 A	SAGA , 146 evt	7	All

Table 2. Ablation study variants. SAGA (the proposed model) uses

K{=}7

fields per event, dual-head pre-training (

\mathcal{L}_{\text{NTL}}+\mathcal{L}_{\text{MTL}}+\lambda_{c}\mathcal{L}_{\text{con}}

), and all interaction surfaces. Each row modifies exactly one dimension; all other settings are held constant. Set A fixes events; Set B fixes the token budget.

		TP-A		TP-B		TP-C
ID	Objective	Clk	Conv	Clk	Conv	Clk	Conv
A3	SAGA (dual-head)	+6.5	+3.3	+1.7	+0.6	-1.4	—
A1	LM only ( $\lambda_{c}{=}0$ )	+3.5	+1.6	+0.4	+0.3	-4.2	—
A2	Contr. only ( $\alpha_{\text{ntl}}{=}\alpha_{\text{mtl}}{=}0$ )	+5.8	+3.9	+0.5	+0.4	-8.6	—

Table 3. Training objective ablation. Values are relative % improvement in NDCG@3 over no-embedding baseline.

Set A — Fixed 146 events:
			TP-A		TP-B		TP-C
ID	Tokenization	$K$	Clk	Conv	Clk	Conv	Clk	Conv
B3 A	SAGA (round-robin)	7	+6.5	+3.3	+1.7	+0.6	-1.4	—
B1 A	Fused (PinFM)	1	+7.3	+4.5	+0.6	+0.4	-3.9	—
B2 A	Interleaved (HSTU)	2	+3.7	+2.9	-1.8	-0.7	-12.6	—
Set B — Fixed 1024-token budget:

Table 4. Tokenization granularity ablation. Set A : fixed 146 events; Set B : fixed 1024-token budget.

		TP-A		TP-B		TP-C
ID	Training surfaces	Clk	Conv	Clk	Conv	Clk	Conv
C3	SAGA (All surfaces)	+6.5	+3.3	+1.7	+0.6	-1.4	—
C2	Engage. + Transactions	+2.5	+0.7	-1.7	-0.7	-4.7	—
C1	Engagement only	+3.1	+1.4	-0.9	-0.3	-5.6	—

Table 5. Training data composition ablation. All variants use SAGA (

K{=}7

, dual-head).

Touchpoint	Clk	Conv
TP-B	+5.05%	+1.53%
TP-C	$-$ 0.06%

Table 6. Online A/B test results. Values are relative % lift over the control group (prior deep learning ranker).

User Modeling

User modeling represents people's interests, knowledge, goals, and changing behavior to personalize predictions and AI assistance.

See all 151 papers

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds
Apr 15, 2026Yifeng Zhou, Yuehong Hu, Zhixiang Feng +9Recommender Systems Sequential Recommendation
Tencent Inc., Shenzhen, China
From Clicks to Intent: Cross-Platform Session Embeddings with LLM-Distilled Taxonomy for Financial Services Recommendations
Jun 24, 2026Dianjing Fan, Yao Li, Kyaw Hpone Myint +4Sequential Recommendation Representation Learning
Capital One, McLean, VA, USA · Capital One, New York, NY, USA · Capital One, Boston, MA, USA
TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems
Jun 23, 2026Qingyun Liu, Bo Yan, Yang Liu +16Semantic ID Recommendation Representation Learning
Google DeepMind · YouTube

Apr 15, 2026J/K move · Enter open · S save

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds

Yifeng Zhou, Yuehong Hu, Zhixiang Feng, Junwei Pan+8

Tencent Inc., Shenzhen, China

Recommender Systems Sequential Recommendation Feature Interaction Modeling

Open paper AskPDF arXiv

Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling.

Abstract

User Modeling

Explore similar work

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds

From Clicks to Intent: Cross-Platform Session Embeddings with LLM-Distilled Taxonomy for Financial Services Recommendations

TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems