User Embeddings are Superpositions of Interpretable Behavioral Modes

Jonathan Kozaczuk; Zheng Dong; Noyan Evirgen; Keivan Majidi; Eddie K. H. Ng; Chris Porter; Qi Zhao; Jia Geng

cs.LGNov 5, 2026

User Embeddings are Superpositions of Interpretable Behavioral Modes

Authors: Jonathan Kozaczuk, Zheng Dong, Noyan Evirgen, Keivan Majidi, Eddie K. H. Ng, Chris Porter, Qi Zhao, Jia Geng,

Organizations: Amazon

Abstract

User embeddings are central components of modern recommendation, personalization, and behavioral modeling systems. Though these embeddings can compactly encode rich behavioral information, they are inherently dense and opaque, limiting interpretability. To address this gap, we argue that user embeddings can be understood as sparse superpositions of interpretable behavioral modes. This structure is directly predicted by generative models of user behavior for linear embeddings and extends approximately to deep-learning representations by the superposition hypothesis. Superimposed modes can naturally be recovered by sparse dictionary learning methods, such as sparse autoencoders (SAEs), and recovered modes can be automatically named and queried in natural language. We validate mode recovery against known ground truth on synthetic data, where SAEs recover modes better than classical sparse baselines, including on transformer-encoded embeddings. We then apply the method to classical and transformer-based user embeddings from two real-world datasets. We also show how the corresponding natural-language labels enable zero-shot audience retrieval and characterization over embeddings never trained on text. The framework is post-hoc and model-agnostic, requiring no modification to the upstream embedding system.

Figures & tables

Figure 1: Mode recovery vs. embedding dimension $m$ (mean $\pm$ 1 std over 5 random projections). Left: median per-feature Pearson $r$ with best-matching ground-truth mode. Center: mean per-user cosine similarity between projected codes and ground-truth weight vector. Right: fraction of modes recovered at decoder–centroid cosine similarity ${\geq}\,0.7$ . The vertical line marks $m=75$ , the ground-truth mode count.

Task type	Method	FVU $\downarrow$	Lift	AUROC	AUROC ${}_{\text{scored}}$	Coverage
Category	SAE (ours)	0.075	1.93 $\times$	0.705	0.678	0.735
	FastICA	0.083	1.86 $\times$	0.695	0.660	0.770
	K-Means encoding	0.388	1.85 $\times$	0.680	0.663	0.614
	Text-embedding baseline	—	1.39 $\times$	0.633	0.633	1.000
Life-event	SAE (ours)	0.075	5.30 $\times$	0.715	0.741	0.876
	FastICA	0.083	4.75 $\times$	0.713	0.717	0.932

Table 1: Open e-commerce zero-shot NL audience retrieval, metrics averaged per task block (6 category tasks and 2 life-event tasks, see Appendix C.2 ). AUROC

{}_{\text{scored}}

restricts AUROC to users with nonzero scores. Text baseline is the matched-input category-pooled variant (Section 5 ). Text baseline has no FVU.

Encoder	Method	FVU $\downarrow$	Lift	AUROC	AUROC ${}_{\text{scored}}$	Coverage
SASRec	SAE (ours)	0.041	2.53	0.692	0.687	0.977
SASRec	K-Means encoding	0.136	2.16	0.678	0.663	0.811
BERT4Rec	SAE (ours)	0.032	2.38	0.690	0.688	0.984
BERT4Rec	K-Means encoding	0.108	2.41	0.697	0.676	0.848
—	Text-embedding baseline	—	1.64	0.701	0.701	1.000

Table 2: MovieLens zero-shot NL audience retrieval, averaged over 2 SAE-training seeds

\times

32 non-compositional queries (query list in Appendix E.2 ). FVU is from a single SAE seed per encoder. Text baseline is encoder-independent. Bold: SAE significantly better than K-Means encoding (

p{<}0.05

, cross-seed paired Wilcoxon, see Appendix D ).

K-Means encoding top-5 for Jurassic Park		SAE (ours) top-5 for Jurassic Park
5/5 paraphrase a comedy concept		5 distinct concepts
comedy	Late-1990s indie/quirky dramedies	thriller	1990s B-movie action-thriller
comedy	Early-2000s indie/quirky comedy	action	Low-quality 1990s action blockbuster
comedy	Low-budget 1990s B-movie comedy	comedy	1990s low-quality mainstream comedy
comedy	1990s low-quality mainstream comedy	sequel	1980s-90s action-comedy sequel
comedy	LGBT romantic comedy and drama	sci-fi	1990s B-grade sci-fi action

Figure 2: Top-5 features describing the Jurassic Park audience (a sci-fi action franchise) under K-Means encoding and SAE on SASRec, ranked by lift at a 10% top-activator threshold. All five K-Means features paraphrase a mainstream-comedy concept, while the SAE top-5 span five distinct themes. Small-caps tags are our one-word summaries of the LLM-assigned feature names.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Items	Interactions per item	Top 1%	Top 10%	Singletons
Synthetic	778,639	7.5	50.9%	79.7%	62.3%
Open e-commerce (ASIN)	939,082	2.0	19.1%	45.8%	75.6%
MovieLens 1M	3,706	270	8.7%	44.4%	3.1%

Appendix

Table 3: Item popularity in the synthetic data and both real datasets: share of interactions carried by the most popular 1% and 10% of items, and the fraction of items that appear exactly once.

Figure 3: Item popularity and user activity for the synthetic data and both real datasets. (a) Share of interactions carried by the most popular items, with dots at the top 1% and 10% of items (cf. Table 3 ). (b) Interactions per user divided by each dataset’s median. Synthetic histories are bounded at about 100 interactions by the generation budget, which truncates the upper tail.

Technology & Gaming
ID	Behavioral mode	Domain
1	Tech Enthusiast	Tech/Electronics
2	Mobile Gamer	Gaming
3	PC Gamer	Gaming/Hardware
4	Smart Home Builder	IoT/Automation
5	Audiophile	Audio

Appendix

Table 4: Complete taxonomy of the 75 synthetic behavioral modes used as ground truth for the synthetic experiments of Section 4 . Each behavioral mode was generated with a name, description, product categories, and example products (not shown for compactness); only the name and domain grouping are displayed here.

Lifestyle & Hobbies
ID	Behavioral mode	Domain
46	Book Lover	Books/Literature
47	Board Game Enthusiast	Gaming/Tabletop
48	Puzzle Solver	Puzzles/Games
49	Craft Enthusiast	Crafts/DIY
50	Knitter/Crocheter	Yarn/Crafts

Appendix

Table 5: Complete taxonomy of the 75 synthetic behavioral modes used as ground truth (continued).

Method	Code–GT alignment	Median per-feature $r$	Modes recovered (cos ${\geq}\,0.7$ )
Oracle (linear ceiling)	0.78	—	—
SAE	0.58	0.28	0.97
SAE (merged)	0.54	0.54	0.95
FastICA	0.53	0.62	0.59
K-Means encoding	0.45	0.30	0.52
PCA	0.25	0.17	0.00

Appendix

Table 6: Mode recovery on mean-pooled SASRec embeddings (

d{=}256

) of the synthetic data, all methods at matched

L_{0}

. LDA is fit to raw user–item counts and has no decoder directions.

$K^{\prime}$	Users	Oracle	SAE	Best classical	SAE / oracle
15	4,591	0.943	0.695	0.667	0.737
20	8,394	0.931	0.683	0.598	0.734
25	13,245	0.918	0.679	0.643	0.740
30	19,880	0.897	0.651	0.631	0.725
40	36,737	0.871	0.642	0.623	0.737
50	58,248	0.838	0.619	0.617	0.739

Appendix

Table 7: Sensitivity to the number of modes

K^{\prime}

on the

d{=}256

transformer embeddings. Code–GT alignment for the oracle, the SAE, and the best of PCA, FastICA, and K-Means encoding at each

K^{\prime}

Dataset	Method	FVU $\downarrow$	Lift	AUROC	Coverage
Open e-commerce	PCA	0.388	1.28 $\times$	0.579	0.797
Open e-commerce	ICA	0.083	1.86 $\times$	0.695	0.770
MovieLens (SASRec)	PCA	0.189	1.65	0.576	0.815
MovieLens (SASRec)	ICA	0.343	1.99	0.610	0.910
MovieLens (BERT4Rec)	PCA	0.200	1.60	0.592	0.904
MovieLens (BERT4Rec)	ICA	0.357	2.08	0.580	0.833

Appendix

Table 8: Additional baselines on both datasets. Metrics averaged across tasks. SAE significantly exceeds every classical baseline on Lift at matched

L_{0}

(cross-seed Wilcoxon, Appendix D ); ICA is SAE’s closest competitor on AUROC (tied on life-event AUROC, SAE wins elsewhere) and achieves higher Coverage by construction (denser activations). NMF is excluded as it requires non-negative preprocessing that doesn’t naturally arise in either embedding space and underperforms classical baselines at all

m

(Section 4 ).

Task type	Task name	$n_{\text{pos}}$	Base rate
Category retrieval	Musician	345	8.4%
	Baby products	839	20.5%
	Board gamer	1421	34.8%
	Coffee enthusiast	1482	36.3%
	Gardener	1539	37.7%
	Pet owner	2196	53.8%

Appendix

Table 9: Open e-commerce task definitions.

n_{\text{pos}}

is the number of users satisfying the positive definition out of the evaluated user set for each task block (4085 users with at least one post-2021 window for category retrieval; 4373 users with a 2021 survey response for life-event retrieval).

Open e-commerce category retrieval (cross-seed paired Wilcoxon, 5 seeds $\times$ 6 tasks)
	Comparison	One-sided $p$
Lift@ $n_{\text{pos}}$	SAE vs K-Means encoding	$\mathbf{p=0.0003}$
Lift@ $n_{\text{pos}}$	SAE vs ICA	$\mathbf{p=0.0002}$
Lift@ $n_{\text{pos}}$	SAE vs PCA	$\mathbf{p<10^{-4}}$
AUROC	SAE vs K-Means encoding	$\mathbf{p<10^{-4}}$
AUROC	SAE vs ICA	$\mathbf{p=0.004}$

Appendix

Table 10: Consolidated paired Wilcoxon one-sided

p

-values. Default alternative H1: SAE

>

baseline on the named metric. Bold indicates

p{<}0.05

in the SAE-wins direction ; reverse-direction losses are reported unbolded for clarity. MovieLens retrieval columns report per-encoder and Stouffer-combined

p

; Open e-commerce and audience-characterization sections are single-dataset (e-commerce) or Stouffer-combined across MovieLens encoders (audience). Stouffer across encoders assumes independence; here the two encoders share users and queries, so combined

p

-values should be read as descriptive summaries rather than independence-based combinations (per-encoder columns are the primary quantity). Note on pairing: Open e-commerce and MovieLens SAE-vs-K-Means rows are cross-seed (Open e-commerce: 5 seeds

\times

task; MovieLens: 2 SAE-training seeds

\times

32 queries). MovieLens rows vs. PCA/ICA/text are single-seed paired across 32 queries. † For top-5 redundancy (median pairwise cosine among the top-5 features), smaller is more diverse; alternative hypothesis is H1: K-Means redundancy

>

SAE redundancy. ‡ For tests where the named comparison wins in the reverse direction, the reported

p

is the reverse-direction one-sided value (equal to

1-p_{\text{primary}}

) rather than the alternative-hypothesis

p

$\lambda$	mean $L_{0}$	$c_{\text{dec}}$	% eligible	n eligible	recon. MSE
$5.70{\times}10^{-3}$	68.7	0.095	24.5%	251	0.060
$3.62{\times}10^{-3}$	65.9	0.080	24.6%	252	0.044
$1.98{\times}10^{-3}$	82.8	0.081	37.1%	380	0.026
$1.46{\times}10^{-3}$	102.3	0.083	43.6%	446	0.021
$8.30{\times}10^{-4}$	147.7	0.086	38.0%	389	0.014

Appendix

Table 11: Width-and-sparsity sweep on MovieLens SASRec embeddings at

d_{\text{hidden}}{=}1024

. Selected operating point (bold) sits at the

c_{\text{dec}}

elbow with the highest eligible-feature count. “Eligible” counts features with 30–500 activating users.

Epochs	mean $L_{0}$	$c_{\text{dec}}$	n eligible	n too-rare	recon. MSE
125	101.0	0.054	653	0	0.091
250	80.2	0.059	572	125	0.052
500	74.3	0.076	498	165	0.033

Appendix

Table 12: Training-length sensitivity at

d_{\text{hidden}}{=}1024

\lambda{=}1.98{\times}10^{-3}

on SASRec embeddings. Selected training length (bold) balances feature count against within-theme redundancy. Metrics at 250 epochs differ slightly from the corresponding row in Table 11 because the two tables report independent training runs; seed-level variation on MovieLens SAE training can reach

{\sim}10\%

on L0 and reconstruction MSE at a fixed operating point (see seed-variance discussion in the supplementary notes).

Query	Title patterns (representative)	Genre	$n_{\text{+}}$
Broad audiences (21 queries)
Fans of Quentin Tarantino films	Pulp Fiction, Reservoir Dogs, Jackie Brown, Kill Bill, Four Rooms	—	517
Fans of Steven Spielberg adventure and sci-fi films	Raiders of the Lost Ark, Indiana Jones, Jurassic Park, E.T., Close Encounters, …(9 total)	—	3099
Fans of classic Disney animated films	Lion King, Little Mermaid, Beauty and the Beast, Aladdin, Snow White, …(21 total)	—	1682
Viewers of dark, psychological thrillers	Silence of the Lambs, Se7en, Fight Club, American Psycho, Memento, …(13 total)	—	1748
Fans of romantic comedies from the 1990s	Pretty Woman, Sleepless in Seattle, You’ve Got Mail, Notting Hill, Four Weddings, …(10 total)	Romance/Comedy	1237

Appendix

Table 13: Natural-language audience queries on MovieLens 1M with title-pattern and genre-constraint ground-truth definitions.

n_{\text{+}}

is the resulting number of positive users out of 6040.

Encoder	Threshold	$\Delta$ Lift	$\Delta$ AUROC	$\Delta$ AUROC ${}_{\text{scored}}$	Coverage (SAE / K-Means)
SASRec	0.0	$+6.3\%$	$-0.016$	$-0.004$	0.973 / 0.818
	0.5	$+10.8\%$	$+0.000$	$+0.011$	0.976 / 0.817
	0.6	$+11.2\%$	$+0.009$	$+0.018$	0.975 / 0.808
	0.7	$+7.4\%$	$+0.019$	$+0.019$	0.983 / 0.795
BERT4Rec	0.0	$-5.8\%$	$-0.019$	$+0.003$	0.971 / 0.847
	0.5	$-4.2\%$	$-0.016$	$+0.006$	0.973 / 0.846

Appendix

Table 14: Sensitivity of MovieLens retrieval to the auto-interpretation confidence threshold (operating point 0.6). SAE minus K-Means encoding: Lift as a percentage of the K-Means value, AUROC and AUROC

{}_{\text{scored}}

as absolute differences, and coverage as the raw SAE / K-Means pair. Computed over 3 SAE seeds per encoder, a different seed set from Table 2 , so the 0.6 row differs slightly from it.

User Modeling

User modeling represents people's interests, knowledge, goals, and changing behavior to personalize predictions and AI assistance.

See all 157 papers

UEmbed: Unified Sparse and Dense Multimodal Embeddings
Aug 3, 2026Tingyu Song, Mingxin Li, Yanzhao Zhang +5Multimodal IR Learned Sparse Retrieval
CASIA · University of Chinese Academy of Sciences · Alibaba Group +1
Bridging Textual Profiles and Latent User Embeddings for Personalization
May 7, 2026Zhaoxuan Tan, Xiang Zhai, Yan Zhu +2Sequential Recommendation RL for Recommendation
University of Notre Dame · 2Google · * This work was done while Zhaoxuan was a student researcher at Google.
Query as Anchor: Scenario-Adaptive User Representation via Large Language Model
Feb 16, 2026Jiahao Yuan, Yike Xu, Jinyong Wen +9User Modeling Representation Learning
Ant Group · East China Normal University · Zhejiang University

Aug 3, 2026J/K move · Enter open · S save

UEmbed: Unified Sparse and Dense Multimodal Embeddings

Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long+4

CASIA · University of Chinese Academy of Sciences · Alibaba Group+1

Multimodal IR Learned Sparse Retrieval Multimodal Embedding Information Retrieval Hybrid Retrieval

Open paper AskPDF arXiv

Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.

Abstract

User Modeling

Explore similar work

UEmbed: Unified Sparse and Dense Multimodal Embeddings

Bridging Textual Profiles and Latent User Embeddings for Personalization

Query as Anchor: Scenario-Adaptive User Representation via Large Language Model