User representation learning in real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated by tokenization. (ii) Lack of quantitative analysis of how tokenization configurations should scale with data size. In this report, we propose User Behavioral Densing Law for characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled by tokenization. To derive the scaling pattern governing the minimum sufficient tokenization configuration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens, and the scaling slope varies systematically with the tokenization method and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation. Extensive experiments across diverse data sources, tokenization methods, and downstream tasks demonstrate the generalizability and reliability of the User Behavioral Densing Law, providing practical guidance for tokenization configuration selection in large-scale user representation learning. Moreover, ALGN outperforms existing baselines.
Figures & tables
Figure 1: (a) User representation learning pipeline. Once trained on user behavioral sequences, a representation model can support various downstream applications, including text-based retrieval for user targeting, U2U retrieval for recommendation system and classification for advertising and risk control scenarios. In industrial settings with billion-scale users, when scaling up input data volume, commonly via increasing behavioral timespan and user amount, models directly consume raw data (e.g., raw user behavioral text description), will suffer from performance saturation beyond practical scaling thresholds while tokenization will alleviate the limitation and enable sustained performance gains, as shown in (b).
Research direction
Raw Data
Tokenized Data
Capacity Configuration
Capacity Allocation
User data scaling ( 1 ; 31 ; 38 ; 47 ; 12 ; 37 ; 30 ; 42 )
Table 1: Comparison of related work across the dimensions considered in this study.
Notation
Description
Statistics
N
Number of users
100M–2B
D
Length of the user behavior sequence
30-270 days
P
Model size of the pretrained user embedding model
0.05B–0.4B
und
Behavior information of user n within the d -th time window, including Paybill Bnd , SPM Snd , and Miniprogram Mnd
Daily events 0–10 per user; median ∼ 6
s
Behavior text tokens used for training
∼2200 per user
Dtest
Test dataset for downstream evaluation
Classification: 50 datasets, 0.5M users per set Text retrieval: 22 datasets, 0.5M users per set U2U retrieval: 22 datasets, 0.5M users per set
Table 2: Notations, corresponding descriptions, and key statistics. Pretraining statistics are shown above the dashed line, and downstream evaluation settings are shown below.
Figure 2: Pretraining framework for general-purpose user representation. Each user’s behavioral sequence is split into a past and future segment, as the former part is encoded by a Transformer encoder and the latter is fed to a LoRA-tuned LLM-based model to generated embeddings, which are further aligned via a contrastive loss.
Figure 3: Three downstream tasks for user representation evaluation.
Figure 4: Raw behavioral scaling saturates beyond approximately 0.03B users and 60 days. All three probing metrics exhibit a flattening trend around the saturation thresholds, indicating that additional raw data volume yields diminishing downstream returns.
Figure 5: Scaling model parameters under fixed behavioral data exhibits strong diminishing returns. Larger models continue to achieve lower training loss, but downstream evaluation accuracy rapidly saturates beyond moderate model scales. This indicates that representation quality is constrained by information density rather than model capacity alone.
Figure 6: Overview of the RQ-VAE based tokenization method used for behavioral densing.
Figure 7: Tokenized representations outperform raw sequences at matched data scales with the gap widening where raw modeling saturates. Subplot (a) shows the full two dimensional scaling surface. Subplots (b) and (c) show cross sections at fixed user population and fixed temporal horizon.
Figure 8: Downstream task performance with data size & training computes scaling.
Figure 9: Behavioral Densing Law across different data sources and tokenization methods, derived from three tasks.
Figure 10: Adaptive variable-length tokenization method allocates more codes to high-information behavioral periods and fewer codes to routine or redundant periods, achieving comparable or better accuracy with fewer tokens on average than fixed-length baselines.
Method
Capacity (%) ↓
AUC (%) ↑
KS (%) ↑
Acc (%) ↑
RQ-VAE ( 19 )
100.00
74.56
39.02
82.44
VQ-VAE ( 34 )
316.23
73.45
37.95
82.47
RQ-Kmeans ( 8 )
100.00
73.77
38.34
82.85
SARQ ( 35 )
76.24
75.36
41.28
83.16
ALGN
63.47
76.43
43.31
83.52
Table 3: Comparison of different tokenization methods. SID capacity usage is used to measure efficiency, with classification metrics reported for performance evaluation.
Method
Capacity(%) ↓
AUC (%) ↑
KS (%) ↑
Acc (%) ↑
RQ-VAE ( 19 )
100.00
74.56
39.02
82.44
Heuristic Variable-Length Baseline
86.91
74.43
38.10
82.39
w/o Uncertainty Signal
77.83
75.11
40.21
82.77
w/o Residual Signal
76.24
75.36
41.28
83.16
w/o Regularization
73.42
76.01
42.21
83.35
w/ SID Length Regularization
63.16
76.22
42.99
83.47
Table 4: Ablation study on each component of ALGN.
Figure 11: Effect on densing law for ALGN under classification task.
Figure 12: Contrastive alignment loss over 100,000 training steps for models pre-trained on behavioral sequences spanning D∈{30,60,90,120,180} days. While all configurations converge, the reduction in final loss diminishes sharply as D increases. The curves for D≥90 days nearly overlap, indicating that additional historical days contribute minimal new optimization signal, consistent with the downstream metric saturation observed in Section 4 .