User representation learning in real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated by tokenization. (ii) Lack of quantitative analysis of how tokenization configurations should scale with data size. In this report, we propose User Behavioral Densing Law for characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled by tokenization. To derive the scaling pattern governing the minimum sufficient tokenization configuration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens, and the scaling slope varies systematically with the tokenization method and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation. Extensive experiments across diverse data sources, tokenization methods, and downstream tasks demonstrate the generalizability and reliability of the User Behavioral Densing Law, providing practical guidance for tokenization configuration selection in large-scale user representation learning. Moreover, ALGN outperforms existing baselines.
Figures & tables
Figure 1: (a) User representation learning pipeline. Once trained on user behavioral sequences, a representation model can support various downstream applications, including text-based retrieval for user targeting, U2U retrieval for recommendation system and classification for advertising and risk control scenarios. In industrial settings with billion-scale users, when scaling up input data volume, commonly via increasing behavioral timespan and user amount, models directly consume raw data (e.g., raw user behavioral text description), will suffer from performance saturation beyond practical scaling thresholds while tokenization will alleviate the limitation and enable sustained performance gains, as shown in (b).
Research direction
Raw Data
Tokenized Data
Capacity Configuration
Capacity Allocation
User data scaling ( 1 ; 31 ; 38 ; 47 ; 12 ; 37 ; 30 ; 42 )
Table 1: Comparison of related work across the dimensions considered in this study.
Notation
Description
Statistics
N
Number of users
100M–2B
D
Length of the user behavior sequence
30-270 days
P
Model size of the pretrained user embedding model
0.05B–0.4B
und
Behavior information of user n within the d -th time window, including Paybill Bnd , SPM Snd , and Miniprogram Mnd
Daily events 0–10 per user; median ∼ 6
s
Behavior text tokens used for training
∼2200 per user
Dtest
Test dataset for downstream evaluation
Classification: 50 datasets, 0.5M users per set Text retrieval: 22 datasets, 0.5M users per set U2U retrieval: 22 datasets, 0.5M users per set
Table 2: Notations, corresponding descriptions, and key statistics. Pretraining statistics are shown above the dashed line, and downstream evaluation settings are shown below.
Figure 2: Pretraining framework for general-purpose user representation. Each user’s behavioral sequence is split into a past and future segment, as the former part is encoded by a Transformer encoder and the latter is fed to a LoRA-tuned LLM-based model to generated embeddings, which are further aligned via a contrastive loss.
Figure 3: Three downstream tasks for user representation evaluation.
Figure 4: Raw behavioral scaling saturates beyond approximately 0.03B users and 60 days. All three probing metrics exhibit a flattening trend around the saturation thresholds, indicating that additional raw data volume yields diminishing downstream returns.
Figure 5: Scaling model parameters under fixed behavioral data exhibits strong diminishing returns. Larger models continue to achieve lower training loss, but downstream evaluation accuracy rapidly saturates beyond moderate model scales. This indicates that representation quality is constrained by information density rather than model capacity alone.
Figure 6: Overview of the RQ-VAE based tokenization method used for behavioral densing.
Figure 7: Tokenized representations outperform raw sequences at matched data scales with the gap widening where raw modeling saturates. Subplot (a) shows the full two dimensional scaling surface. Subplots (b) and (c) show cross sections at fixed user population and fixed temporal horizon.
Figure 8: Downstream task performance with data size & training computes scaling.
Figure 9: Behavioral Densing Law across different data sources and tokenization methods, derived from three tasks.
Figure 10: Adaptive variable-length tokenization method allocates more codes to high-information behavioral periods and fewer codes to routine or redundant periods, achieving comparable or better accuracy with fewer tokens on average than fixed-length baselines.
Method
Capacity (%) ↓
AUC (%) ↑
KS (%) ↑
Acc (%) ↑
RQ-VAE ( 19 )
100.00
74.56
39.02
82.44
VQ-VAE ( 34 )
316.23
73.45
37.95
82.47
RQ-Kmeans ( 8 )
100.00
73.77
38.34
82.85
SARQ ( 35 )
76.24
75.36
41.28
83.16
ALGN
63.47
76.43
43.31
83.52
Table 3: Comparison of different tokenization methods. SID capacity usage is used to measure efficiency, with classification metrics reported for performance evaluation.
Method
Capacity(%) ↓
AUC (%) ↑
KS (%) ↑
Acc (%) ↑
RQ-VAE ( 19 )
100.00
74.56
39.02
82.44
Heuristic Variable-Length Baseline
86.91
74.43
38.10
82.39
w/o Uncertainty Signal
77.83
75.11
40.21
82.77
w/o Residual Signal
76.24
75.36
41.28
83.16
w/o Regularization
73.42
76.01
42.21
83.35
w/ SID Length Regularization
63.16
76.22
42.99
83.47
Table 4: Ablation study on each component of ALGN.
Figure 11: Effect on densing law for ALGN under classification task.
Figure 12: Contrastive alignment loss over 100,000 training steps for models pre-trained on behavioral sequences spanning D∈{30,60,90,120,180} days. While all configurations converge, the reduction in final loss diminishes sharply as D increases. The curves for D≥90 days nearly overlap, indicating that additional historical days contribute minimal new optimization signal, consistent with the downstream metric saturation observed in Section 4 .
Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this work, we systematically investigate how the information granularity of tokens, controlled by the compression rate (i.e., average bytes of text per token), affects scaling trends. We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate. This flexibility allows us to study the role of compression rate well beyond 4.57 bytes per token obtained with a popular BPE tokenizer. Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived (Kaplan et al., 2020; Hoffmann et al., 2022). Furthermore, we discover that the optimal compression rate differs from the one obtained with BPE and decreases with compute. These findings generalize to both latent and subword tokenization, as well as to languages other than English, guiding language model developers on tokenization scheme selection for maximal compute efficiency.
Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6
Language model loss follows remarkably regular scaling laws over model and data size, yet it remains unclear why the aggregate loss should exhibit a power-law form. Existing explanations often attribute this regularity to a heavy-tailed spectrum of pattern difficulty in natural language, but this view has not been directly validated at token-level granularity in large-scale real-data training. We present a token-level framework that decomposes scaling laws into localized learning events of individual contextualized tokens. By fitting token loss trajectories with sigmoids, we show that token learning is concentrated in localized transitions, giving rise to a learning-time spectrum that dominates the scaling-law shape. Across more than one hundred pre-training runs on large and diverse real-language corpora with modern LLM architectures, scaling up to 6B parameters and 300B training tokens, the measured learning-time spectrum quantitatively reconstructs the validation loss derivative along the training-step T, data-scale D, and model-scale M axes. We further show that the same signal is actionable: by reshaping the training distribution according to when tokens become learnable, we alter the optimization trajectory and achieve 11% faster validation-loss reduction. These results provide direct empirical evidence that scaling laws are governed primarily by the distribution of token-level learning times, and that this distribution can be used not only to explain scaling behavior but also to improve training performance.
Pingjie Wang, Zechen Hu, Peiru Yang +2
Dots Studio, Xiaohongshu Inc. · Shanghai Jiao Tong University · Tsinghua university
We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law). Fitting the proposed law on a large set of training runs, we find that it correctly recovers the scaling of the optimal batch size. Moreover, because it makes use of training runs with suboptimal batch size, our proposed law can be robustly fit with a significantly smaller amount of training runs. We further show that the three-term law can be used to derive scaling laws for suboptimal batch sizes, and that it matches previous empirical findings related to the critical batch size.
Fabian Schaipp
Inria, ´Ecole Normale Sup´erieure, PSL Research University, Paris