Transaction event sequences support diverse prediction tasks in online commerce and digital financial services. The same transaction records can yield prediction targets with different semantics, including values, changes, and relations to past events. These targets impose different constraints on history representations, but their predictability alone does not establish the downstream utility of a target set. Moreover, target-wise prediction losses do not explicitly model conditional co-occurrence among targets. To address these challenges, we propose TAPES, a task-aligned joint predictive pretraining framework for transaction sequences. TAPES constructs a finite collection of candidate target sets from admissible field-operator combinations. It selects a target set according to the downstream utility of representations learned through short joint pretraining trials under matched budgets. It then reinitializes the encoder and pretrains it on the selected targets through a joint likelihood parameterized by a shared discrete latent variable, before downstream supervised adaptation. Experiments in a real-world transaction-risk setting and on multiple public financial benchmarks spanning diverse downstream tasks demonstrate that TAPES outperforms representative baselines. Ablation studies further validate the contributions of predictive target selection and joint modeling.
Figures & tables
Figure 1. Motivation for TAPES. (a) Different transformations of the same transaction field yield prediction targets with distinct semantics, illustrated by amount level, adjacent change, and historical deviation. (b) Two historical contexts share the same target-wise conditional marginals but exhibit different joint outcome patterns, motivating joint prediction. Event values, target combinations, and distributions are illustrative; bin denotes target-specific discretization.
Figure 2. Overview of Task-Aligned Joint Predictive Pretraining for Transaction Event Sequences (TAPES). (1) Stage 1: Task-guided target-set selection. Field–operator combinations capture predictive semantics beyond raw field values, and short pretraining trials identify a target set useful for the downstream task. (2) Stage 2: Shared-latent joint pretraining. The selected targets are jointly modeled through a shared discrete latent variable to learn representations of transaction histories. Candidate sets and probabilities are illustrative.
Method
AlfaBattle
Churn
Taobao
MBD-mini
Retail X5
Avg.Rank ↓
AUROC ↑
AP ↑
AUROC ↑
AP ↑
AUROC ↑
AP ↑
mAUROC ↑
mAP ↑
Macro F1 ↑
Acc. ↑
AUROC ↑
Tree ensembles on aggregated features
CatBoost
76.64
12.80
81.60
84.39
70.66
75.61
77.60
1.81
49.00
51.44
77.17
5.40
XGBoost
76.95
12.87
81.82
84.05
70.65
75.30
79.33
1.75
49.86
51.85
77.53
4.00
Supervised sequence models
GRU
74.33
10.27
81.48
84.52
68.08
72.85
78.05
1.31
53.79
54.80
80.17
6.00
Table 1. Development results for the evaluated configurations on five public benchmarks (%). PANTHER uses the SPRM adaptation specified in Appendix A.2 . Bold and underline indicate the best and second-best scores in each column. Avg. Rank is the mean rank over the five primary metrics.
Backbone
d
L
AlfaBattle
Retail X5
MBD
Churn
Taobao
GRU
512
2
74.59
54.34
78.05
81.48
68.08
+ TAPES(Ours)
76.05 (+1.96%)
53.85 (-0.90%)
75.93 (-2.72%)
81.56 (+0.10%)
71.03 (+4.33%)
LSTM
512
2
74.78
52.97
77.15
80.90
67.15
+ TAPES(Ours)
75.51 (+0.98%)
53.37 (+0.76%)
77.41 (+0.34%)
81.78 (+1.09%)
70.36 (+4.78%)
Transformer
256
4
75.57
53.60
79.27
80.90
69.85
+ TAPES(Ours)
78.09 (+3.33%)
54.30 (+1.31%)
79.95 (+0.86%)
82.99 (+2.58%)
70.55 (+1.00%)
Table 2. Development performance across encoder configurations (%). Parentheses show relative percentage changes from the displayed supervised score. Retail uses Macro-F1, MBD-mini uses macro-AUROC, and the remaining datasets use AUROC. d and L denote hidden dimension and depth. Bold marks the higher score within each two-row group.
Figure 3. Development performance at different labeled-data fractions for downstream fine-tuning on AlfaBattle, Retail X5, MBD-mini, and Rosbank Churn. Curves show means over fine-tuning seeds; shaded regions indicate ± one sample standard deviation.
Method
AUROC (%)
Passed-CB AP (%)
Relative AddCB change (%)
Scratch
90.88
5.80
0.00
CoLES
89.66
4.35
-0.93
TabBERT
90.83
4.95
-0.93
MLEM
90.13
5.47
+0.28
TAPES (Ours)
92.04
4.48
+0.30
Table 3. Validation results on TransRisk after 25k fine-tuning updates. Bold indicates the best value.
Figure 4. Industrial fine-tuning evaluations on TransRisk. (a) AUROC of TAPES and the supervised control without history pretraining after 25k fine-tuning updates. (b) Comparison with baseline using 24k pretraining updates and 25k fine-tuning updates. (c) Relative AddCB recall change at a 0.5% intervention budget, with the business baseline selected at 10k updates as reference. Stars identify checkpoints selected by the validation business criterion.
Figure 5. Frozen-representation evaluation on TransRisk. Bars show Recall@top1% for TAPES and the business baseline (left axis). The dashed curve shows TAPES’s relative recall gain over an architecture-matched random encoder (right axis).
Ablation Variant
S
J
AlfaBattle
Retail X5
AUROC ↑
AP ↑
Macro-F1 ↑
Acc. ↑
AUROC ↑
w/o Pretraining
–
–
75.25
10.80
53.18
54.17
79.66
w/o Both
×
×
76.98 (+1.73)
12.01 (+1.21)
53.32 (+0.14)
54.15 (-0.02)
79.78 (+0.12)
w/o Selection
×
✓
77.09 (+1.84)
12.18 (+1.38)
53.40 (+0.22)
54.25 (+0.08)
79.82 (+0.16)
w/o Joint Prediction
✓
×
77.12 (+1.87)
12.35 (+1.55)
53.79 (+0.61)
54.47 (+0.30)
79.90 (+0.24)
TAPES (Ours)
✓
✓
77.38 (+2.13)
12.52 (+1.72)
54.08 (+0.90)
54.60 (+0.43)
79.90 (+0.24)
Table 4. Component ablation under full fine-tuning. S : target-set selection; J : joint likelihood. All metrics within each configuration–dataset pair come from the same checkpoint. Parenthesized values indicate changes in percentage points relative to w/o Pretraining. Green and red indicate increases and decreases, respectively. AP denotes average precision; Retail AUROC is macro one-vs-rest AUROC. Bold marks the best scores at the reported precision.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Value
Training examples
24,698,621
Validation examples
4,825,198
Total examples
29,523,819
Categorical attributes per historical event
11
Maximum historical events
300
Current-transaction feature dimension
652
Appendix
Table 5. TransRisk data used in the experiments.
Dataset
Task
# Train sequences
# Train events
Primary metric
Alpha
Credit default
674,667
106,567,621
AUROC
Retail X5
Age classification
222,822
25,324,802
Macro-F1
MBD-mini
Product purchase, four labels
499,930
10,917,772
Macro-AUROC
Rosbank Churn
Churn prediction
3,475
333,261
AUROC
Taobao
Future purchase
12,837
3,220,892
AUROC
Appendix
Table 6. Dataset summary. Public sequence and event counts refer to the full downstream training sets after eligibility filtering and history truncation. Event counts exclude padding; validation and test partitions are not included in these counts.
Setting
Configuration
Optimizer
AdamW
Weight decay
0.01
Gradient-norm clipping
1.0
Effective batch size
At most 128 ; final batch retained
Task-head learning rate
10−3
Maximum/minimum epochs
10/3
Appendix
Table 7. Reference configuration for standard sequence-classification adaptation.
Figure 6. Effect of the Stage 2 pretraining budget on Rosbank Churn with the CT256 × 4 backbone. (a) Training negative log-likelihood per target over 60 epochs. (b) Stage 2 optimizer updates (bars, left axis) and development AUROC after fine-tuning (line, right axis) at 5, 10, 20, 40, and 60 epochs. The Stage 1-selected target set, fitted transformations, and fine-tuning protocol are fixed across budgets. Each score uses the fine-tuning checkpoint selected by early-stopping AUROC.