CARTA: Self-Supervised Reusable Transaction Embeddings via Heterogeneous Tokenization and Cross-Attention

Shreyas K. Chandrahas; Jose Munoz; Satish Babu Bellapu

cs.LGOct 17, 2026

CARTA: Self-Supervised Reusable Transaction Embeddings via Heterogeneous Tokenization and Cross-Attention

Authors: Shreyas K. Chandrahas, Jose Munoz, Satish Babu Bellapu

Organizations: Visa Inc. Foster City, CA, USA · Massachusetts Institute of Technology Cambridge, MA, USA

Abstract

Acquiring fraud labels for payment transactions is expensive and inconsistent across tasks, and high-cardinality identifiers such as merchant names make one-hot feature pipelines infeasible at scale. We present CARTA, a self-supervised encoder that learns a compact 16-dim embedding from raw transactions without labels; the frozen embedding is then reused for label-efficient fraud probing and exploratory unsupervised risk stratification without retraining the encoder. CARTA uses type-specific projections with learned embedding tables for high-cardinality identifiers, trained with token excision and InfoNCE contrastive loss. On IEEE-CIS, its linear probe at 1% of labels exceeds a raw-feature probe at 100%; on IBM TabFormer (2M rows), its frozen embedding reaches GBDT AUPRC 0.508, above frequency encoding (0.411) and the merchant-dropping baseline (0.022), though fully-supervised CatBoost is stronger. CARTA is a label-free, reusable representation that complements task-specific supervised models.

Figures & tables

Figure 1. CARTA system overview. Raw transaction features are projected into a uniform token space via type-specific transformations (continuous features to a dense projection, low-cardinality categoricals to a one-hot projection, and high-cardinality identifiers to learned embedding tables). Token excision randomly masks a subset of feature tokens per view as a self-supervised corruption. The Perceiver encoder compresses tokens into a 16-dim Variational Autoencoder (VAE) ( Kingma and Welling, 2014 ) latent. Training uses a composite loss (

\mathcal{L}_{\text{VAE}}+\mathcal{L}_{\text{InfoNCE}}

with cyclical KL). At inference the frozen embedding is reused across downstream tasks.

		IEEE-CIS (590,540 rows, 3.50% fraud)			TabFormer (2,000,000 rows, 0.13% fraud)
Method	Dim	Lin. AUPRC	GBDT AUPRC	GBDT AUC	Lin. AUPRC	GBDT AUPRC	GBDT AUC
Raw †	30/68	0.134 ±0.002	0.616 ±0.006	0.923	0.013 ±0.001	0.022 ±0.002	0.835
Freq. enc. ‡	35/69	—	—	—	0.023 ±0.001	0.411 ±0.056	0.871
PCA-16	16	0.112 ±0.001	0.455 ±0.005	0.882	0.012 ±0.000	0.017 ±0.001	0.818
CARTA-16 (ours)	16	0.217 ±0.034	0.465 ±0.013	0.868	0.008 ±0.009	0.508 ±0.063	0.876
CatBoost §	all	—	0.508	0.907	—	0.711	0.982

Table 1. Full-label probe performance (mean

\pm

std); Dim = representation dimensionality. Bold marks the best frozen representation per column; the fully-supervised CatBoost baseline (

\lx@sectionsign

) is excluded.

\dagger

Raw drops the high-cardinality columns (OHE infeasible);

\ddagger

frequency encoding applies only to TabFormer. Subscript stds: CARTA-16 on IEEE-CIS is the mean over 5 independent training seeds (population std; matched inner probe scores averaged per run); Raw/PCA representations are deterministic, so their stds reflect 5 probe seeds. TabFormer results are from a single training run (5 probe seeds).

Method	1%	5%	10%	25%	100%
Raw	0.162	0.129	0.131	0.136	0.134
PCA-16	0.105	0.111	0.110	0.114	0.112
UMAP-16	0.046	0.049	0.049	0.050	0.050
CARTA-16	0.206	0.219	0.215	0.219	0.217

Table 2. Linear-probe AUPRC on IEEE-CIS vs. fraction of training labels (mean; 5 training seeds for CARTA-16, 5 probe seeds otherwise). CARTA-16 at 1% labels exceeds raw features at 100% labels; the margin (CARTA-16 at 1%

=0.206

vs. raw at 100%

=0.134

) exceeds the full-label stds (

\pm 0.034

over training seeds for CARTA-16 and

\pm 0.002

over probe seeds for raw, respectively).

Figure 2. Linear-probe AUPRC vs. fraction of training labels; error bars are

\pm\sigma

over 5 training seeds for CARTA-16 and 5 probe seeds for Raw and PCA-16. On IEEE-CIS, CARTA-16 at 1% of labels (

0.206

) exceeds raw features at 100% of labels (

0.134

), so self-supervised pretraining reaches a given representation quality with far fewer labels.

	IEEE-CIS		TabFormer
Variant	Lin. AUPRC	GBDT AUPRC	Lin. AUPRC	GBDT AUPRC
Full (cell + InfoNCE + cycl. KL)	0.199	0.492	0.014	0.401
$-$ Cyclical KL ( $\beta{=}1$ )	0.208	0.481	0.219	0.431
$-$ InfoNCE (single view)	0.151	0.467	0.005	0.365
$-$ Token excision	0.186	0.425	0.031	0.513
Feature-wise excision (default)	0.241	0.469	0.009	0.533

Table 3. Ablation results (linear probe / GBDT AUPRC; each variant is a single trained encoder, Perceiver unless noted, evaluated as the mean over 5 probe seeds; up to 300 epochs with early stopping, batch 16,384 throughout). “Full” denotes the ablation reference configuration (cell-mode excision + InfoNCE + cyclical KL). Rows are intended for within-table comparison; the canonical CARTA numbers are those in Table 1 , whose CARTA row instead averages 5 independent training runs with per-run inner-probe averaging. The protocols therefore differ by seed population (e.g., the feature-wise row here is one training run scoring 0.241, while the Table 1 5-run mean is

0.217\pm 0.034

). TabFormer values are especially sensitive: the linear probe operates near the 0.13% base rate, and for the merchant-aware variants GBDT AUPRC varies by 0.02–0.08 (std) across the 5 probe seeds even on a fixed embedding.

Representation	Lin. AUPRC	LE@1%	GBDT AUPRC	Silhou.	Train (s)
PCA-16	0.112	0.105	0.455	0.152	$<1$
UMAP-16	0.050	0.046	0.357	0.202	1,318
Autoencoder	0.179	0.147	0.468	0.215	319
Sparse AE	0.139	0.135	0.446	0.225	303
MLP-VAE	0.083	0.083	0.443	0.717	286
SCARF	0.214	0.217	0.491	0.086	788

Table 4. Unsupervised representation comparison on IEEE-CIS. Every method produces a 16-dim label-free embedding scored with the same probes, K-means silhouette (best

k\in[3,7]

), and training wall-time. Lin. AUPRC is the mean over 5 probe seeds; LE@1% is that probe at 1% of training labels; each embedding is a single timed training run (UMAP timed multi-threaded). In this single-run comparison CARTA’s full-label linear AUPRC (

0.212

) is similar to SCARF’s (

0.214

) with a shorter wall-clock time (64% of SCARF’s), while SCARF leads on LE@1% (

0.217

vs.

0.187

) and GBDT AUPRC (

0.491

vs.

0.450

); the high-silhouette methods (MLP-VAE, UMAP) have weak probes, so silhouette does not track downstream usefulness here.

Figure 3. Representation quality (linear-probe AUPRC, 5 probe seeds) versus training wall-clock time for seven label-free 16-dim encoders on IEEE-CIS (one timed training run per method). CARTA’s (blue) full-label linear AUPRC is similar to SCARF’s at 64% of SCARF’s single-run training time. Error bars are

\pm\sigma

over probe seeds.

Figure 4. Unsupervised behavioral archetypes discovered by K-means (

k=3

, silhouette

=0.085

) on the full IEEE-CIS CARTA latent space (590K transactions, no fraud labels used). The highest-incidence archetype (103K transactions) shows an observed fraud rate of 8.4%, nearly 4.0

\times

the rate of the lowest-incidence archetype (2.1%), despite being discovered without any labeled supervision.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 798 papers

Interpretable vs Learned Encoders for High-Cardinality Fraud Detection
Jul 1, 2026Xiao Han, Jingjing Liu, Moxuan Zheng +2Financial Fraud Detection Tabular ML
Goizueta Business School Emory University Atlanta, GA, USA · Electrical Engineering and Computer Sciences University of California, Berkeley Kirkland, WA · Stern School of Business New York University New York, NY, USA +2
Temporal Heterogeneous Graph Transformer for Credit Card Fraud Detection
Sep 7, 2026Qinwen YanFinancial Fraud Detection Transformer
University of California, Los Angeles Los Angeles, California, USA
Topological Fraud Detection in Latent Transaction Spaces
Sep 8, 2026Avraham BourlaFinancial Fraud Detection Privacy-Preserving ML

Jul 1, 2026J/K move · Enter open · S save

Interpretable vs Learned Encoders for High-Cardinality Fraud Detection

Xiao Han, Jingjing Liu, Moxuan Zheng, Zhen Zhang+1

Goizueta Business School Emory University Atlanta, GA, USA · Electrical Engineering and Computer Sciences University of California, Berkeley Kirkland, WA · Stern School of Business New York University New York, NY, USA+2

Financial Fraud Detection Tabular ML Automated Feature Engineering

Open paper AskPDF arXiv

A total of seven categorical encoding methods were tested on the IEEE-CIS fraud benchmark dataset (590,540 records, 3.5% positives, 8 high-cardinality columns). The encoders were evaluated using a stratified 5-fold cross-validation (CV) with three repetitions. Five of the encoders had identical frozen LightGBM learners in the downstream phase, allowing for controlled comparisons of their performance to each other. CatBoost and TabNet were included as comparisons across paradigms using different learners. The entity embeddings produced the highest AUC-ROC (0.9612), with a statistically significant tie with that of CatBoost (0.9602) and statistically superior to tier group encoding (0.9548), whereas target encoding was only 0.0023 worse than tier group encoding and the auditor-friendly tier boundaries were maintained. Off-the-shelf TabNet did not outperform tree-based pipelines and collapsed under data scarcity. On AUC-PR, CatBoost leads (0.822 vs. 0.793); no encoder dominated both metrics. Per-column analysis confirmed the embedding advantage arises from joint multi-column representation.

Abstract

Representation Learning

Explore similar work

Interpretable vs Learned Encoders for High-Cardinality Fraud Detection

Temporal Heterogeneous Graph Transformer for Credit Card Fraud Detection

Topological Fraud Detection in Latent Transaction Spaces