Large Language Models (LLMs) have shown strong performance on text, but whether pretrained LLMs are effective for large-scale tabular classification remains unclear, as many previous works demonstrate that LLMs' tabular data classification performance falls behind tree-based models or neural architectures designed for tabular data. At Visa Research, we managed to fine-tune a pretrained Llama 3.2 1B LLM on billions of rows of labeled tabular data and make it outperform a wide array of tabular classifiers and even our competitive production deep learning model. For ease of reference, we refer to the model as the Visa Research LLM (VRLLM). In this paper, we share how we achieved this exciting result, covering data preparation, training recipe, ablation studies of design choices and deployment validation. We also benchmark frontier LLMs on the same task and analyze their prediction behavior across model families and generations. Our interpretability experiments suggest that VRLLM learns feature-specific predictive associations, while component ablations indicate that the earliest two FFN layers contribute to the emergence of linearly decodable task information. Our paper provides an engineering recipe and a focused empirical case study on adapting compact LLMs to industrial tabular prediction tasks.
Figures & tables
Figure 1: Classification performance of frontier models, tabular classifiers, the Visa production model, and VRLLM. Performance is reported as recall at the selected operating false-positive rate (OFPR). All experiments were conducted on offline data in a non-production environment for research purposes. Models marked ∗ were evaluated zero-shot on 10,000 examples; all others were evaluated on a larger dataset. Models marked † were evaluated few-shot without fine-tuning, while the remaining models were trained or fine-tuned on the same source data. See Sec. 4 for more evaluation details and performance comparisons.
Figure 2: Three different tabular data serialization templates. The data and templates are used for illustration purposes and do not reflect the appearance or contents of the production data. The label token is masked during fine-tuning and inference.
Table 2: Performance on a large-scale temporal holdout test set. Models are grouped by whether they undergo task-specific training or fine-tuning. Bold and underlined values indicate the best and second-best results, respectively.
Figure 3: Zero-shot classification performance on a 10,000-transaction subset across frontier model families and generations.
Table 6Figure 7Figure 8
Relative PR-AUC (%)
Relative Recall@OFPR (%)
Population
VRLLM
Production Model
VRLLM
Production Model
Matched by Transaction Day
100.0
100.0
100.0
100.0
Matched by Product Code
99.4
99.9
97.8
97.4
Matched by Dollar Bucket
98.8
99.1
93.5
93.8
Matched by MCC
92.1
86.7
87.9
83.7
Unseen Merchant
86.7
73.8
81.5
68.0
Table 5: VRLLM vs. the production model on the unseen-merchant test set and four seen-merchant control populations. PR-AUC and Recall@OFPR are each normalized to that model’s own Matched-by-Transaction-Day population (100%) so the relative drop for each population is directly comparable.
Model
Unique true positives (% of all positives)
Feature A
Feature B
Feature C
VRLLM
8.0%
53.2
8.7
5.5
Production Model
7.9%
33.6
2.7
13.1
Table 6: Disagreement analysis of VRLLM and production model on the same evaluation data. The number under each proprietary feature (A, B, and C) represents its average value in the corresponding true-positive group.
Source
Target
Target association
Typical prediction probability response
Code-A
Code-B
Much higher training-sample positive rate
Increases for both ground-truth classes
Code-B
Code-A
Much lower training-sample positive rate
Decreases for both ground-truth classes
Table 7: A summary of the prior learned for a product-code feature. Code names and prior levels are illustrative for compliance; the response directions summarize the feature-swap analysis.
Source
Target
Training data positive rate
Typical prediction probability response
Low bucket
High bucket
Similar across buckets
Increases for both ground-truth classes
High bucket
Low bucket
Similar across buckets
Decreases for both ground-truth classes
Table 8: A summary of the prior learned for the dollar-bucket feature. Buckets are described only by relative direction (low to high) to avoid revealing production boundaries.
Figure 8: Balanced binary linear-probe accuracy by decoder layer under the baseline (unablated) model, with layer-0 and layer-1 FFN ablation shown individually and the range spanning single-head ablation (layers 0–15, 512=32×16 heads in total) and layer-2–15 FFN ablation shown as bands.
Product Code
I
K
Θ
Ξ
E
N
Λ
M
O
Σ
Positive Rate
45.20%
42.67%
38.29%
35.56%
34.23%
33.27%
30.57%
26.71%
26.25%
24.99%
Product Code
P
T
Z
Π
Δ
Γ
B
Υ
A
Positive Rate
16.70%
15.76%
0.50%
0.24%
0.00%
0.00%
0.00%
0.00%
0.00%
Table 9: Training-set positive rate for a 19-code subset of the product-code feature (sorted by positive rate, descending). Each code is replaced by a fixed pseudonymous symbol since the underlying product-code values cannot be disclosed; the encoding is arbitrary but matches Figures 9 – 10 . Negative-class records were downsampled so these rates are not representative of the true population rate.
Figure 9: Product-code counterfactual swap, true-positive source instances: mean signed change in predicted positive-class probability (post-swap minus original) when the source product code (rows) is replaced by the target product code (columns), restricted to a 12-code (rows) by 20-code (columns) subset; rows are a subset of the codes in Table 9 , and columns additionally include code H (never seen during training, absent from Table 9 ). Axis labels give, in parentheses, the number of true-positive transactions actually sampled for each code as a source (up to 10,000 ). Diverging color scale centered at zero.
Figure 10: Same construction as Figure 9 , restricted to true-negative source instances.
Dollar Bucket
Unknown
I
II
III
IV
V
VI
VII
VIII
IX
Positive Rate
0.00%
31.65%
33.01%
29.70%
31.50%
31.07%
32.74%
36.83%
34.89%
32.44%
Table 10: Training-set positive rate for the 10 dollar-buckets (ordered by increasing bucket). Buckets are encoded as Roman numerals in their true increasing order since the actual dollar ranges cannot be disclosed; bucket Unknown is a non-numeric category outside this ordering. Negative-class records were downsampled, so these rates are not representative of the true population rate.
Figure 11: Dollar-bucket counterfactual swap: mean signed change in predicted positive-class probability (post-swap minus original) when the source bucket (rows) is replaced by the target bucket (columns), restricted to ten observed training categories of Table 10 , arranged in their true increasing order. Left: true-positive source instances; right: true-negative source instances, sharing the same diverging color scale (centered at zero) for direct comparison. Axis labels give, in parentheses, the number of transactions actually sampled for each bucket as a source, for that panel’s ground-truth class (up to 10,000 ).
Figure 12: Spearman rank correlation of zero-shot scores across frontier LLMs on the 10K transaction classification sample. Models are ordered by hierarchical clustering on 1−ρ so related models appear as contiguous blocks. The gold box marks a tightly coupled cluster whose within-group correlations range from 0.63 to 0.86 . The red outline highlights Gemini-4-argon, whose outputs are the most decoupled from the rest (mean correlation 0.26 , maximum 0.44 ).