Large Language Models (LLMs) have shown strong performance on text, but whether pretrained LLMs are effective for large-scale tabular classification remains unclear, as many previous works demonstrate that LLMs' tabular data classification performance falls behind tree-based models or neural architectures designed for tabular data. At Visa Research, we managed to fine-tune a pretrained Llama 3.2 1B LLM on billions of rows of labeled tabular data and make it outperform a wide array of tabular classifiers and even our competitive production deep learning model. For ease of reference, we refer to the model as the Visa Research LLM (VRLLM). In this paper, we share how we achieved this exciting result, covering data preparation, training recipe, ablation studies of design choices and deployment validation. We also benchmark frontier LLMs on the same task and analyze their prediction behavior across model families and generations. Our interpretability experiments suggest that VRLLM learns feature-specific predictive associations, while component ablations indicate that the earliest two FFN layers contribute to the emergence of linearly decodable task information. Our paper provides an engineering recipe and a focused empirical case study on adapting compact LLMs to industrial tabular prediction tasks.
Figures & tables
Figure 1: Classification performance of frontier models, tabular classifiers, the Visa production model, and VRLLM. Performance is reported as recall at the selected operating false-positive rate (OFPR). All experiments were conducted on offline data in a non-production environment for research purposes. Models marked ∗ were evaluated zero-shot on 10,000 examples; all others were evaluated on a larger dataset. Models marked † were evaluated few-shot without fine-tuning, while the remaining models were trained or fine-tuned on the same source data. See Sec. 4 for more evaluation details and performance comparisons.
Figure 2: Three different tabular data serialization templates. The data and templates are used for illustration purposes and do not reflect the appearance or contents of the production data. The label token is masked during fine-tuning and inference.
Table 2: Performance on a large-scale temporal holdout test set. Models are grouped by whether they undergo task-specific training or fine-tuning. Bold and underlined values indicate the best and second-best results, respectively.
Figure 3: Zero-shot classification performance on a 10,000-transaction subset across frontier model families and generations.
Table 6Figure 7Figure 8
Relative PR-AUC (%)
Relative Recall@OFPR (%)
Population
VRLLM
Production Model
VRLLM
Production Model
Matched by Transaction Day
100.0
100.0
100.0
100.0
Matched by Product Code
99.4
99.9
97.8
97.4
Matched by Dollar Bucket
98.8
99.1
93.5
93.8
Matched by MCC
92.1
86.7
87.9
83.7
Unseen Merchant
86.7
73.8
81.5
68.0
Table 5: VRLLM vs. the production model on the unseen-merchant test set and four seen-merchant control populations. PR-AUC and Recall@OFPR are each normalized to that model’s own Matched-by-Transaction-Day population (100%) so the relative drop for each population is directly comparable.
Model
Unique true positives (% of all positives)
Feature A
Feature B
Feature C
VRLLM
8.0%
53.2
8.7
5.5
Production Model
7.9%
33.6
2.7
13.1
Table 6: Disagreement analysis of VRLLM and production model on the same evaluation data. The number under each proprietary feature (A, B, and C) represents its average value in the corresponding true-positive group.
Source
Target
Target association
Typical prediction probability response
Code-A
Code-B
Much higher training-sample positive rate
Increases for both ground-truth classes
Code-B
Code-A
Much lower training-sample positive rate
Decreases for both ground-truth classes
Table 7: A summary of the prior learned for a product-code feature. Code names and prior levels are illustrative for compliance; the response directions summarize the feature-swap analysis.
Source
Target
Training data positive rate
Typical prediction probability response
Low bucket
High bucket
Similar across buckets
Increases for both ground-truth classes
High bucket
Low bucket
Similar across buckets
Decreases for both ground-truth classes
Table 8: A summary of the prior learned for the dollar-bucket feature. Buckets are described only by relative direction (low to high) to avoid revealing production boundaries.
Figure 8: Balanced binary linear-probe accuracy by decoder layer under the baseline (unablated) model, with layer-0 and layer-1 FFN ablation shown individually and the range spanning single-head ablation (layers 0–15, 512=32×16 heads in total) and layer-2–15 FFN ablation shown as bands.
Product Code
I
K
Θ
Ξ
E
N
Λ
M
O
Σ
Positive Rate
45.20%
42.67%
38.29%
35.56%
34.23%
33.27%
30.57%
26.71%
26.25%
24.99%
Product Code
P
T
Z
Π
Δ
Γ
B
Υ
A
Positive Rate
16.70%
15.76%
0.50%
0.24%
0.00%
0.00%
0.00%
0.00%
0.00%
Table 9: Training-set positive rate for a 19-code subset of the product-code feature (sorted by positive rate, descending). Each code is replaced by a fixed pseudonymous symbol since the underlying product-code values cannot be disclosed; the encoding is arbitrary but matches Figures 9 – 10 . Negative-class records were downsampled so these rates are not representative of the true population rate.
Figure 9: Product-code counterfactual swap, true-positive source instances: mean signed change in predicted positive-class probability (post-swap minus original) when the source product code (rows) is replaced by the target product code (columns), restricted to a 12-code (rows) by 20-code (columns) subset; rows are a subset of the codes in Table 9 , and columns additionally include code H (never seen during training, absent from Table 9 ). Axis labels give, in parentheses, the number of true-positive transactions actually sampled for each code as a source (up to 10,000 ). Diverging color scale centered at zero.
Figure 10: Same construction as Figure 9 , restricted to true-negative source instances.
Dollar Bucket
Unknown
I
II
III
IV
V
VI
VII
VIII
IX
Positive Rate
0.00%
31.65%
33.01%
29.70%
31.50%
31.07%
32.74%
36.83%
34.89%
32.44%
Table 10: Training-set positive rate for the 10 dollar-buckets (ordered by increasing bucket). Buckets are encoded as Roman numerals in their true increasing order since the actual dollar ranges cannot be disclosed; bucket Unknown is a non-numeric category outside this ordering. Negative-class records were downsampled, so these rates are not representative of the true population rate.
Figure 11: Dollar-bucket counterfactual swap: mean signed change in predicted positive-class probability (post-swap minus original) when the source bucket (rows) is replaced by the target bucket (columns), restricted to ten observed training categories of Table 10 , arranged in their true increasing order. Left: true-positive source instances; right: true-negative source instances, sharing the same diverging color scale (centered at zero) for direct comparison. Axis labels give, in parentheses, the number of transactions actually sampled for each bucket as a source, for that panel’s ground-truth class (up to 10,000 ).
Figure 12: Spearman rank correlation of zero-shot scores across frontier LLMs on the 10K transaction classification sample. Models are ordered by hierarchical clustering on 1−ρ so related models appear as contiguous blocks. The gold box marks a tightly coupled cluster whose within-group correlations range from 0.63 to 0.86 . The red outline highlights Gemini-4-argon, whose outputs are the most decoupled from the rest (mean correlation 0.26 , maximum 0.44 ).
Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)-(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.
Large language models (LLMs) have recently been adapted to tabular prediction by serializing structured features into natural language, but their performance in low-data regimes remains limited compared to gradient-boosted decision trees (GBDTs). In this work, we revisit the boosting paradigm, traditionally associated with tree ensembles, and ask whether it can be applied as a general training principle for LLM fine-tuning. We propose BoostLLM, a framework that transforms parameter-efficient fine-tuning into a multi-round residual optimization process by training sequential PEFT adapters as weak learners. To incorporate tabular inductive bias, BoostLLM integrates decision-tree paths as a second input view alongside raw features; analysis reveals that the path view acts as a structured teacher in early training steps before the model shifts toward feature-driven representations. Empirically, BoostLLM achieves consistent improvements over standard fine-tuning across multiple LLM backbones and datasets, matching or surpassing XGBoost across a wide range of shot counts and outperforming GPT-4o-based methods with a 4B model. We further show that the framework scales: pairing with stronger tree models and extended boosting horizons yields additional gains under appropriate stabilization. These results suggest that boosting can serve as a general training principle for LLM fine-tuning, particularly in low-data regimes for structured data.
Tabular prediction has long been dominated by gradient-boosted decision trees and specialized deep tabular models, while large language models (LLMs) remain difficult to make competitive despite their cross-task adaptability and transparent reasoning traces. We address this gap by incorporating tabular structural priors into LLM post-training. Specifically, we propose Permutation Relative Policy Optimization (PRPO), which operationalizes column-permutation invariance through label-preserving column permutations and two-level advantage estimation. This design converts sparse outcome rewards into denser and more stable optimization signals. Extensive experiments on 139 OpenML datasets show that our 8B model reaches a genuinely competitive regime against strong specialized tabular baselines. It achieves strong fully supervised performance, dominates zero-shot settings, and performs on par with 32-shot strong baselines. Moreover, it substantially outperforms much larger general-purpose and reasoning LLMs, including up to a 53.17% improvement over DeepSeek-R1 (685B). These results show that structural-prior RL post-training is an effective route for making LLMs competitive in tabular prediction.
Pengxiang Cai, Zihao Gao, Wanchen Lian +2
The Hong Kong University of Science and Technology (Guangzhou) · Zhejiang University