cs.LGFeb 23, 2026
SaveMultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning
Organizations: Samsung Electronics, Hwaseong, South Korea · Seoul National University of Science and Technology, Seoul, South Korea
Abstract
Recently, TabPFN has gained attention as a foundation model for tabular data. However, it struggles to integrate heterogeneous modalities such as images and text, which are common in domains like healthcare and marketing, thereby limiting its applicability. To address this, we present the Multi-Modal Prior-data Fitted Network (MMPFN), which extends TabPFN to handle tabular and non-tabular modalities in a unified manner. MMPFN comprises per-modality encoders, modality projectors, and pre-trained foundation models. The modality projectors serve as the critical bridge, transforming non-tabular embeddings into tabular-compatible tokens for unified processing. To this end, we introduce a multi-head gated MLP and a cross-attention pooler that extract richer context from non-tabular inputs while mitigates attention imbalance issue in multimodal learning. Extensive experiments on medical and general-purpose multimodal datasets demonstrate that MMPFN consistently outperforms competitive state-of-the-art methods and effectively exploits non-tabular modalities alongside tabular features. These results highlight the promise of extending prior-data fitted networks to the multimodal setting, offering a scalable and effective framework for heterogeneous data learning. The source code is available at https://github.com/too-z/MultiModalPFN.
Figures & tables
Figure 1: An overview of MMPFN. MMPFN extends TabPFN by incorporating per-modality encoders and a modality projector to extract features from non-tabular data. Newly developed components are highlighted in color, while existing ones appear in gray. Layers marked as ‘frozen’ remain fixed during fine-tuning, whereas all others are trainable. Encoded target labels are part of the training inputs but are omitted from the diagram for clarity.
| Dataset | # train samples | # test samples | # features | # numeric features | # categorical features | # images | # text | # classes |
|---|---|---|---|---|---|---|---|---|
| PAD-UFES-20 [ 45 ] | 1838 | 460 | 21 | 3 | 18 | 1 | 0 | 6 |
| CBIS-DDSM(Mass) [ 49 ] | 1318 | 378 | 8 | 3 | 5 | 3 | 0 | 2 |
| CBIS-DDSM(Calc) [ 49 ] | 1545 | 326 | 8 | 3 | 5 | 3 | 0 | 2 |
| Airbnb [ 30 ] | 18316 | 4579 | 50 | 27 | 23 | 0 | 1 | 10 |
| Salary [ 33 ] | 15841 | 3961 | 4 | 1 | 3 | 0 | 3 | 6 |
| Cloth [ 32 ] | 18788 | 4698 | 5 | 2 | 3 | 0 | 3 | 5 |
Table 1: Statistics of the multimodal datasets used in our experiments. “# images” and “# text” denote the number of image and text fields per sample, respectively. PetFinder variants indicate which modalities (tabular (T), image (I), text (t)) are used.
| Method | PU20 | Mass | Calc | Petfinder | Avg. |
|---|---|---|---|---|---|
| TabPFN [ 25 ] | 82.17 (2) | 71.27 (5) | 73.31 (2) | 36.33 (8) | 4.25 |
| Catboost [ 47 ] | 80.43 (4) | 78.31 (1) | 72.09 (4) | 38.69 (4) | 3.25 |
| AutoGluon [ 57 ] | 81.09 (3) | 76.28 (2) | 71.04 (6) | 38.81 (3) | 3.50 |
| MMCL [ 20 ] | 76.61 (7) | 57.62 (7) | 60.12 (8) | 36.61 (7) | 7.25 |
| TIP [ 14 ] | 78.75 (6) | 73.12 (4) | 67.96 (7) | 37.28 (5) | 5.50 |
| HEALNet [ 24 ] | 74.65 (8) | 68.10 (6) | 71.83 (5) | 37.03 (6) | 6.25 |
Table 2: Comparison with state-of-the-art on tabular-image multimodal datasets . Results are reported as accuracy (rank), averaged over five random seeds, where lower rank indicates better performance. “Avg.” denotes the mean accuracy across datasets. Best and second-best results are marked in bold and underline .
| Method | Airbnb | Salary | Cloth | Petfinder | Avg. |
|---|---|---|---|---|---|
| TabPFN [ 25 ] | 46.96 (2) | 44.96 (6) | 55.07 (9) | 36.33 (7) | 6.00 |
| Catboost [ 47 ] | 43.56 (4) | 40.36 (9) | 59.24 (8) | 35.47 (8) | 7.25 |
| AutoGluon [ 57 ] | 44.60 (3) | 45.24 (5) | 72.07 (1) | 37.96 (4) | 3.25 |
| AllTextBERT [ 3 ] | 30.9 (9) | 44.0 (7) | 68.0 (3) | 34.6 (9) | 7.00 |
| TFN [ 61 ] | 35.7 (8) | 45.8 (3) | 60.1 (7) | 36.8 (6) | 6.00 |
| MulT [ 59 ] | 36.3 (7) | 45.4 (4) | 63.6 (6) | 37.6 (5) | 5.50 |
Table 3: Comparison with state-of-the-art on tabular-text multimodal datasets . Results are reported as accuracy (rank), averaged over five random seeds, where lower rank indicates better performance. “Avg.” denotes the mean accuracy across datasets. Best and second-best results are marked in bold and underline .
Figure 2: Performance on PU20 versus the number of non-tabular tokens. (a) Image-only results with DINOv2 and an MLP baseline. (b) Multimodal results under token imbalance. The y-axis shows accuracy and the x-axis shows the number of MGM heads. In (b), MGM+CAP uses 24 CAP heads.
Figure 3: Token count and attention mass . Attention mass for tabular and non-tabular tokens is measured as the number of non-tabular tokens varies, without CAP. Values are averaged over 12 self-attention layers in TabPFN. PU20 and Salary use 11 and 4 tabular tokens. The x-axis is in log scale.
| Method | 32 | 4 32 | 128 |
|---|---|---|---|
| MGM | 84.17 | 82.43 | 83.91 |
| MGM+CAP | 84.10 | 83.57 | 84.59 |
Table 4: Token count and representation quality on PU20 and Cloth . We compare non-tabular token configurations on PU20 (left) and Cloth (right). Columns 32 and 128 denote and MGM heads, and 4 32 denotes four repetitions of the setting. For MGM+CAP, is fixed to 24 and 4, respectively.
| Category | Method | PU20 | Mass | Calc | Cloth | Salary | Airbnb | PetFinder-I | PetFinder-T | PetFinder-A | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Single-head | Linear | 83.48 | 66.19 | 74.17 | 58.06 | 44.67 | 47.02 | 37.22 | 36.64 | 37.30 | 53.86 |
| MLP | 83.78 | 64.87 | 73.87 | 60.44 | 44.33 | 46.85 | 37.22 | 36.64 | 37.30 | 54.14 | |
| Multi-head | MLP | 84.39 | 67.99 | 73.99 | 64.39 | 45.93 | 46.79 | 40.64 | 38.12 | 40.04 | 55.81 |
| MoE | 83.22 | 66.67 | 73.13 | 55.07 | 44.25 | 46.12 | 36.82 | 36.83 | 36.94 | 53.23 | |
| MGM | 85.22 | 74.53 | 75.40 | 66.26 | 46.17 | 47.78 | 40.70 | 39.04 | 41.19 | 57.37 |
Table 5: Ablation of the design of MGM. We compare single-head and multi-head feature extraction mechanisms (Linear/MLP, MoE, and the proposed MGM) across all datasets. MoE denotes Mixture-of-Experts. Best results are highlighted in bold .
| Method | PU20 | Mass | Calc | Cloth | Salary | Airbnb | PetFinder-I | PetFinder-T | PetFinder-A | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| FiLM [ 46 ] | 80.00 | 73.02 | 73.25 | 65.82 | 42.27 | 46.06 | 39.84 | 38.31 | 40.77 | 55.48 |
| CAP | 85.22 | 74.53 | 75.40 | 66.26 | 46.17 | 47.78 | 40.70 | 39.04 | 41.19 | 57.37 |
Table 6: Ablation of the design of CAP. We compare two mechanisms for incorporating non-tabular information (FiLM and the proposed CAP) on all datasets. FiLM denotes Feature-wise Linear Modulation. Best results are highlighted in bold .
Figure 4: Cosine similarity between multimodal feature embeddings . Axes denote all tabular and text/image features. From left to right and top to bottom, it shows the correlations between features in the experiments on the PU20, Calc, Cloth, Mass, Petfinder, and Airbnb datasets.
| PU20 | Mass | Calc | PetFinder | Avg. | |
|---|---|---|---|---|---|
| TIP [ 14 ] 10% | 70.44 | 68.31 | 62.27 | 34.86 | 58.97 |
| (-10.6) | (-6.58) | (-8.37) | (-6.49) | (-8.00) | |
| MMPFN 10% | 72.87 | 76.13 | 72.09 | 35.73 | 64.21 |
| (-14.14) | (+0.75) | (-5.23) | (-12.30) | (-10.27) |
Table 7: Performance in Low-data Regime . Each method uses two rows: accuracy (top) and percentage change vs. full-data (bottom). The ‘Avg.’ column averages percentage changes only. Best results are highlighted in bold .
Figure 5: Accuracy of AutoGluon vs. MMPFN on PetFinder under different modality combinations: tabular, +text, +image, +image+text.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Effect of the ratio between tabular and non-tabular features. The black dots and vertical lines show the mean and variance across five random seeds. Darker black dots correspond to a larger number of MGM heads (i.e., more non-tabular features generated by MGM), ranging from 8 to 128. The red dot indicates the average result across all MGM-head settings. The x-axis shows the number of non-tabular features generated by CAP, and the y-axis denotes accuracy. The blue line represents the number of tabular features. Dataset names are shown above each subfigure.
| CBIS–DDSM (MASS) | Salary | |||
|---|---|---|---|---|
| Activation | Accuracy | Orthogonality | Accuracy | Orthogonality |
| GELU | 72.09 | 0.0565 | 45.04 | 0.04876 |
| GLU | 75.10 | 0.0913 | 45.87 | 0.05831 |
Table S1 : Effect of activation choice in MGM. Comparison of accuracy and mean output-vector orthogonality when using GELU versus GLU, with and without orthogonality loss, on CBIS–DDSM (MASS) and Salary.
| Encoder | PU20 | Mass | Calc | Petfinder |
|---|---|---|---|---|
| ResNet50 | 83.26 | - | 73.94 | - |
| DINOv2 | 85.22 | 74.53 | 75.40 | 40.74 |
| DINOv3 | 85.61 | 75.48 | 76.75 | 40.57 |
Table S2 : Effect of replacing the image encoder. Performance of MMPFN when substituting DINOv2 with ResNet50 and DINOv3. Results are reported as averaged accuracy over five random seeds.
| Encoder | Airbnb | Salary |
|---|---|---|
| Electra | 47.78 | 46.17 |
| DeBERTa | 47.82 | 45.69 |
Table S3 : Effect of replacing the text encoder. Performance of MMPFN when using different pretrained text encoders. Results are reported as averaged accuracy over five random seeds.
Figure S2 : Additional results of token-count and attention mass . We extend the analysis in Figure 3 to additional datasets. For Mass and Calc, each sample contains three images, and thus the number of non-tabular tokens is given by the number of mgm heads times 3. The number of tabular tokens is 25, 4 and 4 for Airbnb, Mass, and Calc, respectively.
Figure S3 : Effect of distribution-alignment and orthogonality regularization. Under the same experimental conditions as Figure 2 , we compare the performance of variants that incorporate orthogonality and MMD-based regularization constraints to the MGM baseline, for both image-only and tabular+image settings.
Figure S4 : Fine-tuning loss curves of MMPFN across datasets. Each subfigure shows the validation loss over training steps for four different datasets, illustrating stable convergence behavior under our fine-tuning setup.
More in this topic
Tabular Foundation Models
Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.
See all 188 papers