MotherTree: Meta-learning on synthetic data improves decision tree training

Ziyuan Wang; Fredrik D. Johansson

cs.LGOct 7, 2026

MotherTree: Meta-learning on synthetic data improves decision tree training

Authors: Ziyuan Wang, Fredrik D. Johansson

Organizations: Chalmers University of Technology University of Gothenburg

Abstract

Conventional decision tree algorithms produce effective, transparent models that can be audited, communicated, and deployed independently of the training data, but require learning every new task from scratch. In contrast, tabular foundation models demonstrate that meta-learning from a synthetic prior distribution enables strong in-context prediction for previously unseen tasks, especially in small-sample regimes. However, this approach does not produce a standalone model that can be inspected in isolation. We introduce MotherTree, a tabular transformer that meta-learns decision tree induction: given a training set for a new task, it outputs a hard, axis-aligned decision tree, equivalent in form to classically trained trees, in a single forward pass. MotherTree is pre-trained on a synthetic prior using stochastic gradient descent without requiring reference trees for supervision. On established benchmarks with controlled sample size, the approach is competitive with size-matched trees from common algorithms: recursive partitioning, gradient-based tree learning, globally optimal trees, and distillation from tabular foundation models. Notably, MotherTree consistently improves over from-scratch gradient-based learning and acts as a strong initializer: task-specific tuning of the generated tree outperforms the corresponding from-scratch learner on all benchmarks and sample sizes. These results show that meta-learning can provide effective inductive biases for learning stand-alone, small decision tree classifiers.

Figures & tables

Figure 1: When applying MotherTree to a new task, labeled training data is fed to the data set encoder to generate sample embeddings that are aggregated by class, forming a single data set embedding. The tree decoder maps the data set embedding to one embedding per node in the generated tree using a transformer, ultimately predicting the node split

(\iota_{i},\tau_{i})

(for internal nodes) or value

L_{j}

(for leaves). These define an axis-aligned, deterministic tree for stand-alone inference.

Panel of Figure 2 — Figure 2: Mean AUC for varying training set sample budget, $B$ . See also Appendix Table 5 .

	TabArena $(B=200)$		OpenML-CC18-Small $(B=200)$
Method	Binary	Multiclass	Binary	Multiclass
CART	$0.670_{\pm 0.107}$	0.779 ±0.170	$0.759_{\pm 0.123}$	0.802 ±0.110
CART ←TabPFN	$0.659_{\pm 0.099}$	0.780 ±0.165	$0.748_{\pm 0.125}$	$0.795_{\pm 0.105}$
$(\alpha^{},\beta^{})$ Tsallis	$0.620_{\pm 0.108}$	$0.762_{\pm 0.167}$	$0.680_{\pm 0.155}$	$0.780_{\pm 0.115}$
DNDT	$0.595_{\pm 0.079}$	$0.595_{\pm 0.055}$	$0.650_{\pm 0.108}$	$0.643_{\pm 0.076}$
RADDT	$0.627_{\pm 0.100}$	$0.744_{\pm 0.152}$	$0.710_{\pm 0.135}$	$0.798_{\pm 0.124}$

Table 1: Experimental results (avg. AUC

\pm

std.) across tasks with training set budget

B=200

on two benchmarks. Tables for all sample size budgets can be viewed in Appendix A.5 . GradTree and MotherTree share tree parameterization. Best results shown in bold , second-best results underlined .

Panel of Figure 3 — Figure 3: Critical Difference Diagram across both benchmarks at different training set sample budgets, $B$ . More comparisons across other context budgets are reported in Appendix A.7

Figure 4: Decoder ablation on OpenML-CC18-Small and TabArena. Bars show the change in mean ROC-AUC relative to GradTree at each context size. We compare the MotherNet-style MLP decoder with our tree-structured Transformer decoder, both before and after task-specific finetuning.

Panel of Figure 5 — Figure 5: Learning parity with distractors at fixed maximum tree depth, $D=5$ . Right: $\lambda=m-(D-k)$ is the number of distractors that must be ignored to learn an optimal tree. Green ( $\lambda\leq 0$ ): the depth budget has slack and splitting on a distractor does not rule out an optimal model. Red ( $\lambda>0$ ): the budget is exhausted, so each distractor used takes a level away from the parity tree; at least $\lambda$ of them must be refused. Results across all settings could be seen in Appendix A.8.3 .

Figure 6: Number of uninformative paths (lower is better) and leaf utilization (higher is better).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Context	TabICL Encoder	MotherNet Encoder
100	0.632	0.639
200	0.652	0.657
400	0.671	0.675
800	0.679	0.685
1000	0.681	0.687

Appendix

Table 2: Mean AUC for different encoders on TabArena.

Training Scheme	Mean AUC
Stage 2 only	0.685
Stage 1 + Stage 2	0.752

Appendix

Table 3: Impact of Stage 1 pre-training(

400k

steps) on validation sets from OpenML-CC18 ( Müller et al., 2025 ) after

125k

steps.

Model	Hyperparameters
CART	criterion $\in\{$ gini, entropy $\}$ ; min_samples_split $\in\{2,5,10\}$ ; min_samples_leaf $\in\{1,2,5\}$ .
GradTree	feature selector LR $\in\{0.001,0.005,0.010,0.050\}$ ; threshold LR $\in\{0.010,0.050,0.100,0.250\}$ ; leaf node LR $\in\{0.001,0.005,0.010,0.100\}$ ; epochs $=1000$ ; patience $=50$ .
DL85	time limit $=600$ s; max depth $=5$ ; quantile binning $=5$ .
$(\alpha^{},\beta^{})$ Tsallis	$a\in\mathrm{linspace}(1,10,10)$ ; $b\in\mathrm{linspace}(1,10,10)$ .
DNDT	learning rate $=0.05$ ; batch size $=256$ ; epochs $=300$ ; patience $=20$ ; max features $=10$ ; Gumbel temperature $1.0\rightarrow 0.1$ over 200 epochs.
RADDT	tree depth $=5$ ; epochs $=300$ ; num scale $=5$ ; start num $=1$ ; warm start: CART (1 initialization).

Appendix

Table 4: Hyperparameter search spaces and fixed settings for each model.

Benchmark	Model	100	200	400	800	1000
	CART	0.668 ±0.121	0.693 ±0.128	0.720 ±0.131	0.743 ±0.130	0.751 ±0.126
	CART ←TabPFN	0.655 ±0.114	0.684 ±0.124	0.715 ±0.132	0.751 ±0.128	0.756 ±0.126
	$(\alpha^{},\beta^{})$ Tsallis	0.645 ±0.123	0.650 ±0.133	0.655 ±0.141	0.654 ±0.141	0.654 ±0.141
	DNDT	0.629 ±0.089	0.647 ±0.094	0.659 ±0.096	0.665 ±0.094	0.662 ±0.098
	RADDT	0.642 ±0.116	0.651 ±0.121	0.670 ±0.125	0.686 ±0.132	0.688 ±0.135
	DL85	0.611 ±0.089	0.629 ±0.111	0.659 ±0.124	0.708 ±0.126	0.714 ±0.127

Appendix

Table 5: Overall mean AUC across benchmarks — model comparison per budget.

Model	100	200	400	800	1000
CART	0.647 ±0.101	0.670 ±0.107	0.697 ±0.106	0.722 ±0.108	0.731 ±0.104
CART ←TabPFN	0.634 ±0.092	0.659 ±0.099	0.690 ±0.108	0.732 ±0.107	0.738 ±0.104
$(\alpha^{},\beta^{})$ Tsallis	0.619 ±0.100	0.620 ±0.108	0.624 ±0.118	0.623 ±0.116	0.623 ±0.117
DNDT	0.586 ±0.071	0.595 ±0.079	0.607 ±0.079	0.614 ±0.088	0.616 ±0.089
RADDT	0.620 ±0.095	0.627 ±0.100	0.646 ±0.108	0.660 ±0.115	0.662 ±0.114
DL85	0.594 ±0.073	0.604 ±0.088	0.631 ±0.096	0.685 ±0.103	0.692 ±0.106

Appendix

Table 6: Mean AUC for binary classification tasks on TabArena and OpenML-CC18

Model	100	200	400	800	1000
CART	0.748 ±0.161	0.779 ±0.170	0.808 ±0.180	0.819 ±0.178	0.824 ±0.176
CART ←TabPFN	0.736 ±0.155	0.780 ±0.165	0.806 ±0.177	0.822 ±0.179	0.826 ±0.180
$(\alpha^{},\beta^{})$ Tsallis	0.744 ±0.156	0.762 ±0.167	0.769 ±0.168	0.772 ±0.171	0.771 ±0.170
DNDT	0.578 ±0.044	0.595 ±0.055	0.605 ±0.068	0.622 ±0.057	0.608 ±0.073
RADDT	0.724 ±0.153	0.744 ±0.152	0.760 ±0.149	0.780 ±0.159	0.785 ±0.171
DL85	0.675 ±0.117	0.722 ±0.142	0.763 ±0.164	0.792 ±0.174	0.798 ±0.170

Appendix

Table 7: Mean AUC for multi-class tasks on TabArena and OpenML-CC18.

		Training set budget, $B$
Hardware	Model	100	200	400	800	1000
GPU	CART ←TabPFN	0.72	0.66	0.82	0.96	1.02
	MotherTree	0.15	0.15	0.18	0.19	0.20
	MotherTree-Tune	1.57	2.15	3.15	3.82	4.75
CPU	CART ←TabPFN	1.07	1.64	2.85	4.33	4.90
CPU	MotherTree	0.21	0.17	0.24	0.33	0.35

Appendix

Table 8: Average runtime (tree fitting and test-set inference) comparison on GPU and CPU for OpenML-CC18-Small over varying training set size.

Panel of Figure 7 — Figure 7: Critical difference diagrams with different maximum context lengths for two joint benchmarks

$k$ (bits)	$m$ (distractors)	$\lambda$	$q$	$d$	#data sets
2	2, 3, 4	$-1,\;0,\;+1$	$0.50$ – $0.70$	4, 5, 6	15
3	1, 2, 3	$-1,\;0,\;+1$		4, 5, 6	15
4	1, 2, 3	$0,\;+1,\;+2$		5, 6, 7	15
5	1, 2	$+1,\;+2$		6, 7	10
2, 3, 4, 5	0 (parity control)	$-3,\;-2,\;-1,\;0$	$0.55$	$k$	4
total					59

Appendix

Table 9: XOR/parity data set configurations. The tree depth is fixed at

D=5

throughout.

k

is the number of parity bits (

y=\mathrm{XOR}

of the

k

bits),

m

the number of distractor features (independent noisy copies of

y

, each agreeing with it with probability

q

d=k+m

the feature dimensionality (all features binary) and

\lambda=m-(D-k)

is the number of distractors a model must leave unused for the parity tree to still fit below them.

\lambda\leq 0

indicates model can incorporate all bits even if all distractors are captured. Every

(k,m)

pair is generated for

q\in\{0.50,0.55,0.60,0.65,0.70\}

; the entries of the

m

\lambda

and

d

columns correspond positionally. One seed per configuration gives

59

data sets in total.

Panel of Figure 8 — Figure 8: Parity with distractors at a fixed depth budget $D=5$ . $\lambda=m-(D-k)$ is the number of distractors that must be left to learn the parity tree. Green ( $\lambda\leq 0$ ): the depth budget has slack. Splitting on a distractor does not destroy the label. Red ( $\lambda>0$ ): the budget is exhausted, so each distractor used takes a level away from the parity tree; at least $\lambda$ of them must be refused.

Explore similar work

Nov 6, 2025cs.LG

Towards Scalable Meta-Learning of near-optimal Interpretable Models via Synthetic Model Generations

Decision trees are widely used in high-stakes fields like finance and healthcare due to their interpretability. This work introduces an efficient, scalable method for generating synthetic pre-training data to enable meta-learning of decision trees. Our approach samples near-optimal decision trees synthetically, creating large-scale, realistic datasets. Using the MetaTree transformer architecture, we demonstrate that this method achieves performance comparable to pre-training on real-world data or with computationally expensive optimal decision trees. This strategy significantly reduces computational costs, enhances data generation flexibility, and paves the way for scalable and efficient meta-learning of interpretable decision tree models.

Kyaw Hpone Myint, Zhe Wu, Alexandre G. R. Day +1

Capital One

Decision Tree Learning Meta-Learning Interpretable ML Synthetic Data Pretraining Synthetic Data Generation

Ask PDF

Feb 11, 2026cs.LG

TabICLv2: A better, faster, scalable, and open tabular foundation model

Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50 GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and foster open research by releasing code for inference, pretraining, and synthetic data generation at https://github.com/soda-inria/tabicl.

Jingang Qu, David Holzmüller, Gaël Varoquaux +1

SODA Team, INRIA Saclay, Palaiseau, France · Probabl, France

Synthetic Data Pretraining Tabular Foundation Models Synthetic Tabular Data Generation Tabular ICL Tabular Prediction

Ask PDF

May 1, 2026cs.LG

Trees to Flows and Back: Unifying Decision Trees and Diffusion Models

Decision trees and diffusion models are ostensibly disparate model classes, one discrete and hierarchical, the other continuous and dynamic. This work unifies the two by establishing a crisp mathematical correspondence between hierarchical decision trees and diffusion processes in appropriate limiting regimes. Our unification reveals a shared optimization principle: \emph{Global Trajectory Score Matching (GTSM)}, for which gradient boosting (in an idealized version) is asymptotically optimal. We underscore the conceptual value of our work through two key practical instantiations: \treeflow, which achieves competitive generation quality on tabular data with higher fidelity and a 2\times computational speedup, and \dsmtree, a novel distillation method that transfers hierarchical decision logic into neural networks, matching teacher performance within 2% on many benchmarks.

Sai Niranjan Ramachandran, Suvrit Sra

School of Computation, Information and Technology, Technical University of Munich, Germany · Munich center for machine learning (MCML)

Decision Tree Learning Diffusion Models

Ask PDF