Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles

Srikumar Krishnamoorthy

cs.LGAug 16, 2026

Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles

Authors: Srikumar Krishnamoorthy

Organizations: Information Systems Area, Indian Institute of Management Ahmedabad, India

Abstract

Predictive models in clinical and regulated settings must be accurate and fully auditable. Tree ensembles deliver strong accuracy on tabular data, but their sequential boosting couples structure discovery with coefficient estimation, making compact per-prediction auditing difficult. Interpretable alternatives impose structural constraints that limit expressiveness: generalized additive models typically restrict interactions to pairwise terms and post-hoc rule extractors produce overlapping rules that hinder compact interpretation. We introduce Residual Pattern Tree Ensemble (RPTE), a three-stage learning approach, that is built on three key principles: bounded feature budget, source disjointness, and separate coefficient estimation. Stage~~1 builds a supervised symbolic feature vocabulary. Stage~~2 grows shallow trees under a source-disjointness constraint, where each raw variable is allocated to at most one tree, and retains only the discovered tree structures. Stage~3 solves a single $\ell_1$ -regularized logistic regression over leaf-region indicators, yielding jointly optimal sparse coefficients. This learning approach ensures that every prediction decomposes into an algebraic sum of named, non-overlapping rule contributions, enabling full auditability by design. Empirical evaluation on twelve clinical-domain binary classification benchmarks using repeated stratified 5-fold cross-validation shows that RPTE performs competitively against tuned opaque ensembles and interpretable baselines. RPTE reduces model inspection units by 9 $\times$ to 87 $\times$ relative to XGBoost and maintains lower audit complexity than EBM on all 12 datasets. RuleFit requires comparable or fewer inspection units on three datasets where its rule count is small, but without source-disjointness guarantees. The source code is available at this https URL.

Figures & tables

Property	RPTE	RuleFit	EBM	FIGS	SIRUS	CORELS	XGBoost
Source disjointness	Enforced	No	No	No	No	No	No
Exact source decomposition	Yes	No	Term wise	No	No	N/A	No
Separate structure/ weight fit	Yes	Yes	No	Optional refit	Fixed average	N/A	No
Discovered interactions	max depth	Tree size	Pairwise ∗	max depth	max depth	Pre-mined	max depth
Bounded audit load	Hard	L1/soft	Interaction cap	Hard split cap	Hard rule cap	$\lambda$ -implied	Indirect
Native contribution unit	Leaf/ direct term	Rule/ linear term	Term score	Leaf	Rule output	First matching rule	Leaf

Table 1: Comparison of structural properties across tabular classification methods.

Symbol	Description
$n,p_{0}$	Number of observations and raw variables
$K$	User-specified feature budget
$\mathcal{H}$	Feature vocabulary, $V=\|\mathcal{H}\|\leq p_{0}+2K$
$\omega(j)$	Source set of column $j$ , $\omega(j)\subseteq\mathcal{R}_{0}$ , $\|\omega(j)\|\leq 2$
$T$	Maximum boosting rounds
$L_{\max}$	Maximum leaves of tree

Table 2: Summary of notation.

Dataset	Short	$n$	$p$	Balance	Ratio	Domain
lupus	lupus	87	3	0.40	1.5:1	SLE diagnosis
postop. patient data	postop.	88	8	0.27	2.7:1	Discharge planning
appendicitis	append.	106	7	0.20	4.0:1	Appendicitis diagnosis
corral	corral	160	6	0.44	1.3:1	Structured interaction
hepatitis	hepatitis	155	19	0.21	3.8:1	Hepatitis survival
prnn_crabs	crabs	200	7	0.50	1.0:1	Morphological class.

Table 3: Dataset summary with class balance. Balance shows the minority class proportion; ratio shows the majority-to-minority class ratio.

Dataset	RPTE	XGBoost	LightGBM	RF	EBM	RuleFit
lupus	0.768	0.772	0.775	0.754	0.806	0.730
postop.	0.432	0.440	0.426	0.333	0.407	0.443
append.	0.797	0.806	0.797	0.818	0.847	0.774
corral	1.000	1.000	1.000	0.998	1.000	1.000
crabs	0.974	0.976	0.977	0.967	0.983	0.997
hepatitis	0.790	0.793	0.782	0.868	0.848	0.793

Table 4: Predictive performance: ROC AUC, mean over 15 outer splits. The final row is the unweighted mean of the 12 dataset-level means. Bold marks the highest AUC within each row.

Comparison	$W$	$p$	$p_{\mathrm{Holm}}$	Med. $\Delta$ AUC	Record
RPTE vs XGBoost	0	0.001	0.005 ∗	$-0.006$	0-1-11
RPTE vs LightGBM	16	0.148	0.442	$-0.004$	3-1-8
RPTE vs RF	23	0.233	0.467	$-0.006$	4-0-8
RPTE vs EBM	5	0.010	0.039 ∗	$-0.020$	1-1-10
RPTE vs RuleFit	28	0.700	0.700	$-0.001$	5-1-6

Table 5: Panel-level Wilcoxon signed-rank tests on ROC AUC across 12 dataset means. Holm correction is applied across the five prespecified RPTE-versus-baseline comparisons.

Dataset	RPTE	XGBoost	LightGBM	RF	EBM	RuleFit
lupus	10	760 (80 $\times$ )	744 (78 $\times$ )	2,807 (294 $\times$ )	159 (17 $\times$ )	59 (6 $\times$ )
postop.	16	211 (13 $\times$ )	73 (5 $\times$ )	4,587 (294 $\times$ )	33 (2 $\times$ )	98 (6 $\times$ )
append.	20	660 (34 $\times$ )	992 (51 $\times$ )	1,607 (82 $\times$ )	657 (34 $\times$ )	75 (4 $\times$ )
corral	18	1,600 (87 $\times$ )	3,074 (167 $\times$ )	3,455 (188 $\times$ )	32 (2 $\times$ )	51 (3 $\times$ )
crabs	39	1,894 (49 $\times$ )	4,544 (118 $\times$ )	7,557 (196 $\times$ )	1,033 (27 $\times$ )	55 (1.44 $\times$ )
hepatitis	40	600 (15 $\times$ )	497 (13 $\times$ )	3,761 (95 $\times$ )	368 (9 $\times$ )	80 (2 $\times$ )

Table 6: Audit-load metrics: MIU, mean over 15 splits. Bold marks the lowest MIU per row. Parentheses report the MIU ratio baseline/RPTE; values above one favor RPTE.

Metric	RPTE	XGBoost	LightGBM	RF	EBM	RuleFit
AUC	0.830 (.850)	0.838 (.856)	0.836 (.851)	0.837 (.889)	0.853 (.881)	0.831 (.845)
MIU	72 (37)	1,773 (1,034)	3,244 (1,245)	8,759 (4,844)	13,597 (514)	78 (68)
IIU	30.0 (14.4)	279.2 (216.9)	317.5 (180.3)	341.7 (321.5)	28.7 (14.5)	78.1 (67.5)

Table 7: Accuracy–auditability tradeoff across 12 datasets. Entries are mean (median) of the dataset-level results over 15 splits.

Dataset	AUC	AUC	$\Delta$ AUC	MIU	MIU	$\Delta$ MIU
	default	strict		default	strict
lupus	0.768	0.750	$-0.018$	10	11	$+1$
postop.	0.432	0.454	$+0.022$	16	13	$-3$
append.	0.797	0.751	$-0.046$	20	16	$-4$
corral	1.000	1.000	0.000	18	10	$-8$
crabs	0.974	0.927	$-0.047$	39	40	$+1$

Table 8: Strict source-disjointness analysis over 15 outer splits. Default RPTE is compared with the independently nested-CV-retuned strict variant.

Configuration	AUC	$\Delta$ AUC	MIU	IIU
RPTE estimator
Full RPTE (HUG + augmented pairs)	0.830	–	72	30.0
HUG, no augmentation	0.819	$-$ 0.011	62	18.8
Frequent top- $K$ vocabulary	0.825	$-$ 0.005	131	25.7
Original features only	0.805	$-$ 0.025	127	15.8
No leaf reassignment	0.809	$-$ 0.022	120	12.6

Table 9: Ablation analysis across eight configurations and 12 datasets: mean ROC AUC,

\Delta

AUC relative to full RPTE, and audit-load metrics.

Comparison	$W$	$p$	$p_{\mathrm{Holm}}$	Record
Full RPTE vs original only	8	0.0122	0.0977	11–0–1
Full RPTE vs no reassignment	5	0.0098	0.0879	10–1–1
Full RPTE vs HUG no augmentation	23	0.2334	0.9336	8–0–4
HUG no augmentation vs frequent RPTE	33	0.6772	1.000	7–0–5
Frequent RPTE vs frequent XGBoost	15	0.1230	0.6152	2–1–9
HUG no-augmentation RPTE vs corresponding XGBoost	16	0.0771	0.4629	2–0–10

Table 10: Panel-level Wilcoxon signed-rank tests for the nine prespecified AUC contrasts. Holm correction is applied across all nine tests. Record denotes wins–ties–losses for the first-named configuration.

Fig. 1 : Linear CKA between HUG and frequent representations at Stage 1 (mined-pattern matrices) and immediately before logistic-regression fitting (full RPTE design matrices).

Fig. 2 : Impact of HUG vocabulary transfer on XGBoost.

	RPTE	XGBoost	LightGBM	RF	EBM	RuleFit
Tuning	26.2 (62.4)	16.0 (29.2)	11.7 (36.8)	9.7 (17.3)	382.1 (688.4)	112.2 (111.5)
Fit	1.987 (5.431)	0.783 (1.510)	0.464 (1.452)	0.727 (1.786)	24.551 (52.971)	6.009 (9.672)
Predict	0.019 (0.041)	0.003 (0.003)	0.005 (0.013)	0.012 (0.009)	0.004 (0.012)	0.090 (0.049)

Table 11: Execution time: mean (SD) of the 12 dataset-level mean times, in seconds.

Dataset	Pattern signature	Folds	Domain concepts matched
Heart-c	{cp, oldpeak, thal}	4/15	Ischaemia + anatomical evidence
	{ca, cp, oldpeak, sex, slope}	3/15	Ischaemia + anatomical + sex
	{ca, cp, oldpeak, thal, thalach, trestbps}	2/15	Ischaemia + anatomical + baseline
WDBC	{mean_concavity, worst_area}	3/15	Nuclear size + boundary
	{worst_concave_pts, worst_perim., worst_texture}	3/15	Nuclear size + boundary
	{mean_concave_pts, worst_perim.}	3/15	Nuclear size + boundary

Table 12: Top recurring RPTE pattern signatures and domain alignment. Folds = number of folds (out of 15) in which the exact variable combination is prediction-active.

Dataset	Folds	Rule A	Rule B	Cov. J
WDBC	0 vs 9	worst_perim $\leq$ 106.0 AND worst_concave_pts $\leq$ 0.135 AND radius_err $\leq$ 0.596 3 features, coeff $=-2.52$	worst_perim $\leq$ 106.0 AND worst_compact $\times$ mean_area $\leq$ 201.0 AND area_err $\leq$ 46.9 AND worst_concave_pts $\leq$ 0.135 5 features, coeff $=-1.64$	.987
	5 vs 14	worst_radius $\leq$ 16.8 AND worst_concave_pts $\leq$ 0.136 AND area_err $\leq$ 38.6 AND smooth_err $>$ 0.003 4 features, coeff $=-2.84$	worst_perim $\leq$ 111.5 AND worst_concave_pts $\leq$ 0.136 AND area_err $\leq$ 38.6 AND smooth_err $>$ 0.003 4 features, coeff $=-2.43$	.984
Heart-c	1 vs 4	$\|$ cp $-$ thal $\|$ $>$ 2.5 AND oldpeak $>$ 0.65 3 features, coeff $=-1.57$	cp $=$ [0,1), oldpeak $=$ [0.8, 6.2) AND trestbps $\times$ thal $>$ 290 4 features, coeff $=-1.42$	.918
	9 vs 11	cp $-$ oldpeak $>-0.45$ AND ca $\leq$ 0.5 AND chol $\times$ thal $\leq$ 628.5 5 features, coeff $=+1.20$	cp $-$ oldpeak $>-0.45$ AND ca $\leq$ 0.5 AND trestbps $\times$ thal $\leq$ 317.5 AND chol $\leq$ 323 6 features, coeff $=+1.74$	.902

Table 13: Cross-fold alternate explanation pairs. Each row shows two rules from different folds that identify nearly the same patient subgroup (coverage Jaccard

\geq 0.90

) using different features. Features unique to each rule are in bold .

Dataset	Removed feature	Reference rule	Alternative rule	Cov. $J$
WDBC	worst_ symmetry	worst_perimeter $\leq 105.95$ AND worst_concave_points $\leq 0.13505$ AND radius_error $\leq 0.59555$ coefficient $=-2.52$	worst_perimeter $\leq 105.95$ AND worst_concave_points $\leq 0.13505$ AND area_error $\leq 44.13$ coefficient $=-2.45$	1.000
	worst_ perimeter	worst_perimeter $\leq 105.95$ AND worst_concave_points $\leq 0.13505$ AND radius_error $\leq 0.59555$ coefficient $=-2.52$	worst_radius $\leq 16.805$ AND worst_concave_points $\leq 0.13505$ AND area_error $\leq 48.7$ AND smoothness_error $>0.003294$ AND worst_texture $\leq 33.27$ coefficient $=-2.77$	.857
Heart-c	fbs	cp $-$ oldpeak $\leq-0.7$ AND thal $>2.5$ coefficient $=-1.90$	cp $-$ oldpeak $\leq-0.7$ AND trestbps $\times$ thal $>315$ coefficient $=-1.93$	.944
	exang	cp $-$ oldpeak $\leq-0.7$ AND thal $\leq 2.5$ AND NOT(ca $=[1,4)$ , cp $=[0,1)$ ) AND trestbps $-$ thalach $\leq-8.5$ coefficient $=+0.17$	cp $-$ oldpeak $\leq-0.7$ AND thal $\leq 2.5$ AND ca $=[0,1)$ AND trestbps $-$ thalach $\leq-8.5$ coefficient $=+0.46$	1.000

Table 14: Same-fold alternate explanations. Reference and alternative models are trained on the same outer-training fold using different raw feature sets. Differing conditions are shown in bold .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

RuleFit
Conflicting overlap	prnn_crabs : index $\leq 23.5$ , FL $\leq 17.3$ , BD $\leq 11.45$ (coef. $=-1.23$ , train $=38$ ) and index $\leq 26.0$ , FL $\leq 17.75$ , BD $\leq 11.3$ (coef. $=+0.17$ , train $=38$ ). The rules overlap on 36 training and 10 held-out observations (train/test Jaccard $=.90/.91$ ), so both opposing contributions apply to nearly the same subgroup.
Nested thresholds	prnn_crabs : seven { FL , index } rules form 10 nested pairs. Example: index $\leq 23.5$ , FL $\leq 17.1$ ( $-0.87$ ) within index $\leq 24.5$ , FL $\leq 18.05$ ( $-0.11$ ); train/test Jaccard $=.949/.833$ .
Activation multiplicity	Mean active rules per held-out observation: $6.7$ – $19.2$ ; maximum $=35$ ( prnn_crabs ).
FIGS
Conflicting overlap	Hepatitis: BILIRUBIN $\leq 13.5$ , ALBUMIN $\leq 28.5$ (Tree 1, $+0.192$ , $n=87$ ) and BILIRUBIN $\leq 16.5$ , ALBUMIN $\leq 28.5$ (Tree 3, $-0.022$ , $n=91$ ); train/test Jaccard $=.956/.958$ .
Nested thresholds	Hepatitis: ASCITES $>0.5$ , BILIRUBIN $>18$ , $29<\texttt{SGOT}\leq 82.5$ contains $0.5<\texttt{ASCITES}\leq 1.5$ , BILIRUBIN $>18$ , $37.5<\texttt{SGOT}\leq 81.5$ ; 10 shared training observations, Jaccard $=.909$ .

Appendix

Table 15: Interpretability issues observed in fitted additive rule models (outer fold 0, seed 42). For SIRUS, contrast denotes the true-branch output minus the false-branch output.

Method	Tuned parameters	Fixed settings
RPTE	$L\in\{1,2\}$ , $\mathrm{topK}\in\{50,100\}$ , $G\in\{0.01,0.001\}$ , Sequential RPTE estimator	Package defaults
XGBoost	$\mathrm{max\_depth}\in\{2,4\}$ ; $\mathrm{learning\_rate}\in\{0.03,0.1\}$ ; $\mathrm{min\_child\_weight}\in\{1,5\}$	$\mathrm{reg\_lambda}=1$ ; $\mathrm{n\_estimators}=200$ ; early stopping after 20 rounds
LightGBM	$\mathrm{learning\_rate}\in\{0.03,0.1\}$ ; $\mathrm{num\_leaves}\in\{15,31\}$ ; $\mathrm{min\_child\_samples}\in\{10,30\}$	$\mathrm{reg\_lambda}=1$ ; $\mathrm{n\_estimators}=200$
Random Forest	$\mathrm{n\_estimators}\in\{50,100\}$ ; $\mathrm{max\_depth}\in\{4,8\}$ ; $\mathrm{min\_samples\_leaf}\in\{1,5\}$	$\mathrm{max\_features}=\sqrt{p}$
EBM	$\mathrm{max\_rounds}\in\{200,500\}$ ; $\mathrm{interactions}\in\{0,10\}$ ; $\mathrm{learning\_rate}\in\{0.015,0.05\}$	Package defaults
RuleFit	$\mathrm{n\_estimators}\in\{50,100\}$ ; $\mathrm{max\_rules}\in\{30,60\}$ ; $\mathrm{tree\_size}\in\{3,4\}$	Package defaults

Appendix

Table 16: Hyperparameter search spaces. Each model has eight candidate configurations evaluated by inner three-fold stratified cross-validation.

	Full RPTE		HUG no aug.		Frequent		Original only		No reassign.
Dataset	AUC	MIU	AUC	MIU	AUC	MIU	AUC	MIU	AUC	MIU
lupus	0.768	10	0.758	8	0.755	39	0.767	36	0.754	39
postop.	0.432	16	0.479	6	0.475	35	0.467	35	0.388	39
append.	0.797	20	0.780	16	0.780	60	0.760	61	0.756	53
corral	1.000	18	0.883	24	1.000	13	0.883	43	1.000	13
crabs	0.974	39	0.934	31	0.909	86	0.904	85	0.928	55

Appendix

Table 17: Per-dataset ablation results: mean ROC AUC and MIU over 15 outer splits. Bold denotes the highest AUC and lowest MIU within each dataset, determined from unrounded values.

Explore similar work

Jun 30, 2026cs.LG

Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again

Recent work has shown that well-optimized individual decision trees can match complex black box models in some settings, primarily in noisy domains. For the remaining settings, however, complex ensembled compositions of trees often achieve higher accuracy at the cost of interpretability, leaving practitioners with difficult modeling decisions along an accuracy-interpretability tradeoff. Ideally, we would like to classify as much of the data as possible with one or a small number of trees, achieving interpretability for most samples while maintaining state-of-the-art accuracy. We introduce Multistage Defer Trees: a sequence of sparse decision trees that each make predictions for most samples, while deferring a small proportion to the next tree in the sequence or, ultimately, to a black box. We demonstrate that we can train this model class to match the performance of complex tree-based ensembles while routing most samples through only one or a small number of sparse decision trees. We discuss a range of techniques for training these models while maintaining simplicity. Our method expands the accuracy--interpretability frontier in settings where single-tree methods remain insufficient, demonstrating that even when complex models are necessary, they need not be fully opaque.

Zakk Heile, Hayden McTavish, Margo Seltzer +1

Department of Computer Science · Duke University · Durham, USA +2

Ensemble Learning Learning to Defer Interpretable ML Decision Tree Learning Classification

Ask PDF

Sep 6, 2026cs.LG

Sparse Oblique Rule Boosting for Simpler Additive Rule Ensembles

Small additive ensembles of symbolic rules offer interpretable prediction models. Traditionally, these ensembles use rule conditions based on conjunctions of simple threshold propositions

x \geq t

on a single input variable

x

and threshold

t

, resulting geometrically in axis-parallel polytopes as decision regions. While this form ensures a high degree of interpretability for individual rules and can be learned efficiently using the gradient boosting approach, it relies on having access to a curated set of expressive input features so that a small ensemble of axis-parallel regions can describe the target variable well. Absent such features, reaching sufficient accuracy requires increasing the number and complexity of individual rules, which diminishes the interpretability of the model. Here, we extend classical rule ensembles by introducing logical propositions with learnable sparse linear transformations of input variables, i.e., propositions of the form

\mathbf{x}^T\mathbf{w} \geq t

, where

\mathbf{w}

is a learnable sparse weight vector, enabling decision regions as general polyhedrons with oblique faces. We propose a learning method using gradient boosting based on a weighted logistic regression. Empirical results across 14 regression and classification tasks demonstrate that the proposed method achieves lower model complexity than competitive baselines while maintaining similar or better predictive accuracy. Hence, the approach provides a favorable trade-off between interpretability and accuracy and reduces the reliance on manual feature engineering.

Shahrzad Behzadimanesh, Pierre Le Bodic, Geoffrey I. Webb +1

Department of Data Science and Artificial Intelligence, Monash University, Melbourne, VIC, Australia. · Department of Information Systems, University of Haifa, Haifa, Israel.

Interpretable ML Ensemble Learning

Ask PDF

Aug 19, 2026cs.LG

Leaf Values as Coordinates: Exact Contrastive Explanation for Gradient-Boosted Ensembles

A gradient-boosted ensemble predicts by summing one leaf value per tree. Read those values as coordinates rather than as intermediate results, and every instance becomes a point in R^M on which the model acts linearly: the score is the sum of the coordinates. This small change of view makes contrastive explanation exact. The difference between two instances is a vector that is identically zero wherever they share a leaf, so the gap between a rejected applicant and an accepted one is carried by a handful of coordinates, each traceable to a real split in a real tree. Nothing is fitted, sampled, or assumed additive in features -- the additivity is already there, in the right space. We build a recourse method on this representation and evaluate it on five tabular datasets under repeated cross-validation. Its recommendation reconstructs the model's own decision to 6.2 x 10^-15, so an auditor can re-check the arithmetic without the model. On the credit datasets it is Pareto-non-dominated on effort against realism. And when recommendations are restricted to changes the subject could actually make -- not their age, not a settled delinquency -- it retains 58% of its validity where the strongest baseline retains 41%, a distinction the standard evaluation cannot see because it never asks whether a recommendation can be carried out.

Emanuele Luzio

Independent Researcher

Counterfactual Explanations Algorithmic Recourse Gradient-Boosted Decision Trees Tabular ML

Ask PDF