An Enhanced Projection Pursuit Tree Classifier with Visual Methods for Assessing Algorithmic Improvements

Natalia da Silva; Dianne Cook; Eun-Kyung Lee

stat.MLFeb 24, 2026

An Enhanced Projection Pursuit Tree Classifier with Visual Methods for Assessing Algorithmic Improvements

Authors: Natalia da Silva, Dianne Cook, Eun-Kyung Lee

Organizations: Instituto de Estadística (IESTA), Universidad de la República, Montevideo, Uruguay · Department of Econometrics and Business Statistics, Monash University, Melbourne, Australia · Department of Statistics, Ewha Womans University, Seoul, Republic of Korea

Abstract

This paper presents enhancements to the projection pursuit tree classifier and visual diagnostic methods for assessing their impact in high dimensions. The original algorithm uses linear combinations of variables in a tree structure where depth is constrained to be less than the number of classes -- a limitation that proves too rigid for complex classification problems. Our extensions improve performance in multi-class settings with unequal variance-covariance structures and nonlinear class separations by allowing more splits and more flexible class groupings in the projection pursuit computation. Proposing algorithmic improvements is straightforward; demonstrating their actual utility is not. We therefore develop two visual diagnostic approaches to verify that the enhancements perform as intended. Using high-dimensional visualization techniques, we examine model fits on benchmark datasets to assess whether the algorithm behaves as theorized. An interactive web application enables users to explore the behavior of both the original and enhanced classifiers under controlled scenarios. The enhancements are implemented in the R package PPtreeExt.

Figures & tables

Figure 1: Illustration of the original PPtree algorithm for $G=3$ . In step 1, PP optimization decides on the best linear projection. Step 2 combines two groups to make a two-group classification. The PP optimization is computed for two group problem, and an appropriate split is decided. This is repeated for the remaining two groups to get the final model containing two splits. The locations where modifications will be made are circled.

Rule	Name	Formula
1	mean	$c=\frac{1}{2}\bar{\mathbf{x}}_{1}+\frac{1}{2}\bar{\mathbf{x}}_{2}$
2	sample size weighted mean	$c=\left(\frac{n_{2}}{n_{1}+n_{2}}\right)\bar{\mathbf{x}}_{1}+\left(\frac{n_{1}}{n_{1}+n_{2}}\right)\bar{\mathbf{x}}_{2}$
3	standard deviation weighted mean	$c=\left(\frac{s_{2}}{s_{1}+s_{2}}\right)\bar{\mathbf{x}}_{1}+\left(\frac{s_{1}}{s_{1}+s_{2}}\right)\bar{\mathbf{x}}_{2}$
4	standard error weighted mean	$c=\frac{s_{2}/\sqrt{n_{2}}}{s_{1}/\sqrt{n_{1}}+s_{2}/\sqrt{n_{2}}}\bar{\mathbf{x}}_{1}+\frac{s_{1}/\sqrt{n_{1}}}{s_{1}/\sqrt{n_{1}}+s_{2}/\sqrt{n_{2}}}\bar{\mathbf{x}}_{2}$
5	median	$c=\frac{1}{2}med_{1}+\frac{1}{2}med_{2}$
6	sample size weighted median	$c=\frac{n_{2}}{n_{1}+n_{2}}med_{1}+\frac{n_{1}}{n_{1}+n_{2}}med_{2}$

Table 1: Rules for computing the split value,

c

, between two groups, on a data projection, using means, weighted mean, medians, standard deviations, and interquartile ranges.

Figure 2: Comparison of decision boundaries produced by the rpart (left) and PPtree (right) algorithms on the LC data. The boundaries generated by PPtree are oblique to the coordinate axes, appropriately capturing the separations as linear combinations of the two variables. Even with many splits, the rpart algorithm cannot elegantly classify this data.

Figure 3: Comparison of decision boundaries produced by the rpart (left) and PPtree (right) algorithms on the MC data. The T class (orange triangles) cannot be separated using a single linear partition, and PPtree fails to model it accurately because each class must be assigned to a single terminal node. This illustrates the need for allowing more splits in PPtree to capture unusually shaped clusters and separations involving combinations of variables.

Figure 4: Illustration of the algorithm with modification 2 applied to a three-class problem. Projections of the data are computed at each node, and multiple splits per class are allowed. An impurity-based criterion, such as entropy, is used to determine when and how often to split observations within a node.

Figure 5: Comparing the fits for three-class data with a strong correlation between variables. Modification 1 (different subsetting rules) of PPtree is shown. The modification produces lower error and decision boundaries that are more symmetrically placed between the class clusters.

Figure 6: Comparing the fits for data where one class has two distinct clusters. Modification 2 (multiple splits) of PPtree is used. The modification produces lower error and better decision boundaries.

Figure 7: Comparing the fits for five-cluster Gaussian mixtures. Modification 2 of PPtree is used. This produces lower error and reasonable decision boundaries.

Figure 8: Average test error with 95% error bars from 200 training/test splits of benchmark data for four models: rpart , PPtree, PPtreeExt modifications 1 and 2, where the latter performs better than the others. The order of the plots is based on the pattern of errors: at top left PPtreeExt (mod 2) is best but only slightly better than rpart , and at bottom right each of the PPtree versions iteratively performs better than rpart .

dataset	Cases	Predictors	Groups	Imbalance	Correlation
data20-3	8143	5	2	0.58	0.45
data27-a11	1747	18	5	0.38	0.24
data27-a31	1834	18	5	0.28	0.27
data27-b31	1424	18	5	0.21	0.24
data27-c11	1111	18	5	0.14	0.27
data27-c31	1448	18	5	0.17	0.27

Table 2: Overview of benchmark datasets where the proposed tree-based extensions outperform the selected tree methods. The table summarizes the number of observations, predictors, number of classes, class imbalance, and absolute average correlation among predictors. Imbalance quantifies the disparity in class sizes (0 indicates perfectly balanced classes), while higher correlation suggests a greater potential for class separation through linear combinations of variables.

Panel of Figure 9 — Figure 9: Two projections from a grand tour of data20-3, from which we can see why the PPtreeExt method is the most effective. Class 0 is split into two clusters on either side of group 1, which means that the original algorithm is not optimal. Secondly, the separation between the two groups is primarily V3 with small contributions from other variables, which gives PPtreeExt a slight edge over rpart .

Panel of Figure 10 — Figure 10: Examining why data49-2 is best classified using PPtreeExt . Because there are 100 predictors, for visualisation it is useful to reduce the dimension using principal component analysis, and then examine the separation between groups. PC1 explains 99.8% of the overall variability among predictors, but it is not useful for separating the classes (a). PC2 and PC3 (b) have an interesting difference between the two classes, which shows why multiple linear splits might be useful. A guided projection pursuit tour on the first 26 principal components (c) shows that these classes could be separated with a linear combination. This is why the original PPtree classifier does better than the tree algorithm and also why the extension improves the performance.

Panel of Figure 11 — Figure 11: Examining why data69-1 is best classified using PPtreeExt : (a) best projection from a guided tour shows that the data has three similar linear pieces oriented to form a triangle, (b) boundary produced by the PPtreeExt (mod 2) fitted model, and (c) boundary produced by rpart fitted model. The same projection is used in all plots to allow comparison. The boundary is examined by predicting the class of a uniform random sample of points in the data space, and viewed using a slice tour. The boundary for PPtreeExt (mod 2) mirrors what we learn from the data, that it is carving the data space into three equiangular chunks, but rpart is not capturing these linear combinations.

Figure 12: Average test error from 200 training/test splits of benchmark data for: SVM, Random Forest, PPtree (pda), PPtree (LDA), PPtreeExt modifications 1 and 2, PPforest and rpart .

Figure 13: Average test error from 200 training/test splits of benchmark data for: SVM, Random Forest, PPtree (pda), PPtree (LDA), PPtreeExt modifications 1 and 2, PPforest and rpart .

Figure 14: Average test error from 200 training/test splits of benchmark data for: SVM, Random Forest, PPtree (pda), PPtree (LDA), PPtreeExt modifications 1 and 2, PPforest and rpart .

Figure 15: Average test error from 200 training/test splits of benchmark data for: SVM, Random Forest, PPtree (pda), PPtree (LDA), PPtreeExt modifications 1 and 2, PPforest and rpart .

Name	URL
data12	https://archive.ics.uci.edu/dataset/518/speaker+accent+recognition
data13-1	https://archive.ics.uci.edu/dataset/547/algerian+forest+fires+dataset
data13-2	https://archive.ics.uci.edu/dataset/547/algerian+forest+fires+dataset
data20-1	https://archive.ics.uci.edu/dataset/357/occupancy+detection
data20-2	https://archive.ics.uci.edu/dataset/357/occupancy+detection
data20-3	https://archive.ics.uci.edu/dataset/357/occupancy+detection

Table 3: Source data information

dataset	Cases	Predictors	Groups	Imbalance	Correlation
data12	329	12	6	0.41	0.38
data13-1	122	13	2	0.03	0.41
data13-2	121	13	2	0.29	0.37
data20-1	2665	5	2	0.27	0.81
data20-2	9752	5	2	0.58	0.29
data20-3	8143	5	2	0.58	0.45

Table 4: Overview of benchmark datasets

Multiclass Classification

Multiclass classification assigns inputs to one of several labels while addressing uncertainty, class imbalance, and errors across classes.

See all 97 papers

Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles
Aug 16, 2026Srikumar KrishnamoorthyRule-Based Classification Interpretable ML
Information Systems Area, Indian Institute of Management Ahmedabad, India
Optimal hypersurface decision trees
Sep 15, 2025Xi HeDecision Tree Learning
Department of Computer Science, Peking University, Beijing, China, 1000871
PINE: Pruning Boosted Tree Ensembles with Conformal In-Distribution Prediction Equivalence
May 27, 2026Haruki Yajima, Yusuke MatsuiModel Compression Ensemble Learning
The University of Tokyo.

Aug 16, 2026J/K move · Enter open · S save

Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles

Srikumar Krishnamoorthy

Information Systems Area, Indian Institute of Management Ahmedabad, India

Rule-Based Classification Interpretable ML Ensemble Learning

Open paper AskPDF arXiv

Predictive models in clinical and regulated settings must be accurate and fully auditable. Tree ensembles deliver strong accuracy on tabular data, but their sequential boosting couples structure discovery with coefficient estimation, making compact per-prediction auditing difficult. Interpretable alternatives impose structural constraints that limit expressiveness: generalized additive models typically restrict interactions to pairwise terms and post-hoc rule extractors produce overlapping rules that hinder compact interpretation. We introduce Residual Pattern Tree Ensemble (RPTE), a three-stage learning approach, that is built on three key principles: bounded feature budget, source disjointness, and separate coefficient estimation. Stage~~1 builds a supervised symbolic feature vocabulary. Stage~~2 grows shallow trees under a source-disjointness constraint, where each raw variable is allocated to at most one tree, and retains only the discovered tree structures. Stage~3 solves a single

\ell_1

-regularized logistic regression over leaf-region indicators, yielding jointly optimal sparse coefficients. This learning approach ensures that every prediction decomposes into an algebraic sum of named, non-overlapping rule contributions, enabling full auditability by design. Empirical evaluation on twelve clinical-domain binary classification benchmarks using repeated stratified 5-fold cross-validation shows that RPTE performs competitively against tuned opaque ensembles and interpretable baselines. RPTE reduces model inspection units by 9

\times

to 87

\times

relative to XGBoost and maintains lower audit complexity than EBM on all 12 datasets. RuleFit requires comparable or fewer inspection units on three datasets where its rule count is small, but without source-disjointness guarantees. The source code is available at this https URL.

Abstract

Multiclass Classification

Explore similar work

Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles

Optimal hypersurface decision trees

PINE: Pruning Boosted Tree Ensembles with Conformal In-Distribution Prediction Equivalence