Tree Depth and Sample-Quality-Associated Features: Two Hidden Factors in Boruta Selection for RNA-Seq Data Classification

Tareque Mohmud Chowdhury; Abu Raihan Mostofa Kamal

cs.LGOct 23, 2026

Tree Depth and Sample-Quality-Associated Features: Two Hidden Factors in Boruta Selection for RNA-Seq Data Classification

Authors: Tareque Mohmud Chowdhury, Abu Raihan Mostofa Kamal

Organizations: Department of Computer Science and Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh

Abstract

Random Forest--based selectors such as Boruta depend on tree depth, yet Boruta's most used implementation, BorutaPy, recommends depths of 3 to 7 without published evidence. We tested the effect of depth using pan-cancer data from The Cancer Genome Atlas (10,469 tumors, 33 cancer types) and breast cancer PAM50 subtyping (981 tumors, 5 subtypes). Label-dependent steps were fitted inside training folds, effects were measured against seed-to-seed variation, and each task had a locked test set. In pan-cancer data, depth 5 changed the selection far more than a new seed did: the top-500 Jaccard overlap with unlimited depth was 0.37, against 0.84 (depth 5) and 0.93 (unlimited) between seeds. Depth-5 trees confirmed 472 instead of 1,659 genes, covered fewer large-class markers (16% vs 62%) and gave less accurate 50-gene panels (balanced accuracy 0.809 vs 0.889). The effect vanished once the cap reached 61 to 76% of the median depth of unrestricted trees, a depth set mainly by the number of classes, not sample size. In PAM50 subtyping (median depth 12) the effect was small, and leaving depth uncapped cost no accuracy in either task. The effect replicated in a second cross-validation repeat and under permutation importance. About a third of pre-filtered candidates correlated within class with the number of detected genes, a sample-quality measure; a within-class filter removed them without loss of test accuracy (0.943 vs 0.942 balanced accuracy, 1,000 genes). We recommend uncapped trees, or a cap above 75% of the realized median depth, and a within-class quality filter before Boruta.

Figures & tables

Figure 1: Study design. Two tasks from TCGA RNA-seq data: pan-cancer classification (10,469 tumors, 33 classes) and breast cancer PAM50 subtyping (981 tumors, 5 classes). Each task has a locked test set (20% of patients). Inside every training fold: expression filter, within-class quality filter (quality-aware pipeline only), category-aware ANOVA F pre-filter to 3,000 candidates, and Boruta at each depth setting with three random seeds. Selections are compared with seed-to-seed variation; panels of the top

K

genes are evaluated with logistic regression (LR) and LightGBM.

	Pan-cancer	PAM50
Classes	33 cancer types	5 subtypes
Tumors after QC	10,469	981
Development set	8,374	797
Locked test set	2,095	184
Training samples per CV fold	about 6,700	637
Smallest class	CHOL (35)	Normal-like (36)

Table 1: Cohorts and Splits

Figure 2: Effect of Random Forest maximum depth on Boruta feature selection in TCGA pan-cancer data (33 classes; 5-fold cross-validation, 3 seeds per depth). Shaded: depth range recommended in the BorutaPy documentation (3–7); “unl.” = unlimited depth. (a) Genes confirmed by Boruta (mean

\pm

SD over folds and seeds). (b) Jaccard overlap of the top 500 genes: blue, between each depth and unlimited depth, compared across different seeds; dashed, between two seeds at the same depth. (c) Stability of the top 500 genes across folds (Nogueira index). (d) Validation balanced accuracy of logistic regression and LightGBM trained on the top 50 genes (mean

\pm

SD over folds); differences between depths are consistent within folds (Section 4.2 ). (e) Share of each class’s 20 strongest one-vs-rest marker genes found among the top 500, averaged over the 8 smallest and the 8 largest classes. (f) Share of non-coding “other” gene types, mostly processed pseudogenes, among the top 500.

Figure 3: The depth effect follows the depth that unrestricted trees reach. (a) Depth gap, defined as the seed-to-seed overlap of the top 500 genes minus their overlap with unlimited-depth selections from different seeds, in the pan-cancer and PAM50 tasks; 0 means changing the depth alters the selection no more than changing the seed. Shaded: BorutaPy-recommended range. (b) Realized depth of unrestricted Random Forest trees (100 trees per training fold, 5 folds), on the same depth scale as (a); boxes show quartiles, whiskers the 5th–95th percentiles.

Depth	3	5	8	10	15	20	25	30	Unlimited
Confirmed genes	249	472	911	1,224	1,550	1,638	1,661	1,660	1,659
Seed-to-seed Jaccard	0.823	0.842	0.831	0.779	0.906	0.923	0.931	0.924	0.930
Jaccard vs unlimited (cross-seed)	0.322	0.370	0.479	0.581	0.782	0.911	0.931	0.926	–
Nogueira $\Phi$	0.849	0.863	0.869	0.858	0.888	0.900	0.897	0.898	0.900
Balanced accuracy, LR, $K=50$	0.806	0.809	0.840	0.858	0.881	0.886	0.888	0.893	0.889
Balanced accuracy, LightGBM, $K=50$	0.827	0.831	0.838	0.851	0.876	0.872	0.874	0.868	0.870

Table 2: Pan-Cancer Selections by Random Forest Depth (5 Folds, Seeds 0–2; Top 500 Genes Unless Stated)

Figure 4: Candidate genes associated with sample quality in the pan-cancer task. A gene is quality-associated when its within-class correlation with

\log_{10}

of the number of detected genes exceeds 0.3 in absolute value (training fold). (a) Share of quality-associated candidates by RNA category (blue) and by the most affected gene types (gray), over 5 folds. (b) Share among the top 500 Boruta genes at each depth; dashed, share among all 3,000 candidates. (c) Share among the top 500 genes re-measured on validation samples, without and with the within-class quality filter.

Task, classifier, $K$	Quality-aware (95% CI)	Baseline	Depth 5	PAM50 genes	Quality-aware minus baseline (95% CI)
Pan-cancer, LR, 1,000	0.943 (0.927–0.958)	0.942	–	–	$+0.001$ ( $-0.006$ to 0.007)
Pan-cancer, LightGBM, 1,000	0.928 (0.912–0.945)	0.923	–	–	$+0.006$ ( $-0.001$ to 0.013)
Pan-cancer, LR, 100	0.921 (0.904–0.936)	0.925	–	–	$-0.005$ ( $-0.013$ to 0.002)
Pan-cancer, LightGBM, 100	0.905 (0.887–0.923)	0.904	–	–	$+0.001$ ( $-0.012$ to 0.013)
PAM50, LR, 50	0.889 (0.814–0.951)	0.897	0.902	0.894	$-0.008$ ( $-0.044$ to 0.017)
PAM50, LightGBM, 20	0.833 (0.741–0.920)	0.834	0.811	–	$-0.001$ ( $-0.039$ to 0.031)

Table 3: Balanced Accuracy on the Locked Test Sets

Figure 5: Balanced accuracy on the locked test sets, with 95% confidence intervals from 2,000 bootstrap resamples drawn within class. (a) Pan-cancer task (2,095 samples, 33 classes), panels of 1,000 genes (chosen by the one-standard-error rule) and 100 genes. (b) PAM50 task (184 samples, 5 subtypes), panels of 20 or 50 genes; the 50 PAM50 genes are an upper reference, as the labels were derived from them.

Values: repeat 0 / repeat 1 (5 folds $\times$ 3 seeds each; top-500 Jaccard; depth gap measured across different seeds). Unl.: unlimited depth.
Depth	Depth gap	Seed Jaccard	Nogueira $\Phi$	Confirmed
5	0.468 / 0.468	0.838 / 0.839	0.863 / 0.872	472 / 469
8	0.353 / 0.352	0.831 / 0.831	0.869 / 0.877	911 / 920
10	0.199 / 0.203	0.779 / 0.784	0.858 / 0.859	1,224 / 1,227
15	0.121 / 0.130	0.904 / 0.909	0.888 / 0.893	1,550 / 1,554
Unl.	0.000 / 0.000	0.927 / 0.932	0.900 / 0.907	1,659 / 1,670

Table 4: Pan-Cancer Selections in Two Independent CV Repeats

Figure 6: Realized depth and the depth effect versus sample size and number of classes (pan-cancer subsamples). (a) Median realized depth of unrestricted trees (100 trees per training fold) against training-set size, all 33 classes; mean

\pm

SD over 5 folds. (b) Median realized depth against the number of randomly drawn cancer types at 637 training samples (3 draws per fold). (c) Depth gap of the top 500 genes for the full pan-cancer data, for a 637-sample subsample with all 33 classes (depths 5–20 and unlimited), and for PAM50 (637 training samples, 5 subtypes). At 637 samples fewer than 500 genes were confirmed, so the top 500 also includes tentative genes ranked by hit rate.

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 200 papers

Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery
Apr 15, 2026Pushpa Kumar Balan, Aijing FengFeature Selection Scientific Discovery
Department of Computer Science and Cybersecurity, University of Central Missouri
Novel GPU Boruta algorithms for feature selection from high-dimensional data
May 11, 2026Xurui Li, Zhiguo Gan, Jiaming Zhang +2GPU Acceleration Feature Selection
Department of Chemical Engineering, Tsinghua University, 2026 Shuangqing Road, Beijing, 100084, China.
When Does Deep Representation Learning Help Single-Cell Clustering? A Sensitivity-Aware Diagnostic Benchmark for Biomedical AI Pipelines
Jul 28, 2026Nguyen Thanh Phong, Truong Viet Vu, Nguyen Ha Thu +4Representation Learning Clustering
Faculty of Information Technology, Van Lang School of Technology, Van Lang University, Ho Chi Minh City, Vietnam · Faculty of Law, Van Lang University, Ho Chi Minh City, Vietnam

Apr 15, 2026J/K move · Enter open · S save

Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery

Pushpa Kumar Balan, Aijing Feng

Department of Computer Science and Cybersecurity, University of Central Missouri

Feature Selection Scientific Discovery Cancer Genomics LLM Reasoning

Open paper AskPDF arXiv

Gradient saliency from deep sequence models surfaces candidate biomarkers efficiently, but the resulting gene lists can be contaminated by tissue-composition confounders that degrade downstream classifiers. We study whether LLM chain-of-thought (CoT) reasoning can filter these confounders, and whether reasoning quality is associated with downstream performance. We train a Mamba SSM on TCGA-BRCA RNA-seq and extract the top-50 genes by gradient saliency; DeepSeek-R1 evaluates every candidate with structured CoT to produce a final 17-gene set. On the held-out test split, the raw 50-gene saliency set (no LLM) performs worse than a 5,000-gene variance baseline (AUC 0.832 vs. 0.903), while the LLM-filtered set surpasses it (AUC 0.927), using 294x fewer features. A faithfulness audit (COSMIC CGC, OncoKB, PAM50) shows that 6 of 17 selected genes (35.3%) are validated BRCA biomarkers, while 10 of 16 known BRCA genes present in the input were missed - including FOXA1. This divergence between downstream performance and reasoning faithfulness suggests selective faithfulness in this setting: targeted confounder removal can improve predictive performance without comprehensive recall.

Abstract

Feature Selection

Explore similar work

Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery

Novel GPU Boruta algorithms for feature selection from high-dimensional data

When Does Deep Representation Learning Help Single-Cell Clustering? A Sensitivity-Aware Diagnostic Benchmark for Biomedical AI Pipelines