Ensemble Classification and Cancer-Specific Gene Signatures from Quality-Aware Hybrid Gene Selection in TCGA RNA-Seq data

Tareque Mohmud Chowdhury; Abu Raihan Mostofa Kamal

cs.LGOct 31, 2026

Ensemble Classification and Cancer-Specific Gene Signatures from Quality-Aware Hybrid Gene Selection in TCGA RNA-Seq data

Authors: Tareque Mohmud Chowdhury, Abu Raihan Mostofa Kamal

Organizations: Department of Computer Science and Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh

Abstract

Gene-expression classifiers of cancer type are usually developed and validated within one cohort using random splits. Starting from a quality-aware hybrid Boruta selector from earlier work, we developed and evaluated ensemble classifiers and cancer-specific gene signatures. Using 10,469 TCGA tumours of 33 types, we compared five learners and three ensembles under decision rules fixed in advance, reran the selection in site-grouped cross-validation, transferred a breast-cancer subtype panel to 3273 tumours of the independent SCAN-B cohort, and built five-gene stageI tumour versus adjacent-normal panels for nine cancer types. Logistic regression (LR) was the most accurate and best-calibrated single learner (balanced accuracy 0.943 on the locked test set). Stacking added at most 0.006 with 1000 genes but improved on LR in every fold with a minimal 100-gene panel; a hierarchical model did not reduce within-group confusion. Site-grouped validation lowered balanced accuracy by 0.027, almost entirely in colon and rectal cancer (0.008 without them and the untestable LAML). On SCAN-B, the subtype panel reached 0.855, close to the 50 PAM50 genes (0.876). StageI panels reached ROC AUC >= 0.99 in held-out patients and detected most later-stage tumours. Ensembles help most with a minimal panel; with 1000 genes, a calibrated linear model suffices.

Figures & tables

Figure 1: Study design. The gene panels of the quality-aware hybrid selector feed four analyses, one per aim: (i) model comparison under pre-declared rules; (ii) robustness checks with the selector rerun inside site-grouped folds; (iii) external transfer of a breast-cancer subtype panel to SCAN-B; (iv) stage I tumour vs. adjacent-normal signatures, with patients held out from all model building. QC, quality control; CV, cross-validation; DE, differential expression.

Task	Training data	Evaluation	Classes	Section
Cancer type	TCGA development, 8374 tumours	5-fold CV; locked test set (2095)	33	2.3 , 2.4
Breast cancer subtype	TCGA-BRCA development, 797 tumours	SCAN-B (GSE96058), 3273 tumours	5 (PAM50)	2.5
Stage I vs. adjacent normal	9 TCGA types, development patients	patient-grouped CV; held-out patients	2 per type	2.6

Table 1: Cohorts and tasks.

Model	CV, $K=1000$	CV, $K=100$	Test, $K=1000$	Test log loss	Test ECE
Logistic regression (reference)	0.947 $\pm$ 0.011	0.930	0.943 (0.927–0.958)	0.126	0.007
Linear SVM	0.940 $\pm$ 0.012	0.890	0.928 (0.911–0.944)	0.180	0.046
MLP	0.933 $\pm$ 0.008	0.904	0.936 (0.919–0.952)	0.149	0.018
LightGBM	0.941 $\pm$ 0.014	0.916	0.928 (0.912–0.945)	0.197	0.027
Random forest	0.905 $\pm$ 0.018	0.890	0.893 (0.876–0.911)	0.287	0.109
Soft voting	0.946 $\pm$ 0.010	0.922	0.937 (0.921–0.952)	0.145	0.039

Table 2: Model comparison, cancer type (33 classes). CV, mean

\pm

SD over five folds; test, locked test set (2095 tumours) with 95% bootstrap CI. ECE, expected calibration error.

Figure 2: Model comparison. ( a ) CV balanced accuracy (mean

\pm

SD) with the 1000- and 100-gene panels. ( b ) Locked test set, 1000 genes, 95% bootstrap CIs; dashed line, reference LR. ( c ) Expected calibration error in CV; LR highlighted.

Figure 3: Hard groups and site effects. ( a ) Within-group confusion of the flat LR and the hierarchical model (CV; the single-type lung group is omitted). ( b ) Recall per cancer type under random-split vs. site-grouped CV; types losing more than 0.1 are labelled. LAML came from a single site and cannot be tested under site grouping. ( c ) Balanced accuracy under both CV schemes; GBM, LightGBM.

Figure 4: External transfer, TCGA-BRCA to SCAN-B. ( a ) Balanced accuracy on SCAN-B with 50 genes (95% bootstrap CI); the PAM50 genes are an upper reference because SCAN-B subtypes are PAM50 calls. ( b ) Share of each panel’s TCGA top-

K

genes measured in SCAN-B. ( c ) Recall per subtype.

Figure 5: Stage I tumours vs. adjacent normal tissue (development patients). ( a ) DE genes per cancer type (adjusted $p<0.05$ , $|\log_{2}\mathrm{FC}|\geq 1$ ). ( b ) Number of cancer types in which a gene is DE in the same direction (log scale). ( c ) The GO Biological Process and KEGG terms significant in the most cancer types; shading, $-\log_{10}$ adjusted $p$ .

Type	Dev.	Selector	Panel (rank order)	CV AUC	Held-out AUC	Sens.	Spec.
BRCA	148/94	DE	COL10A1, MMP11, COL11A1, HSD17B6, LINC01614	0.996	0.994	23/26	19/19
COAD	49/33	DE	FOXQ1, ETV4, KRT80, CDH3, SIM2	1.000	1.000	19/19	8/8
KIRC	183/61	hybrid	FAM245A, PIK3C2G, GPC5, LINC01983, AC104237.2	0.993	1.000	50/50	11/11
KIRP	123/24	DE	AQP6, UMOD, DDN, HEPACAM2, CRHBP	1.000	1.000	29/29	8/8
LIHC	107/39	DE	PLVAP, COL15A1, GABRD, CDKN3, THBS4	0.997	1.000	26/26	11/11
LUAD	178/49	DE	PYCR1, FAM83A, ETV4, STX11, OTUD1	1.000	1.000	41/42	10/10

Table 3: Five-gene stage I tumour vs. adjacent-normal panels. Dev., development patients (stage I / normal); AUC, ROC AUC; held-out, patients from the locked test set of the cancer-type task (sensitivity and specificity as counts); –, fewer than five held-out samples of one kind.

Figure 6: Stage I classifiers. ( a ) CV ROC AUC at

K=5

with genes ranked by DE or by the quality-aware hybrid selector. ( b ) Sensitivity of each stage I panel for development stage II and stage III–IV tumours, never used for fitting (95% exact CI). ( c ) Sensitivity and specificity in held-out patients (n stage I / n normal).

Cancer Genomics

Cancer genomics uses molecular data to understand tumor biology, identify biomarkers, and guide diagnosis, prognosis, and treatment.

See all 36 papers

Tree Depth and Sample-Quality-Associated Features: Two Hidden Factors in Boruta Selection for RNA-Seq Data Classification
Oct 23, 2026Tareque Mohmud Chowdhury, Abu Raihan Mostofa KamalFeature Selection Random Forest
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh
Label-Space Mismatch Distorts Cross-Cohort Evaluation in Breast Cancer Molecular Subtyping
Oct 18, 2026Suxing Liu, Byungwon Min, Zhijian Zheng +1Cancer Subtype Classification Cross-Dataset Generalization
Department of IT Engineering, Mokwon University, Daejeon 35349, Republic of Korea · School of Digital Media Technology, Jiangxi Arts & Ceramics Technology Institute, Jingdezhen 333001, China · University of Alabama, Tuscaloosa, AL 35487, USA +1
Feature Dimensionality Outweighs Model Complexity in Breast Cancer Subtype Classification Using TCGA-BRCA Gene Expression Data
May 7, 2026Meena Al HasaniBreast Cancer Multiclass Classification
Independent Researcher

Oct 23, 2026J/K move · Enter open · S save

Tree Depth and Sample-Quality-Associated Features: Two Hidden Factors in Boruta Selection for RNA-Seq Data Classification

Tareque Mohmud Chowdhury, Abu Raihan Mostofa Kamal

Department of Computer Science and Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh

Feature Selection Random Forest

Open paper AskPDF arXiv

Random Forest--based selectors such as Boruta depend on tree depth, yet Boruta's most used implementation, BorutaPy, recommends depths of 3 to 7 without published evidence. We tested the effect of depth using pan-cancer data from The Cancer Genome Atlas (10,469 tumors, 33 cancer types) and breast cancer PAM50 subtyping (981 tumors, 5 subtypes). Label-dependent steps were fitted inside training folds, effects were measured against seed-to-seed variation, and each task kept a locked test set. In pan-cancer data, depth 5 changed the selection far more than a new seed did: the top-500 Jaccard overlap with unlimited depth was 0.37, against 0.84 (depth 5) and 0.93 (unlimited) between seeds. Depth-5 trees confirmed 472 instead of 1,659 genes, covered fewer large-class markers (16% vs 62%) and gave less accurate 50-gene panels (balanced accuracy 0.809 vs 0.889). The effect vanished once the cap reached 61 to 76% of the median depth of unrestricted trees, a depth that depends mainly on the number of classes, not sample size. In PAM50 subtyping (median depth 12) the effect was small, and leaving depth uncapped cost no accuracy in either task. It replicated in a second cross-validation repeat and under permutation importance. About a third of pre-filtered candidates correlated within class with the number of detected genes, a sample-quality measure; a within-class filter removed them without loss of test accuracy (0.943 vs 0.942 balanced accuracy, 1,000 genes). We recommend uncapped trees, or a cap above 75% of the realized median depth, and a within-class quality filter before Boruta.

Abstract

Cancer Genomics

Explore similar work

Tree Depth and Sample-Quality-Associated Features: Two Hidden Factors in Boruta Selection for RNA-Seq Data Classification

Label-Space Mismatch Distorts Cross-Cohort Evaluation in Breast Cancer Molecular Subtyping

Feature Dimensionality Outweighs Model Complexity in Breast Cancer Subtype Classification Using TCGA-BRCA Gene Expression Data