cs.LGApr 13, 2026

Interpretable DNA Sequence Classification via Dynamic Feature Generation in Decision Trees

Authors: Nicolas Huynh, Krzysztof Kacprzyk, Ryan Sheridan, David Bentley, Mihaela van der Schaar

Organizations: University of Cambridge · CU Anschutz Medical Campus

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The analysis of DNA sequences has become critical in numerous fields, from evolutionary biology to understanding gene regulation and disease mechanisms. While deep neural networks can achieve remarkable predictive performance, they typically operate as black boxes. Contrasting these black boxes, axis-aligned decision trees offer a promising direction for interpretable DNA sequence analysis, yet they suffer from a fundamental limitation: considering individual raw features in isolation at each split limits their expressivity, which results in prohibitive tree depths that hinder both interpretability and generalization performance. We address this challenge by introducing DEFT, a novel framework that adaptively generates high-level sequence features during tree construction. DEFT leverages large language models to propose biologically-informed features tailored to the local sequence distributions at each node and to iteratively refine them with a reflection mechanism. Empirically, we demonstrate that DEFT discovers human-interpretable and highly predictive sequence features across a diverse range of genomic tasks.

Figures & tables

Figure 1: Limitations of conventional trees. Training and test accuracies versus tree depth for motif detection, mean and confidence intervals at 95%95\% reported for 55 seeds. Conventional decision trees must grow deep to achieve a high training accuracy, and they do not learn patterns which generalize at test time.
Figure 2: DEFT is a tree-based method for interpretable DNA sequence analysis. Left : DEFT discovers high level sequence features that can consider multiple positions simultaneously, contrasting CART . For example, the feature upstream_G_content_20_49 operates on a window of positions highlighted in orange (non-G nucleotides) and blue (G nucleotides). Right : It leverages LLMs to generate candidate features at each node in a two-step process: first outputting semantic representations, then converting them into Python executable code. DEFT conditions the generation of features on the partial tree structure and task-specific metadata.
Figure 3: Decision tree constructed by DEFT for the Pol II dataset. DEFT discovers high-level sequence features . We also report the leaves’ predictions . Values along the edges correspond to the threshold at each split.
Figure 4: Performance comparison against tree-based baselines. Training and test accuracies across varying depths (mean ±\pm 95% CIs over 5 seeds). DEFT consistently outperforms the tree-based baselines.
 Pol II PausingPromotersEnhancers
MethodAcc.AUPRCAcc.AUPRCAcc.AUPRC
CNN0.868(0.007)0.868{\scriptstyle\,(\,0.007\,)}0.933(0.006)0.933{\scriptstyle\,(\,0.006\,)}0.828(0.005)0.828{\scriptstyle\,(\,0.005\,)}0.931(0.002)0.931{\scriptstyle\,(\,0.002\,)}0.790(0.065)0.790{\scriptstyle\,(\,0.065\,)}0.964(0.006)0.964{\scriptstyle\,(\,0.006\,)}
Transformer0.876(0.016)0.876{\scriptstyle\,(\,0.016\,)}0.942(0.002)0.942{\scriptstyle\,(\,0.002\,)}0.835(0.006)0.835{\scriptstyle\,(\,0.006\,)}0.933(0.005)0.933{\scriptstyle\,(\,0.005\,)}0.704(0.045)0.704{\scriptstyle\,(\,0.045\,)}0.952(0.007)0.952{\scriptstyle\,(\,0.007\,)}
XGBoost0.759(0.005)0.759{\scriptstyle\,(\,0.005\,)}0.848(0.002)0.848{\scriptstyle\,(\,0.002\,)}0.822(0.000)0.822{\scriptstyle\,(\,0.000\,)}0.925(0.000)0.925{\scriptstyle\,(\,0.000\,)}0.777(0.000)0.777{\scriptstyle\,(\,0.000\,)}0.965(0.000)0.965{\scriptstyle\,(\,0.000\,)}
Log. Reg.0.769(0.002)0.769{\scriptstyle\,(\,0.002\,)}0.857(0.001)0.857{\scriptstyle\,(\,0.001\,)}0.796(0.000)0.796{\scriptstyle\,(\,0.000\,)}0.901(0.000)0.901{\scriptstyle\,(\,0.000\,)}0.730(0.000)0.730{\scriptstyle\,(\,0.000\,)}0.945(0.000)0.945{\scriptstyle\,(\,0.000\,)}
CART0.750(0.007)0.750{\scriptstyle\,(\,0.007\,)}0.830(0.003)0.830{\scriptstyle\,(\,0.003\,)}0.793(0.000)0.793{\scriptstyle\,(\,0.000\,)}0.897(0.000)0.897{\scriptstyle\,(\,0.000\,)}0.674(0.000)0.674{\scriptstyle\,(\,0.000\,)}0.939(0.000)0.939{\scriptstyle\,(\,0.000\,)}
Table 1: Performance comparison against black boxes and ensemble baselines.
 TrainTest
ddDEFTDEFTperf\texttt{DEFT}_{\texttt{perf}}DEFTDEFTperf\texttt{DEFT}_{\texttt{perf}}
10.769(0.004){0.769}_{\scriptscriptstyle(0.004)}0.820(0.012){0.820}_{\scriptscriptstyle(0.012)}0.762(0.004){0.762}_{\scriptscriptstyle(0.004)}0.810(0.012){0.810}_{\scriptscriptstyle(0.012)}
20.836(0.017){0.836}_{\scriptscriptstyle(0.017)}0.872(0.020){0.872}_{\scriptscriptstyle(0.020)}0.823(0.024){0.823}_{\scriptscriptstyle(0.024)}0.855(0.014){0.855}_{\scriptscriptstyle(0.014)}
30.872(0.009){0.872}_{\scriptscriptstyle(0.009)}0.893(0.006){0.893}_{\scriptscriptstyle(0.006)}0.856(0.014){0.856}_{\scriptscriptstyle(0.014)}0.867(0.007){0.867}_{\scriptscriptstyle(0.007)}
Table 2: Performance comparison at different maximum tree depths, showing training and test accuracies for both methods. We report mean(std)\text{mean}_{(\text{std})} for 5 seeds.
ModelHal. Vol.Hal. EffortHal. Dif.
DEFT15.515.50.875
DEFTperf\texttt{DEFT}_{\texttt{perf}}19.717.31.0
Table 3: Halstead metrics : Volume, effort and difficulty (lower is better). Removing the intepretability constraints leads to more complex features.
Figure 5: Ablations. Mean and confidence intervals at 95%95\% for 55 seeds.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

CharacteristicValue
Total number of samples60006000
Training set size40004000
Test set size20002000
Dimensionality of the sequences101101
Label distribution (training set)Pausing: 51%51\% , Non-pausing: 49%49\%
Appendix
Table 4: Pol II pausing dataset characteristics
CharacteristicValue
Total number of samples3613136131
Training set size2709727097
Test set size90349034
Dimensionality of the sequences251251
Label distribution (training set)1: 54%54\% , 0: 46%46\%
Appendix
Table 5: Non-TATA Promoters dataset characteristics
CharacteristicValue
Total number of samples1639016390
Training set size1610816108
Test set size282282
Dimensionality of the sequences145145
Label distribution1: 50%50\% , 0: 50%50\%
Appendix
Table 6: Enhancers dataset characteristics
ParameterValue
Model namegpt-4o
Version1001
Temperature11
Top p0.950.95
Appendix
Table 7: LLM parameters
HyperparameterValues
Conv1 filters{32, 64}
Conv1 kernel size{2, 12, 16}
Conv2 filters{64, 128}
Dense units{64, 128}
Dropout rate{0, 0.2, 0.5}
Learning rate{1×10−3,1×10−4}\{1\times 10^{-3},1\times 10^{-4}\}
Appendix
Table 8: CNN grid-search hyperparameters and their ranges.
HyperparameterValues
Hidden dimension{32, 64, 128}
Attention heads{2, 4}
Encoder layers{1, 2, 4}
Dropout rate{0, 0.2, 0.5}
Learning rate{1×10−3,1×10−4}\{1\times 10^{-3},1\times 10^{-4}\}
Batch size{32, 128}
Appendix
Table 9: Transformer grid-search hyperparameters and their ranges.
Section / ReferenceSummaryResults
Section C.1Feature found without interpretability constraints.—
Section C.2Unregularized CART increasingly overfits as depth grows.Fig. 6
Section C.3F1, precision, recall, and AUPRC across datasets.Fig. 7
Section C.4DEFT vs. Nucleotide Transformer (fine tuned with LoRA) for context.Tab. 12
Section C.5Specialized CNN/attention baselines: PEPMAN (Pol II), DeePromoter (Promoters).Tab. 13 , Tab. 14
Section C.6DEFT instantiated with an open-source LLM ( gpt-oss ).Tab. 15 ,
Appendix
Table 10: Summary of additional results.
Figure 6: CART trees can overfit. Deep trees constructed with CART overfit the training set when there is no explicit regularization mechanism. We report the mean and confidence intervals at the 95%95\% level for 55 seeds.
Figure 7: Performance comparison across datasets (rows) and metrics (columns). Each plot shows mean with 95%95\% CIs over 55 seeds.
HyperparameterValue
LoRA rank ( rr )1
LoRA alpha ( α\alpha )32
LoRA dropout0.1
OptimizerAdamW
Learning rate5×10−45\times 10^{-4}
Train batch size8
Appendix
Table 11: Nucleotide Transformer fine-tuning hyperparameters.
 Pol II PausingPromotersEnhancers
MethodAcc.AUPRCAcc.AUPRCAcc.AUPRC
CART ( 22 -mers)0.750(0.007)0.750{\scriptstyle\,(\,0.007\,)}0.830(0.003)0.830{\scriptstyle\,(\,0.003\,)}0.793(0.000)0.793{\scriptstyle\,(\,0.000\,)}0.897(0.000)0.897{\scriptstyle\,(\,0.000\,)}0.674(0.000)0.674{\scriptstyle\,(\,0.000\,)}0.939(0.000)0.939{\scriptstyle\,(\,0.000\,)}
DEFT0.869(0.007)0.869{\scriptstyle\,(\,0.007\,)}0.912(0.006)0.912{\scriptstyle\,(\,0.006\,)}0.833(0.004)0.833{\scriptstyle\,(\,0.004\,)}0.926(0.004)0.926{\scriptstyle\,(\,0.004\,)}0.767(0.049)0.767{\scriptstyle\,(\,0.049\,)}0.948(0.012)0.948{\scriptstyle\,(\,0.012\,)}
Nucleotide Transformer0.885(0.012)0.885{\scriptstyle\,(\,0.012\,)}0.948(0.009)0.948{\scriptstyle\,(\,0.009\,)}0.872(0.007)0.872{\scriptstyle\,(\,0.007\,)}0.960(0.004)0.960{\scriptstyle\,(\,0.004\,)}0.771(0.035)0.771{\scriptstyle\,(\,0.035\,)}0.919(0.040)0.919{\scriptstyle\,(\,0.040\,)}
Appendix
Table 12: Performance comparison with the Nucleotide Transformer.
ModelAccuracyAUPRC
CART (2-mers)0.750(0.007)0.750{\scriptstyle\,(\,0.007\,)}0.830(0.003)0.830{\scriptstyle\,(\,0.003\,)}
DEFT0.869(0.007)0.869{\scriptstyle\,(\,0.007\,)}0.912(0.006)0.912{\scriptstyle\,(\,0.006\,)}
PEPMAN0.883(0.008)0.883{\scriptstyle\,(\,0.008\,)}0.944(0.007)0.944{\scriptstyle\,(\,0.007\,)}
Appendix
Table 13: Pol II pausing task
ModelAccuracyAUPRC
CART (2-mers)0.793(0.000)0.793{\scriptstyle\,(\,0.000\,)}0.897(0.000)0.897{\scriptstyle\,(\,0.000\,)}
DEFT0.833(0.004)0.833{\scriptstyle\,(\,0.004\,)}0.926(0.004)0.926{\scriptstyle\,(\,0.004\,)}
DeePromoter0.802(0.007)0.802{\scriptstyle\,(\,0.007\,)}0.914(0.005)0.914{\scriptstyle\,(\,0.005\,)}
Appendix
Table 14: Promoter identification task
 Pol II PausingPromotersEnhancers
MethodAcc.AUPRCAcc.AUPRCAcc.AUPRC
CART ( 22 -mers)0.750(0.007)0.750{\scriptstyle\,(\,0.007\,)}0.830(0.003)0.830{\scriptstyle\,(\,0.003\,)}0.793(0.000)0.793{\scriptstyle\,(\,0.000\,)}0.897(0.000)0.897{\scriptstyle\,(\,0.000\,)}0.674(0.000)0.674{\scriptstyle\,(\,0.000\,)}0.939(0.000)0.939{\scriptstyle\,(\,0.000\,)}
DEFT ( gpt-4o )0.869(0.007)0.869{\scriptstyle\,(\,0.007\,)}0.912(0.006)0.912{\scriptstyle\,(\,0.006\,)}0.833(0.004)0.833{\scriptstyle\,(\,0.004\,)}0.926(0.004)0.926{\scriptstyle\,(\,0.004\,)}0.767(0.049)0.767{\scriptstyle\,(\,0.049\,)}0.948(0.012)0.948{\scriptstyle\,(\,0.012\,)}
DEFT ( gpt-oss )0.864(0.009)0.864{\scriptstyle\,(\,0.009\,)}0.908(0.006)0.908{\scriptstyle\,(\,0.006\,)}0.829(0.001)0.829{\scriptstyle\,(\,0.001\,)}0.921(0.002)0.921{\scriptstyle\,(\,0.002\,)}0.724(0.031)0.724{\scriptstyle\,(\,0.031\,)}0.938(0.010)0.938{\scriptstyle\,(\,0.010\,)}
Appendix
Table 15: Performance comparison with an open-source LLM.
ParameterValue
Model namegpt-oss 120b
Temperature11
Top p0.950.95
Reasoning effortMedium
Appendix
Table 16: LLM parameters
 Pol II PausingPromotersEnhancers
MethodAcc.AUPRCAcc.AUPRCAcc.AUPRC
DEFT0.8700.9170.8300.9260.7130.943
DEFT (Ensemble)0.8840.9370.8480.9420.8090.969
Appendix
Table 17: Ensembling trees found by DEFT improves performance. Results reported for a given split per dataset.
Figure 8: The reflection mechanism effectively refines the features.
 Pol II PausingPromotersEnhancers
MethodAcc.AUPRCAcc.AUPRCAcc.AUPRC
DNABERT-20.795(0.006)0.795{\scriptstyle\,(\,0.006\,)}0.881(0.005)0.881{\scriptstyle\,(\,0.005\,)}0.937(0.007)0.937{\scriptstyle\,(\,0.007\,)}0.967(0.007)0.967{\scriptstyle\,(\,0.007\,)}0.855(0.025)0.855{\scriptstyle\,(\,0.025\,)}0.979(0.003)0.979{\scriptstyle\,(\,0.003\,)}
HyenaDNA0.777(0.018)0.777{\scriptstyle\,(\,0.018\,)}0.868(0.005)0.868{\scriptstyle\,(\,0.005\,)}0.909(0.009)0.909{\scriptstyle\,(\,0.009\,)}0.932(0.008)0.932{\scriptstyle\,(\,0.008\,)}0.808(0.045)0.808{\scriptstyle\,(\,0.045\,)}0.965(0.009)0.965{\scriptstyle\,(\,0.009\,)}
CART0.750(0.007)0.750{\scriptstyle\,(\,0.007\,)}0.830(0.003)0.830{\scriptstyle\,(\,0.003\,)}0.793(0.000)0.793{\scriptstyle\,(\,0.000\,)}0.897(0.000)0.897{\scriptstyle\,(\,0.000\,)}0.674(0.000)0.674{\scriptstyle\,(\,0.000\,)}0.939(0.000)0.939{\scriptstyle\,(\,0.000\,)}
DEFT0.869(0.007)0.869{\scriptstyle\,(\,0.007\,)}0.912(0.006)0.912{\scriptstyle\,(\,0.006\,)}0.833(0.004)0.833{\scriptstyle\,(\,0.004\,)}0.926(0.004)0.926{\scriptstyle\,(\,0.004\,)}0.767(0.049)0.767{\scriptstyle\,(\,0.049\,)}0.948(0.012)0.948{\scriptstyle\,(\,0.012\,)}
Appendix
Table 18: Comparison with DNABert-2 and HyenaDNA .
More in this topic

Interpretable ML

Interpretable machine learning develops models and explanations that make predictions, evidence, and decision processes understandable to people.

See all 306 papers

Explore similar work

CardsList

  1. SeqMaestro: From nucleotide sequences to biological hypotheses through interpretable machine learning

    Sep 14, 2026Evgeny S. Saveliev, Krzysztof Kacprzyk, Charlotte Capitanchik +7Interpretable MLGenomics

    Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · The Francis Crick Institute, London, UK · UK Dementia Research Institute at King’s College London, London, UK +1

  2. AttnGen: Attention-Guided Saliency Learning for Interpretable Genomic Sequence Classification

    May 13, 2026Rayhaneh Shabani Nia, Ali KarkehabadiNeural Network InterpretabilityAttention Mechanisms

    University of California, Davis California, USA

  3. Biological Reasoning-Informed Regression for Interpretable Regulatory DNA Activity Prediction

    Jun 6, 2026Yi Duan, Zhao Yang, Jiwei Zhu +3Interpretable ML

    Gaoling School of Artificial Intelligence · Renmin University of China · Beijing, China +1