Large language models (LLMs) are increasingly used as classifiers, yet they operate as opaque systems whose decisions are difficult to interpret, which complicates their use in regulated domains such as credit scoring or medical diagnosis. We propose an evolutionary framework that iteratively discovers natural language feature definitions (rubrics) for interpretable classification. An LLM generates candidate binary features, evaluates each sample against them, and the resulting vectors can be used to train a transparent classifier such as logistic regression. The feature set evolves over multiple iterations guided by classification errors, per-class activation rates, and feature ablation scores. We evaluate across three benchmarks, including a credit risk dataset representative of regulated domains, comparing single-shot LLM rubrics, evolved rubrics, and direct zero-shot LLM classification. Evolved features improve over single-shot rubrics by +2.9 pp on average and outperform zero-shot LLM classification on two of three tasks, while providing fully auditable decision logic. On the credit risk task, the zero-shot LLM performs at chance (50.7%) with a strong bias toward a single class, whereas evolved features achieve balanced, interpretable predictions. Crucially, this failure is invisible in aggregate accuracy and surfaces only under per-class auditing. Analysis reveals that evolution is most effective when label boundaries cannot be inferred from category names alone or when the LLM lacks reliable domain-specific reasoning.
Figures & tables
Figure 1: Overview of the evolutionary feature discovery framework. An LLM generates binary feature definitions (the rubric), evaluates each training sample against them, and a logistic regression classifier is trained on the resulting feature matrix. Classification errors, per-class activation rates, and feature ablation scores are fed back to guide the next rubric proposal (dashed arrows).
Figure 2
Figure 2: Comparison of three LLM classification approaches. (a) An LLM generates a static rubric of feature definitions in a single pass, evaluated by an LLM. (b) Our framework uses an LLM to iteratively generate and refine features via evolutionary feedback from classifier performance. (c) A zero-shot LLM classifies directly without explicit features. Only (a) and (b) produce transparent, auditable classifiers; only (b) adapts features to the data.
Dataset
TF-IDF
Zero-shot features
Evolved features
Zero-shot LLM
AG News
80.0 ± 4.1
82.7 ± 0.5
81.3 ± 2.6
89.7 ± 1.2
GoEmotions
31.3 ± 8.2
34.7 ± 6.2
41.0 ± 1.6
38.7 ± 3.7
HELOC
62.3 ± 6.0
65.3 ± 5.7
69.0 ± 2.2
50.7 ± 1.7
Average
57.9
60.9
63.8
59.7
Table 1: Test accuracy (%) across 3 evaluation splits (mean ± std). Best per dataset in bold .
Figure 3: Cross-validation accuracy over evolutionary iterations. Solid lines show best-accepted CV score; shaded regions show the range across 3 seeds. Evolution makes decisive improvements early in the search on tasks where the initial rubric is suboptimal.
Method
Classifier training calls
LLM inference calls
TF-IDF + logistic regression
0
0
Zero-shot features
501
1
Evolved features (ours)
5,010
1
Zero-shot LLM classifier
0
1
Table 2: LLM calls required for training (feature discovery) and inference (per sample). Evolved features assume T=10 iterations with n=500 training samples.
Figure 4: Aggregate accuracy hides a per-class failure on HELOC. (a) Both methods sit near or above the majority baseline, so neither looks broken. (b) On the balanced 50/50 test set, the zero-shot LLM predicts at_risk for 81% of applicants (a systematic single-class bias), whereas evolved features stay balanced. The failure surfaces only under per-class auditing.
#
Feature definition
∣wˉ∣
1
Net Fraction Of Revolving Burden > 40 (high revolving debt)
0.652
2
Number Of Satisfactory Trades < 15 (few satisfactory trades)
0.539
3
Months Since Last Illegal Trade between 0 and 12 (recent illegal-trade activity)
0.505
4
Percentage Of Legal Trades < 90 (low legal-trade share)
0.399
5
Months Since First Trade < 100 and Nr Trades Initiated In Last Year ≥ 3 (short history with recent activity)
0.374
6
Nr Banks With High Ratio ≥ 2 (multiple lenders stressed)
0.314
Table 3: Evolved rubric for HELOC, the task on which the zero-shot LLM performs at chance with a strong single-class bias. Evolution discovers numerical thresholds that act as decision stumps expressed in natural language, each checkable against a credit report.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Evolved rubrics for all three datasets, showing which features were retained from the initial rubric (blue) and which were newly discovered by evolution (green). Evolution replaces generic features with task-specific discriminators, with the degree of replacement reflecting task difficulty.
#
Feature definition
∣wˉ∣
1
Contains sports-specific terminology (athlete names, team abbreviations, sport league names like NHL, NBA, or position/play descriptions)
Table 4: Final evolved rubric for AG News (4-class topic classification) after 15 iterations. The initial rubric (iteration 0) was never improved upon. Features ordered by average absolute classifier coefficient.
School of Management Yale University New Haven, CT, US, 06511 · Department of Accountancy, Economics and Finance Hong Kong Baptist University Kowloon, Hong Kong SAR
Citigroup, Irving, TX, United States · University of North Carolina at Chapel Hill, Chapel Hill, NC, United States · Georgia Institute of Technology, Atlanta, GA, United States +1