Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models
Authors: Baptiste Arnaudo, Amine Bennouna, Sébastien Martin
Organizations: École polytechnique, Palaiseau, France · Now at the Institute for Computational and Mathematical Engineering (ICME), Stanford University. · Operations Department, Kellogg School of Management, Northwestern University, Evanston, IL, USA
Feature extraction from unstructured data typically relies on opaque embeddings unsuited to interpretable decision-making. Interpretable features, on the other hand, typically require costly, and often infeasible, hand-engineering. Large language models (LLMs) offer a natural bridge, as they can read raw data and describe it in interpretable terms. One could simply ask an LLM to extract features from a large sample of the dataset, but context limitations and diluted attention can lead to unsatisfactory results. We introduce a method that automatically discovers interpretable, decision-relevant features from unstructured data. Our key idea is to perform pairwise comparisons between items, asking an LLM to articulate, in interpretable terms, the features distinguishing each pair. We treat these explanations, or rationales, as data: we develop a statistical choice model that combines these rationales across the comparisons to identify relevant features and estimate per-item, per-feature scores. We first test our method on a synthetic hiring dataset of cover letters with known ground-truth features and show that it significantly outperforms direct LLM feature extraction in feature recovery. We then apply it to the ChangeMyView (Reddit) debate dataset, where it discovers several interpretable features associated with persuasion while reaching predictive accuracy comparable to established baselines. We also show how the same rationales can be used to audit LLM decisions, yielding a holistic score whose decomposition over interpretable features reveals what the LLM rewards.
Figures & tables
Figure 1: Overview of the pipeline described in Section 3 . Items are first compared in pairs by the LLM, cited features are then clustered, and a statistical model learns the relevant features from the LLM’s choices. Finally, statistical diagnostics identify the reliable scores. The orange running example follows one pair of cover letters (the setup of Section 4 ) through the pipeline.
Method
Cost
K
Tracked
Stat.R.
Sem.R.
Attrib.
LK ↓
Ours
3.45
21
100.0%
0.850
100.0%
76.2%
0.090
Benchmarks
Direct LLM
0.26
21
81.2%
0.732
81.2%
61.9%
0.109
Direct LLM–Multi-score
3.42
21
81.2%
0.787
81.2%
66.7%
0.118
Local LLMs
2.35
23
75.0%
0.702
68.8%
47.8%
0.110
Local LLMs–Multi-score
3.27
23
75.0%
0.737
68.8%
47.8%
0.113
Ablations
Pairwise comparisons + LLM scoring
3.67
32
93.8%
0.763
87.5%
40.6%
0.093
Table 1: Per-method estimates on the single full-data run. Costs are Batch-API prices in USD; K is the number of features the method reports. Tracked, Sem.R., Attrib. in %; Stat.R. and LK are correlations.
Figure 2: Discovered features span more effective dimensions than prior ones while retaining predictive accuracy. In both panels, each point is an average over random k -subsets of the feature set, so that sets of different sizes are compared at matched size. Left: noise-corrected participation ratio of the feature-score correlation matrix (Spearman) over the 1,600 held-out exchanges; dotted: y=k . Right: held-out accuracy of the ℓ1 logistic regression on the 800 held-out pairs; chance =50% . D: discovered ( 29 features); D r : reliable subset of D ( 16 ); P: prior ( 16 ); S: sample-based ( 42 ); BT/PW: Bradley–Terry/pointwise scoring; #words: reply length. Bands: 95% bootstrap percentile intervals over held-out exchanges (left) and exchanges + pairs (right). Shaded: k>16 , where P and D r are not defined, and k>29 , where only S is defined. Details in Appendix D.3.1 .
Figure 4
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Which features matter for each judge on each dataset. Radial axis is σk (with estimation noise removed) on each judge’s reliable features ( ρ>0.5 , κ<100 ). Bars are 95% delta-method intervals from the sandwich covariance. Axes are cross-judge clusters formed by meaning ( GPT-5.4 (high reasoning) ), restricted to constructs at least three of the four judges discovered (see Figures 6 and 7 for the full plots). An open marker denotes a construct absent from that judge’s reliable basis (never surfaced, or removed by the identifiability screen), not a zero estimate.
Figure 5: Per-judge overall score distribution and its agreement (Spearman rank correlation) with human scores.
Figure 6: Denoised per-feature spread by judge on ELLIPSE. Axes are cross-judge constructs formed by meaning ( GPT-5.4 (high reasoning) ); annotations give how many of the four judges discovered each construct. The radial coordinate is the denoised spread σk of each judge’s feature mapped to that construct, on per-judge standardized reliable feature sets ( ρ>0.5 , κ<100 ); bars are 95% delta-method intervals from the sandwich covariance. An open marker denotes a construct absent from that judge’s reliable basis (never surfaced, or removed by the identifiability screen), not a zero estimate.
Figure 7: Denoised per-feature spread by judge on ASAP 2.0. Same construction, quantities, and conventions as Figure 6 , on the ASAP 2.0 corpus.
Figure 8: Out-of-fold reliability diagrams for the winner gate (left; unit: comparison) and the citation gate (right; unit: cell) of the model on ELLIPSE, one row per judge. Units are grouped into ten equal-width bins B1,…,B10 of predicted probability p^i ; points show the observed event rate yˉb=∣Bb∣1∑i∈Bbyi against the mean prediction pˉb=∣Bb∣1∑i∈Bbp^i , with 95% bootstrap intervals (resampling comparisons, resp. cells). Faint line: in-sample fit; dashed: perfect calibration ( yˉb=pˉb ). The expected calibration error is ECE=∑bN∣Bb∣∣yˉb−pˉb∣ , with N=∑b∣Bb∣ . Bottom strips: ∣Bb∣ per bin, log scale.
Figure 9: Out-of-fold Brier skill of the two gates, per judge and corpus, with in-sample values overlaid. With p^i the predicted probability and yi∈{0,1} the gated event, the Brier score is BS=N−1∑i(p^i−yi)2 and the skill score against a reference forecast qi is BSS=1−BS/BSref , where BSref=N−1∑i(qi−yi)2 ; BSS=1 is perfect, 0 matches the reference, and negative values are worse than it. Two references: chance ( qi=21 ) and the empirical event rate of the training fold ( qi=yˉtrain ). Intervals as in Figure 8 .
Figure 10: As Figure 8 , on ASAP 2.0.
Figure 11: The job advertisement used as the fixed evaluation context. Colored tags mark which requirement grounds which dial, using the same visual vocabulary as the letter template (Figure 12 ): the qualifications ground the education and numbered dials, and the preferred qualifications and responsibilities ground the six role-skill dials. Go is a mandatory requirement realized identically in every letter, not a dial. The remaining dials of Table 3 (school and employer prestige, prose fluency, and the four niche dials) have no anchor in the ad and are described in Appendix C.1 .
Figure 12: Example cover-letter template. Roman text is fixed template prose; angle-bracketed slots are filled from the ground-truth dial vector. Skill coverage sets how many core units are capability sentences; paragraph count is constant. Fluency is imposed afterward by injecting 0/5/12 surface errors.
Dial
Type
Levels
Group
Experience (years)
ordered
3 / 5 / 7
primary
Education level
ordered
BS / MS / PhD
primary
School selectivity
ordered
low / mid / high
primary
Employer prestige
ordered
startup / mid / big tech
primary
Languages known
ordered
2 / 5 / 9
primary
Production launches
ordered
2 / 4 / 6
primary
Appendix
Table 3: The 18 engineered ground-truth dials of the cover-letter corpus.
Figure 13: Out-of-fold reliability of the citation gate (left, all cells) and direction gate (right, cited cells) of the feature-discovery model. Points: observed rate per bin of predicted probability, with 95% bootstrap intervals over comparisons; faint line: in-sample; dashed: perfect calibration; bottom: cells per bin (log scale).
Figure 14: Out-of-fold Brier skill of the two gates against chance and against the training-fold empirical rates; intervals as in Figure 13 .
feature
Cite rate
γk
κ
ρ
Kept
AI experience
0.613
−1.36[−1.45,−1.27]
3.6
0.937
✓
Security assessment experience
0.596
−1.37[−1.46,−1.28]
2.9
0.935
✓
Security automation experience
0.562
−1.31[−1.40,−1.22]
2.4
0.917
✓
Agentic systems experience
0.544
−1.52[−1.61,−1.43]
2.7
0.924
✓
Professional experience
0.531
−2.43[−2.55,−2.31]
3.8
0.960
✓
Algorithmic strength
0.524
−1.48[−1.57,−1.39]
2.7
0.917
✓
Appendix
Table 4: Features surfaced by our comparative method in the point estimate of Table 1 . Of 32 surfaced features, we keep the features that have ρ≥ρmin:=0.75 and κ≤κmax:=10 , leaving 21 features. Fitted β^=+0.005 (95% CI [−0.033,+0.043] ). CIs for β^ and γ coefficients were computed using ( B.3 ).
Feature
Description
Scoring guidelines
Academic degree
Captures the formal level of academic preparation or equivalent technical grounding the applicant presents.
1 No clear qualifying degree or equivalent preparation is shown. 2 Shows a bachelor’s degree or equivalent practical preparation. 3 Shows a master’s degree in a relevant technical field. 4 Shows a doctoral degree in a relevant technical field.
Adversarial practice
Captures ongoing offensive-security practice through CTF participation and competitive results.
1 No ongoing offensive-security practice is shown. 2 Mentions occasional CTF participation or light adversarial practice. 3 Shows regular or serious CTF participation as continuing skill development. 4 Shows strong competitive results, such as near-top or top placements in substantial CTF fields.
Agent boundary design
Captures experience designing agentic systems with explicit control boundaries between Python agents and Go services.
1 No agentic tooling or agent architecture is described. 2 Mentions agents or agentic systems only at a high level. 3 Describes building agentic systems with Python agents and Go back-end support. 4 Describes explicit guardrails such as bounded actions, permissions, auditing, or enforced boundaries between agents and services.
Algorithmic depth
Captures depth of data-structure and algorithm reasoning as evidenced by concrete redesign or optimization examples.
1 Does not provide evidence of algorithmic or data-structure skill. 2 Claims strong algorithms knowledge, but with little concrete illustration. 3 Provides a clear example of using better algorithms or data structures to improve a system. 4 Provides a strong example with technical specificity and visible impact, such as removing a bottleneck or making a previously hard problem tractable.
Applied AI deployment
Captures whether the applicant has shipped AI or LLM features in production and understands reliability or evaluation concerns.
1 No meaningful AI or LLM experience is shown. 2 Mentions AI interest or exposure without a clear shipped production example. 3 Shows direct experience shipping AI or LLM-backed functionality in production. 4 Shows shipped AI or LLM systems plus concrete attention to evaluation, reliability, or failure modes.
Developer enablement mindset
Captures whether the applicant frames security tooling as a way to help developers do the right thing with less friction.
1 Does not frame security work in terms of helping other engineers. 2 Mentions helping teams only in broad or generic language. 3 Clearly emphasizes making secure behavior easier or more natural for developers. 4 Explicitly focuses on reducing friction, building good defaults, or serving busy developer teams through usable tooling.
Appendix
Table 5: The rubric elicited by Direct LLM. Descriptions and the per-score guidelines are reproduced verbatim as the judge produced them; the open elicitation lets the judge choose its own scale, and it selected a 1 – 4 scale for every feature.
Figure 15: Paired comparisons over B=50 bootstrap draws (with replacement) of the 200 -letter corpus; every arm sees the same draws, and differences are taken within replicate. Each row shows the mean paired difference with its 95% percentile CI ( ⋆ : excludes zero); the right column gives Ours better , the share of replicates our method wins ( ΔK , the difference of discovered features, is annotated below each panel and carries no share). The panels bracket discovery variance from opposite sides and are not comparable to one another : (a) re-runs discovery per replicate on both sides, (b) freezes discovery on both sides, pooled over fusion and corpus draws. Both panels score our method at a reduced budget (average degree 25.3 against the designed 40 ); it leads on several metrics regardless. Exact values: Table 7 .
Method
Cost
K
Tracked
Stat.R.
Sem.R.
Attrib.
LK ↓
Ours
3.45
21
100.0%
0.850
100.0%
76.2%
0.090
resampling matchings
[16.2, 22.0]
[87.5, 93.8]
[0.737, 0.805]
[81.2, 93.8]
[71.4, 87.5]
[0.087, 0.095]
Direct LLM
0.26
21
81.2%
0.732
81.2%
61.9%
0.109
resampling letters
[17.2, 25.0]
[68.8, 87.5]
[0.649, 0.825]
[62.5, 87.5]
[40.6, 69.6]
[0.109, 0.135]
Direct LLM–Multi-score
3.42
21
81.2%
0.787
81.2%
66.7%
0.118
resampling letters
[21.0, 21.0]
[81.2, 81.2]
[0.788, 0.807]
[81.2, 81.2]
[57.1, 66.7]
[0.119, 0.145]
Appendix
Table 6: Per-method point estimates with run-variance bands. Tracked, Sem.R., Attrib. in %; Stat.R. and LK are correlations; bands are in the same units as the column. Below each method’s point estimates, the gray row gives that method’s own run-variance band ( [2.5,97.5] percentiles across replicates), obtained by resampling the unit named at the left of the row. These bands are not confidence intervals for the point estimates above them – the point estimates come from the single full-data run, the bands from re-runs on resampled data, so a point estimate may fall outside its band – and they must not be compared across methods , since the arms resample different units. Paired, comparable contrasts: Table 7 .
Difference
95% CI
Ours better
(a) Ours − Direct LLM (discovery re-run per replicate on both sides)
ΔK
−4.90
[−9.78,−1.00]
—
Δ Tracked ∗
+0.138
[+0.014,+0.250]
96%
Δ Stat.R.
+0.051
[−0.038,+0.132]
88%
Δ Sem.R. ∗
+0.122
[+0.000,+0.250]
76%
Δ Attrib. ∗
+0.349
[+0.233,+0.543]
100%
Appendix
Table 7: Paired comparisons over B=50 bootstrap draws (with replacement) of the 200 -letter corpus; every arm sees the same draws, and differences are taken within replicate. Each row reports the mean difference, its 95% percentile CI ( ∗ : excludes zero), and Ours better : the share of replicates in which our method wins in the metric’s preferred direction ( ΔK is descriptive and carries no share). The panels bracket discovery variance from opposite sides and are not comparable to one another . Both panels score our method at a reduced budget (average degree 25.3 against the designed 40 ) ; it leads on several metrics regardless. Design rationale, the conditional-fusion alternative, and budget details: Appendix C.4.2 .
Figure 16: Dial-recovery heatmap yielded by the features discovered by our method. The rows are the discovered features and the columns are the dials.
Figure 17: Dial-recovery heatmap yielded by the features discovered by Direct LLM. The rows are the discovered features and the columns the dials. The scores leak more between one another and the dial recovery is less clean than in Figure 16 as some dials are not recovered.
Figure 18: Left: probability that a dial is recovered as a function of the acceptance threshold used to merge similar features across the E=27 elicitations of Local LLMs (dials are matched to the merged features by thresholded Hungarian assignment at the calibrated η ). Right: number of features surviving the merge, against the number of pooled elicitations, one curve per acceptance threshold. At E=27 , recovering every dial demands a high threshold, which leaves an unusable number of near-duplicate features, hence the need for an LLM taxonomy.
Framing
Accuracy
P1 (unnamed, scrubbed)
53.6%[51.2,56.1]
P2 (named)
53.6%[51.1,56.1]
Lift P2−P1
−0.1 pp [−2.4,+2.3]
Appendix
Table 8: Memorization probe on 1,600 CMV exchanges pinned to the data of Table 2 (judge: GPT-5.4 mini ; chance =50% ). The lift is assessed by a paired McNemar test on the same exchanges ( b/c=185/184 , p=1.000 ).
Figure 19: The reliable discovered features are more predictive, feature for feature, than the full discovered set and than the sample-elicited set. Each panel shows, as a function of the number of features k , the difference in held-out accuracy between D r (the 16 reliable discovered features) and another feature set. At each k , an ℓ1 -regularized logistic regression is fitted on up to 50 random k -subsets of each set (all subsets when there are fewer), and accuracy is averaged over subsets; PW scores are averaged over two scoring runs. Shaded: 95% bootstrap interval at each k ( 4,000 resamples of the 800 held-out pairs and of the drawn subsets). (a, b) Against the full discovered set D, with the same scoring: D r is ahead at every k , by 1.0 point on average over k under both scorings ( 95% CIs [−0.1,2.2] for PW, [0.0,2.1] for BT). (c, d) Against the sample-elicited set S, scored pointwise only: D r + PW is ahead at every k , by 3.6 points on average ( [1.8,5.4] ); D r + BT’s advantage grows with k , reaching 3.5 points at k=16 ( [0.5,6.6] ).
Arm
Dims
Accuracy
95% CI
Emb. (full)
3,072
64.88%
[61.50,68.00]
Tan
3,205
63.75%
[60.38,67.00]
D + PW
29
62.63%
[59.38,66.00]
D r + PW
16
62.13%
[58.88,65.50]
Emb.
3,072
61.63%
[58.25,65.00]
D r + BT
16
61.50%
[58.13,64.88]
Appendix
Table 9: Held-out accuracy of each method. D r : the reliable subset of the discovered features D. Emb. denotes text-embedding-3-large embeddings. Arms marked (full) are trained on all 3,411 training pairs; all others on 400 . Accuracies for PW arms are obtained from the average of two scoring runs. Zero-shot asks the judge directly to predict the Δ -winning reply, with the query issued in both presentation orders and the two predictions averaged. Chance =50% ; 95% bootstrap percentile intervals ( 4,000 resamples of the 800 held-out pairs, shared across arms).
Figure 20: Both dimensionality measures agree on the ordering of the feature sets. Noise-corrected effective dimensionality against feature set size k , averaged over k -subsets of each feature set. Left: participation ratio, as in Figure 2 . Right: effective rank [ Roy and Vetterli, 2007 ] , expH(p) with pi=σi/∑jσj the normalized singular values. Both are computed on the disattenuated correlation matrix. The order holds under both measures. Bands, shading and dotted line as in Figure 2 .
Figure 21: Effectiveness of the discovered argumentation techniques (D + PW). The figure is the complete version of Figure 3 and shows each discovered feature’s effectiveness , i.e. the fraction of the test pairs in which the reply that earned the Δ scores higher on that feature. 0.5 is chance. Brackets give 95% Wald intervals and q is the Benjamini–Hochberg false-discovery rate across the 29 tests. The discovered features are gathered by category. Exact values of the BH q and each feature’s description can be found in Table 10 .
Feature
Description
Win rate (95% CI)
q
Concrete illustration ∗
The argument makes its case through specific, vivid, or worked-through examples rather than abstraction alone.
0.604 [0.570,0.638]
< 0.001
Qualified rebuttal ∗
The challenger narrows the claim, marks exceptions, or emphasizes contextual dependence instead of arguing absolutely.
0.596 [0.562,0.630]
< 0.001
Structured rebuttal development ∗
The challenger builds an organized, multi-step, or multi-pronged case rather than a single bare point.
0.583 [0.548,0.617]
< 0.001
Scope broadening
The exchange expands or contracts the discussion by moving between narrow cases and wider principles or issue areas.
0.583 [0.548,0.617]
< 0.001
Counterexample use ∗
The challenger undermines the OP with exceptions, alternative cases, or edge-case scenarios that break the generalization.
0.581 [0.547,0.616]
< 0.001
Challenger evidentiary support ∗
The challenger backs the rebuttal with external evidence, citations, factual corrections, or expert authority.
0.575 [0.541,0.609]
< 0.001
Appendix
Table 10: Which argumentation techniques separate winners from losers (D + PW). This table details the results shown in Figures 3 and 21 . It presents each discovered feature (its name and description) together with its effectiveness , i.e. the fraction of the test pairs in which the reply that earned the Δ scores higher on that feature. 0.5 is chance. Brackets give 95% Wald intervals and q is the Benjamini–Hochberg false-discovery rate across the 29 tests; features in bold satisfy q<0.05 . 19 of the 29 features separate winners, all of them positively. Features marked ∗ form the reliable subset D r ( 16 features, κ≤50 and ρ≥0.75 ).
Feature
Description
Core Claim Relevance
It measures the degree to which the reply focuses on the central claim under dispute rather than drifting to side issues.
Reason Engagement
It measures the degree to which the reply directly addresses the original poster’s stated reasons, premises, or assumptions.
View Understanding
It measures the degree to which the reply accurately represents the other person’s position before challenging it.
Logical Coherence
It measures the degree to which the reply’s conclusions follow clearly and consistently from its premises.
Evidence Strength
It measures the degree to which the reply supports its claims with substantial, pertinent evidence or examples.
Source Trustworthiness
It measures the degree to which the reply relies on credible, authoritative, and verifiable sources when offering support.
Appendix
Table 11: Prior-elicited features (P). The 16 features named by GPT-5.4 (high reasoning) for assessing the persuasiveness of a reply, with no data shown (Prompt ), in the order produced.
Feature
Description
Topical responsiveness
How directly a reply addresses the poster’s actual view rather than pivoting to meta-issues or adjacent topics.
Premise targeting
Whether a reply attacks the core assumption or inference supporting the view instead of focusing on side details.
Clarification demand
How much a reply advances the discussion by asking the poster to specify vague claims, terms, or examples.
Conceptual reframing
Whether a reply changes the frame of the debate by redefining the key concept or explaining that the dispute is really about something else.
Thought experiment
Whether a reply uses an invented scenario to test the implications or consistency of the poster’s view.
Analogy use
How much a reply explains its point by mapping the issue onto a different familiar domain.
Appendix
Table 12: Sample-elicited features (S). The 42 features named by GPT-5.4 (high reasoning) after reading a sample of 752 exchange pairs, without their outcomes.
Feature discovery from complex unstructured data is fundamentally a reasoning problem: it requires identifying abstractions that are predictive of a target outcome while avoiding leakage, proxies, and post-outcome signals. With the introduction of ever-improving Large Language Models (LLMs), our method provides a structured method for addressing this challenge. LLMs are well suited for this task by being able to process large amounts of information, but unconstrained feature generation can lead to weak features. In this work, we study reasoning control in LLMs by inducing cognitive behaviors for improving feature discovery. We introduce CoFEE (Cognitive Feature Engineering Engine), a reasoning control framework that enforces cognitive behaviors in how the LLM reasons during feature discovery. From a machine learning perspective, these cognitive behaviors act as structured inductive biases over the space of candidate features generated by the model. These behaviors have been exploited with success in ML models, and include backward chaining from outcomes, subgoal decomposition, verification against observability and leakage criteria, and explicit backtracking of rejected reasoning paths. In a controlled comparison, we show that enforcing cognitive behaviors yields features with higher empirical predictability than those under unconstrained vanilla LLM prompts. CoFEE achieves an average Success Rate Score that is 15.2% higher than the vanilla approach, while generating 29% fewer features and reducing costs by 53.3%. Using held-out feature evaluation, we assess whether cognitively induced features generalize beyond the data used for discovery. Our results indicate that, in our evaluated setting, reasoning control is associated with improvements in quality and efficiency of LLM-based feature discovery.
Maximilian Westermann, Ben Griffin, Aaron Ontoyin Yin +6
We introduce an iterative framework that automates the extraction of interpretable, schema-bound categorical features from unstructured text for tabular prediction models. To navigate the feature space, a generator LLM proposes semantic definitions, a separate extractor LLM materializes the features, and a downstream tabular model evaluates their predictive performance. We optimize this search by translating explicit model errors, such as AUC ranking inversions, into natural-language feedback, steering the LLM to resolve specific predictive failures. Evaluated across three public datasets, this error-driven loop accelerates feature discovery by up to 3× compared to unguided search. Empirically, the generated features demonstrate strong multi-view complementarity, strictly outperforming any subset when combined with TF-IDF and dense embeddings. Finally, the framework guarantees instance-level interpretability: the discovered features dominate SHAP importance rankings and provide a fully transparent, semantic audit trail for every prediction.
Large language models (LLMs) are increasingly used as classifiers, yet they operate as opaque systems whose decisions are difficult to interpret, which complicates their use in regulated domains such as credit scoring or medical diagnosis. We propose an evolutionary framework that iteratively discovers natural language feature definitions (rubrics) for interpretable classification. An LLM generates candidate binary features, evaluates each sample against them, and the resulting vectors can be used to train a transparent classifier such as logistic regression. The feature set evolves over multiple iterations guided by classification errors, per-class activation rates, and feature ablation scores. We evaluate across three benchmarks, including a credit risk dataset representative of regulated domains, comparing single-shot LLM rubrics, evolved rubrics, and direct zero-shot LLM classification. Evolved features improve over single-shot rubrics by +2.9 pp on average and outperform zero-shot LLM classification on two of three tasks, while providing fully auditable decision logic. On the credit risk task, the zero-shot LLM performs at chance (50.7%) with a strong bias toward a single class, whereas evolved features achieve balanced, interpretable predictions. Crucially, this failure is invisible in aggregate accuracy and surfaces only under per-class auditing. Analysis reveals that evolution is most effective when label boundaries cannot be inferred from category names alone or when the LLM lacks reliable domain-specific reasoning.