Large language models (LLMs) are increasingly used for high-stakes decision-making, yet existing approaches struggle to reconcile scalability, interpretability, and reproducibility. Black-box models obscure their reasoning, while recent LLM-based rule systems rely on per-sample evaluation, causing costs to scale with dataset size and introducing stochastic, hallucination-prone outputs. We propose reframing LLMs as code generators rather than per-instance evaluators. A single LLM call generates executable, human-readable decision logic that runs deterministically over structured data, eliminating per-sample LLM queries while enabling reproducible and auditable predictions. We combine code generation with automated statistical validation using precision lift, binomial significance testing, and coverage filtering, and apply cluster-based gap analysis to iteratively refine decision logic without human annotation. We instantiate this framework in venture capital founder screening, a rare-event prediction task with strong interpretability requirements. On VCBench, a benchmark of 4,500 founders with a 9% base success rate, our approach achieves 37.5% precision and an F0.5 score of 25.0%, outperforming GPT-4o (at 30.0% precision and an F0.5 score of 25.7%) while maintaining full interpretability. Each prediction traces to executable rules over human-readable attributes, demonstrating verifiable and interpretable LLM-based decision-making in practice.
Figures & tables
Method
Cost
Speed
Interpret.
Stat. valid.
Determin.
CAAFE Hollmann et al. (2023)
Low
Fast
Low
CV only
Yes
FeatLLM Balek et al. (2024)
High
Slow
High
None
No
RRF Griffin et al. (2025)
High
Slow
High
None
No
Ours
Low
Fast
High
Built-in
Yes
Table 1: Comparison with related work on automated feature engineering across five key dimensions.
Figure 1: Overview of our LLM-as-code-generator pipeline. The LLM is called once to generate executable Python rules, which then evaluate deterministically across all founders without further LLM involvement. Statistical validation filters low-quality rules, and cluster-based gap analysis guides iterative refinement.
Field
Type
Example Values
industry
string
“Software Development”
educations
list
degree, field, qs_ranking
jobs
list
role, company_size, industry, duration
ipos
list
prior IPO events
acquisitions
list
prior acquisition events
Table 2: Founder profile data structure.
Figure 2: Rule generation process. The LLM receives structured founder profiles and generates Python lambda expressions that encode binary screening rules. These rules execute deterministically across all 4,500 founders without additional LLM calls.
Metric
Threshold
Rationale
Precision lift
>1.0
Above-baseline success rate
p-value
<0.05
Statistical significance
Coverage
1%–50%
Sufficient power, meaningful discrimination
Table 3: Statistical validation criteria for rule filtering.
Method
Precision
Recall
F 0.5
Random Baseline
9.0%
9.0%
9.0%
Y Combinator
14.0%
6.9%
8.6%
Tier-1 VCs
23.0%
5.2%
10.7%
Claude-3.5-Haiku
15.8%
46.4%
18.2%
GPT-4o
30.0%
16.3%
25.7%
o3
43.2%
7.4%
21.5%
Table 4: Performance comparison on VCBench.
Figure 3: Rule quality distribution. Each point is a generated rule. Significant rules (green) cluster above the lift=1.0 threshold, with a tradeoff between lift and coverage.
Iter
Rule
Lift
Cov.
1
prior_exit
3.05 ×
3.16%
1
founder_scaled_company_200plus
2.80 ×
2.56%
1
worked_in_vc_pe
2.35 ×
3.47%
1
executive_chair_or_operating_partner
2.28 ×
2.71%
1
elite_qs_top10
2.18 ×
11.91%
2
cybersecurity_domain_depth
6.54 ×
0.38%
Table 5: Top rules by precision lift per iteration.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Rule Name
Description
Lift
Coverage
p-value
Iteration 1: Initial generation (no feedback)
prior_exit
Founder has ≥ 1 prior IPO or acquisition
3.05 ×
3.16%
< .001
founder_scaled_200plus
Founded/co-founded company that grew to 200+ employees
Security industry + technical security role experience
6.54 ×
0.38%
< .001
Appendix
Table 6: Representative generated rules with validation statistics. Iter 1 rules were generated without feedback; Iter 2 rules were guided by cluster-based gap analysis targeting underperforming subpopulations.
Metric
Iter 1
Iter 2
Rules generated
30
30
Significant (p < 0.05)
19 (63%)
21 (70%)
Mean lift (sig. rules)
1.70 ×
2.58 ×
Median lift (sig. rules)
1.46 ×
1.88 ×
Rules with lift > 2 ×
5
10
Rules with lift > 3 ×
1
6
Appendix
Table 7: Rule generation statistics per iteration. Feedback-guided generation (Iter 2) produces more significant rules with higher mean precision lift, though with lower average coverage as the LLM targets more specialized subpopulations.
Many high-stakes screening tasks require predicting rare outcomes from unstructured text, where errors are costly and decisions must be auditable. We introduce Random Rule Forest (RRF), an interpretable ensemble that uses a large language model (LLM) not as an end-to-end predictor but as a generator of simple YES/NO questions. Each question acts as a weak learner, and their responses are combined by a plain unit-weight vote into an auditable ``green-flags'' scorecard: enough independent positive signals indicate a higher chance of success. We argue this deliberate simplicity is a robust default when positives are scarce and learned weights are hard to estimate. We evaluate RRF in two low-base-rate domains. On early-stage startup screening from founder profiles, RRF produces a transparent scorecard whose precision is several times the base rate (with light expert input raising it further) and, unlike direct prompting, its operating point can be controlled directly. On an established Phase~I clinical-trial benchmark, RRF outperforms published baselines on the threshold-independent metrics PR-AUC and ROC-AUC. Together these show that LLMs can serve as auditable feature generators for high-stakes text-based decisions, combining transparency with competitive predictive performance.
Ben Griffin, Aaron Ontoyin Yin, Diego Vidaurre +4
University of Oxford, United Kingdom · Vela Research, San Francisco, United States · Aarhus University, Aarhus, Denmark +2
Large Language Models are increasingly deployed for decision-making, yet their adoption in high-stakes domains remains limited by miscalibrated probabilities, unfaithful explanations, and inability to incorporate expert knowledge precisely. We propose IDEA, a framework that extracts LLM decision knowledge into an interpretable parametric model over semantically meaningful factors. Through joint learning of verbal-to-numerical mappings and decision parameters via EM, correlated sampling that preserves factor dependencies, and direct parameter editing with mathematical guarantees, IDEA produces calibrated probabilities while enabling quantitative human-AI collaboration. Experiments across five datasets show IDEA with Qwen-3-32B (78.6%) outperforms DeepSeek R1 (68.1%) and GPT-5.2 (77.9%), achieving perfect factor exclusion and exact calibration -- precision unattainable through prompting alone. The implementation is publicly available at https://github.com/leonbig/IDEA.
Yanji He, Yuxin Jiang, Yiwen Wu +3
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd · The Hong Kong University of Science and Technology
Large language models make decisions through parallel prefill and sequential decoding. Although effective, this approach incurs decoding costs and does not explicitly expose decision probabilities. Existing task-specific classifiers require per-task training and offer limited generalization to unseen decision schemas. We propose Large Decision Models (LDMs), repurposing language models to return explicit probabilities over labels specified at run time using one prefill forward pass. We first adapt models to this interface through Diverse Decision Supervision (DDS), covering varied domains, rules and labels. To further narrow the performance gap to a reasoning model, Self-Posed Reasoning Distillation (SPRD) adds tasks generated by a frozen teacher, filters them for answer consistency, and trains on label distributions obtained from repeated teacher reasoning. We evaluate on DecisionVista, pairing new-schema tests with held-out human-labelled sources. Across three base models, the recipe improves accuracy over a step-matched baseline by 1.8 and 1.5 percentage points, respectively, achieving a 182-fold speedup in latency over reasoning rollout on Gemma-4-31B. Reasoning remains stronger on the new-schema test, motivating a confidence router for uncertain decisions. On development sets, routing recovers a median 92% of reasoning's accuracy advantage while generating 27-58% of its tokens. Code is available at https://github.com/JianhuiWei7/LDM.
Ziyi Chen, Yan Zhang, Jianhui Wei +2
University of Illinois Urbana-Champaign · National University of Singapore · Zhejiang University +2