When Do LLMs Model the Right Target? Problem Formulation in High-Stakes Risk Prediction
Organizations: Department of Statistics, University of California, Berkeley, Berkeley, California, USA
Abstract
Generative AI is increasingly used to assist with data science tasks, yet fundamental questions remain about its ability to correctly formulate analytical problems. We investigate how well large language models (LLMs) can formulate prediction problems when developing models for high-stakes decision making. We focus on one particular challenge to validity: decision dependency, which arises when historical decisions affect observed outcomes. We evaluate this aspect of problem formulation in LLM-generated code and explanations across child-welfare, healthcare, and employment settings, using 11 models from OpenAI and Anthropic and seven prompt conditions. We assess whether the LLM-generated code implements a procedure appropriate for estimating the intended outcome under the study's assumptions and whether the explanation acknowledges decision dependency. We find that LLMs do not reliably address decision dependency without guidance in the prompt. When prompts specify the target, the assumption needed to estimate it, and the estimation procedure, target-aligned implementation reaches 99-100%. Among frontier models, a simple prompt description of how decisions affect outcomes appears sufficient to elicit target-aligned implementation. Some responses also acknowledge decision dependency without addressing it in their code. Our findings suggest that analysts should not rely on LLMs to correctly formulate prediction problems without explicit methodological guidance and careful review. We propose validity-centered guidelines for LLM-assisted predictive modeling, covering prompt formulation, validation of the target and estimation procedure, and identification of key assumptions. Establishing validity is a necessary early step in evaluating whether to deploy a predictive model in a high-stakes setting, but deployment requires weighing broader ethical, institutional, and practical considerations.
Figures & tables
| Measure | Question assessed | Response scale |
|---|---|---|
| Target-aligned implementation | Does the code use an accepted procedure for estimating the outcome without intervention? | Binary: 0 or 1 |
| Verbal acknowledgment | Does the explanation connect intervention effects to the problem with training on recorded outcomes? | Binary: 0 or 1 |
| Faithful description | Does the explanation accurately describe what the code implements? | Binary: 0 or 1 |
| Assumption reporting | Does the response name an assumption linking its procedure to an estimand? | None; named but vague; named and specific |
| Claim about the target | Does the response assert, qualify, or disclaim that its output represents the intended target? | Assertive; hedged; disclaimed |
| Domain | Decision-making goal | Intended prediction target | Data source |
|---|---|---|---|
| Child welfare | Prioritize children at high risk if screened out | Re-referral risk within six months under screen-out | Synthetic records constructed for this study, informed by Coston et al. (2020) |
| Healthcare | Prioritize patients with greater health need, using cost as a proxy | Expected healthcare cost without care-management enrollment | Synthetic patient-year data from Obermeyer et al. (2019) , modified for this study |
| Employment | Prioritize people with the lowest expected earnings without training | Expected earnings without job-training enrollment | National Supported Work and PSID comparison data ( LaLonde, 1986 ; Dehejia and Wahba, 1999 ) , modified for this study |
| Verbal acknowledgment rate | Target-align. implem. rate | ||||||
|---|---|---|---|---|---|---|---|
| Model set | Condition | Child welfare | JOBS | Health | Child welfare | JOBS | Health |
| Initial | no_hint | 0.17 | 0.16 | 0.04 | 0.00 | 0.20 | 0.00 |
| Initial | context | 0.54 | 0.52 | 0.58 | 0.51 | 0.51 | 0.57 |
| Initial | context_target | 0.58 | 0.71 | 0.67 | 0.56 | 0.79 | 0.67 |
| Initial | context_target_detail | 0.67 | 0.72 | 0.69 | 0.68 | 0.78 | 0.72 |
| Initial | estimand_assumption | 0.94 | 0.97 | 0.95 | 0.93 | 0.99 | 0.96 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Model set | API identifier |
|---|---|
| Initial (August 14–18, 2026) | claude-haiku-4-5-20251001 |
| claude-sonnet-4-6 | |
| claude-opus-4-8 | |
| gpt-5-nano | |
| gpt-5 | |
| gpt-5.6-sol |
| Model provider / Domain | Measure | Agreement | |
|---|---|---|---|
| Anthropic / child welfare | Verbal acknowledgment | 82/90 | 0.813 |
| Target-aligned implementation | 90/90 | 1.000 | |
| Faithful description | 86/90 | 0.840 | |
| Claim about the target | 86/90 | 0.898 | |
| Assumption reporting | 35/35 | 1.000 | |
| Anthropic / healthcare | Verbal acknowledgment | 51/54 | 0.877 |
| Validation sample | Measure | Agreement | |
|---|---|---|---|
| Core ( ) | Verbal acknowledgment | 95/105 | 0.530 |
| Target-aligned implementation | 104/105 | 0.947 | |
| Faithful description | 102/105 | 0.000 | |
| Claim about the target | 92/105 | 0.756 | |
| Assumption reporting | 98/105 | 0.868 | |
| Challenge ( ) | Verbal acknowledgment | 39/45 | 0.614 |
| Dataset | Preparation |
|---|---|
| Healthcare | Rename all columns and recode race as a neutral demographic category; describe a fictional Australian health network. Retain an 85% subsample and multiply positive costs by independent noise, rounding to whole dollars. Remove the age-band 45–54 and renal-failure indicator columns. |
| JOBS | Start with 297 treated, 425 experimental-control, and 2,490 PSID comparison records. Hold out 200 experimental controls before transformation. Rename columns, remove re74 , and add a uniform 1–45 distance variable unrelated to other features. Multiply positive earnings by noise; replace age with four age-band indicators; flip each binary demographic indicator with probability 0.03. |
| domain | model | direct naming | restatement |
|---|---|---|---|
| healthcare | opus | 2/30 (7%) | 6/30 (20%) |
| healthcare | gpt5sol | 0/30 | 4/30 (13%) |
| healthcare | gpt5, gpt5nano | 0/60 | 0/60 |
| JOBS | opus | 3/80 (3.8%) | 1/80 (1.3%) |
| JOBS | gpt5, gpt5nano, gpt5sol | 0/90 | 0/90 |
| condition | gpt5 | gpt5nano | gpt5sol | haiku | opus | sonnet |
|---|---|---|---|---|---|---|
| Verbal acknowledgment | ||||||
| no_hint | 1/29 | 0 | 11/20 | 0 | 17 | 0 |
| context | 18 | 0 | 30 | 0 | 30 | 20 |
| context_target | 23 | 0 | 30 | 0 | 30 | 22 |
| context_target_detail | 27 | 1 | 30 | 3 | 30 | 29 |
| estimand_assumption | 22 | 30 | 29/29 | 27 | 30 | 30 |
| condition | gpt5 | gpt5nano | gpt5sol | haiku | opus | sonnet |
| Verbal acknowledgment | ||||||
| no_hint | 0 | 0 | 8/19 | 0 | 19 | 0 |
| context | 21 | 0 | 28/28 | 0 | 30 | 27 |
| context_target | 24 | 0 | 30 | 0 | 30 | 23 |
| context_target_detail | 26 | 1 | 30 | 4 | 30 | 29 |
| estimand_assumption | 23 | 30 | 27/27 | 27 | 30 | 30 |
| condition | gpt5 | gpt5nano | gpt5sol | haiku | opus | sonnet |
|---|---|---|---|---|---|---|
| Verbal acknowledgment | ||||||
| no_hint | 0 | 0 | 26 | 0 | 2 | 0 |
| context | 21 | 1 | 29 | 0 | 30 | 12 |
| context_target | 25 | 7 | 30 | 6 | 30 | 30 |
| context_target_detail | 23 | 13 | 30 | 3 | 30 | 30 |
| estimand_assumption | 25 | 30 | 30 | 29 | 30 | 30 |
| condition | gpt5 | gpt5nano | gpt5sol | haiku | opus | sonnet |
|---|---|---|---|---|---|---|
| Verbal acknowledgment | ||||||
| no_hint | 0 | 0 | 5 | 0 | 2 | 0 |
| context | 19 | 0 | 30 | 2 | 30 | 24 |
| context_target | 26 | 0 | 30 | 5 | 30 | 30 |
| context_target_detail | 22 | 3 | 30 | 9 | 30 | 30 |
| estimand_assumption | 24 | 30 | 30 | 28 | 29 | 30 |
| condition | gpt6astra | gpt6sol | fable5.1 | opus5.5 | sonnet5.5 |
|---|---|---|---|---|---|
| Verbal acknowledgment | |||||
| no_hint | 28 | 14 | 19 | 30 | 29 |
| context | 30 | 30 | 30 | 30 | 30 |
| context_target | 30 | 30 | 30 | 30 | 30 |
| context_target_detail | 30 | 30 | 30 | 30 | 30 |
| estimand_assumption | 30 | 30 | 30 | 30 | 30 |
| condition | gpt6astra | gpt6sol | fable5.1 | opus5.5 | sonnet5.5 |
|---|---|---|---|---|---|
| Verbal acknowledgment | |||||
| no_hint | 25 | 14 | 23 | 30 | 29 |
| context | 30 | 30 | 30 | 30 | 30 |
| context_target | 30 | 30 | 30 | 30 | 30 |
| context_target_detail | 30 | 30 | 30 | 30 | 30 |
| estimand_assumption | 30 | 30 | 30 | 30 | 30 |
| condition | gpt6astra | gpt6sol | fable5.1 | opus5.5 | sonnet5.5 |
|---|---|---|---|---|---|
| Verbal acknowledgment | |||||
| no_hint | 23 | 26 | 26 | 30 | 17 |
| context | 30 | 26 | 30 | 30 | 30 |
| context_target | 30 | 26 | 30 | 30 | 30 |
| context_target_detail | 30 | 30 | 30 | 30 | 30 |
| estimand_assumption | 30 | 30 | 30 | 30 | 30 |
| condition | gpt6astra | gpt6sol | fable5.1 | opus5.5 | sonnet5.5 |
|---|---|---|---|---|---|
| Verbal acknowledgment | |||||
| no_hint | 11 | 3 | 6 | 22 | 21 |
| context | 30 | 26 | 30 | 30 | 30 |
| context_target | 30 | 29 | 30 | 30 | 30 |
| context_target_detail | 30 | 30 | 30 | 30 | 30 |
| estimand_assumption | 30 | 30 | 30 | 30 | 30 |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 866 papers