cs.AIOct 1, 2026
SaveAnswering clinicians' questions over trial evidence tables with verifiable, feedback-driven language models
Organizations: Arizona State University, Tempe, Arizona, USA · Mayo Clinic, Phoenix, Arizona, USA
Abstract
Systematic reviews condense clinical trials into evidence tables, yet clinicians can interrogate these tables only through database queries, and many questions concern attributes that the table does not record, such as a drug's target class or a harmonised endpoint. Here we introduce FD-SCoPE, a language-model framework that answers both kinds of question, exposes the query, the selected trials and the derivation rule behind every answer, and learns from expert corrections. On an oncology evidence table of 159 immune checkpoint inhibitor trial records, FD-SCoPE completed all 140 clinician-style tasks (alternatives, 90.7-97.9%). For questions needing derived attributes it retrieved 99.3% of relevant trial records at a positive predictive value of 89.8% and outperformed four alternative approaches (derived-value F1 77.7% versus 64.8-73.4%). Corrections on 299 questions, simulated from reference answers, raised F1 on 1,201 unseen questions from 77.9% to 84.9%. Language models coupled with executable queries, verified programs and expert feedback can give clinicians auditable access to trial evidence.
Figures & tables
Figure 1: FD-SCoPE architecture. Following a safety check, clinical questions are routed through direct SQL retrieval or a hybrid pathway that combines trial selection with derivation of variables not explicitly stored in the database. Expert feedback updates reusable SQL examples and derivation programs to improve subsequent answers.
| Question set | What it tests | Example | |
|---|---|---|---|
| Recorded fields (benchmark) | 1,500 | Retrieving trials and fields stored in the table | “Show trials of CTLA-4 for non-small cell lung cancer that began in 2018 or later.” |
| Derived attributes (benchmark) | 1,500 | Selecting trials, then deriving a value that is not stored (31 families) | “In small cell lung trials, classify the ICI by target class from its generic name as PD-1, PD-L1 or CTLA-4.” |
| Clinician-style tasks | 140 | Look-ups, counts, evidence landscapes and trial listings | “Which bladder cancer trials published in 2018 or later enrolled more than 500 patients?” |
Table 1: Question sets used in the evaluation.
Figure 2: Clinician-style tasks and benchmark questions about recorded fields. a , Share of 140 clinician-style tasks answered correctly by FD-SCoPE and seven alternatives using the same language model; few-shot, zero-shot and chain-of-thought are the prompting baselines. b , The same result by task type, with the number of tasks in brackets; each grey dot is one alternative, and ‘Control’ denotes listings of trials with a comparator other than chemotherapy. c , Exact-match accuracy on the 1,500 benchmark questions. d , Execution F1 on the same questions. e , Exact-match accuracy with two language models. Bars are 95% confidence intervals (Wilson in a , bootstrap in c – e ). Values are in Supplementary Tables 4 and 4 .
Figure 3: Questions that require an attribute not stored in the table (n = 1,500). a , Derived-value F1 for FD-SCoPE and four alternatives using the same language model. b , Share of questions for which every row of the answer was correct. c , Share of questions for which exactly the reference set of trial records was returned. d , Sensitivity and positive predictive value for retrieving the relevant trial records. e , Flow of the 1,500 questions through FD-SCoPE: the value was computed by a verified program reused from an earlier question, derived row by row by the model, or computed by a newly induced and verified program, or no trial matched the question; band width is proportional to the number of questions, and blue bands lead to answers that were correct in every row. Bars in a – d are bootstrap 95% confidence intervals. Values are in Supplementary Tables 4 and 4 .
Figure 4: Effect of corrections on held-out questions. Corrections on 299 questions were simulated from reference answers; results are for the 1,201 questions not used for feedback. a , Question-level measures before and after feedback. b , Accuracy of individual derived values for all trials, and for trials excluded from learning when corrections were drawn from half of the trials. Numbers beside the points are the change in percentage points. Values are in Supplementary Table 4 .
Figure 5: Derived-value F1 by derivation family and approach. Each cell is the derived-value F1 (%) on the same 1,201 held-out questions, for the four alternative approaches and for FD-SCoPE before and after feedback; darker cells are higher and the number of questions in each family is in brackets. Families are ordered by the change that feedback produced. Questions in the endpoint-role family do not state which endpoint field is meant and were not answered correctly by FD-SCoPE in either condition. Values are in Supplementary Table 4 .
Figure 6: Component analysis and robustness to phrasing. a , b , Derived-attribute questions (304-question subset): derived-value F1 and share of questions with every row correct for the full system and with one component removed or replaced. c , Recorded-field questions (300 held-out benchmark questions): exact-match accuracy with one component removed. d , Exact-match accuracy as a function of the share of the approved-example bank that is available; the band is the bootstrap 95% confidence interval. e , f , Exact-match accuracy and execution F1 on original and reworded questions with the reference answer unchanged; numbers in brackets are question counts and numbers beside the points are the change in percentage points. Bars in a – c are bootstrap 95% confidence intervals. Values are in Supplementary Tables 4 , 4 , 4 and 4 .
| Exact match | Execution F1 | Query | Executable | Tokens per | |
| System | (95% CI) | (95% CI) | similarity | queries | question |
| FD-SCoPE | 58.9 (56.3–61.5) | 67.4 (65.1–69.6) | 89.8 | 100.0 | 5,864 |
| DAIL-SQL style | 55.2 (52.4–57.7) | 67.1 (64.8–69.3) | 89.9 | 100.0 | 3,004 |
| Zero-shot prompting | 49.9 (47.2–52.4) | 57.8 (55.4–60.1) | 55.3 | 100.0 | 2,099 |
| Few-shot prompting | 49.3 (46.7–51.9) | 63.3 (61.0–65.5) | 84.3 | 100.0 | 2,491 |
| DIN-SQL style | 48.1 (45.6–50.5) | 62.6 (60.3–64.8) | 86.4 | 100.0 | 6,508 |
Table S1: Accuracy on the 1,500 benchmark questions about recorded fields
| Sensitivity | Positive predictive value | Exact trial set | |
| System | (95% CI) | (95% CI) | (% of questions) |
| FD-SCoPE | 67.4 (65.0–69.5) | 73.7 (71.5–75.8) | 63.3 |
| DAIL-SQL style | 68.6 (66.4–70.7) | 74.7 (72.6–76.8) | 64.6 |
| Zero-shot prompting | 64.0 (61.6–66.2) | 69.5 (67.2–71.6) | 57.3 |
| Few-shot prompting | 65.2 (62.8–67.4) | 71.5 (69.3–73.7) | 59.9 |
| DIN-SQL style | 65.5 (63.3–67.7) | 71.9 (69.5–73.9) | 60.9 |
Table S2: Trial-level retrieval on the benchmark questions about recorded fields
| Derived-value F1 | F1, | Every row correct | Table | Exact | Tokens per | |
|---|---|---|---|---|---|---|
| System | (95% CI) | exact | (95% CI) | F1 | table | question |
| FD-SCoPE | 77.7 (75.7–79.4) | 62.1 | 67.0 (64.6–69.2) | 87.6 | 52.3 | 8,592 |
| BlendSQL style | 73.4 (71.4–75.3) | 56.6 | 59.1 (56.6–61.5) | 83.9 | 44.1 | 3,827 |
| Code agent | 67.9 (65.8–69.8) | 51.8 | 50.6 (48.1–53.0) | 79.7 | 38.5 | 6,329 |
| Row-retrieval QA | 66.1 (64.1–67.8) | 50.9 | 35.1 (32.6–37.4) | 74.0 | 25.4 | 7,515 |
| Whole-table prompting | 64.8 (62.8–66.7) | 49.5 | 35.5 (33.1–37.7) | 73.8 | 25.7 | 28,503 |
Table S3: Accuracy on the 1,500 questions that require a derived attribute
| Row-selection | Sensitivity | Positive predictive | Exact trial set | |
|---|---|---|---|---|
| System | F1 | (95% CI) | value (95% CI) | (95% CI) |
| FD-SCoPE | 88.1 | 99.3 (98.9–99.7) | 89.8 (88.4–91.1) | 84.3 (82.3–86.2) |
| BlendSQL style | 84.8 | 97.0 (96.1–97.7) | 87.0 (85.6–88.6) | 76.6 (74.4–78.7) |
| Code agent | 81.7 | 94.4 (93.1–95.5) | 82.7 (81.0–84.3) | 70.9 (68.5–73.1) |
| Row-retrieval QA | 75.7 | 87.6 (86.3–88.8) | 75.8 (74.4–77.4) | 44.0 (41.5–46.4) |
| Whole-table prompting | 77.6 | 90.5 (89.2–91.8) | 74.4 (72.6–76.1) | 46.1 (43.5–48.4) |
Table S4: Trial-level retrieval on the questions that require a derived attribute
| Before | After feedback on | After feedback from | |
| Measure | feedback | 299 questions | half of the trials |
| Derived-value F1 (95% CI) | 77.9 (75.8–80.0) | 84.9 (83.2–86.6) | 83.4 (81.4–85.1) |
| Derived-value F1, exact | 62.6 | 77.6 | 78.6 |
| Every row correct (95% CI) | 67.1 (64.4–69.7) | 75.8 (73.3–78.2) | 73.8 (71.3–76.2) |
| Evidence-table F1 | 87.7 | 89.5 | 89.3 |
| Exact table match | 52.6 | 68.5 | 68.5 |
Table S5: Effect of feedback on the 1,201 held-out questions
| Alternative approaches | FD-SCoPE | |||||||
| Derivation family | Whole | Row | Code | Blend- | Before | After | ||
| table | retrieval | agent | SQL | feedback | feedback | Change | ||
| Dosing schedule | 31 | 0.0 | 0.0 | 23.6 | 0.4 | 21.0 | 93.1 | +72.1 |
| Named control regimen | 31 | 6.3 | 5.2 | 6.9 | 5.7 | 18.6 | 71.7 | +53.1 |
| Chemotherapy backbone | 31 | 10.8 | 7.5 | 9.2 | 2.7 | 13.9 | 66.1 | +52.2 |
| Composite endpoint components | 32 | 48.3 | 56.3 | 51.3 | 53.2 | 52.0 | 91.9 | +39.9 |
Table S6: Derived-value F1 (%) by derivation family on the held-out questions
| Exact match | Execution F1 | Tokens per | |
| Configuration | (95% CI) | (95% CI) | question |
| Full system | 62.0 (56.7–67.3) | 71.5 (66.9–75.9) | 5,859 |
| Without catalogue grounding | 64.0 (58.7–69.3) | 73.0 (68.4–77.2) | 5,703 |
| Without model decomposition | 60.0 (54.7–65.3) | 71.8 (67.2–76.0) | 3,271 |
| Without exemplar retrieval | 56.0 (50.3–61.3) | 69.9 (65.3–74.3) | 5,139 |
| Without context-field completion | 55.0 (49.7–60.3) | 69.1 (64.7–73.6) | 5,785 |
Table S7: Component analysis for questions about recorded fields (300 held-out benchmark questions)
| Derived-value F1 | Every row correct | Exact | Trial | Exact | |
|---|---|---|---|---|---|
| Configuration | (95% CI) | (95% CI) | table | PPV | trial set |
| Full system | 78.0 (73.7–81.8) | 66.8 (61.2–71.7) | 52.3 | 92.4 | 87.5 |
| Without program induction | 77.9 (73.7–81.8) | 66.8 (61.5–71.7) | 53.6 | 92.4 | 87.5 |
| Without output normalisation | 76.7 (72.4–80.6) | 62.2 (56.6–67.4) | 45.4 | 91.8 | 82.2 |
| Without planner | 72.0 (67.3–76.3) | 61.2 (55.9–66.4) | 47.7 | 92.4 | 87.5 |
| Model selects the rows | 67.6 (63.2–71.6) | 37.8 (32.6–42.8) | 28.9 | 77.6 | 53.0 |
Table S8: Component analysis for questions that require a derived attribute (304-question subset)
| All (140), | Look-up | Count | Landscape | Enrolment | Control | PD-L1 | ||
|---|---|---|---|---|---|---|---|---|
| System | (%) | 95% CI | (40) | (40) | (24) | (23) | (9) | (4) |
| FD-SCoPE | 140 (100.0) | 97.3–100.0 | 40 | 40 | 24 | 23 | 9 | 4 |
| DAIL-SQL style | 137 (97.9) | 93.9–99.3 | 37 | 40 | 24 | 23 | 9 | 4 |
| MAC-SQL style | 136 (97.1) | 92.9–98.9 | 39 | 38 | 24 | 23 | 8 | 4 |
| Few-shot | 136 (97.1) | 92.9–98.9 | 37 | 39 | 24 | 23 | 9 | 4 |
| Zero-shot | 134 (95.7) | 91.0–98.0 | 36 | 38 | 24 | 23 | 9 | 4 |
Table S9: Clinician-style tasks answered correctly
| Rewording | Questions | Original | Reworded | Change |
|---|---|---|---|---|
| Lower case | 300 | 62.0 | 61.7 | 0.3 |
| Brand name for generic drug name | 86 | 61.6 | 61.6 | 0.0 |
| Disease abbreviation | 76 | 59.2 | 60.5 | +1.3 |
| Drug-class shorthand | 81 | 67.9 | 67.9 | 0.0 |
| Typing error in a clinical term | 187 | 61.0 | 55.1 | 5.9 |
Table S10: Exact-match accuracy (%) of FD-SCoPE on reworded questions
| Difference, points | Questions | Questions | ||
| Comparison | (95% CI) | better | worse | |
| Recorded-field questions: FD-SCoPE vs an alternative | ||||
| DAIL-SQL style, exact match | 1,500 | +3.7 (+1.9 to +5.5) | 123 | 67 |
| DAIL-SQL style, execution F1 | 1,500 | +0.3 ( 0.5 to +1.1) | 162 | 83 |
| Few-shot prompting, exact match | 1,500 | +9.6 (+7.7 to +11.5) | 193 | 49 |
| Few-shot prompting, exact match (Gemma-3-27B) | 1,500 | +5.1 (+2.7 to +7.6) | 227 | 150 |
Table S11: Paired differences between two systems on the same questions
| Scope check on | Scope check off | Unsafe | ||||
|---|---|---|---|---|---|---|
| Category | Prompts | Declined | Answered | Declined | Answered | statements |
| Request to modify the table | 12 | 12 | 0 | 0 | 12 | 0 |
| Prompt injection | 10 | 10 | 0 | 0 | 9 | 0 |
| Request for clinical advice | 8 | 8 | 0 | 0 | 8 | 0 |
| Request for patient-level data | 6 | 6 | 0 | 0 | 6 | 0 |
| Valid evidence question (control) | 10 | 0 | 10 | 0 | 10 | 0 |
Table S12: Safety set of 46 prompts
| Setting | Qwen3.8-27B | Gemma-3-27B-it |
|---|---|---|
| Numerical precision | bfloat16 | bfloat16 |
| Served context limit | 40,000 tokens | 24,000 tokens |
| Temperature | 0.0 | 0.0 |
| Top- | 1.0 | 1.0 |
| Thinking mode | Disabled | Not applicable |
| JSON-constrained output | Enabled | Enabled |
Table S13: Language-model serving and inference settings
| Approach | Settings |
|---|---|
| Zero-shot prompting | One call; schema and up to 24 observed values per field; 400 new tokens. |
| Few-shot prompting | As zero-shot, with four fixed expert examples; 400 new tokens. |
| Chain-of-thought prompting | Reasoning scaffold with worked examples; 2,500 new tokens. |
| DIN-SQL style | Four calls: schema linking (400 tokens), difficulty classification (10), generation (500) and self-correction (1,200); the corrected query is kept only if it executes. |
| DAIL-SQL style | Eight examples chosen by similarity of the question with literals masked, most similar last; schema given as a table definition; 500 tokens. |
| C3 style | Zero-shot prompt with clear layout and tips; ten samples at temperature 0.7; the result shared by most executable samples is returned. |
Table S14: Settings of the alternative approaches
More in this topic
Clinical QA
Clinical question answering uses AI to answer medical and patient-specific questions from clinical knowledge, records, guidelines, and evidence.
See all 63 papers