cs.AIOct 27, 2026
SaveOpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents
Organizations: Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan, USA · Department of Radiology, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
Abstract
Background and purpose: Radiotherapy planning agents based on large language models (LLMs) decide when to submit a plan or request review but are evaluated mainly by dose. This study developed OpenGray, a benchmark of planning tasks with known correct decisions. Materials and methods: OpenGray adds tasks with known answers to the OpenKBP-Opt head-and-neck dataset, including two contradictions with dose goals proven incompatible and a feasible control with a stored plan meeting every hard goal. Four systems, from a rule-based planning script to a full LLM agent, were evaluated on 30 patients, the LLM-based ones with a commercial (Terra) and a local (Ministral 3 8B) LLM. Decisions, achieved dose, problem recognition, and technical failures were recorded and also read from logs of 39 stereotactic radiosurgery runs of SAGE, a clinical LLM agent. Results: With Terra, the script preceded by an LLM review and the full agent had decision accuracies of 95.4% and 92.1%, respectively, yet escalated only 50% (95% confidence interval, 25-75%) and 25% (10-40%) of overlap contradictions, which required arithmetic on the anatomy. All four systems submitted every feasible control, but 24% of these plans missed a hard goal that the stored plan had met. In four SAGE runs, one goal failed at every attempt; the agent identified it in every reply but had no action for stopping. Conclusions: Favorable aggregate scores concealed specific decision failures. Planning agents should be evaluated on their decisions as well as their dose and given an explicit action for stopping and requesting review.
Figures & tables
Fig. 1: OpenGray and its two applications. (a) Data: of the 100 OpenKBP-Opt cases, 10 were used to develop the systems, 30 to evaluate them, and 60 only to calibrate a plan-quality score; the 39 SAGE runs were read from their planning logs. (b) Planning interface: each planning system acts only through the harness, which carries out the requested planning functions and returns the case summary and dose. (c) The four outcomes and how each was obtained on the benchmark and from the planning logs. The correct decision is known for each benchmark task but not for a logged clinical request.
| Task | Construction | Verified answer | Expected decision | Patients |
|---|---|---|---|---|
| Routine planning a | Default goals | Not applicable | Submit | 30 |
| Conflicting note a | A physician note contradicts a formal goal | The formal goal takes precedence | Flag the note, then submit or escalate | 30 |
| Missing structure | A hard goal names the left optic nerve, which is absent | Structure list of the case | Escalate | 30 |
| Coverage/cap contradiction | Target D99 goal of 78.0 or 78.5 Gy, above the 77 Gy External maximum | Comparison of the two stated goals | Escalate | 29 |
| Overlap contradiction | High target dose required in voxels shared with the mandible | Proof that | Escalate | 10 |
| Feasible control | Spinal-cord D0.1cc limit tightened to 33.6–39.4 Gy in 24 of 25 tasks | Stored plan meeting every hard goal | Submit | 25 |
Table 1: The six tasks of the benchmark and their expected decisions.
Fig. 2: Decision accuracy on the 30 evaluation patients. (a) Decision accuracy over the four scored tasks, averaged within each patient with patients weighted equally. (b) Decision accuracy on each scored task, with the number of eligible patients; error bars show 95% patient-bootstrap confidence intervals, and 0 marks a system that never made the expected decision. The LLM-based systems were run with Terra (solid bars) and with Ministral 3 8B (hatched bars), and technical failures, including replies that did not follow the required format, received no credit.
Fig. 3: Feasible controls and overlap contradictions. (a) Outcome of every feasible-control attempt: submitted with every hard dose goal met, submitted with a hard goal unmet, or not submitted, with the number that met every hard goal. The LLM-based systems made two attempts per control, systems that passed a task to the controller submitted its plans, and the controller with 2,000 iterations gave the same result as with 5,000 (Supplementary Table S17 ). (b) Final decision in every attempt on the 10 overlap contradictions, with the number escalated as a contradiction. (c) Percentage of attempts escalated as a contradiction on each overlap task, with the tasks ordered by how far the shared voxels exceeded the combined allowance (count margin); 0 marks no such escalation.
Fig. 4: The SAGE planning logs, read with the outcomes of OpenGray: measured value of the failing goal at each attempt in (a) a run that met every goal and (b–d) three of the four runs that reached the 10-attempt limit. Dashed lines mark the strict goal thresholds. In (b), crosses mark the goal values of attempt 8, which were reported again after the two attempts in which no dose was calculated. In (d), the shaded region marks the attempts at which the instructions of the run required the flag “not feasible with priorities,” which was never written.
Figure S1: Development T4 grade by evaluable scenario: mean grade of each system in each scenario. Scripted controls cover 10 patients per scenario (the priority heuristic, 50 episodes), and the rotated language-system scenarios contain six or seven episodes. An appropriate escalation earns 0.9, so the grade combines action credit and dose outcomes, and 0 marks a grade of zero. The scripted systems ignore the note, and a submission is graded by its PlanScore whether or not the note is flagged, so their conflicting-note grades equal their grades on the unaltered control. The tightened-cord scenario, which had no proof of infeasibility, is excluded.
Figure S2: Development T5 escalation frequency and grade. (a) Percentage of episodes escalated for any reason in each task, with the counts beside the bars; coverage/cap contradictions involve 10 patients and overlap contradictions 5, and no system escalated on a feasible control. (b) Patient-mean track grade with 95% patient-bootstrap confidence intervals.
Figure S3: Sensitivity to fluence-complexity objectives. (a, b) Patient-mean PlanScore without (dark) and with (light) the added complexity objectives in T1 and T2; hatched bars are the historical Sonnet and GLM T2 baselines with the rules withheld. (c) Percentage of the 240 final normalized fluences that met the intended limits, which normalization could violate.
Figure S4: Planning performance on the 10 development patients: patient-mean PlanScore in initial planning (T1; one optimization, rules disclosed) and iterative planning (T2; three optimizations), with 95% patient-bootstrap confidence intervals. Hatched bars are the historical Sonnet and GLM T2 runs with the rules withheld, which are not directly comparable with the others. The score-visible search had access to PlanScore during planning.
| Set | Task | Responses | Recognition yes / applicable | Action yes / responses |
|---|---|---|---|---|
| Development | Conflicting instructions | 12 | 6/12 | 6/12 |
| Development | Missing structure | 12 | 12/12 | 12/12 |
| Development | Coverage/cap contradiction | 58 | 36/58 | 40/58 |
| Development | Overlap contradiction | 46 | 16/46 | 14/46 |
| Development | Feasible control | 10 | Not applicable | 10/10 |
| Reserved cohort | Routine planning | 10 | Not applicable | 10/10 |
Table S1: Reviewed responses by task and cohort. Counts use the cohort-specific rubrics, and these stratified samples do not provide pooled population estimates.
| Case | Set | N | m | Cap E | Coverage c | Threshold | Margin |
|---|---|---|---|---|---|---|---|
| pt_258 | D | 4635 | 48 | 77.00 | 78.00 | 77.9063 | 0.0937 |
| pt_269 | D | 14079 | 142 | 77.00 | 78.00 | 77.8850 | 0.1150 |
| pt_289 | D | 4997 | 51 | 77.00 | 78.00 | 77.8945 | 0.1055 |
| pt_302 | D | 6440 | 66 | 77.00 | 78.00 | 77.8978 | 0.1022 |
| pt_306 | D | 4430 | 46 | 77.00 | 78.00 | 77.9085 | 0.0915 |
| pt_307 | D | 4726 | 49 | 77.00 | 78.00 | 77.9072 | 0.0928 |
Table S2: Coverage/cap certificates. D denotes development and R denotes reserved patients. All doses are in Gy. Margins are calculated using unrounded thresholds.
| Case | Set | Target | O | m | n | Voxel margin | Dose margin |
|---|---|---|---|---|---|---|---|
| pt_258 | D | 7000 | 128 | 48 | 3 | 77 | 6.70 |
| pt_306 | D | 7000 | 104 | 46 | 3 | 55 | 6.70 |
| pt_307 | D | 7000 | 111 | 49 | 3 | 59 | 6.70 |
| pt_320 | D | 7000 | 127 | 80 | 3 | 44 | 6.70 |
| pt_339 | D | 7000 | 247 | 64 | 3 | 180 | 6.70 |
| pt_248 | R | 5600 | 631 | 286 | 3 | 342 | 5.40 |
Table S3: Overlap certificates. All organ constraints concern the mandible. denotes overlap, the cold-target allowance, and the hot-organ allowance. The dose margin is Gy.
| Case | Hard goals | Minimum margin (Gy) |
|---|---|---|
| pt_242 | 5 | 2.5102 |
| pt_245 | 6 | 0.5938 |
| pt_248 | 6 | 2.3075 |
| pt_249 | 5 | 2.5491 |
| pt_251 | 7 | 3.5477 |
| pt_256 | 6 | 2.9016 |
Table S4: Stored feasible witnesses. The minimum margin is the smallest signed distance to a tolerated hard-goal boundary. A positive value indicates that every hard goal is met.
| Endpoint (Gy) | Controller | Arithmetic | Interpreter | Full agent |
|---|---|---|---|---|
| Mandible D0.1cc | 69.33 (19) | 69.83 (19) | 69.43 (20) | 69.30 (20) |
| Brainstem D0.1cc | 43.90 (25) | 43.89 (25) | 44.13 (26) | 40.01 (26) |
| Esophagus Dmean | 39.24 (13) | 38.56 (14) | 38.56 (14) | 32.40 (14) |
| External Dmax | 75.89 (29) | 75.90 (29) | 75.92 (30) | 76.76 (30) |
| Larynx Dmean | 42.88 (16) | 43.00 (17) | 43.00 (17) | 40.97 (17) |
| PTV56 D99 | 54.92 (26) | 54.97 (26) | 54.92 (27) | 55.00 (27) |
Table S5: Reserved routine planning. Patient-mean dose endpoints among submitted plans, reported as mean (patients).
| Endpoint (Gy) | Controller | Arithmetic | Interpreter | Full agent |
|---|---|---|---|---|
| Mandible D0.1cc | 68.60 (15) | 68.60 (15) | 68.60 (15) | 68.38 (15) |
| Brainstem D0.1cc | 41.79 (23) | 41.79 (23) | 41.79 (23) | 38.03 (23) |
| Esophagus Dmean | 36.71 (10) | 36.71 (10) | 36.71 (10) | 31.61 (10) |
| External Dmax | 75.93 (25) | 75.93 (25) | 75.93 (25) | 76.74 (25) |
| Larynx Dmean | 42.34 (13) | 42.34 (13) | 42.34 (13) | 39.45 (13) |
| PTV56 D99 | 54.78 (22) | 54.78 (22) | 54.78 (22) | 54.99 (22) |
Table S6: Reserved feasible controls. Patient-mean dose endpoints among submitted plans, reported as mean (patients).
| Population | System | Eligible | Submitted | All hard goals | Rejected |
|---|---|---|---|---|---|
| Reserved routine | Controller | 30 | 29 | 21 | 0 |
| Reserved routine | Interpreter + controller | 60 | 60 | 42 | 0 |
| Reserved routine | Full agent | 60 | 59 | 47 | 0 |
| Reserved routine | Arithmetic + controller | 30 | 29 | 20 | 0 |
| Reserved feasible control | Controller | 25 | 25 | 19 | 0 |
| Reserved feasible control | Interpreter + controller | 50 | 50 | 38 | 0 |
Table S7: Attempt-level dose accounting. All-hard-goal and rejection counts concern submitted plans, and the eligible denominators also include attempts without a submission.
| Case | System | Scenario | Failure | Primary |
|---|---|---|---|---|
| pt_304 | Controller | Routine | Interrupted attempt | No |
| pt_248 | Arithmetic + controller | Routine | Interrupted attempt | No |
| pt_245 | Arithmetic + controller | Conflicting note | Interrupted attempt | No |
| pt_251 | Full agent | Routine | Connection failure | No |
| pt_267 | Full agent | Conflicting note | Connection failure | No |
| pt_275 | Full agent | Coverage/cap | Request timeout | Yes |
Table S8: The six technical failures and their membership in the primary endpoint.
| Attempt | PTV V21Gy (%) | PTV Dmax (Gy) | Brainstem Dmax (Gy) | Optic-chiasm Dmax (Gy) | Brain-GTV V12Gy (cm 3 ) |
|---|---|---|---|---|---|
| 1 | 21.75 | 22.18 | 0.81 | 0.049 | 3.25 |
| 2 | 60.71 | 23.03 | 1.82 | 0.058 | 3.43 |
| 3 | 87.97 | 23.61 | 1.59 | 0.061 | 3.32 |
| 4 | 96.62 | 25.66 | 0.91 | 0.057 | 3.79 |
Table S9: Dose metrics of Patient 4 at its four new dose calculations. Values are rounded for display; the goals were evaluated with strict operators on the unrounded values. The retry after an unreadable reply in attempt 3 preceded the dose calculation of that attempt and added no dose state.
| System | Missing structure (30) | Coverage/cap (29) | Overlap (10) | Feasible control (25) | Primary | Equal family | Equal task |
|---|---|---|---|---|---|---|---|
| Controller | 100 | 0 | 0 | 100 | 60.0 | 50.0 | 58.5 |
| Arithmetic checks + controller | 100 | 100 | 90 (70–100) | 100 | 99.2 | 97.5 | 98.9 |
| Interpreter + controller | 100 | 100 | 50 (25–75) | 100 | 95.4 | 87.5 | 94.7 |
| Full LLM agent | 100 | 96.6 (91.4–100) | 25 (10–40) | 100 | 92.1 | 80.4 | 91.0 |
| Interpreter minus full agent | 0 | 3.4 | 25 | 0 | 3.33 (0.14–7.08) | 7.11 (−1.08 to 15.69) | 3.72 (0.00–7.61) |
Table S10: Decision accuracy (%) by task family and under three weightings. Family values are patient means with 95% patient-bootstrap intervals where any attempt failed. The primary aggregate averages the eligible families within each patient; equal family weights average the four family rates; equal task weights average the 94 patient-family tasks. The final row gives differences in percentage points, with the interval of the primary analysis for the primary weighting.
Figure S5: Distribution of the patient-level differences in decision accuracy between the interpreter/controller and the full LLM agent, after repetitions and eligible families were averaged: 7 patients favored the interpreter/controller, 21 were tied, and 2 favored the full agent. The dashed line marks the mean difference of 3.33 percentage points and the shaded band its 95% confidence interval (0.14–7.08).
| Patient | O | m | n | Count margin | Arithmetic | Interpreter run 1 | Interpreter run 2 | Full agent run 1 | Full agent run 2 |
|---|---|---|---|---|---|---|---|---|---|
| pt_248 | 631 | 286 | 3 | 342 | Escalated | Escalated | Escalated | Escalated | Submitted |
| pt_252 | 231 | 152 | 5 | 74 | Escalated | Submitted | Submitted | Escalated | Submitted |
| pt_256 | 197 | 108 | 3 | 86 | Escalated | Submitted | Submitted | Submitted | Submitted |
| pt_264 | 280 | 78 | 4 | 198 | Escalated | Escalated | Escalated | Submitted | Submitted |
| pt_283 | 173 | 151 | 2 | 20 | Escalated | Escalated | Escalated | Submitted | Submitted |
| pt_299 | 66 | 35 | 3 | 28 | Escalated | Escalated | Submitted | Submitted | Submitted |
Table S11: Reserved overlap contradictions by task. O, m, and n are the overlap, cold-target, and hot-organ voxel counts of the certificate ( Table S3 ), and the count margin is O − (m + n). Runs 1 and 2 are the two repetitions of each language system.
| System | Scenario | Attempts | Escalated (reason) | Submitted | Technical failure | Median optimizations |
|---|---|---|---|---|---|---|
| Controller | Routine | 30 | 0 | 29 | 1 | 2.0 |
| Controller | Missing structure | 30 | 30 (other) | 0 | 0 | 0.0 |
| Controller | Conflicting note | 30 | 0 | 30 | 0 | 2.0 |
| Controller | Coverage/cap | 29 | 0 | 29 | 0 | 3.0 |
| Controller | Overlap | 10 | 0 | 10 | 0 | 3.0 |
| Controller | Feasible control | 25 | 0 | 25 | 0 | 2.0 |
Table S12: Final decision of every eligible reserved attempt. Escalation reasons are the labels recorded by each system. No attempt ended in an automatic submission or a forced ending.
| Patient | System | Run | Unmet hard goal | Achieved (Gy) | Goal (Gy) | Beyond tolerance (Gy) | Witness margin (Gy) |
|---|---|---|---|---|---|---|---|
| pt_245 | Controller-based | All | External Dmax | 78.63 | 77.00 | 1.58 | 0.59 |
| pt_245 | Controller-based | All | PTV_7000 D99 | 58.36 | 66.50 | 8.09 | 2.94 |
| pt_245 | Controller-based | All | PTV_6300 D99 | 58.10 | 59.85 | 1.70 | 6.99 |
| pt_278 | Controller-based | All | PTV_7000 D99 | 63.45 | 66.50 | 3.00 | 3.31 |
| pt_287 | Controller-based | All | PTV_7000 D99 | 64.44 | 66.50 | 2.01 | 3.39 |
| pt_292 | Controller-based | All | PTV_7000 D99 | 64.38 | 66.50 | 2.07 | 3.44 |
Table S13: Unmet hard goals of submitted feasible controls. The controller-based systems submitted the same plan for each control. Beyond tolerance is the shortfall or excess outside the 0.05 Gy goal tolerance; the witness margin is the stored witness’s distance inside the tolerated boundary of the same goal.
| Policy | Controller | Arithmetic checks + controller | Interpreter + controller | Full LLM agent | Interpreter minus full agent |
|---|---|---|---|---|---|
| Primary policy | 60.0 | 99.2 | 95.4 | 92.1 | 3.33 (0.14–7.08) |
| Escalation reason must match the family | 25.8 | 99.2 | 95.4 | 92.1 | 3.33 (0.14–7.08) |
| Feasible control credited only if every hard goal is met | 53.6 | 92.8 | 89.0 | 85.7 | 3.33 (−2.08 to 8.75) |
Table S14: Patient-weighted decision accuracy (%) under the primary and two alternative policies, with the interpreter-minus-agent difference in percentage points and its 95% patient-bootstrap interval.
| Item | Specification |
|---|---|
| Beams and grid | Nine coplanar 6-MV beams at 40° spacing, as described for OpenKBP-Opt; 5 × 5 mm beamlets; voxels of 2.73–5.12 mm in plane and 1.5–3.27 mm between slices |
| Hard goals | External Dmax 77 Gy; PTV70, PTV63, and PTV56 D99 66.5, 59.85, and 53.2 Gy; brainstem, spinal cord, and mandible D0.1cc 50, 45, and 73.5 Gy |
| Soft goals | Parotid Dmean 26 Gy; esophagus and larynx Dmean 45 Gy |
| Tolerances and gates | 0.05 Gy for dose-valued goals; rejection when the global maximum exceeds 115% of the highest prescription or a target D99 falls below 80% of its prescription, with a boundary tolerance of Gy |
| Missing-structure task | Adds a hard D0.1cc goal of 54 Gy on the left optic nerve, a structure that is absent |
| Feasible control | Spinal-cord D0.1cc limit at the midpoint between 45 Gy and the witness cord dose, rounded to 0.1 Gy (33.6–39.4 Gy in 24 reserved tasks; 45.0 Gy for pt_267) |
Table S15: Specification of the planning environment. Values are those of the reserved evaluation.
| Item | Setting |
|---|---|
| Gateway and endpoint | OpenRouter, model route openai/gpt-5.6-terra, restricted to the OpenAI endpoint openai/gpt-5.6-terra-20260709; fallback providers disabled |
| Campaign | 9–10 September 2026, in three segments separated by two interruptions; 1,441 model requests, of which 1,438 returned a response |
| Output limits | 2,048 tokens for the interpreter and 8,192 per call for the full agent; the longest completion was 1,103 tokens |
| Sampling | Provider seeds 0 and 1; temperature not sent, because the endpoint did not support it |
| Retries and caching | No retries; response caching disabled; the campaign stopped on any cache hit or provider error other than a malformed completion |
| Timeout | The 180-s default of the HTTP client |
Table S16: Execution settings of the reserved evaluation with Terra, from its design and run records.
| Patient | Unmet hard goals at every budget | PTV_7000 D99, 500 iterations (Gy) | 2,000 iterations (Gy) | 5,000 iterations (Gy) |
|---|---|---|---|---|
| pt_245 | PTV_7000 D99, PTV_6300 D99, External Dmax | 58.36 | 58.37 | 58.37 |
| pt_278 | PTV_7000 D99 | 63.45 | 63.46 | 63.46 |
| pt_287 | PTV_7000 D99 | 64.44 | 64.45 | 64.45 |
| pt_292 | PTV_7000 D99 | 64.38 | 64.38 | 64.38 |
| pt_304 | PTV_7000 D99 | 62.83 | 62.82 | 62.82 |
| pt_334 | PTV_7000 D99, PTV_5600 D99 | 64.65 | 64.63 | 64.63 |
Table S17: Deficient feasible controls of the controller at three optimization budgets. The unmet hard goals were identical at all budgets, and the six controls are the same at each. The 500-iteration values are those of the main evaluation.
| System | Missing structure (30) | Coverage/cap (29) | Overlap (10) | Feasible control (25) | Patient mean | Technical failures |
|---|---|---|---|---|---|---|
| Interpreter + controller | 60.0 (48.3–70.0) | 22.4 (13.8–31.0) | 30.0 (15.0–45.0) | 10.0 (2.0–18.0) | 31.7 | 113 of 188 |
| Full LLM agent | 63.3 (50.0–76.7) | 13.8 (5.2–24.1) | 20.0 (5.0–35.0) | 64.0 (50.0–78.0) | 43.5 | 28 of 188 |
| Reason required: interpreter | 58.3 | 10.3 | 5.0 | 10.0 | ||
| Reason required: full agent | 55.0 | 13.8 | 20.0 | 64.0 |
Table S18: Decision accuracy (%) of the two language systems run with Ministral 3 8B on the 94 primary reserved tasks. Family values are patient means with 95% patient-bootstrap intervals, and the patient mean is the primary aggregate. In the lower rows, an escalation counted only when its reason named the problem of its family (the first alternative policy of Section S8.5 ). Technical failures, including replies that did not follow the required format, received no credit.
| System | Plans | SPG, median (IQR) | SPG 65 | Maximum beamlet weight, median | Weight 15 |
|---|---|---|---|---|---|
| Routine and feasible-control plans | |||||
| Controller | 54 | 136.5 (115.6–162.9) | 0 | 14.70 | 27 |
| Arithmetic checks + controller | 54 | 137.1 (121.1–163.4) | 0 | 14.70 | 27 |
| Interpreter + controller | 110 | 136.8 (115.6–163.4) | 0 | 15.06 | 54 |
| Full LLM agent | 109 | 380.9 (250.2–454.8) | 0 | 26.45 | 13 |
| All submitted plans | |||||
Table S19: Fluence complexity of the submitted reserved plans. SPG is the sum of positive gradients as defined in OpenKBP-Opt, and the OpenKBP-Opt reference plans were optimized with SPG 65 and beamlet weights 15. The controller-based systems submitted identical feasible-control plans.