Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning

Li-Ching Chen; Andrés Franco Valiente; Yuta Ishiyama; Zachary R. Chalmers; William Chen; Kelsey Natsuhara; Taema Brinjikji; Madhumita Sushil

cs.LGOct 12, 2026

Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning

Authors: Li-Ching Chen, Andrés Franco Valiente, Yuta Ishiyama, Zachary R. Chalmers, William Chen, Kelsey Natsuhara, Taema Brinjikji, Madhumita Sushil,

Organizations: University of California, San Francisco · Weill Cancer Hub West · Vokram Inc. · OpenEvidence

Abstract

Clinical decision-making is a sequential process of information collection, clinical inference, and forecasting under partial observability, often without a uniquely correct treatment decision. Large language models (LLMs) have demonstrated strong medical question-answering capabilities but extending these advances to clinical reasoning remains challenging because many clinical tasks are only partially verifiable. Recent clinical reinforcement learning (RL) work has therefore largely focused on diagnostic endpoints, where correctness can be more directly verified. Oncology treatment decision making is a particularly challenging instance of this problem that extends beyond medical question-answering in which multiple decision trajectories can be clinically defensible for the same partially observed patient state. We formulate oncology treatment planning as an interactive RL problem under partial observability and construct patient-specific, process-aware rewards from decision processes observed in real world clinical records. The agent must learn to sequentially acquire information from a structured patient representation before committing to a first-line treatment recommendation, and is evaluated on information gathering, treatment selection, and reasoning alignment. Using GRPO and GDPO to optimize Llama-3.1-8B-Instruct within this environment, RL substantially improves the base policy beyond frontier model capabilities. The GDPO-trained policy increases Mean@3 scalar clinical reward by $79.60\%$ over base Llama and exceeds GPT-5.2 by $18.82\%$ under the same interactive environment, while reducing query count during training. Overall, our results show that patient-specific, partially verifiable rewards can provide an effective learning signal for interactive clinical RL when both the patient state and the underlying clinical decision process are only partially observed.

Figures & tables

Figure 1: The system consists of five functional components: a patient environment constructed from retrospective clinical data, a trainable clinical agent, a deterministic action parser, a frozen patient-record mediator, and a frozen judge.

	Mean@3	Best@3	SD	95% CI
GRPO Llama-3.1-8B-Instruct	0.5712	$0.6446$	$0.1465$	[0.547, 0.595]
GDPO Llama-3.1-8B-Instruct	0.6085	0.6718	$0.1264$	[0.588, 0.629]
GRPO Qwen3.5-9B	$0.1994$	$0.4878$	$0.2096$	[0.165, 0.234]
GDPO Qwen3.5-9B	$0.3429$	$0.5808$	$0.2174$	[0.307, 0.379]
GRPO GLM-4-9B-0414	$0.5313$	$0.6263$	$0.1321$	[0.510, 0.553]
GDPO GLM-4-9B-0414	$0.5105$	$0.6192$	$0.1474$	[0.486, 0.535]

Table 1: Overall reward performance across different agent policies. Results are evaluated on

n=142

cases with three generations per case. Mean@3 is the mean scalar reward across three generations; Best@3 is the mean of the highest scalar reward among the three generations for each case. Best results are shown in bold and second-best results are underlined .

Policy	Information $(0$ – $5)$	Regimen $(0$ – $5)$	Reasoning $(0$ – $5)$	Complete $(0$ – $1)$	Format $(0$ – $1)$
GRPO Llama-3.1-8B-Instruct	$2.754$	3.366	$2.688$	$0.960$	$1.000$
GDPO Llama-3.1-8B-Instruct	2.953	3.453	2.791	0.988	$1.000$
GRPO Qwen3.5-9B	$2.606$	$1.453$	$2.298$	$0.439$	$1.000$
GDPO Qwen3.5-9B	$2.725$	$2.033$	$2.399$	$0.664$	$1.000$
GRPO GLM-4-9B-0414	$2.728$	$2.967$	$2.444$	0.972	$1.000$
GDPO GLM-4-9B-0414	$2.629$	$2.854$	$2.455$	$0.953$	$1.000$

Table 2: Mean@3 performance on each reward axis. Information, treatment, and reasoning are scored from 0–5; completeness and format are scored from 0–1. Best results are shown in bold and second-best results are underlined .

Panel of Figure 2 — Figure 2: Learning dynamics of GRPO and GDPO with Llama-3.1-8B-Instruct. Dark curves show a 15-step moving average and faint curves show the unsmoothed training trajectory. Each curve corresponds to a single training run.

Policy	$n$	Scalar Reward	Information $(0$ – $5)$	Regimen $(0$ – $5)$	Reasoning $(0$ – $5)$	Complete $(0$ – $1)$	Queries per case
Oncologists	$34$	$0.573$	$3.000$	$2.912$	$2.676$	$1.000$	$9.412$
GRPO Llama	$34$	$0.537$	$2.647$	$3.206$	$2.608$	$0.931$	$4.598$
GDPO Llama	$34$	$0.600$	$2.931$	$3.382$	$2.804$	$0.980$	$11.275$
GPT-5.2	$34$	$0.544$	$2.971$	$2.951$	$3.000$	$0.873$	$37.824$

Table 3: Comparison between oncologists and language-model policies in the same interactive treatment-planning environment. Four oncologists each evaluated 6 to 10 unique cases from the held-out test set, for a total of 34 clinician trajectories. Agent results are evaluated on the same 34 cases under the same five-turn interaction protocol and averaged over three generations per case. Information, treatment, and reasoning rewards are scored from 0–5; completeness is scored from 0–1.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Illustrative example of trajectories from the clinical agent given a patient case. The clinical agent interacts the patient environment in order to retrieve relevant information necessary for selecting the treatment. In the process, they will describe the reasoning behind the retrieval certain information, and the inference on the retrieved information.

Panel of Figure 4 — Figure 4: Learning dynamics of GRPO and GDPO with Llama-3.1-8B-Instruct. We report the total scalar reward and the average number of queries issued by the policy. Dark curves show a 15-step moving average and faint curves show the corresponding unsmoothed training trajectory. Each curve corresponds to a single training run.

		Information $(0$ – $5)$	Regimen $(0$ – $5)$	Reasoning $(0$ – $5)$	Complete $(0$ – $1)$	Format $(0$ – $1)$
GRPO Llama-3.1-8B-Instruct	Mean@3	$2.754$	3.366	$2.688$	$0.960$	$1.000$
	Best@3	$3.000$	3.789	$2.944$	$1.000$	$1.000$
GDPO Llama-3.1-8B-Instruct	Mean@3	2.953	3.453	2.791	0.988	$1.000$
	Best@3	3.176	3.845	3.077	$1.000$	$1.000$
GRPO Qwen3.5-9B	Mean@3	$2.606$	$1.453$	$2.298$	$0.439$	$1.000$
	Best@3	$3.007$	$2.796$	$2.768$	$0.824$	$1.000$

Appendix

Table 4: Performance on each reward axis. Information, treatment, and reasoning are scored from 0–5; completeness and format are scored from 0–1. Best results are shown in bold and second-best results are underlined .

Panel of Figure 5 — Figure 5: GRPO and GDPO training dynamics on Qwen3.5-9B policy backbone. Dark curves show a 15-step moving average and faint curves show the corresponding unsmoothed training trajectory. Each curve corresponds to a single training run.

Panel of Figure 6 — Figure 6: GRPO and GDPO training dynamics on GLM-4-9B-0414 policy backbone. Dark curves show a 15-step moving average and faint curves show the corresponding unsmoothed training trajectory. Each curve corresponds to a single training run.

Policy	Queries/case	Unique fields	Information Reward
Llama-3.1-8B (Base)	$8.98$ $[8.42,9.57]$	$6.21$	$2.12$
GRPO-Llama	$4.33$ $[4.17,4.51]$	$4.12$	$2.75$
GDPO-Llama	$11.20$ $[10.84,11.57]$	$8.86$	$2.95$
Qwen (Base)	$7.26$ $[6.87,7.65]$	$5.20$	$2.41$
GRPO-Qwen	$6.96$ $[6.66,7.25]$	$5.18$	$2.61$
GDPO-Qwen	$7.96$ $[7.57,8.35]$	$5.73$	$2.73$

Appendix

Table 5: Patient-level changes in information-gathering behavior on the held-out test set. Values report the mean paired difference between each RL-trained policy based on Llama-3.1-8B-Instruct and the reference policy, with 95% bootstrap confidence intervals.

Panel of Figure 7 — Figure 7: Policy performance across primary-oncologist subgroups. (a) Per-patient reward distributions for each policy. (b) Per-patient reward difference between the specialized GDPO policy and GPT-5.2. Positive values indicate higher reward for GDPO.

Provider	Policy	$n$	Mean $\Delta$	95% CI	Median $\Delta$	Holm $p$
A	GRPO Llama	47	0.071	[0.009, 0.137]	0.044	0.109
	GDPO Llama	47	0.100	[0.036, 0.169]	0.067	0.018
	Llama base	47	-0.165	[-0.239, -0.087]	-0.200	$7.6\times 10^{-4}$
	Qwen base	47	-0.544	[-0.619, -0.463]	-0.556	$1.6\times 10^{-8}$
B	GRPO Llama	33	0.032	[-0.055, 0.124]	0.000	0.674
	GDPO Llama	33	0.063	[-0.015, 0.149]	0.044	0.396

Appendix

Table 6: Paired comparisons of policy reward against GPT-5.2 within physician-associated patient subgroups. Differences are computed per patient using Mean@3 scalar reward. Confidence intervals are 95% bootstrap confidence intervals for the mean paired difference.

p

-values are from paired Wilcoxon signed-rank tests with Holm correction within each provider subgroup.

Policy	Provider A	Provider B	Provider C	$H$	$p$
GRPO Llama	0.599	0.519	0.604	5.400	0.067
GDPO Llama	0.627	0.550	0.644	8.585	0.014
GPT-5.2	0.528	0.487	0.549	1.784	0.410
Llama base	0.363	0.329	0.311	1.426	0.490
Qwen base	-0.017	-0.032	0.002	0.248	0.883

Appendix

Table 7: Per-patient Mean@3 scalar reward across physician-associated patient subgroups. Kruskal–Wallis comparisons are descriptive because provider subgroups may differ in patient case mix.

Clinical Decision-Making

Clinical decision-making studies how evidence, patient preferences, risks, and resource constraints can guide diagnosis, treatment, and care.

See all 133 papers

CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics
May 10, 2026Aishik Nagar, Arun-Kumar Kaliya-Perumal, Yu-Hsuan Han +5Healthcare Clinical Decision Support
ASUS Intelligent Cloud Services (AICS) · Rehabilitation Research Institute of Singapore, Nanyang Technological University · Department of Family Medicine,2026 Taipei Veterans General Hospital +2
Learning Clinical-Trial Strategy: Offline Policy Training for Decision Agents
Aug 4, 2026William Bolton, Philip TorrClinical Decision-Making Imitation Learning
Department of Engineering Science, University of Oxford, Oxford, United Kingdom.
Reinforcement Learning for Evidence-Seeking Diagnostic Reasoning with Large Language Models
Jul 3, 2026Shengyi Hua, Kangzhe Hu, Conghui He +2Medical Diagnosis RL for Language Model Reasoning
Qing Yuan Research Institute, Shanghai Jiao Tong University · Shanghai AI Laboratoty · Shanghai Innovation Institute

May 10, 2026J/K move · Enter open · S save

CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics

Aishik Nagar, Arun-Kumar Kaliya-Perumal, Yu-Hsuan Han, Andrew Sheng-Han Huang+4

ASUS Intelligent Cloud Services (AICS) · Rehabilitation Research Institute of Singapore, Nanyang Technological University · Department of Family Medicine,2026 Taipei Veterans General Hospital+2

Healthcare Clinical Decision Support RL for Language Model Reasoning Clinical Reasoning Partially Observable Markov Decision Processes Clinical Language Model Evaluation+1

Open paper AskPDF arXiv

Inpatient clinical reasoning is a sequential decision under partial observability: the clinician sees the admission so far and must choose the next action whose downstream consequences are not yet visible. Existing clinical-LLM evaluations and RL rewards signals collapse this into closed-form retrieval, clinical journey leakage, or unanchored LLM-as-judge scoring. We introduce CLR-voyance, a framework that reformulates inpatient reasoning as a Partially Observable Markov Decision Process (POMDP) and supervises it with rewards that are simultaneously outcome-grounded and clinician-validated. We instantiate the formulation as CLR-POMDP, which partitions successful patient journeys into a policy-visible past and an oracle-only future. Using the past information, an oracle LLM generates a case-specific query-answer pair, and the first adaptive rubric for clinical reasoning which is verifiable in the future of the patient journey. These rubrics are used for both post-training and evaluation of models for inpatient clinical reasoning. We post-train Qwen3-8B and MedGemma-4B with GRPO followed by model merging, yielding state-of-the-art inpatient clinical reasoning while retaining generalist capabilities. CLR-voyance-8B achieves 84.91% on CLR-POMDP, ahead of frontier medical reasoning models like GPT-5 (77.83%) and MedGemma-27B (66.66%) and has comparable or better performance on existing medical benchmarks. To ensure a clinically meaningful setting, we conduct a large-scale clinician alignment study, where physicians curate per-case rubrics, grade candidate responses, and provide blinded pairwise preferences of model reasoning. This study provides insights on clinical LLM-as-a-judge and clinical preference-model selection, which can inform the community at large. CLR-voyance has been deployed for 6+ months at a partner public hospital, drafting thousands of reasoning-heavy inpatient notes.

Abstract

Clinical Decision-Making

Explore similar work

CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics

Learning Clinical-Trial Strategy: Offline Policy Training for Decision Agents

Reinforcement Learning for Evidence-Seeking Diagnostic Reasoning with Large Language Models