Reinforcement learning with verifiable rewards is increasingly used to improve language model reasoning, yet its effects on sequential decision procedures remain poorly understood. We study this problem in clinical multi-turn agents that must decide what information to request, when to stop gathering evidence, and which action to take. We introduce three controlled environments: medication refill triage, acute diagnostic work-up, and dosing titration, each with hidden clinical states, query costs, exact adaptive oracles, and deterministic outcome verifiers. Across these tasks, outcome-only group relative policy optimization often improves aggregate reward while producing undesirable procedures, including premature commitment, one-question collapse, and failures on rare high-risk cases. In diagnostics, assigning dense credit to questions that reduce uncertainty improves held-out value and moderate-case support across three training seeds relative to outcome-only reinforcement learning, without the high-risk seed collapse observed under outcome-only training. Separating exploration from exploitation further increases high-risk support at its best checkpoint, but at the cost of excessive information gathering on low-acuity cases. Additional experiments show that the optimal strength of process credit is task-dependent; that after a supervised warmup that already gathers carefully, reinforcement learning raises aggregate reward by trimming questions while degrading high-risk performance; and that reinforcement learning does not reliably repair rare behavior gaps inherited from biased supervised data. Verifiable final outcomes are therefore insufficient for training reliable multi-turn agents: reward design must account for the procedure used to reach an answer, and evaluation must distinguish adaptive information gathering from reward-exploiting shortcuts.
Figures & tables
Figure 1: Experimental framework and effects of reward design on clinical multi-turn decision procedures. A, The LoRA-adapted MedGemma-4B-it policy interacts with a partially observed patient simulator across medication-refill triage, acute diagnostics and dosing titration. Each environment provides an exact adaptive oracle and deterministic outcome verifier. The three GRPO methods use different training reward formulations or exploration strategies, with policy optimization and rollout generation distributed across two compute nodes. All policies are evaluated on held-out cases using the same unshaped terminal verifier. B, Selected trajectories for the same simulated high-risk abdominal-pain case illustrate different information-gathering procedures. The exact adaptive oracle gathers four targeted findings before emergency treatment. Outcome-only GRPO commits prematurely after one finding. Credit-assignment GRPO reaches the correct terminal action after four requests and receives an oracle-grade reward despite following a different evidence sequence. Decoupled exploration reaches the correct terminal action after eight requests, including additional non-decisive findings. The learned trajectories are selected illustrative examples, not representative estimates of typical policy behavior. Oracle-grade reward does not necessarily establish sufficient gathered clinical evidence. C, Held-out diagnostics performance measured by high-risk verifier value and mean questions asked. Outcome-only and credit-assignment GRPO show mean ± standard deviation across three training seeds; the base model and decoupled GRPO are single-seed references (hatched bars). The dashed line indicates the exact adaptive oracle’s mean question count. All methods are scored using the same unshaped evaluation reward. Decoupled GRPO is shown at its step-40 checkpoint, selected on held-out data; its final checkpoint does not retain the high-risk advantage (Table 3 ). The held-out diagnostics set contains only two high-risk cases.
Task
Findings
Commit actions
Families
Latent state
Turn cap
Refill triage
12
5
2 (ACEi/ARB, NSAID)
severity (none/mod/high)
10
Diagnostics
18
5
2 (chest, abdominal)
severity (none/mod/high)
12
Titration
6
3 (ordered)
1 (anticoagulation)
joint clot × bleed (9 states)
10
Table 1: Clinical environment dimensions. “Findings” is the set of binary clinical features available for the agent to request. “Commit actions” is the number of terminal decisions available to the policy. All environments use a question cost of 0.03 and an oracle-grade tolerance of 0.03 . Diagnostics contains the largest finding space and is designed to reveal hedging and premature commitment on rare emergency cases.
Figure 2: Agent–simulator dialogues rendered in plain language. The dialogues are simplified simulator traces, not real clinical conversations. The replying party is a rule-based simulator, not a patient or clinician. It looks up each requested finding in the case’s hidden record and reports whether it is present. Findings are binary, so the simulator returns no numeric values; each question and answer paraphrases one requested finding and its revealed value. (A) The same held-out high-risk diagnostics case as Figure 1 . The exact adaptive oracle gathers four findings before emergency treatment (reward +0.83 ). The outcome-only GRPO rollout shown is that policy’s most common behaviour on this case (6 of 12 samples). It asks one question and admits the patient (reward +0.17 ). (B) Exact adaptive oracle trajectories for one held-out case in each task, matching Appendix B . Rewards are terminal utility minus 0.03 per question. Hidden risk is never shown to the agent.
Task
Method
overall val
overall supp
high val
high supp
asks
Diagnostics
base
+0.251
0.35
+0.160
0.50
2.5
plain GRPO ‡
+0.285
0.41
+0.098
0.33
1.8
credit-assignment GRPO ‡
+0.313
0.59
+0.157
0.50
2.1
decoupled GRPO (step 40 § )
+0.329
0.42
+0.248
1.00
4.4
Titration
base
−0.022
0.52
+0.396
0.67
2.8
plain GRPO
−0.012
0.57
+0.627
1.00†
1.4
Table 2: Final held-out results across tasks and training methods. All policies are evaluated in the hidden condition using the pure verifiable reward and k=12 sampled rollouts per case. High-risk value and high-risk oracle-grade support are the primary safety-focused metrics. Overall value, overall support, and mean questions describe aggregate performance and interaction burden. Bold marks the strongest non-exploitative high-risk result within each task. Apparent support produced by a degenerate under-gathering policy is marked exploit , because support@ 12 can remain high even when typical trajectories do not gather the evidence required for the decision. ‡ Diagnostics plain GRPO and credit-assignment GRPO are reported as means across three seeds; the corresponding standard deviations appear in Table 3 . All other rows are single-seed results. High-risk subgroup sizes are small (diagnostics n=2 , titration high-clot n=6 , refill n=7 cases); with k=12 rollouts per case, diagnostics high-risk support can take only the values 0 , 0.5 , and 1 .
Policy
overall value
high-value
high supp
moderate supp
mean asks
base (untrained)
+0.251
+0.160
0.50
0.26
2.5
plain GRPO (control)
+0.285±0.027
+0.098±0.037
0.33±0.29
0.38±0.18
1.82±0.11
credit-assignment GRPO ( λ=0.2 )
+0.313±0.024
+0.157±0.057
0.50±0.00
0.67±0.10
2.08±0.30
decoupled GRPO (step 40 § )
+0.329
+0.248
1.00
0.63
4.4
decoupled GRPO (final, step 60)
+0.304
+0.158
0.50
0.74
5.0
Table 3: Diagnostics results in the held-out hidden condition using the pure verifiable reward. Plain GRPO and credit-assignment GRPO are reported as mean ± sample standard deviation over three training seeds. Base and decoupled GRPO are single-seed reference policies. Credit assignment improves overall value, high-risk value, and moderate oracle-grade support relative to the outcome-only control. One plain GRPO seed loses the supported high-risk case, whereas credit assignment retains it in each tested seed. “supp” denotes oracle-grade support@12 and should be interpreted as sampled capability rather than single-trajectory reliability. The held-out high-risk subgroup contains only two cases; every policy with high-risk support of 0.50 supports the same abdominal case, so high-risk values in this table are illustrative rather than precise estimates. § The step-40 checkpoint was selected on the held-out evaluation set and is therefore an optimistic estimate; the final checkpoint is shown for comparison.
Figure 3: Diagnostics performance by risk subgroup in the held-out hidden condition. Every method is evaluated with the pure verifier reward. Plain GRPO and credit-assignment GRPO show mean ± standard deviation over three training seeds; base and decoupled GRPO are single-seed references. Outcome-only GRPO raises aggregate value but reduces and destabilizes high-risk performance. Credit assignment improves the moderate subgroup and maintains high-risk support of 0.50 (one of two high-risk cases) across all three seeds. Decoupled GRPO produces the highest high-risk value and support, but requests substantially more information.
Policy
overall value
overall asks
one-ask rate
high-clot value
high-clot supp
base (untrained)
−0.022
2.76
0.74
+0.396
0.67
plain GRPO (control)
−0.012
1.40
0.91
+0.627
1.00
+ credit-assignment ( λ=0.03 )
−0.010
1.24
0.93
+0.825
1.00
+ credit-assignment ( λ=0.1 )
−0.036
2.67
0.62
+0.556
1.00
+ credit-assignment ( λ=0.2 )
−0.091
4.10
0.31
+0.446
0.67
Table 4: Titration results in the held-out hidden condition using the pure verifier reward. The information credit coefficient controls the depth of gathering. Plain GRPO and λ=0.03 produce high one-question rates and under-gathering. At λ=0.2 , the policy asks more than the oracle and loses aggregate value. The analytically motivated setting λ=0.1 places mean questioning close to the oracle level and preserves high clot support. The absence of an overall value gain is consistent with the small best constant gap of +0.075 , which leaves limited adaptive headroom.
Figure 4: Information credit coefficient sweep on the titration task. Mean questions remain below the oracle level when λ≤0.03 , approach the oracle at λ=0.1 , and exceed it at λ=0.2 . The coefficient therefore behaves as a task-specific control parameter for gathering depth, rather than as a constant that can be transferred without calibration.
Policy
overall val
overall supp
high val
high supp
benign supp
mean asks
biased SFT (init)
+0.337
0.93
−0.285
1.00
0.92
3.0
+ credit-assignment GRPO
+0.236
0.45
+0.092
0.14
0.45
2.2
+ decoupled-GRPO
+0.292
0.41
+0.040
0.36
0.16
4.3
Table 5: Attempted repair of a mildly biased diagnostics SFT policy ( 30% of high-risk SFT examples retained), in the held-out hidden condition with n=200 cases ( 14 high-risk) and the pure verifier reward. The SFT initialization samples an oracle-grade trajectory for every high-risk case (support 1.00 ) but performs poorly on a typical high-risk rollout (value −0.285 ). Both RL methods raise high-risk value but sharply reduce high-risk support, and both reduce overall support. Credit-assignment GRPO under-gathers; decoupled GRPO over-gathers and damages benign support. Neither converts the sampled behaviour into a reliable procedure. “supp” denotes oracle-grade support@12.
Figure 5: Effect of supervised coverage bias on aggregate performance and latent support. Left, the biased SFT retains high overall support despite poor high-risk value. Right, severe bias removes the high-risk action emergent_tx from sampled trajectories and fails Gate 2. Mild bias retains a 21% sampling rate and passes Gate 2. The subsequent repair experiment shows that nonzero latent support is necessary but not sufficient for RL to restore the missing behaviour.
causal-LM cross-entropy on every assistant turn (system/user masked)
Optimizer
AdamW, lr 1×10−4 , linear warmup 5% then constant
Epochs / batch / max seq len
2 / 4 / 2048
Training trajectories
800 (oracle-adaptive; generation seed 7 )
Seed / checkpoint
1 / final
Appendix
Table 6: Hyperparameters for the supervised protocol warmup. Loss is computed on every assistant turn; system messages, user messages, and simulator responses are excluded from the supervised objective.
Algorithm
GRPO (TRL), custom on-policy rollout_func
Initialization
base model, or merged SFT adapter
Adapter
LoRA r=32 , α=64 (q,k,v,o,gate,up,down)
Group size (rollouts/prompt)
8
Per-device batch / grad-accum
8 / 4
Training cases / steps / save
300 / 60 / every 20
Learning rate / KL β
1×10−5 / 0.02
Appendix
Table 7: Hyperparameters for GRPO training. Outcome-only GRPO uses the terminal utility minus the accumulated question cost. Credit assignment adds λΦ to the rollout reward during training only. Decoupled GRPO instead alternates exploration updates based on state recoverability with exploitation updates based on env.score . Every policy is evaluated using the same unshaped terminal verifier.
Figure 6: Training diagnostics derived from the run logs. Top, diagnostics GRPO over 60 optimisation steps. The panels show the optimised reward, KL divergence from the initialization, policy entropy, and mean questions per rollout for outcome-only GRPO, credit-assignment GRPO, and decoupled GRPO. The question trajectory reveals the characteristic procedural differences among the methods; outcome-only GRPO and credit-assignment GRPO both settle well below the oracle gathering level of approximately 3.9 , with credit assignment slightly higher, while decoupled GRPO rises above it. Bottom, training loss for the biased SFT warmup. Loss converges to approximately 0.002 , showing that the supervised trajectories are closely fit before RL begins.
Figure 7: Checkpoint trajectory for decoupled GRPO on diagnostics. Evaluation uses the held-out hidden condition and the pure verifier reward. High-risk oracle-grade support and high-risk value peak at checkpoint 40, reaching 1.00 and +0.248 , respectively. The same checkpoint also shows increased gathering; mean questions approach and then exceed the oracle level of approximately 3.9 . The trajectory illustrates the method’s central tradeoff between high-risk coverage and information burden.
Figure 8: Refill triage performance by risk subgroup in the held-out hidden condition, evaluated with the pure verifier reward. The figure compares the base model, the SFT v3 warmup, outcome-only GRPO, and credit-assignment GRPO at their final checkpoints. The SFT warmup already achieves the strongest high-risk result. Both RL policies subsequently reduce high-risk value and move toward shorter, under-gathered trajectories, despite preserving high support under stochastic sampling.
Figure 9: Attempted reinforcement learning repair of the biased SFT policy. Credit assignment and decoupled GRPO move away from the initialization in opposite directions. Credit assignment reduces gathering and loses high-risk support; decoupled GRPO increases gathering and damages benign support. Both methods reduce the SFT policy’s strong aggregate support, and neither restores the missing high-risk procedure.
Figure 10: Attempted repair of a mildly biased diagnostics SFT policy by risk subgroup, using the held-out hidden condition and n=200 cases. Both RL methods lose most of the initialization’s high-risk oracle-grade support; support falls from 1.00 for the SFT initialization to 0.14 with credit-assignment GRPO and 0.36 with decoupled GRPO, even as high-risk value rises. The methods fail in different directions. Credit assignment reduces gathering, while decoupled GRPO increases question burden and lowers benign support from 0.92 to 0.16 . This figure complements Table 5 and Figure 9 .