Behavioral Cloning Outperforms Entropy-Regularized RL: Critic-Driven Failure of Actor-Critic Methods on Adaptive Tumor Treatment

Aleksandar Dimitrov; Giacomo Spigler

cs.LGSep 6, 2026

Behavioral Cloning Outperforms Entropy-Regularized RL: Critic-Driven Failure of Actor-Critic Methods on Adaptive Tumor Treatment

Authors: Aleksandar Dimitrov, Giacomo Spigler

Organizations: Department of Cognitive Science & Artificial Intelligence, Tilburg University, Tilburg, The Netherlands

Abstract

Adaptive dosing requires policies that reduce tumor burden without excessive toxicity. Learned dosing policies are typically judged against historical or heuristic comparators, which cannot show whether a policy has found the best behavior available. We instead study a three-population tumor-control ODE in which optimal-control analysis fixes the form of a good schedule -- bang-bang dosing punctuated by a singular arc -- and construct a numerical controller of that form as a proxy for near-optimal behavior. Judged against this reference under a sustained-cure criterion -- 200 consecutive days below 5% carrying capacity -- Soft Actor-Critic (SAC) trained from scratch never reaches cure. Behavioral cloning (BC) of the reference reproduces it (100% sustained cure, 30/30 seeds), but SAC fine-tuning of the cloned policy destroys it across five entropy coefficients, and TD3 and BC-regularized SAC fail identically; the pattern persists under multiplicative pharmacokinetic action noise. Along curative trajectories the post-collapse critic ranks the collapsed-policy action above the reference action in 96% of states, concentrated in the maintenance phase, and the policy settles into a non-curative adaptive-therapy equilibrium. The reference is what makes this legible: against a heuristic comparator the fine-tuned policy would read as a competent controller rather than a failure.

Figures & tables

Policy	Cure rate	95% CI	Final $S$	Final $R$
MTD ( $u=1$ )	0%	[0, 12]%	$1.91\times 10^{7}$	$2.07\times 10^{7}$
Periodic 5/5	0%	[0, 12]%	$2.38\times 10^{7}$	$2.47\times 10^{7}$
SAC from scratch	0%	[0, 12]%	$8.24\times 10^{8}$	$2.18\times 10^{4}$
Teacher ( SmartProxy v3 )	100%	[88, 100]%	$2.81\times 10^{7}$	$4.28\times 10^{6}$
BC, at fine-tuning onset	100%	[88, 100]%	$3.07\times 10^{7}$	$4.23\times 10^{6}$
BC $+$ SAC, after 30k updates	0%	[0, 12]%	$2.92\times 10^{8}$	$2.76\times 10^{5}$

Table 1: Sustained-cure metric (

n=30

seeds; 95% Clopper-Pearson exact binomial confidence intervals). BC is evaluated at fine-tuning onset, before any gradient updates; the BC

+

SAC row is the same run after 30k updates.

Figure 1: Entropy-coefficient sweep. All five $\alpha$ values show collapse from 100% to 0% cure within 1,000 gradient steps of training onset. The curves coincide exactly and are drawn with distinct dash patterns so that the overlap remains visible.

Figure 2: Along the teacher’s curative trajectory, the collapsed critic assigns a higher estimated value to the collapsed-policy action than to the teacher action in 96% of states (shaded), with the largest gap during the singular-arc maintenance phase.

Figure 3: Failure mode of BC+SAC versus the teacher and BC-only policies from a shared initial condition. The collapsed policy holds a high- $S$ /low- $R$ equilibrium but never enters the cure basin.

Actor-Critic Methods

Actor-critic methods train a policy alongside a value estimator to guide decisions in complex control and decision-making tasks.

See all 107 papers

Adaptive Chemotherapy Control under Tumor Heterogeneity via Reinforcement Learning
Sep 10, 2026Bereket Sitotaw Kidane, Md Samiul Haque Motayed, Shuo WangRL Control Reinforcement Learning
Department of Mechanical and Aerospace Engineering, University of Texas at Arlington, Arlington, TX 76019, USA
Recurrent Deep Reinforcement Learning for Chemotherapy Control under Partial Observability
May 4, 2026Firas Mohamed Elamine Kiram, Imane Youkana, Rachida Saouli +2Reinforcement Learning Healthcare
Dept. of Information Engineering University of Padova Padova, Italy · LINFI Lab, Dept. of Computer Science University of Biskra Biskra, Algeria
Rank Collapse Is Recoverable, Growing $|Q|$ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic
Sep 26, 2026Tianqi Bu, YuXuan Peng, Junteng Tu +1Off-Policy RL Soft Actor-Critic

Sep 10, 2026J/K move · Enter open · S save

Adaptive Chemotherapy Control under Tumor Heterogeneity via Reinforcement Learning

Bereket Sitotaw Kidane, Md Samiul Haque Motayed, Shuo Wang

Department of Mechanical and Aerospace Engineering, University of Texas at Arlington, Arlington, TX 76019, USA

RL Control Reinforcement Learning Deep Q-Learning Oncology Generalization in RL

Open paper AskPDF arXiv

Designing effective chemotherapy regimens is hindered by tumor heterogeneity and drug resistance, which complicate the deployment of patient-specific model-based optimal control across diverse populations. We develop and compare closed-loop deep reinforcement learning (DRL) dosing policies with continuous (TD3) and discrete (DQN) action spaces trained on a high-dimensional heterogeneous tumor model. The DRL policies are benchmarked against a Pontryagin's Maximum Principle (PMP)-derived open-loop benchmark. We assess generalization under parametric heterogeneity using a 100-patient virtual cohort with plus or minus 10 percent uniform perturbations in growth and drug-sensitivity parameters. Across this cohort, TD3 achieves higher average tumor reduction, while DQN yields tighter inter-patient dosing consistency, revealing a clear efficacy-consistency trade-off in this study. Our simulations assume full observation of all tumor subpopulations; translation to sparse and noisy clinical measurements will require partial-observability formulations and/or state estimation. Overall, the results show that simulation-trained DRL can learn state-dependent feedback dosing policies that complement open-loop optimal control benchmarks.

Abstract

Actor-Critic Methods

Explore similar work

Adaptive Chemotherapy Control under Tumor Heterogeneity via Reinforcement Learning

Recurrent Deep Reinforcement Learning for Chemotherapy Control under Partial Observability

Rank Collapse Is Recoverable, Growing ∣Q∣|Q|∣Q∣ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic

Rank Collapse Is Recoverable, Growing $|Q|$ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic