cs.LGSep 6, 2026

Behavioral Cloning Outperforms Entropy-Regularized RL: Critic-Driven Failure of Actor-Critic Methods on Adaptive Tumor Treatment

Authors: Aleksandar Dimitrov, Giacomo Spigler

Organizations: Department of Cognitive Science & Artificial Intelligence, Tilburg University, Tilburg, The Netherlands

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Adaptive dosing requires policies that reduce tumor burden without excessive toxicity. Learned dosing policies are typically judged against historical or heuristic comparators, which cannot show whether a policy has found the best behavior available. We instead study a three-population tumor-control ODE in which optimal-control analysis fixes the form of a good schedule -- bang-bang dosing punctuated by a singular arc -- and construct a numerical controller of that form as a proxy for near-optimal behavior. Judged against this reference under a sustained-cure criterion -- 200 consecutive days below 5% carrying capacity -- Soft Actor-Critic (SAC) trained from scratch never reaches cure. Behavioral cloning (BC) of the reference reproduces it (100% sustained cure, 30/30 seeds), but SAC fine-tuning of the cloned policy destroys it across five entropy coefficients, and TD3 and BC-regularized SAC fail identically; the pattern persists under multiplicative pharmacokinetic action noise. Along curative trajectories the post-collapse critic ranks the collapsed-policy action above the reference action in 96% of states, concentrated in the maintenance phase, and the policy settles into a non-curative adaptive-therapy equilibrium. The reference is what makes this legible: against a heuristic comparator the fine-tuned policy would read as a competent controller rather than a failure.

Figures & tables

PolicyCure rate95% CIFinal SSFinal RR
MTD ( u=1u=1 )0%[0, 12]%1.91×1071.91\times 10^{7}2.07×1072.07\times 10^{7}
Periodic 5/50%[0, 12]%2.38×1072.38\times 10^{7}2.47×1072.47\times 10^{7}
SAC from scratch0%[0, 12]%8.24×1088.24\times 10^{8}2.18×1042.18\times 10^{4}
Teacher ( SmartProxy v3 )100%[88, 100]%2.81×1072.81\times 10^{7}4.28×1064.28\times 10^{6}
BC, at fine-tuning onset100%[88, 100]%3.07×1073.07\times 10^{7}4.23×1064.23\times 10^{6}
BC ++ SAC, after 30k updates0%[0, 12]%2.92×1082.92\times 10^{8}2.76×1052.76\times 10^{5}
Table 1: Sustained-cure metric ( n=30n=30 seeds; 95% Clopper-Pearson exact binomial confidence intervals). BC is evaluated at fine-tuning onset, before any gradient updates; the BC ++ SAC row is the same run after 30k updates.
Figure 1: Entropy-coefficient sweep. All five α\alpha values show collapse from 100% to 0% cure within 1,000 gradient steps of training onset. The curves coincide exactly and are drawn with distinct dash patterns so that the overlap remains visible.
Figure 2: Along the teacher’s curative trajectory, the collapsed critic assigns a higher estimated value to the collapsed-policy action than to the teacher action in 96% of states (shaded), with the largest gap during the singular-arc maintenance phase.
Figure 3: Failure mode of BC+SAC versus the teacher and BC-only policies from a shared initial condition. The collapsed policy holds a high- SS /low- RR equilibrium but never enters the cure basin.
More in this topic

Actor-Critic Methods

Actor-critic methods train a policy alongside a value estimator to guide decisions in complex control and decision-making tasks.

See all 107 papers

Explore similar work

CardsList
Sep 10, 2026

Adaptive Chemotherapy Control under Tumor Heterogeneity via Reinforcement Learning

Designing effective chemotherapy regimens is hindered by tumor heterogeneity and drug resistance, which complicate the deployment of patient-specific model-based optimal control across diverse populations. We develop and compare closed-loop deep reinforcement learning (DRL) dosing policies with continuous (TD3) and discrete (DQN) action spaces trained on a high-dimensional heterogeneous tumor model. The DRL policies are benchmarked against a Pontryagin's Maximum Principle (PMP)-derived open-loop benchmark. We assess generalization under parametric heterogeneity using a 100-patient virtual cohort with plus or minus 10 percent uniform perturbations in growth and drug-sensitivity parameters. Across this cohort, TD3 achieves higher average tumor reduction, while DQN yields tighter inter-patient dosing consistency, revealing a clear efficacy-consistency trade-off in this study. Our simulations assume full observation of all tumor subpopulations; translation to sparse and noisy clinical measurements will require partial-observability formulations and/or state estimation. Overall, the results show that simulation-trained DRL can learn state-dependent feedback dosing policies that complement open-loop optimal control benchmarks.

Bereket Sitotaw Kidane, Md Samiul Haque Motayed, Shuo Wang

Department of Mechanical and Aerospace Engineering, University of Texas at Arlington, Arlington, TX 76019, USA

RL ControlReinforcement LearningDeep Q-LearningOncologyGeneralization in RL
AskPDF
May 4, 2026

Recurrent Deep Reinforcement Learning for Chemotherapy Control under Partial Observability

Chemotherapy dose optimization can be formulated as a dynamic treatment regime, requiring sequential decisions under uncertainty that must balance tumor suppression against toxicity. However, most reinforcement learning approaches assume full observability of the patient state, a condition rarely met in clinical practice. We investigate whether memory-augmented policies can improve chemotherapy control under partial observability. To this end, we employ a recurrent TD3-based approach with separate LSTM actor-critic networks and evaluate it on the AhnChemoEnv benchmark from DTR-Bench, considering both off-policy and on-policy recurrent architectures against feed-forward TD3 and Soft Actor-Critic. Pharmacokinetic and pharmacodynamic variability are held fixed to isolate hidden-state uncertainty and observation noise and to avoid confounding effects from inter-patient variability. Across ten random seeds, recurrence yields modest benefit under full observability but substantially stronger and more stable performance under partial observability, with more consistent tumor suppression and improved normal-cell preservation. These findings indicate that memory-based policies are particularly beneficial when clinically relevant state information is incomplete or noisy.

Firas Mohamed Elamine Kiram, Imane Youkana, Rachida Saouli +2

Dept. of Information Engineering University of Padova Padova, Italy · LINFI Lab, Dept. of Computer Science University of Biskra Biskra, Algeria

Reinforcement LearningHealthcareSequential Decision MakingPartially Observable RLOncology
AskPDF
Sep 26, 2026

Rank Collapse Is Recoverable, Growing ∣Q∣|Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic

Raising the update-to-data (UTD) ratio breaks off-policy critics in two ways grouped as "plasticity loss": collapsing representations and growing value magnitude ∣Q∣|Q|. We separate them in Soft Actor-Critic (SAC) with scaled critics (width 2048, no normalization). Collapse is survivable: at UTD ratio 16, HalfCheetah critics with most units dormant keep learning, and the training guard, which stops runs whose loss or ∣Q∣|Q| explodes, never flags them. Within one high-UTD SAC configuration, runs start close together, and how far a critic's log⁡10∣Q∣\log_{10}|Q| has climbed by step 15k, its early growth, ranks the runs by how soon the guard flags them. At 15k, a flagged run's ∣Q∣|Q| sits a median of over a hundredfold below its flag level, yet the climb's rate already orders the flags (Harrell's C and out-of-sample AUC 0.78 on Walker2d, 0.98 on Ant, at UTD ratio 4). Dormancy does not. Aborting on this rate saves about a tenth of held-out Walker2d compute and stays net-positive live. A LayerNorm critic lowers the rate, removes the flag on Walker2d at UTD ratio 4 and lowers the return.

Tianqi Bu, YuXuan Peng, Junteng Tu +1

Off-Policy RLSoft Actor-CriticRepresentation Collapse
AskPDF