Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving

Mahmoud Selim; Cristina Cipriani; Karl Henrik Johansson

cs.LGOct 7, 2026

Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving

Authors: Mahmoud Selim, Cristina Cipriani, Karl Henrik Johansson

Organizations: KTH Royal Institute of Technology

Abstract

Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as \emph{interaction distribution shift} (IDS), and introduce \emph{Interaction-Constrained Drive Policy} (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/

Figures & tables

Figure 1: Overview of ICDP . The logged scene is encoded into structured scene features used by the diffusion actor and critic. The actor generates candidate ego trajectories, which are evaluated by a pessimistic chunk-level critic and by the interaction-support module. The latter combines a joint ego–agent classifier with an ego-only classifier; their residual score isolates interaction-specific support and yields the IDS estimate used to constrain policy improvement.

	Val14		Test14-Hard		Test14-Random
Planner	NR	R	NR	R	NR	R
PDM-Open ∗	53.53	54.24	33.51	35.83	52.81	57.23
GameFormer w/o refine.	13.32	8.69	7.08	6.69	11.36	9.31
PlanTF	84.27	76.95	69.70	61.61	85.62	79.58
PLUTO w/o refine. ∗	88.89	78.11	70.03	59.74	89.90	78.62
Diffusion Planner	89.87	82.80	75.99	69.22	89.19	82.93

Table 1: Closed-loop performance on nuPlan Val14, Test14-Hard, and Test14-Random under non-reactive (NR) and reactive (R) evaluation. The best result in each column is highlighted in blue.

	Val14		Test14-Hard		Test14-Random
Method	NR	R	NR	R	NR	R
BC	$89.8\pm 1.2$	$82.4\pm 1.09$	$78.42\pm 0.69$	$68.21\pm 0.95$	$91.98\pm 0.62$	$82.71\pm 0.53$
Offline RL	$89.4\pm 1.74$	$83.4\pm 2.11$	$78.75\pm 0.75$	$70.07\pm 1.08$	$90.74\pm 0.70$	$83.02\pm 1.08$
ICDP	$91.3\pm 1.91$	$84.3\pm 2.04$	$78.34\pm 1.03$	$72.87\pm 2.01$	$93.33\pm 0.85$	$86.34\pm 1.18$

Table 2: Controlled comparison of BC, unconstrained Offline RL, and ICDP . Results are averaged over ten evaluation checkpoints spaced by 2k optimization steps and reported as mean

\pm

std.

Panel of Figure 2 — Figure 2: Sensitivity of ICDP to (a) ego-support regularization, (b) interaction-support regularization, and (c) the number of surrounding agents. Scores are averaged across Test14-Hard and Test14-Random under non-reactive and reactive evaluation.

Panel of Figure 3 — Figure 3: Real-world deployment of ICDP on a full-scale autonomous truck. (a–b) Representative multi-agent interaction trial involving an overtaking vehicle, shown from external and onboard-perception viewpoints. (c–d) Standalone driving on curved road segments under real sensing and vehicle dynamics.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Action horizon $H$	$32$ steps ( $3.2$ s)
Sampling interval $\Delta t$	$0.1$ s
Discount factor $\gamma$	$0.99$
Collision / severe off-road return	$-5$
Soft / hard drivable-area threshold	$0.3$ / $0.5$ m
TTC additive-penalty threshold	$0.95$ s

Appendix

Table 3: Principal reward parameters used for training.

Parameter	Value
Planning horizon	$32$ steps ( $3.2\,\mathrm{s}$ )
History horizon	$16$ frames ( $1.5\,\mathrm{s}$ )
Actor/critic agent capacity	$32$
Interaction-support agent context	$48$
Lane / route capacity	$70$ / $25$
Map-query radius	$100\,\mathrm{m}$

Appendix

Table 4: Principal implementation settings used for ICDP .

Panel of Figure 4 — Figure 4: Qualitative trajectory samples from the learned diffusion policy across diverse nuPlan scenes. Each panel shows ten independently generated ego trajectories together with the logged ego trajectory, preferred route, and surrounding traffic. Samples are shown directly without critic-based selection or post-processing.

Panel of Figure 5 — Figure 5: Distribution of scenario types in the processed offline dataset. The $62$ recorded categories are divided into more- and less-frequent groups to make the long-tailed distribution legible.

Planner	Overall	Nudge Around	High Traffic	Jaywalk
PlanTF	47.70	49.40	58.85	33.94
PLUTO w/o refine.	58.47	71.56	67.25	25.48
Diffusion Planner	52.90	60.48	49.71	26.20
Flow Planner	61.82	72.96	67.21	43.57
ICDP	62.7	51.8	70.2	51.1

Appendix

Table 5: Closed-loop performance on InterPlan and selected interaction-critical scenario categories. The best result in each column is highlighted in blue.

Explore similar work

CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies
May 6, 2026Keyu Chen, Nanfei Ye, Yida Wang +4Counterfactual Learning Reinforcement Fine-Tuning
School of Vehicle and Mobility, Tsinghua University · Li Auto Inc
Human-like autonomy emerges from self-play and a pinch of human data
Jun 11, 2026Daphne Cornelisse, Julian Hunt, Zixu Zhang +4Self-Play Random
1NYU Tandon School of Engineering · 2NYU Courant · 3Princeton University +2
Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning
Aug 11, 2026Xincong Hu, Lei Ou, Maosen Li +3Autonomous Driving Adversarial Robustness
Nanjing University · 1Nanjing University, Nanjing, Jiangsu, China · 2Yinwang Intelligent Technology Co., Ltd., China +1

May 6, 2026, cs.LGJ/K move · Enter open · S save

CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies

Keyu Chen, Nanfei Ye, Yida Wang, Wenchao Sun+3

School of Vehicle and Mobility, Tsinghua University · Li Auto Inc

Counterfactual Learning Reinforcement Fine-Tuning Closed Loop

Open paper AskPDF arXiv

Open-loop imitation learning has advanced modern autonomous driving policy architectures, but closed-loop deployment remains vulnerable to policy-induced distribution shift. Existing post-training paradigms exhibit fundamental trade-offs: closed-loop RL fine-tuning provides grounded feedback from executed actions but is constrained by the sparsity of informative events, whereas counterfactual fine-tuning provides dense supervision over candidate futures but inherits bias from imperfect future estimates. We introduce Counterfactual-to-Interactive Reinforcement Fine-Tuning (CRAFT), an on-policy framework that formulates closed-loop post-training as proxy-residual optimization. CRAFT uses group-normalized counterfactual advantages as a dense proxy for real closed-loop advantages and aligns this proxy with the closed-loop world through grounded residual correction from interaction-critical events. To stabilize adaptation, CRAFT regularizes the online policy toward an EMA teacher via asymmetric KL self-distillation. Theoretically, CRAFT decomposes the real closed-loop policy gradient into proxy and residual terms under the same visited-state distribution, reducing residual variance with an aligned proxy while mitigating proxy bias through grounded residual approximation. Empirically, CRAFT achieves the strongest closed-loop gains on Bench2Drive across hierarchical planning, vision-language-action, and vocabulary-scoring architectures. Ablations, scaling behavior, stability analyses, and transfer results further validate the complementary roles of dense counterfactual proxy and grounded residual correction. Project page: https://currychen77.github.io/CRAFT.

Abstract

Explore similar work

CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies

Human-like autonomy emerges from self-play and a pinch of human data

Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning