RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts

Hongbin Lin; Chaoda Zheng; Yiming Yang; Xiangyu Li; Shijia Chen; Jinhao Deng; Kangjie Chen; Dongbin Zhang

cs.CVSep 29, 2026

RoXDrive: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts

Authors: Hongbin Lin, Chaoda Zheng, Yiming Yang, Xiangyu Li, Shijia Chen, Jinhao Deng, Kangjie Chen, Dongbin Zhang, +6 more

Organizations: The Chinese University of Hong Kong, Shenzhen · XPeng Motors · Xidian University

Abstract

End-to-end autonomous driving policies are commonly trained via imitation learning on logged demonstrations without observing the consequences of their own actions, leading to causal confusion in closed-loop real-world deployment. To address this issue, reinforcement learning (RL) post-training offers a promising alternative by leveraging world models as interactive training environments to enable future scene generation for policy improvement. Nevertheless, existing approaches either rely on reconstruction-based simulators, offering limited counterfactual interaction, or adopt synthetic simulators to enable long-horizon closed-loop interaction at the cost of a substantial sim-to-real gap. Recently, video world models have exhibited the ability to generate realistic multi-step future rollouts but may not faithfully reflect action conditions, resulting in action-vision mismatch. In this paper, we introduce RoXDrive, a plug-and-play closed-loop RL framework that enables reliable policy optimization by identifying action-faithful world-model rollouts, consisting of two stages: 1) Model pre-training: In addition to imitation-based policy pre-training, we devise an Action-Vision Faithfulness Evaluator for inverse dynamics estimation with our geometry-aware auxiliary trajectory supervision, enabling long-horizon assessment of whether visual dynamics faithfully reflect the conditioning ego actions. 2) Action-faithful RL post-training: Agents iteratively interact with world models to form long-horizon scene rollouts, retaining only action-faithful ones for dense safety-aware scoring and scene-level closed-loop RL post-training. Extensive experiments on nuScenes and an in-house dataset with over 130K training scenarios demonstrate consistent gains across planners, reducing safety violations by 27.6% with DiffusionDrive on nuScenes and 33.7% with Qwen3-VL on the internal data.

Figures & tables

Figure 1: (a) Conventional synthetic and reconstruction-based simulators trade off interactive freedom and real-scene fidelity. (b) Action-conditioned video world models enable realistic long-horizon interaction, but action–vision mismatch may produce unreliable consequences. (c) RoXDrive explicitly measures action-vision faithfulness, retaining reliable episodes to optimize agents.

Figure 2: Naive fine-tuning of powerful Cosmos 3 ( i.e., Cosmos 3 FT) remains insufficient for long-horizon inverse dynamics, e.g., jitter errors (top) and long-horizon trajectory drifts (bottom). Hence, RoXDrive devises geometry-aware auxiliary supervision to handle these issues.

Figure 3: Illustration of RoXDrive. In stage one (top), we train the action-vision faithfulness evaluator with our geometry-aware auxiliary trajectory supervision beyond policy pre-training. In stage two (bottom), agents iteratively interact with a frozen world model to form long-horizon scene rollouts, retaining only action-faithful rollouts for dense safety-aware scoring and closed-loop RL.

Table 4

Method	RL	Open-loop Col. (%) $\downarrow$			Closed-loop Evaluation (4,675 4s clips, 2 action steps, based on X-World)
		1s	2s	Avg.	Safety		Driving Quality			Driving Score $\uparrow$
		1s	2s	Avg.	Obj. Col. $\downarrow$	Lane Viol. $\downarrow$	Progress $\uparrow$	Comfort $\uparrow$	Clearance $\uparrow$	Driving Score $\uparrow$
ST-P3	✗	0.23	0.62	0.43	268	557	0.397	0.425	0.387	0.341
VAD	✗	0.07	0.17	0.12	820	824	0.255	0.681	0.418	0.271
UniAD	✗	0.62	0.58	0.60	348	870	0.408	0.805	0.415	0.430
CLEAR $\ddagger$	✓	0.11	0.23	0.17	–	–	–	–	–	–

Table 3: Comparison of open-loop (2s, collision) and closed-loop (4s) model performance on nuScenes. RL indicates reinforcement learning and

\ddagger

denotes the official weights are unavailable.

Figure 4: Qualitative results of RoXDrive avoiding risky behaviors (see also Appendix H ).

Planner (10 actions)	Training	Safety		Driving Quality			DS $\uparrow$
Planner (10 actions)	Training	Obj. Col. $\downarrow$	Lane Viol. $\downarrow$	Progress $\uparrow$	Comfort $\uparrow$	Centering $\uparrow$	DS $\uparrow$
Gemma-3-4B	Imitation-only	328	171	0.938	0.952	0.521	0.371
Gemma-3-4B	+ RoXDrive	198	127	0.865	0.940	0.539	0.540
Qwen2.5-VL-3B	Imitation-only	331	141	0.945	0.911	0.538	0.394
Qwen2.5-VL-3B	+ RoXDrive	206	107	0.883	0.936	0.547	0.559
Qwen3-VL-2B	Imitation-only	242	162	0.884	0.938	0.527	0.450

Table 4: Closed-loop performance on 1K in-house test scenes (80 frames, 12 Hz) based on VLAs.

Setting	Obj. Col. $\downarrow$	Lane Viol. $\downarrow$	Progress $\uparrow$	Comfort $\uparrow$	Clearance $\uparrow$	DS $\uparrow$
DiffusionDrive (official)	266	772	0.694	0.866	0.392	0.526
$\bullet~$ RL w. All Rollouts	237	672	0.701	0.881	0.385	0.562
$\bullet~$ RL w. Cosmos3-FT	224	667	0.706	0.882	0.388	0.568
$\bullet~$ RL w. AVFE (Ours)	189	562	0.717	0.883	0.380	0.588
Group Size	Obj. Col. $\downarrow$	Lane Viol. $\downarrow$	Progress $\uparrow$	Comfort $\uparrow$	Clearance $\uparrow$	DS $\uparrow$
DiffusionDrive (official)	266	772	0.694	0.866	0.392	0.526

Table 5: Ablations of our AVFE and group size on nuScenes. More ablations are put in Appendix G .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative inverse dynamics evaluation on representative nuScenes scenes.

Setting	Obj. Col. $\downarrow$	Lane Viol. $\downarrow$	Progress $\uparrow$	Comfort $\uparrow$	Clearance $\uparrow$	DS $\uparrow$
DiffusionDrive	266	772	0.694	0.866	0.392	0.526
$\eta=0.25$	209	624	0.691	0.869	0.382	0.565
$\eta=0.50$	205	630	0.707	0.883	0.384	0.575
$\eta=0.75$ (default)	189	562	0.717	0.883	0.380	0.588
$\eta=1.25$	184	587	0.717	0.882	0.380	0.585
$\eta=1.50$	211	627	0.709	0.884	0.381	0.576

Appendix

Table 6: Ablation studies of the filtering threshold

\eta

on nuScenes.

Figure 6: Quantitative trends on test scenes during RL based on TransFuser.

Model	ADE Median $\downarrow$	ADE Mean $\downarrow$	FDE Median $\downarrow$	FDE Mean $\downarrow$
TransFuser ( Chitta et al., 2022 )	0.060	0.075	0.116	0.146
GTRS ( Li et al., 2025d )	0.052	0.049	0.105	0.104

Appendix

Table 7: Open-loop model performance on 1K test scenes in terms of ADE and FDE.

Model	Collision Scene $\downarrow$	Lane Violation $\downarrow$	Ego Progress $\uparrow$	Comfort $\uparrow$	Centering $\uparrow$	Driving Score $\uparrow$
TransFuser ( Chitta et al., 2022 )	257	178	0.881	0.892	0.540	0.498
+ RoXDrive (Ours)	215	145	0.900	0.932	0.539	0.571
GTRS ( Li et al., 2025d )	264	164	0.930	0.917	0.532	0.522
+ RoXDrive (Ours)	214	126	0.890	0.967	0.551	0.607

Appendix

Table 8: Closed-loop model performance on 1K test scenes with a default frame horizon of 80.

Model	MF	Object Collision	Lane Violation	Progress	Comfort	Centering	Driving Score
GTRS ( Li et al., 2025d )	20	31	19	0.968	0.918	0.564	0.859
+ RoXDrive		31	17	0.944	0.966	0.565	0.866
GTRS	40	73	50	0.952	0.918	0.553	0.791
+ RoXDrive		60	36	0.922	0.966	0.556	0.814
GTRS	60	202	123	0.938	0.917	0.539	0.615
+ RoXDrive		160	97	0.901	0.967	0.555	0.682

Appendix

Table 9: Closed-loop performance of GTRS under different maximum frame horizons (MF).

(a) Reward components $R_{\mathrm{col}}$ , $R_{\mathrm{lane}}$ , and $R_{\mathrm{dq}}$ denote the collision, lane violation, and driving-quality terms.
TransFuser	$R_{\mathrm{col}}$	$R_{\mathrm{lane}}$	$R_{\mathrm{dq}}$	Obj. Col. $\downarrow$	Lane Viol. $\downarrow$	Progress $\uparrow$	Comfort $\uparrow$	Centering $\uparrow$	DS $\uparrow$
$\checkmark$	$\times$	$\times$	$\times$	257	178	0.881	0.892	0.540	0.498
$\checkmark$	$\times$	$\times$	$\checkmark$	372	185	0.943	0.860	0.529	0.396
$\checkmark$	$\checkmark$	$\times$	$\checkmark$	241	164	0.911	0.924	0.537	0.545
$\checkmark$	$\times$	$\checkmark$	$\checkmark$	229	156	0.889	0.899	0.539	0.539
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	215	145	0.900	0.932	0.539	0.571

Appendix

Table 10: Ablation studies on reward components and the horizon lengths on the 1K test scenes.

Figure 7: Qualitative results of RoXDrive avoiding the collision on the in-house dataset.

Figure 8: Qualitative results of RoXDrive avoiding the collision on the in-house dataset.

Figure 9: Qualitative results of RoXDrive avoiding lane violations on the in-house dataset.

Figure 10: Qualitative results of RoXDrive avoiding lane violations on the in-house dataset.

Figure 11: Additional qualitative results of RoXDrive mitigating the causal confusion on nuScenes.

Figure 12: Additional qualitative results of RoXDrive mitigating the causal confusion on nuScenes.

Figure 13: Additional qualitative results of RoXDrive mitigating the causal confusion on nuScenes.

Explore similar work

CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies
May 6, 2026Keyu Chen, Nanfei Ye, Yida Wang +4Counterfactual Learning Reinforcement Fine-Tuning
School of Vehicle and Mobility, Tsinghua University · Li Auto Inc
CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving
Jul 3, 2026Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh +1Causality-Aware End-To-End Autonomous Driving Autonomous Driving
Qualcomm AI Research
NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation
Jun 2, 2026Aarti Basant, Amlan Kar, Despoina Paschalidou +30Autonomous Driving Simulation Naturalistic Driving Data
NVIDIA

May 6, 2026, cs.LGJ/K move · Enter open · S save

CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies

Keyu Chen, Nanfei Ye, Yida Wang, Wenchao Sun+3

School of Vehicle and Mobility, Tsinghua University · Li Auto Inc

Counterfactual Learning Reinforcement Fine-Tuning Closed Loop

Open paper AskPDF arXiv

Open-loop imitation learning has advanced modern autonomous driving policy architectures, but closed-loop deployment remains vulnerable to policy-induced distribution shift. Existing post-training paradigms exhibit fundamental trade-offs: closed-loop RL fine-tuning provides grounded feedback from executed actions but is constrained by the sparsity of informative events, whereas counterfactual fine-tuning provides dense supervision over candidate futures but inherits bias from imperfect future estimates. We introduce Counterfactual-to-Interactive Reinforcement Fine-Tuning (CRAFT), an on-policy framework that formulates closed-loop post-training as proxy-residual optimization. CRAFT uses group-normalized counterfactual advantages as a dense proxy for real closed-loop advantages and aligns this proxy with the closed-loop world through grounded residual correction from interaction-critical events. To stabilize adaptation, CRAFT regularizes the online policy toward an EMA teacher via asymmetric KL self-distillation. Theoretically, CRAFT decomposes the real closed-loop policy gradient into proxy and residual terms under the same visited-state distribution, reducing residual variance with an aligned proxy while mitigating proxy bias through grounded residual approximation. Empirically, CRAFT achieves the strongest closed-loop gains on Bench2Drive across hierarchical planning, vision-language-action, and vocabulary-scoring architectures. Ablations, scaling behavior, stability analyses, and transfer results further validate the complementary roles of dense counterfactual proxy and grounded residual correction. Project page: https://currychen77.github.io/CRAFT.

Abstract

Explore similar work

CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies

CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation