Completion-Aware Cross-Fidelity Offline-to-Online Reinforcement Learning for Multi-Line Bus Holding

Yifan Zhang; Qifan Zhang; Liang Zheng

cs.AISep 30, 2026

Completion-Aware Cross-Fidelity Offline-to-Online Reinforcement Learning for Multi-Line Bus Holding

Authors: Yifan Zhang, Qifan Zhang, Liang Zheng

Organizations: Central South University, Changsha, Hunan, China

Abstract

Exploratory reinforcement learning (RL) on an operating bus fleet is impractical,while policies trained only from historical data cannot acquire new experience. Hybrid Offline-and-Online (H2O) RL combines fixed target replay with simulator interaction, but the inexpensive online simulator can differ from the target in transition and event-duration dynamics. We study this cross-fidelity problem for multi-line bus holding and address a failure mode in which lower generalized passenger time coexists with incomplete passenger journeys.

Figures & tables

Frozen learning evidence	Adaptive development	Fresh formal confirmation
Target-SUMO replay; calibrated online simulator; five IQL-initialized H2O+ actors; 25,000 simulator events; fixed endpoint checkpoints.	Terminal actor taper and deterministic hold transforms were developed over successive preregistered rounds. Development outcomes informed the final completion-safety reserve and therefore are not confirmatory.	The fixed candidate and fixed direct parent were evaluated on 30 untouched SUMO cells: ten independent blocks crossed with three demand levels. Exact paired block tests used all five training seeds within each block.

Table 1: Separation of learning, adaptive development, and formal confirmation. Only the final column supplies confirmatory evidence.

Figure 1 : Topology of the twelve directional bus services represented in both the target SUMO environment

\mathcal{M}_{\mathrm{tgt}}

and the calibrated online simulator

\mathcal{M}_{\mathrm{sim}}

. The two environments share stop sequences, timetables, capacities, and passenger origin-destination journeys; their traffic and event-propagation mechanisms differ. Line 7X is drawn more heavily only for legibility; policy control covers all 12 services.

Policy	Gen. passenger s/departed	Completion	Unfinished
Direct parent	2,484.86	0.96185	655.42
Reserve-augmented candidate	2,392.73	0.96650	578.20
Difference	$-92.12$	$+0.00466$	$-77.22$

Table 2: Formal endpoint means for the preregistered candidate and its direct behavior parent. The difference is candidate minus parent.

Demand	$\Delta Y$ (s/departed)	$\Delta R_{\mathrm{cmp}}$	$\Delta U$ (passengers)
0.75	$-40.29$	$+0.00185$	$-21.00$
1.00	$-87.61$	$+0.00470$	$-70.92$
1.25	$-148.46$	$+0.00741$	$-139.74$

Table 3: Candidate-minus-parent formal differences by demand multiplier. Negative generalized time and unfinished counts are favorable; positive completion is favorable.

Algorithm label	Gen. passenger s/departed	Completion	Unfinished passengers
Reserve-augmented candidate	2,392.73	0.96650	578.20
Direct behavior parent	2,484.86	0.96185	655.42
Raw v14 terminal-quarter actor	2,623.19	0.96453	610.03
Base H2O+	2,696.98	0.96288	637.41
Online-only SAC	2,593.85	0.96232	648.31
Delayed actor (v10)	2,532.20	0.95894	702.57

Table 4: Descriptive formal means for the fixed candidate, direct parent, raw actor, and lineage controls. Bold denotes the best displayed mean.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Canonical state width	229
Maximum event slots	16
Slot embedding and fused widths	128 and 256, respectively
Actor/critic/value hidden widths	256, 256
Critic heads	2
Maximum hold	60 s

Appendix

Table B.1: Final actor-training and architecture values.

Diagnostic	Mean	Minimum	Maximum
Training joint AUC	0.9477	0.9370	0.9580
Held-out joint AUC	0.4520	0.4366	0.4747
Held-out raw ESS fraction	0.0615	0.0100	0.1530

Appendix

Table B.2: Domain-ratio diagnostics across five sealed actor-training seeds.

Seed	$\Delta Y$ (s/departed)	$\Delta R_{\mathrm{cmp}}$	$\Delta U$ (passengers)
6201	$-87.25$	$+0.00457$	$-75.43$
6202	$-97.44$	$+0.00485$	$-80.67$
6203	$-94.28$	$+0.00478$	$-79.63$
6204	$-93.44$	$+0.00465$	$-76.97$
6205	$-88.21$	$+0.00444$	$-73.40$

Appendix

Table B.3: Formal candidate-minus-parent differences by training seed.

Explore similar work

Counterfactual Transport Flows for Offline Conservative Trajectory Refinement
Jun 8, 2026Lena Krieger, Xuan Zhao, Zhuo Cao +3Model-Based Reinforcement Learning Long Refinement Trajectories
1IAS-8, Forschungszentrum J ulich, Germany · 2LMU Munich, Munich Center for Machine Learning (MCML), Germany · Department of Computer Science, Aarhus University, Denmark.
Active Offline-to-Online Reinforcement Learning
Jul 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiOffline Reinforcement Learning Policy Evaluation
Virginia Commonwealth University, USA · University of Virginia, USA
Fully Offline Reinforcement Learning
May 28, 2025Mattie Fellows, Clarisse Wibault, Uljad Berdica +3Model-Based Reinforcement Learning Offline Reinforcement Learning
Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford

Jun 8, 2026, cs.LGJ/K move · Enter open · S save

Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

Lena Krieger, Xuan Zhao, Zhuo Cao, Qin Wang+2

1IAS-8, Forschungszentrum J ulich, Germany · 2LMU Munich, Munich Center for Machine Learning (MCML), Germany · Department of Computer Science, Aarhus University, Denmark.

Model-Based Reinforcement Learning Long Refinement Trajectories Offline Reinforcement Learning Counterfactual Learning World Preference Alignment Learning

Open paper AskPDF arXiv

Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback. A key difficulty is improving observed behavior without extrapolating beyond what the offline data supports. We propose \emph{counterfactual transport flows}, a source-conditioned trajectory refinement framework for offline decision-making guided by world feedback. Given a low-feedback candidate trajectory, we construct local preference pairs from offline data by retrieving nearby trajectories in latent trajectory space with higher task-specific feedback, and use them as weak supervision for conservative refinement. The framework learns instance-specific refinement directions: at inference time, a refinement strength parameter controls how far the candidate trajectory is transported, enabling a trade-off between preserving the original behavior and applying stronger improvement. Experiments on D4RL benchmarks, including AntMaze and MuJoCo tasks, show that our method improves behavior from historical returns as world feedback, while providing interpretable trajectory-level refinement paths.

Abstract

Explore similar work

Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

Active Offline-to-Online Reinforcement Learning

Fully Offline Reinforcement Learning