Atomic Proximal Policy Optimization for Electric Robo-Taxi Dispatch and Charger Allocation

Jim Dai; Manxi Wu; Zhanhao Zhang

cs.AIFeb 19, 2025

Atomic Proximal Policy Optimization for Electric Robo-Taxi Dispatch and Charger Allocation

Authors: Jim Dai, Manxi Wu, Zhanhao Zhang

Organizations: Operations Research and Information Engineering, Cornell University · Department of Civil and Environmental Engineering, University of California, Berkeley

Abstract

Pioneering companies such as Waymo have deployed electric robo-taxi services in several U.S. cities. Operating such fleets requires coordinating ride matching, vehicle repositioning, and battery charging in a stochastic environment while accounting for limited charging infrastructure. Understanding both how to design efficient, scalable control algorithms for large fleets and how charger deployment shapes system performance is therefore critical for robo-taxi operators. We model the operations of an electric robo-taxi system as a discrete-time Markov decision process with a long-run average reward objective. To efficiently evaluate system performance under different configurations, we develop a scalable deep reinforcement learning algorithm, Atomic Proximal Policy Optimization (Atomic-PPO), which reduces the action space using atomic action decomposition. The resulting training procedure requires only modest computational resources, making it practical to evaluate a large number of system configurations. We calibrate a simulation environment using real-world NYC for-hire vehicle trip records and benchmark Atomic-PPO against heuristic policies and a fluid-based reward upper bound. The results show that Atomic-PPO produces near-optimal fleet dispatching policies and serves as a reliable tool for system evaluation. We conduct extensive numerical experiments to evaluate how charging infrastructure and fleet size affect system performance. Our findings highlight the importance of ridership-based charger placement, balanced investments in charger quantity and charging speed, and joint planning of charging infrastructure and fleet size.

Figures & tables

Table 2

Algorithm	Reward/ $\bar{R}$	Avg. Daily Reward
Atomic-PPO	91%	$390K
Atomic-PPO (without repositioning)	89%	$385K
Power of k	81%	$345K
Fluid policy	69%	$295K

Table 3: Reward achieved by Atomic-PPO with and without repositioning, power-of-k policy and fluid policy.

Period	Policy	Fleet Serving Trips (%)	Trips Fulfilled (%)	Reward ($/Hour)
Morning rush	Atomic-PPO	82.7%	58.1%	$20K
Morning rush	Atomic-PPO (without repositioning)	76.7%	55.1%	$18K
Morning rush	Power-of- $k$	63.3%	37.6%	$15K
Morning rush	Fluid policy	47.8%	36.1%	$12K
Evening rush	Atomic-PPO	92.2%	46.4%	$22K
Evening rush	Atomic-PPO (without repositioning)	96.3%	49.9%	$22K

Table 4: Fleet serving trips, trips fulfilled, and mean reward per hour by time of day.

Figure 3: Net passenger-driven redistribution during weekday rush hours under Atomic-PPO with vehicle repositioning disabled and power-of- $k$ with $k=2$ . Values are inbound minus outbound passenger-trip assignments per region. Red indicates net inflow and blue net outflow.

	Residential						Workplace			Restaurants
Region	0	1	2	3	4	5	6	7	8	9	Avg. Daily Reward
Unif. 10 chargers	1	1	1	1	1	1	1	1	1	1	$265K
Unif. 20 chargers	2	2	2	2	2	2	2	2	2	2	$380K
Unif. 30 chargers	3	3	3	3	3	3	3	3	3	3	$385K
Unif. 40 chargers	4	4	4	4	4	4	4	4	4	4	$390K
Midtown Manhattan.	0	0	0	0	0	0	0	15	0	0	$385K

Table 5: Average daily reward under different charging facility allocations.

Figure 4: Average daily rewards (top) and reward gaps (Atomic-PPO minus power-of-

k

; bottom) across fleet sizes, charger quantities, and outlet rates. Solid lines show Atomic-PPO; dashed lines show power-of-

k

Regime	Charger Power Rates	Vehicle Range	Avg. Daily Reward
Benchmark	75 kW	130 miles	$390K
Double Range	75 kW	260 miles	$390K

Table 6: Impact of vehicle range on average reward under abundant chargers.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$V$	The set of all service regions
$N$	Fleet Size
$B$	Vehicle battery size
$\Delta$	Set of charging rates
$T$	Number of time steps of each single day
$\tau_{uv}^{t}$	Trip duration from $u$ to $v$ at time $t$

Appendix

Table 7: Notations for the electric robo-taxi system.

Component	Hyperparameter Description	Value
Training setup	Number of rollout trajectories per policy iteration	30
	Number of operation days per trajectory	8
	Initial PPO clipping rate	0.1
	Exponential decay factor for clipping rate	0.97
Policy network	Time embedding dimension	24
	Number of hidden layers	3

Appendix

Table 8: Hyperparameters used for training Atomic-PPO.

	Trained on July’s Demand		Trained on Corresponding Month’s Demand
	Avg. Daily Revenue	% Fluid UB	Avg. Daily Revenue	% Fluid UB
June	$425K	91%	$425K	91%
August	$380K	95%	$385K	95%
September	$455K	91%	$455K	91%

Appendix

Table 9: Comparison of Atomic-PPO performance when trained on July’s demand profile versus retrained on the corresponding month’s demand profile, reported separately for each month.

	Avg. Daily Reward	% Fluid UB
Deterministic trip duration	$390K	91%
Stochastic trip duration	$390K	91%

Appendix

Table 10: Performance comparison of Atomic-PPO under deterministic v.s. stochastic trip durations

	Avg. Daily Reward	% Fluid UB
10 Regions	$390K	91%
15 Regions	$380K	90%
20 Regions	$355K	N/A

Appendix

Table 11: Performance of Atomic-PPO across different number of regions

Proximal Policy Optimization

Proximal Policy Optimization trains reinforcement-learning policies with conservative updates that balance learning progress and stability.

See all 119 papers

A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch
Apr 28, 2026An Nguyen, Hoang Nguyen, Phuong Le +3Electric Vehicle Charging Constrained RL
College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center for Environmental Intelligence, VinUniversity, Hanoi, Vietnam
Emergent Charging Coordination in Electric Delivery Fleets
Sep 7, 2026Javier Vales-Alonso, Juan J. AlcarazMulti-Agent Coordination Electric Vehicle Charging
Department of Information and Communication Technologies, Universidad Politécnica de Cartagena, Cartagena, Spain
Deep Reinforcement Learning for Dynamic Battery Management of Autonomous Order Pickers
Jul 6, 2026Taniya Shaji, Abhay Sobhanan, Christof DefrynRL for Robotics Mobile Robotics
Indian Institute of Management Bangalore, Bannerghatta Road, Bengaluru 560076, Karnataka, India · University of Antwerp, Prinsstraat 13, Antwerp 2000, Belgium

Apr 28, 2026J/K move · Enter open · S save

A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch

An Nguyen, Hoang Nguyen, Phuong Le, Hung Pham+2

College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center for Environmental Intelligence, VinUniversity, Hanoi, Vietnam

Electric Vehicle Charging Constrained RL Distributionally Robust Optimization Distributionally Robust RL Robust RL

Open paper AskPDF arXiv

We study city-scale control of electric-vehicle (EV) ride-hailing fleets where dispatch, repositioning, and charging decisions must respect charger and feeder limits under uncertain, spatially correlated demand and travel times. We formulate the problem as a hex-grid semi-Markov decision process (semi-MDP) with mixed actions -- discrete actions for serving, repositioning, and charging, together with continuous charging power -- and variable action durations. To guarantee physical feasibility during both training and deployment, the policy learns over high-level intentions produced by a masked, temperature-annealed actor. These intentions are projected at every decision step through a time-limited rolling mixed-integer linear program (MILP) that strictly enforces state-of-charge, port, and feeder constraints. To mitigate distributional shifts, we optimize a Soft Actor-Critic (SAC) agent against a Wasserstein-1 ambiguity set with a graph-aligned Mahalanobis ground metric that captures spatial correlations. The robust backup uses the Kantorovich-Rubinstein dual, a projected subgradient inner loop, and a primal-dual risk-budget update. Our architecture combines a two-layer Graph Convolutional Network (GCN) encoder, twin critics, and a value network that drives the adversary. Experiments on a large-scale EV fleet simulator built from NYC taxi data show that PD-RSAC achieves the highest net profit, reaching $1.22M, compared with $0.58M-$0.70M for strong heuristic, single-agent RL, and multi-agent RL baselines, including Greedy, SAC, MAPPO, and MADDPG, while maintaining zero feeder-limit violations.

Abstract

Proximal Policy Optimization

Explore similar work

A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch

Emergent Charging Coordination in Electric Delivery Fleets

Deep Reinforcement Learning for Dynamic Battery Management of Autonomous Order Pickers