Value and Policy Calibration from Execution Feedback for Socially Compatible Driving
Organizations: Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Department of Electrical, Electronic and Computer Engineering, The University of Western Australia, Perth, Australia.
Abstract
Autonomous vehicle (AV) decisions at intersections affect the delay, braking, and motion smoothness of surrounding human-driven vehicles (HDVs). Learning socially compatible behavior requires connecting imagined interaction outcomes with feedback from executed actions. We propose a world-model reinforcement learning framework combines imagined policy improvement with direct value and policy calibration from finalized interactions, balancing AV progress with HDV interaction burden. An interaction-aware latent model supports imagination, while off-policy correction and replay updates enable observed rewards and transitions to supervise both the critic and actor. Simulation experiments at an unsignalized intersection achieve a 100% success and reductions in following-HDV acceleration variation of 65.5% and 44.0% relative to PIWM and DreamerV3, respectively. Ablations identify complementary roles for value calibration in reducing HDV burden and policy calibration in improving passage efficiency. Tests under out-of-distribution traffic shifts further show lower ego jerk and HDV acceleration variation than PIWM at comparable passage times.
Figures & tables
| Conventional driving metrics | HDV burden (satisfaction proxies) | ||||||
|---|---|---|---|---|---|---|---|
| Method | SR (%) | (%) | AV time ( ) | AV jerk ( ) | Delay ( ) | Braking ( ) | Accel. var. ( ) |
| PIWM | |||||||
| DreamerV3 | |||||||
| CPPO | |||||||
| RD-ACPPO | |||||||
| QR-DQN | |||||||
| Method | AV time | AV jerk | |||
|---|---|---|---|---|---|
| Full | |||||
| w/o Value | |||||
| w/o Policy | |||||
| only | |||||
| only |
Autonomous Driving
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
See all 570 papers