SEG-JPEG: Simple Visual Semantic Communications for Remote Operation of Automated Vehicles over Unreliable Wireless Networks
Authors: Sebastian Donnelly, Ruth Anderson, George Economides, James Broughton, Peter Ball, Alexander Rast, Andrew Bradley
Organizations: Autonomous Driving and Intelligent Transport Group, Oxford Brookes University, Oxford, UK · Oxfordshire County Council, Oxford, UK · Department for Transport, London, UK · School of Engineering, Computing & Mathematics, Oxford Brookes University, Oxford, UK · Artificial Intelligence, Data Analysis and Systems Institute, Oxford Brookes University, Oxford, UK
Remote Operation is touted as being key to the rapid deployment of automated vehicles. Streaming imagery to control connected vehicles remotely currently requires a reliable, high throughput network connection, which can be limited in real-world remote operation deployments relying on public network infrastructure. This paper investigates how the application of computer vision assisted semantic communication can be used to circumvent data loss and corruption associated with traditional image compression techniques. By encoding the segmentations of detected road users into colour coded highlights within low resolution greyscale imagery, the required data rate can be reduced by 50% compared with conventional techniques, while maintaining visual clarity. This enables a median glass-to-glass latency of below 200 ms even when the network data rate is below 500 kbit/s, while clearly outlining salient road users to enhance situational awareness of the remote operator. The approach is demonstrated in an area of variable 4G mobile connectivity using an automated last-mile delivery vehicle. Results indicate that large-scale deployment of remotely operated automated vehicles could be possible even on the often constrained public 4G/5G mobile network, providing the potential to expedite the nationwide roll-out of automated vehicles.
Figures & tables
Figure 1: Augmented greyscale imagery presented to a remote operator, highlighting vehicles (blue), pedestrians (red) and bicycles (green) .
Figure 2: Frame distortion in h.264 imagery caused by a reduced quality mobile network connection 1 1 1 Evaluated using imagery from the Oxford RobotCar dataset [ 19 ] .
Figure 3: The SEG-JPEG approach to augment greyscale imagery 1 1 footnotemark: 1 .
Figure 4: Highly compressed greyscale JPEG imagery with road users highlighted to enhance RO situational awareness. Red indicates a person, green a bicycle, blue a road vehicle 1 1 footnotemark: 1 .
Figure 5: Green-Log automated ROV (top), remote operation hardware installed in the vehicle (bottom).
Figure 7: Latency by mobile network throughput, showing how increasing the data rate of streamed imagery increases latency on a limited 4G connection.
Figure 8: G2G latency distribution for SEG-JPEG imagery sent over a poor quality 4G mobile network connection (10Hz, 500kbit/s). The network latency is the largest contributor to overall (G2G) latency variation.
Figure 9: Original uncompressed imagery (left), distorted 500 kbit/s h.264 imagery exhibiting ghosting where remnants of a moving pedestrian from a previous frame are visible (centre), 500 kbit/s SEG-JPEG imagery (right) 1 1 footnotemark: 1 .
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh · Department of Computer Science and Engineering, Mälardalen University, 721 23 Väster˚as, Sweden · Samsung Research America, Mountain View, CA 94043, USA
Nazarbayev University, School of Engineering & Digital Sciences, Astana, Kazakhstan · Zurich University of Applied Sciences (ZHAW), Winterthur, Switzerland
Department of Information and Communication Engineering, Noakhali Science and Technology University, Bangladesh · Department of Artificial Intelligence, Kyung Hee University, Yongin-si 17104, Republic of Korea · Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004-4005, USA +1