cs.ROMar 6, 2026

Nonlinear Performance Degradation of Vision-Based Teleoperation under Network Latency

Authors: Aws Khalil, Jaerock Kwon

Organizations: Department of Electrical and Computer Engineering, University of Michigan - Dearborn, Dearborn, MI, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Teleoperation is increasingly being adopted as a critical fallback for autonomous vehicles. However, the impact of network latency on vision-based, perception-driven control remains insufficiently studied. The present work investigates the nonlinear degradation of closed-loop stability in camera-based lane keeping under varying network delays. To conduct this study, we developed the Latency-Aware Vision Teleoperation testbed (LAVT), a research-oriented ROS 2 framework that enables precise, distributed one-way latency measurement and reproducible delay injection. Using LAVT, we performed 180 closed-loop experiments in simulation across diverse road geometries. Our findings reveal a sharp collapse in stability between 150 ms and 225 ms of one-way perception latency, where route completion rates drop from 100% to below 50% as oscillatory instability and phase-lag effects emerge. We further demonstrate that additional control-channel delay compounds these effects, significantly accelerating system failure even under constant visual latency. By combining this systematic empirical characterization with the LAVT testbed, this work provides quantitative insights into perception-driven instability and establishes a reproducible baseline for future latency-compensation and predictive control strategies. Project page, supplementary video, and code are available at https://bimilab.github.io/paper-LAVT

Figures & tables

Fig. 1: Architecture of the Latency-Aware Vision Teleoperation testbed (LAVT). The system operates in a distributed client–server configuration. On the vehicle side (server), a forward-facing camera publishes a local ROS 2 image topic. A video bridge node subscribes to this topic and transmits frames via a dedicated RTP/H.264 stream (GStreamer) over UDP. On the remote side (client), a corresponding video bridge decodes the stream and republishes the received frames as a local ROS 2 image topic for downstream processing by either a teleoperator interface or a vision-based autonomy module. Control commands are exchanged as ROS 2 topics over rmw_zenoh , with independent Zenoh routers running on both server and client and connected via configured peer-to-peer sessions. The blue path denotes the perception channel with one-way video latency τv\tau_{v} , while the orange path denotes the actuation channel with one-way control latency τc\tau_{c} . Linux Traffic Control and Network Emulator (TC NetEm) is applied independently on each direction to inject controlled delay. Chrony provides system-level clock synchronization between machines, enabling accurate offset estimation and one-way latency computation using embedded frame timestamps. The resulting architecture forms a fully closed-loop teleoperation system suitable for systematic latency characterization in both simulation and real-vehicle deployments.
Fig. 2: A full-scale drive-by-wire (DBW) research vehicle integrated with LAVT for real-world deployment. The DBW system includes brake, throttle, steering, and shift-by-wire control modules. The platform is equipped with a multi-sensor suite including LiDAR, RGB cameras, and GPS. While quantitative latency experiments in this study are conducted in simulation, the complete teleoperation and streaming stack has been integrated with this vehicle without architectural modification.
Fig. 3: Client-side autonomy module in LAVT. The module receives a compressed video stream and vehicle speed from the server. A deterministic classical vision pipeline estimates lane boundaries in a bird’s-eye-view (BEV) representation, from which a centerline is computed. Lateral control is generated using a speed-adaptive Pure Pursuit controller, and longitudinal motion is regulated by a PI speed controller. Safety mechanisms include frame staleness detection and safe-stop behavior.
Fig. 4: Predefined evaluation routes in CARLA Town04. Route A emphasizes short straight segment followed by 90 ∘ right turn then a long straight segment. Route B starts with a 90 ∘ left turn contains followed by a short straight segment and a sustained left-hand curvature. Route C contains short straight →\rightarrow and right-hand curvature. Colored arrows indicate nominal driving direction. Key subsets (black boxes) isolate steering-intensive segments used for focused stability analysis.
ConditionVideo Delay ( τv\tau_{v} )Control Delay ( τc\tau_{c} )
L00 ms0 ms
L175 ms0 ms
L2150 ms0 ms
L3225 ms0 ms
L4150 ms75 ms
L5225 ms100 ms
TABLE I: Latency Conditions
Fig. 5: Verification of injected one-way latencies across all routes (Town04) for L0–L5. For each condition, colored boxplots show video latency ( τv\tau_{v} ) and control-command latency ( τc\tau_{c} ) on the same axis. L0–L3 increase video latency only. L4 and L5 reuse the video-delay profiles of L2 and L3, respectively, while injecting additional control-command latency (visible as a distinct τc\tau_{c} distribution at L4–L5).
Fig. 6: Trajectory overlays for key route segments (Town04) under latency conditions L0–L5. Top: Route A Key, Middle: Route B Key, Bottom: Route C Key. Thin colored lines show individual runs; the thick black curve denotes the reference trajectory (L0 run05). Red crosses indicate aborted runs due to collision. Increasing perception latency induces growing deviation and phase-lag effects, while additional control-command delay (L4–L5) further amplifies instability and termination frequency, demonstrating nonlinear degradation of closed-loop stability.
Fig. 7: Latency-induced degradation in closed-loop lane keeping (Town04; routes A–C and key subsets). Top: Route-balanced mean 95th-percentile absolute cross-track error computed over runs that successfully complete the route without collision; faint markers indicate per-run values. Error bars denote 95% confidence intervals of the mean. Bottom: Route completion rate over all runs; annotations show completed/total runs (out of 30). Tracking degradation emerges at L2 ( ≈\approx 211 ms one-way perception delay), while completion exhibits a nonlinear collapse beyond this threshold. L4 isolates additional control-command delay under matched perception latency.
Cond.Comp. (%)Coll.LaneInvP95xP95_{x} (m)
L01000.200.402.30
L193.30.233.972.73
L250.00.6715.103.91
L336.70.8322.139.25
L430.00.7718.804.64
L510.00.9716.8313.15
TABLE II: Driving performance under latency conditions (Town04, all routes).
More in this topic

Robot Teleoperation

Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.

See all 206 papers

Explore similar work

CardsList

  1. Driving Through the Network: Performance and Workload Under Latency and Video Impairment

    May 15, 2026Ines Trautmannsheimer, Ahmed Azab, Frank DiermeyerRobot Teleoperation

    Technical University of Munich, School of Engineering and Design, Institute of Automotive Technology and Munich Institute of Robotics and Machine Intelligence (MIRMI), 85748 Munich, Germany

  2. A Neural Network Based Teleoperation for Remote Controlled Vehicles

    Aug 11, 2026Ning Ding, Azim EskandarianRobot TeleoperationRobust Control

    Virginia Polytechnic Institute and State University, Blacksburg, VA 24060 USA · College of Engineering, Virginia Commonwealth University, Richmond, VA 23284 USA

  3. Robust Offset-Free Lane-Keeping Control Framework for Vision-Only Automated Vehicles

    Oct 23, 2026Der-Hau LeeAutonomous DrivingModel Predictive Control

    Department of Electrophysics, National Yang Ming Chiao Tung University, Hsinchu 300, Taiwan