Revisiting Direct Control: A Field Evaluation of Sensor Fusion Efficacy in Teleoperating Full-Scale Autonomous Vehicles via Bird's Eye View

Stanisław Kuczma; Sławomir K. Tadeja; Michał R. Nowicki

cs.ROOct 26, 2026

Revisiting Direct Control: A Field Evaluation of Sensor Fusion Efficacy in Teleoperating Full-Scale Autonomous Vehicles via Bird's Eye View

Authors: Stanisław Kuczma, Sławomir K. Tadeja, Michał R. Nowicki

Organizations: Pozna´n University of Technology. · Massachusetts Institute of Technology.

Abstract

In autonomous driving, robot teleoperation serves as a critical safety fallback mechanism, requiring operators to assume direct control in the event of system failure. While recent research has pivoted toward high-level supervisory control or simulation-based studies, these approaches often fail to capture the physical dynamics and sensory demands of driving a full-scale vehicle. To address this gap, we evaluated the impact of spatial data representation on operator performance using a real car-sized vehicle in a field experiment with $N = 18$ participants. We compared three visualization interfaces: (i) a standard camera feed baseline, with two bird's-eye views (BEV) from (ii) a raw LiDAR point cloud, and (iii) a sensor fusion of LiDAR and camera data streams. Contrary to prior passive observation studies suggesting that sensor-fusion BEV may hinder awareness, our active-control results demonstrate that such representation significantly outperforms the standard LiDAR BEV in perceived usability ( $p \leq 0.0001$ ) and cognitive task load ( $p = 0.01$ , NASA-TLX score: 34.58/100) while also being significantly better than the Camera-only view ( $p = 0.043$ ). The fused interface also led to statistically significant reductions in task completion time ( $p = 0.0461$ ) and superior usability (SUS: 77.9/100) compared to those of the Camera-only and LiDAR-only views. By synthesizing objective performance data with comprehensive subjective assessments, we demonstrate that BEV fusion maximizes safety margins while minimizing the operator's mental demand. These findings challenge existing teleoperation conventions and provide an empirically validated approach for designing robust remote piloting systems.

Figures & tables

Fig. 1: Despite the popularity of the bird’s-eye view for machine learning-based perception, robot teleoperation is usually performed with a camera video stream. We asked 18 participants to teleoperate a car-sized vehicle on the (4) obstacle course with (1–3) varying visualization views to quantify their impact on the operator’s performance. We collected both subjective usability and workload data, and objective performance metrics, including driving time, number of collisions, mean trajectory deviation, and stopping accuracy, to judge each validated visualization view.

Author	Platform	Display Interface	Data Trans.	Representation	N	Control
Simulation, Prototype & Crowdsourcing
Boker & Lanir [ 18 ]	Crowdsourcing	Monitor	N/A	BEV	150	No
Kremer et al. [ 14 ]	Simulation (CARLA)	Monitor	Reduced	HD Map + Object List	-	Yes
Wolf et al. [ 12 ]	Prototype	Monitor	N/A	2D Video + Icons	36	No
Hofbauer et al. [ 15 ]	Simulation (CARLA)	Triple Monitor	Adaptive Bitrate	2D Video	-	Yes
Tener & Lanir [ 9 ]	Simulation	Monitor	N/A	3rd Person View + 2D Map	14	No

TABLE I: Comparison and Review of the existing studies coupling remotely operating vehicles with an experiment in terms of platform type, used display setting, data source, transmission, and its representation, as well as experimental sample size (N) or lack thereof, and use of human control showing limited to now user studies utilizing real vehicles.

Fig. 2: The remote operator was positioned at the control station, with operator-vehicle communication split into two independent data streams. The operator’s view was obtained with a high-resolution video transmitter, while control signals were used by the official radio controller.

Panel of Figure 4 — Fig. 3: Operators’ views during teleoperation experiment: (A) camera-based, (B) LiDAR-based, and (C) fused camera and LiDAR.

Fig. 4: The obstacle course with challenges in anti-clockwise direction: (1) parking, (2) tight turn, (3) obstacle avoidance, and (4) stop line.

Panel of Figure 6 — Fig. 5: Images presenting test circuit challenges in anti-clockwise direction: (1) parking, (2) tight turn, (3) obstacle avoidance, and (4) stop line.

Fig. 6: Perceived usability scores measured with SUS for each interface ( $N=18$ ). Higher scores indicate better perceived usability. The whiskers denote standard error (SE) while asterisks denote statistical significance (one for $p\leq 0.05$ ) and three for $p\leq 0.001$ ).

View	Survey	n	min	max	M	SE
Camera	SUS $\uparrow$	18	30.00	87.50	64.72	4.26
Camera	TLX $\downarrow$	18	20.00	60.83	42.69	2.81
Camera+LiDAR	SUS $\uparrow$	18	47.50	100.0	77.92	3.61
Camera+LiDAR	TLX $\downarrow$	18	5.00	55.83	34.58	3.05
LiDAR	SUS $\uparrow$	18	32.50	80.00	51.11	3.39
LiDAR	TLX $\downarrow$	18	14.17	80.00	47.96	3.50

TABLE II: SUS and NASA-TLX scores for each interface type M = mean, SE = Standard Error

Fig. 7: Perceived taskload levels measured with NASA-TLX showing that the multi-modal view of Camera+LiDAR was the least demanding in terms of workload experienced by the operator. The whiskers denote standard error (SE) while asterisks denote statistical significance (one for $p\leq 0.01$ ).

View	Time [s]	No. of collisions	Stop distance [cm]	Parking [cm]	Mean Trajectory Deviation [m]
Camera	149.2 ± 9.14	1.529 ± .259	44.8 ± 7.8	152.3 ± 18.3	0.665 ± .082
Camera+LiDAR	144.0 ± 10.2	1.647 ± .383	26.7 ± 6.3	145.2 ± 21.1	0.635 ± .063
LiDAR	167.8 ± 9.65	1.875 ± .340	27.9 ± 6.1	186.5 ± 27.5	0.830 ± .090

TABLE III: Obtained quantitative metrics after filling missing data and removing outliers

Fig. 8: Results from the objective data, in their original form, indicate that lower values correspond to better performance. In this form, a clear pattern emerges: the fused BEV enables users to drive more quickly, with fewer collisions and greater accuracy.

Robot Teleoperation

Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.

See all 206 papers

May 13, 2026

Beyond VMAF: Towards Application-Specific Metrics for Teleoperation Video

Automated driving has made remarkable progress, yet situations still arise where human intervention is necessary. Teleoperation provides a scalable solution to address such cases, enabling remote operators to support vehicles without being physically present. In this context, video transmission forms the operator's primary source of situational awareness, making video quality a decisive factor for both safety and task performance. In an online study, participants rated compressed video sequences from the Zenseact Dataset and provided subjective quality ratings. These ratings were then used to retrain the Video Multi-Method Assessment Fusion (VMAF) model, yielding an adapted variant tailored to teleoperation. The retrained model demonstrated improved alignment with human ratings compared to the original 4K VMAF. In particular, RMSE decreased from 10.36 to 8.83, and MAD from 8.71 to 6.38, corresponding to improvements of 15% and 27%, respectively. These results highlight that incorporating domain-specific data can enhance the predictive power of established quality metrics in safety-critical applications. At the same time, Outlier cases emerged in which videos received high objective scores despite noticeable degradations in regions critical for the driving task.

Ines Trautmannsheimer, Richard Grauberger, Frank Diermeyer

Chair of Automotive Technology Technical University of Munich Munich, Germany

Robot Teleoperation Video Quality Assessment

Ask PDF

Mar 6, 2026

Nonlinear Performance Degradation of Vision-Based Teleoperation under Network Latency

Teleoperation is increasingly being adopted as a critical fallback for autonomous vehicles. However, the impact of network latency on vision-based, perception-driven control remains insufficiently studied. The present work investigates the nonlinear degradation of closed-loop stability in camera-based lane keeping under varying network delays. To conduct this study, we developed the Latency-Aware Vision Teleoperation testbed (LAVT), a research-oriented ROS 2 framework that enables precise, distributed one-way latency measurement and reproducible delay injection. Using LAVT, we performed 180 closed-loop experiments in simulation across diverse road geometries. Our findings reveal a sharp collapse in stability between 150 ms and 225 ms of one-way perception latency, where route completion rates drop from 100% to below 50% as oscillatory instability and phase-lag effects emerge. We further demonstrate that additional control-channel delay compounds these effects, significantly accelerating system failure even under constant visual latency. By combining this systematic empirical characterization with the LAVT testbed, this work provides quantitative insights into perception-driven instability and establishes a reproducible baseline for future latency-compensation and predictive control strategies. Project page, supplementary video, and code are available at https://bimilab.github.io/paper-LAVT

Aws Khalil, Jaerock Kwon

Department of Electrical and Computer Engineering, University of Michigan - Dearborn, Dearborn, MI, USA.

Robot Teleoperation Vision-Based Robot Control Feedback Control

Ask PDF

May 29, 2026

Can BEV Perception Gracefully Degrade under Sensor Failures?

Despite the remarkable success of multi-modal bird's-eye view (BEV) perception in autonomous driving, current systems exhibit a critical vulnerability: existing fusion mechanisms are highly brittle to sensor corruptions, often causing catastrophic performance degradation. This vulnerability largely stems from the fact that standard fusion frameworks typically integrate multi-modal representations in a static manner, leading to a precipitous performance collapse under missing or corrupted modalities. In contrast, we show that graceful degradation is achievable through active modality reliability assessment. To this end, we present Grace-BEV, a lightweight and plug-and-play framework that enforces active reliability awareness during multi-modal fusion. Instead of relying on computationally expensive cross-modal interactions, Grace-BEV leverages the aligned BEV space to explicitly assess modality trustworthiness via a TrustGate Router and dynamically recalibrate feature integration using the FailSafe Fusion Block. Furthermore, we devise a Three-Phase Training strategy with Modality Dropout to prevent modality dominance and encourage balanced cross-modal learning under unreliable inputs. Extensive experiments on nuScenes-R and nuScenes-C show that Grace-BEV maintains robust performance across diverse corruption settings. Notably, under catastrophic LiDAR failures where standard baselines collapse to 0.0% mean Average Precision (mAP), Grace-BEV restores performance to as high as 34.7% mAP. Moreover, it improves clean accuracy by up to 1.4%, achieving a strong trade-off between robustness and efficiency.

Haifa Zhang, Yijing Wang, Haoyu Wang +2

Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China

Missing-Modality Learning Multimodal Robustness Camera-LiDAR Fusion Autonomous Driving Perception Bird's-Eye-View Representation

Ask PDF

Abstract

Robot Teleoperation

Explore similar work

Beyond VMAF: Towards Application-Specific Metrics for Teleoperation Video

Nonlinear Performance Degradation of Vision-Based Teleoperation under Network Latency

Can BEV Perception Gracefully Degrade under Sensor Failures?