Single-Controller Teleoperation and Multimodal Dataset Generation for a Mobile Manipulator
Authors: Fardien Azizi, Sitor Sla, Christiaan Peppelman, Dion van Santen, Andrei-Carlo Papuc, Khaled Mustafa, Diego Martinez-Baselga, Javier Alonso-Mora
Organizations: The authors are with the Dept. of Cognitive Robotics, TU Delft, 2628 CD Delft, The Netherlands. · The authors are with RoPeRt Group, DIIS-I3A, University of Zaragoza, 50018 Zaragoza, Spain.
Robot learning systems require synchronized, high-quality demonstrations that combine visual, proprioceptive, and action data. However, collecting such datasets with mobile manipulators is often slowed down by fragmented teleoperation interfaces and manual post-processing workflows. This paper presents an integrated framework for teleoperated multimodal data collection on a Clearpath Dingo-O mobile base equipped with a Kinova Gen3 Lite manipulator and an Intel RealSense RGB-D camera. The framework combines single-controller operation of the base, arm, and recording process with an automated ROS-based pipeline for data capture, image-quality filtering, temporal synchronization, visualization, and export to machine-learning-ready Zarr datasets. We validate the system on a pick-and-place mobile-manipulation task in which the robot transports a bottle between two surfaces. The results show that the unified controller supports complete task execution and recording without external commands, while the data pipeline produces synchronized datasets for short demonstrations and identifies corrupted visual frames. The proposed framework provides a practical basis for producing structured demonstrations for robot-learning research.
Figures & tables
Fig. 1: Mobile manipulator interacting with a bottle.
Fig. 2: A schematic overview of the different modes implemented in the controller.
Fig. 3: A series of poses of the Kinova arm are shown in the MuJoCo viewer with x, y, and z axes on the end effector.
Fig. 4: Controller mappings for the forward kinematics and inverse kinematics modes. The visualization layout was generated using ChatGPT (OpenAI).
Fig. 5: General schematic of the data framework, consisting of data collection, processing and export.
Fig. 6: Experimental sequence of the Dingo-O robot executing the pick-and-place manipulation task.
Fig. 7: Examples of overly dark visual frames rejected by the quality filter.
Fig. 8: Comparison of RViz and Foxglove Studio for visual inspection of the synchronized demonstration.