Vision-Based Robot Control

Overview

The field studies how robots can use camera images to choose and revise actions, allowing them to operate when precise maps or external tracking are unavailable. Approaches range from image-based visual servoing and model-predictive control to imitation learning, reinforcement learning, and vision-language-action or world-model policies. These methods are tested in tasks such as object manipulation, legged locomotion, aerial navigation, and surgical camera control, using simulation benchmarks and real-robot trials. A central challenge is making policies transfer across cameras, robots, and environments while remaining responsive and safe under occlusion, latency, and changing conditions.

Latest papers

CardsList

241 papers25 papers in the last four weeks, down 49% on the four weeks before. 0.2% of all new papers.

  1. RoboTeam: Benchmarking Vision-Based Multi-Agent Control of Many Robot Arms in Real Time

    Nov 17, 2026Changxuan Fan, Zhongwei Xie, Boyi Zhang +18Robot Manipulation BenchmarksMulti-Agent Control

  2. Residual Reinforcement Learning for Visual Servo Control with Zero-Shot Sim-to-Real Transfer

    Nov 14, 2026Mahdi Nobar, Alisa Rupenyan, John Lygeros +1Robot Trajectory TrackingVision-Based Robot Control

  3. Self-Calibrating Vision-Agent-Action: Deploying Pick-and-Place to New Robot Cells without Demonstrations or Fine-Tuning

    Nov 14, 2026Jae Hun Hwang, Yechan Seo, Hong-In WonVision-Based Robot ControlZero-Shot Robotic Manipulation

  4. PROOF: Perceptive Robustness via Out-of-Distribution Fallback for Legged Locomotion

    Nov 13, 2026Siyuan Wang, Dongjie Zhu, Jinxuan Lu +2Vision-Based Robot ControlQuadruped Locomotion

  5. Embedded Sim-to-Real Visual Servoing: Learned Reaching and Force-Guided Placement

    Nov 11, 2026Dezhi Sun, Duy Khang Banh, Kenan NiuVision-Based Robot ControlReal-Time Robot Control

  6. Joint-State Inference for Autonomous Excavators Using Cab-View Vision-Control Fusion

    Nov 10, 2026Junwei Li, Jian Pan, Mengchi Cai +6Robotic ExcavationRobot State Estimation

  7. SPARK: Simple Post-training for Adapting pRetrained Knowledge to Robot Control

    Nov 9, 2026Taekyung Kim, Jeongeun Park, Byeongho Heo +3Vision-Based Robot ControlRobot Foundation Models

  8. Tutorial: Vision-Based Formation Control with Differential-Drive Robots

    Nov 8, 2026Christian Marowski, Byung-Cheol MinMulti-Robot Formation ControlMulti-Robot Coordination

  9. Control-Aware Command Generation for Asynchronous Control and Action Inference with Vision-Based Policies towards Fast Execution

    Nov 8, 2026Takayuki Osa, Kazuki OtaRobotic ControlVision-Based Robot Control

  10. Real-Time Event-Based LED Tracking and Pose Estimation with PWM-Encoded Markers

    Nov 5, 2026Junkai Niu, Sheng Zhong, Delei Kong +5Relative Pose EstimationEvent-Based Vision

  11. Dynamic Watermarking and Action-Conditioned Latent Dynamics for Secure Vision-Based Controllers

    Nov 5, 2026Aditya Parameshwaran, Navid Aftabi, Abhishek Hanchate +3Robust WatermarkingVision-Based Robot Control

  12. Adaptive Hybrid Force-Vision Control with EMG-Derived References for Deformable Bottle Manipulation

    Nov 4, 2026Jeffrey Lee, Geoffrey Hanks, Evangeline Bouganim +1Robotic ControlForce-Controlled Robotic Manipulation

  13. Teleoperation System for Service Robots using Virtual Reality Headset and Pose Estimation

    Nov 1, 2026Eduardo Fernandes, Tiago Ribeiro, Antonio Ribeiro +3Robot TeleoperationHuman-Robot Interaction

  14. Learning Agile Humanoid Dodgeball with Onboard Monocular Vision

    Oct 31, 2026Feng Tian, Yulin An, Chenhao Li +2Humanoid Robot ControlVision-Based Robot Control

  15. Gaze-Assisted Supervisory Control for Lunar Robotic Arm Teleoperation Under Concurrent Monitoring

    Oct 29, 2026Jiayu He, Yifan Li, Oliver R. Runswick +4Lunar RoboticsRobot Teleoperation

  16. Predict the Swing, Match the Speed: Robotic Pipe Stabbing With Learned Stereo Vision and Residual Reinforcement Learning

    Oct 27, 2026Mohamed Ashraf Mostafa, Taher El-Fakharany, Samir KhaledDynamic Robotic ManipulationVision-Based Robot Control

  17. TAPESTRY: A Single Backbone for Video Generation and Robot Control

    Oct 27, 2026Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn +4Vision-Based Robot ControlDiffusion Policy

  18. VisGeo-PipeTrack: A Visual Geometric Servo Control Framework with MPC-RCAPID Control for UAV-Based Dynamic Pipes Tracking

    Oct 26, 2026Zhenyang Gao, Jianlin Chen, Yang Yu +3Quadrotor ControlVision-Based Robot Control

  19. X-World: Transferring Video Priors to Variable-View Robot Control

    Oct 25, 2026Chun Gu, Xiaofei Wei, Yiming Chen +2Robotic ControlVision-Based Robot Control

  20. FocusVLA: Future-Intent Guided Dynamic View Routing for Vision-Language-Action Models

    Oct 22, 2026Huixin Xu, Ziya Yu, Peng LiVision-Language-Action ModelsVision-Based Robot Control

  21. Revisiting Distribution Matching for State-to-Vision Policy Distillation in Laparoscopic Robotic Manipulation

    Oct 22, 2026Mingyu Sheng, Jianan Fan, Dongnan Liu +3Robotic ManipulationVision-Based Robot Control

  22. What Novice Operators See Changes What the Policy Learns: A User Study of Keypoint Hand Visual Representation for Teleoperated Demonstration

    Oct 20, 2026Amane Yamaguchi, Shohei Kamiguchi, Masashi Hamaya +3Robot Imitation LearningLearning from Demonstration

  23. EndoVLA-OOV: Language-Conditioned Out-of-View Target Acquisition in Robotic Endoscopy

    Oct 17, 2026Chi Kit Ng, Yidong Zhang, Tianchun Wu +9Robotic EndoscopyVision-Based Robot Control

  24. GR-VIP: Geometric representation conditioning for precise visuomotor policy

    Oct 16, 2026Chanhwi Lee, Jiwon Seong, Eunje Choi +6Robotic ControlVisuomotor Policy Learning

  25. PASS-VLN: Passability-Aware Local-Command Vision-and-Language Navigation

    Oct 16, 2026Cong Zou, Xiangsheng Shi, Xuanhui Wang +4Robot NavigationVision-Language Navigation