Vision-Based Robot Control

Ask about this topic

Overview

The field studies how robots can use camera images to choose and revise actions, allowing them to operate when precise maps or external tracking are unavailable. Approaches range from image-based visual servoing and model-predictive control to imitation learning, reinforcement learning, and vision-language-action or world-model policies. These methods are tested in tasks such as object manipulation, legged locomotion, aerial navigation, and surgical camera control, using simulation benchmarks and real-robot trials. A central challenge is making policies transfer across cameras, robots, and environments while remaining responsive and safe under occlusion, latency, and changing conditions.

Latest papers

CardsList

222 papers38 papers in the last four weeks, up 12% on the four weeks before. 0.2% of all new papers.

  1. Bio-Inspired Event-Based Visual Servoing for Ground Robots

    Mar 24, 2026Maral Mordad, Kian Behzad, Debojyoti Biswas +2Vision-Based Robot ControlEvent-Based Vision

    Department of Electrical & Computer Engineering, Northeastern University, Boston, MA 02115 USA · Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA · Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218 USA

  2. Strain-Parameterized Coupled Dynamics and Dual-Camera Visual Servoing for Aerial Continuum Manipulators

    Mar 24, 2026Niloufar Amiri, Farrokh Janabi-SharifiAerial ManipulationAerial Robotics

    Department of Mechanical, Industrial, and Mechatronics Engineering, Toronto Metropolitan University, ON, Canada.

  3. Agile-VLA: Few-Shot Industrial Pose Rectification via Implicit Affordance Anchoring

    Mar 24, 2026Teng Yan, Zhengyang Pei, Chengyu Shi +8Affordance-Based Robotic ManipulationRobotic Manipulation

    The Hong Kong University of Science and Technology (Guangzhou)

  4. PhotoAgent: A Robotic Photographer with Spatial and Aesthetic Understanding

    Mar 24, 2026Lirong Che, Zhenfeng Gan, Yanbo Chen +2Multimodal AgentsVision-Based Robot Control

    Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China

  5. HyReach: Vision-Guided Hybrid Manipulator Reaching in Unseen Cluttered Environments

    Mar 22, 2026Shivani Kamtikar, Kendall Koe, Justin Wasserman +5Robotic ManipulationContinuum Robot Control

    University of Illinois at Urbana-Champaign · Skild AI

  6. Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation

    Mar 22, 2026Xuening Zhang, Qi Lv, Xiang Deng +3Vision-Based Robot ControlRobotic Manipulation

    Harbin Institute of Technology (Shenzhen), Shenzhen, Guangdong 518055, China · Shandong Jianzhu University, Jinan, Shandong 250101, China

  7. Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

    Mar 21, 2026Kai Li, Shiyu ZhaoVision-Based Robot ControlMobile Robotics

    College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.

  8. CeRLP: A Cross-embodiment Robot Local Planning Framework for Visual Navigation

    Mar 20, 2026Haoyu Xi, Mingao Tan, Xinming Zhang +5Visual Robot NavigationRobot Navigation

    School of Computer Science, Shanghai Jiao Tong University, Shanghai, P. R. China · College of Information Science and Technology, Eastern Institute of Technology, Ningbo, P. R. China

  9. ReMAP-DP: Reprojected Multi-view Aligned PointMaps for Diffusion Policy

    Mar 16, 2026Xinzhang Yang, Renjun Wu, Jinyan Liu +1Vision-Based Robot ControlDiffusion Policy

    School of Computer Science, Beijing Institute of Technology, China.

  10. CyboRacket: A Perception-to-Action Framework for Humanoid Racket Sports

    Mar 15, 2026Peng Ren, Chuan Qi, Haoyang Ge +6Vision-Based Robot ControlHumanoid Robot Control

    BUAA · BZA · USTC +3

  11. VIP-Loco: A Visually Guided Infinite Horizon Planning Framework for Legged Locomotion

    Mar 15, 2026Aditya Shirwatkar, Satyam Gupta, Shishir KolathayaTerrain-Aware Robot LocomotionVision-Based Robot Control

    Robert Bosch Center for Cyber-Physical Systems, Indian Institute of Science, Bengaluru. · Robert Bosch Center for Cyber-Physical Systems and the Department of Computer Science & Automation, Indian Institute of Science, Bengaluru.

  12. Stiffness Copilot: An Impedance Policy for Contact-Rich Teleoperation

    Mar 14, 2026Yeping Wang, Zhengtong Xu, Pornthep Preechayasomboon +4Robot TeleoperationContact-Rich Robotic Manipulation

    All authors are with Meta Reality Labs Research · Yeping Wang is with University of Wisconsin-Madison, Madison, WI, USA · Zhengtong Xu is with Purdue University, West Lafayette, IN, USA

  13. KoopmanFlow: Spectrally Decoupled Generative Control Policy via Koopman Structural Bias

    Mar 14, 2026Chengsi Yao, Ge Wang, Kai Kang +9Robotic ManipulationFlow-Based Generative Modeling

    ISing AI · Future Network of Intelligence Institution, The Chinese University of Hong Kong, Shenzhen · Guangdong Polytechnic Normal University +1

  14. Learning Visuomotor Policy for Multi-Robot Laser Tag Game

    Mar 12, 2026Kai Li, Shiyu ZhaoVisuomotor Policy LearningMulti-Robot Systems

    College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.

  15. Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics

    Mar 11, 2026Hendrik Chiche, Antoine Jamme, Trevor Rigoberto Martinez +1Motion RetargetingRobotic Manipulation

    OMGrab Inc. and affiliated with the University of California, Berkeley, CA 94720 USA, through the Capstone Project program of the Fung Institute for Engineering Leadership · Department of Mechanical Engineering, University of California, Berkeley, CA 94720 USA

  16. DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control

    Mar 11, 2026Teli Ma, Jia Zheng, Zifan Wang +4Robotic ControlVision-Based Robot Control

    Mondo Robotics · HKUST(GZ) · HKUST

  17. Adaptive Vision-Based Control of Redundant Robots with Null-Space Interaction for Human-Robot Collaboration

    Mar 9, 2026Xiangjie Yan, Chen Chen, Xiang LiRobotic ControlHuman-Robot Collaboration

    Department of Automation, Tsinghua University

  18. See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming

    Mar 9, 2026Petr Vanc, Jan Kristof Behrens, Václav Hlaváč +1Robot Skill LearningRobotic Manipulation

    imitrob.ciirc.cvut.cz/publications/seeandswitch

  19. Long-Short Term Agents for Pure-Vision Bronchoscopy Robotic Autonomy

    Mar 9, 2026Junyang Wu, Mingyi Luo, Fangfang Xie +7Robot NavigationRobotic Endoscopy

    Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, 200240, China. · Shanghai Chest Hospital, Shanghai, 10587, China.

  20. GeoLoco: Leveraging 3D Geometric Priors from Visual Foundation Model for Robust RGB-Only Humanoid Locomotion

    Mar 8, 2026Yufei Liu, Xieyuanli Chen, Hainan Pan +5Humanoid Robot LocomotionVision-Based Robot Control

    College of Intelligence Science and Technology, National University of Defense Technology, China

  21. Vision-Guided MPPI for Agile Drone Racing: Navigating Arbitrary Gate Poses via Neural Signed Distance Fields

    Mar 7, 2026Fangguo Zhao, Hanbing Zhang, Zhouheng Li +2Autonomous Drone RacingAerial Robotics

    College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China

  22. Nonlinear Performance Degradation of Vision-Based Teleoperation under Network Latency

    Mar 6, 2026Aws Khalil, Jaerock KwonRobot TeleoperationVision-Based Robot Control

    Department of Electrical and Computer Engineering, University of Michigan - Dearborn, Dearborn, MI, USA.

  23. X-Loco: Towards Generalist Humanoid Locomotion Control via Synergetic Policy Distillation

    Mar 4, 2026Dewei Wang, Xinmiao Wang, Chenyun Zhang +4Humanoid Robot LocomotionRobot Policy Learning

    University of Science and Technology of China · Institute of Artificial Intelligence (TeleAI), China Telecom · Harbin Engineering University

  24. Look Forward to Walk Backward: Efficient Terrain Memory for Backward Locomotion with Forward Vision

    Mar 3, 2026Shixin Luo, Songbo Li, Yuan Hao +4Robot LocomotionLegged Robotics

    The authors are with Institute of Cyber-Systems and Control, Zhejiang University, 310027, China. · The author is with College of Artificial Intelligence, Nankai University, 300350, China. · The author is with Hangzhou Public Library, 310016, China.

  25. Learning Vision-Based Omnidirectional Navigation: A Teacher-Student Approach Using Monocular Depth Estimation

    Mar 2, 2026Jan Finke, Wayne Paul Martis, Adrian Schmelter +3Robot NavigationVision-Based Robot Control

  26. Shape-Interpretable Visual Self-Modeling Enables Geometry-Aware Continuum Robot Control

    Mar 2, 2026Peng Yu, Xin Wang, Ning TanContinuum Robot ControlVision-Based Robot Control

    School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China.

  27. HierKick: Hierarchical Reinforcement Learning for Vision-Guided Soccer Robot Control

    Mar 1, 2026Yizhi Chen, Zheng Zhang, Zhanxiang Cao +7Robotic ControlRobotic RL

    Tongji University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Shanghai Jiao Tong University, Shanghai, China. +2

  28. Curriculum Reinforcement Learning for Quadrotor Racing with Random Obstacles

    Feb 27, 2026Fangyu Sun, Fanxing Li, Yu Hu +4Autonomous Drone RacingAutonomous Racing

    School of Automation and Perception, Shanghai Jiao Tong University, China · Faculty of Aerospace Engineering, Delft University of Technology, The Netherlands

  29. VCA: Vision-Click-Action Framework for Precise Manipulation of Segmented Objects in Target Ambiguous Environments

    Feb 27, 2026Donggeon Kim, Seungwon Jan, Hyeonjun Park +1Robotic ManipulationVision-Based Robot Control

    ROBROS Inc., Seoul, KS013, Republic of Korea

  30. Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera

    Feb 26, 2026Seongyong Kim, Junhyeon Cho, Kang-Won Lee +1Robotic ManipulationDynamic Robotic Manipulation

    Dongguk University