Robotic Perception

Follow this topic Ask about this topic

Robotic perception studies how robots use sensory data to estimate what is around them, how it is arranged, and how it may change. Approaches include neural networks and vision-language models for recognition and segmentation; SLAM, depth estimation, and 3D maps for geometry; and fusion of cameras with LiDAR, radar, tactile, or proprioceptive sensors. Active viewpoint selection and interaction help resolve occlusion and ambiguity, while scene graphs and affordance models connect perception to navigation and manipulation. A central challenge is making systems reliable outside curated benchmarks—under changing lighting, clutter, unfamiliar environments, and tight limits on latency and onboard computing—while representing uncertainty well enough for safe action.

Latest papers 304

64 papers in the last four weeks, up 191% on the four weeks before. 0.4% of all new papers.

Mar 4, 2026

AMP2026: A Multi-Platform Marine Robotics Dataset for Tracking and Mapping

Marine environments present significant challenges for perception and autonomy due to dynamic surfaces, limited visibility, and complex interactions between aerial, surface, and submerged sensing modalities. This paper introduces the Aerial Marine Perception Dataset (AMP2026), a multi-platform marine robotics dataset collected across multiple field deployments designed to support research in two primary areas: multi-view tracking and marine environment mapping. The dataset includes synchronized data from aerial drones, boat-mounted cameras, and submerged robotic platforms, along with associated localization and telemetry information. The goal of this work is to provide a publicly available dataset enabling research in marine perception and multi-robot observation scenarios. This paper describes the data collection methodology, sensor configurations, dataset organization, and intended research tasks supported by the dataset.

Edwin Meriaux, Shuo Wen, David Widhalm +11

McGill University, Montreal, Canada. · MILA - Quebec AI Institute. · Universit´e Paris-Saclay, CentraleSup´elec. +4

Robotic Data Collection Robotic Perception

Mar 4, 2026

One-Shot Badminton Shuttle Detection for Mobile Robots

This paper presents a robust one-shot badminton shuttlecock detection framework for non-stationary robots. To address the lack of egocentric shuttlecock detection datasets, we introduce a dataset of 20,510 semi-automatically annotated frames captured across 11 distinct backgrounds in diverse indoor and outdoor environments, and categorize each frame into one of three difficulty levels. For labeling, we present a novel semi-automatic annotation pipeline, that enables efficient labeling from stationary camera footage. We propose a metric suited to our downstream use case and fine-tune a YOLOv8 network optimized for real-time shuttlecock detection, achieving an F1-score of 0.86 under our metric in test environments similar to training, and 0.70 in entirely unseen environments. Our analysis reveals that detection performance is critically dependent on shuttlecock size and background texture complexity. Qualitative experiments confirm their applicability to robots with moving cameras. Unlike prior work with stationary camera setups, our detector is specifically designed for the egocentric, dynamic viewpoints of mobile robots, providing a foundational building block for downstream tasks, including tracking, trajectory estimation, and system (re)-initialization.

Florentin Dipner, William Talbot, Turcan Tuna +2

Authors are with Robotic Systems Lab (RSL), ETH Zurich, Zurich 8092, Switzerland.

Object Detection Robotic Perception Real-Time Object Detection Small Object Detection

Mar 4, 2026

DISC: Dense Integrated Semantic Context for Large-Scale Open-Set Semantic Mapping

Open-set semantic mapping enables language-driven robotic perception, but current instance-centric approaches are bottlenecked by context-depriving and computationally expensive crop-based feature extraction. To overcome this fundamental limitation, we introduce DISC (Dense Integrated Semantic Context), featuring a novel single-pass, distance-weighted extraction mechanism. By deriving high-fidelity CLIP embeddings directly from the vision transformer's intermediate layers, our approach eliminates the latency and domain-shift artifacts of traditional image cropping, yielding pure, mask-aligned semantic representations. To fully leverage these features in large-scale continuous mapping, DISC is built upon a fully GPU-accelerated architecture that replaces periodic offline processing with precise, on-the-fly voxel-level instance refinement. We evaluate our approach on standard benchmarks (Replica, ScanNet) and a newly generated large-scale-mapping dataset based on Habitat-Matterport 3D (HM3DSEM) to assess scalability across complex scenes in multi-story buildings. Extensive evaluations demonstrate that DISC significantly surpasses current state-of-the-art zero-shot methods in both semantic accuracy and query retrieval, providing a robust, real-time capable framework for robotic deployment. The full source code, data generation and evaluation pipelines will be made available at https://github.com/DFKI-NI/DISC.

Felix Igelbrink, Lennart Niecksch, Martin Atzmueller +1

the German Research Center for Artificial Intelligence (DFKI), Osnabrück, Germany · the Department of Computer Sciene, Osnabrück University, Osnabrück, Germany

Semantic Mapping Robotic Perception Open-Vocabulary 3D Segmentation VLMs for Robotics Semantic Occupancy Mapping

Mar 3, 2026

From Local Matches to Global Masks: Template-Guided Instance Detection and Segmentation in Open-World Scenes

Detecting and segmenting novel object instances in open-world environments is a fundamental problem in robotic perception. Given only a small set of template images, a robot must locate and segment a specific object instance in a cluttered, previously unseen scene. Existing proposal-based approaches are highly sensitive to proposal quality and often fail under occlusion and background clutter. We propose L2G-Det, a local-to-global instance detection framework that bypasses explicit object proposals by leveraging dense patch-level matching between templates and the query image. Locally matched patches generate candidate points, which are refined through a candidate selection module to suppress false positives. The filtered points are then used to prompt an augmented Segment Anything Model (SAM) with instance-specific object tokens, enabling reliable reconstruction of complete instance masks. Experiments demonstrate improved performance over proposal-based methods in challenging open-world settings.

Qifan Zhang, Sai Haneesh Allu, Jikai Wang +2

Intelligent Robotics and Vision Lab, The University of Texas at Dallas

Instance Segmentation Object Detection Robotic Perception

Mar 2, 2026

DINOv3 Visual Representations for Blueberry Perception Toward Robotic Harvesting

Vision Foundation Models trained via large-scale self-supervised learning have demonstrated strong generalization in visual perception; however, their practical role and performance limits in agricultural settings remain insufficiently understood. This work evaluates DINOv3 as a frozen backbone for blueberry robotic harvesting-related visual tasks, including fruit and bruise segmentation, as well as fruit and cluster detection. Under a unified protocol with lightweight decoders, segmentation benefits consistently from stable patch-level representations and scales with backbone size. In contrast, detection is constrained by target scale variation, patch discretization, and localization compatibility. The failure of cluster detection highlights limitations in modeling relational targets defined by spatial aggregation. Overall, DINOv3 is best viewed not as an end-to-end task model, but as a semantic backbone whose effectiveness depends on downstream spatial modeling aligned with fruit-scale and aggregation structures, providing guidance for blueberry robotic harvesting. Code and dataset will be available upon acceptance.

Rui-Feng Wang, Daniel Petti, Yue Chen +1

Bio-Sensing, Automation, and Intelligence Laboratory, Department of Agricultural and Biological Engineering, Institute of Food and Agricultural Sciences, University of Florida, Gainesville, FL 32611, USA · Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA

Robotic Perception Agricultural Robotics Semantic Segmentation Self-Supervised Visual Representation Learning

Mar 1, 2026

Pro-HOI: Perceptive Root-guided Humanoid-Object Interaction

Executing reliable Humanoid-Object Interaction (HOI) tasks for humanoid robots is hindered by the lack of generalized control interfaces and robust closed-loop perception mechanisms. In this work, we introduce Perceptive Root-guided Humanoid-Object Interaction, Pro-HOI, a generalizable framework for robust humanoid loco-manipulation. First, we collect box-carrying motions that are suitable for real-world deployment and optimize penetration artifacts through a Signed Distance Field loss. Second, we propose a novel training framework that conditions the policy on a desired root-trajectory while utilizing reference motion exclusively as a reward. This design not only eliminates the need for intricate reward tuning but also establishes root trajectory as a universal interface for high-level planners, enabling simultaneous navigation and loco-manipulation. Furthermore, to ensure operational reliability, we incorporate a persistent object estimation module. By fusing real-time detection with Digital Twin, this module allows the robot to autonomously detect slippage and trigger re-grasping maneuvers. Empirical validation on a Unitree G1 robot demonstrates that Pro-HOI significantly outperforms baselines in generalization and robustness, achieving reliable long-horizon execution in complex real-world scenarios.

Yuhang Lin, Jiyuan Shi, Dewei Wang +4

Institute of Artificial Intelligence (TeleAI), China Telecom · Zhejiang University · University of Science and Technology of China +1

Robotic Control Humanoid Loco-Manipulation Human-Object Interaction Generalization in Robotic Manipulation Long-Horizon Robotic Manipulation Robotic Perception

Feb 26, 2026

UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perception

We present UniScale, a unified, scale-aware multi-view 3D reconstruction framework for robotic applications that flexibly integrates geometric priors through a modular, semantically informed design. In vision-based robotic navigation, the accurate extraction of environmental structure from raw image sequences is critical for downstream tasks. UniScale addresses this challenge with a single feed-forward network that jointly estimates camera intrinsics and extrinsics, scale-invariant depth and point maps, and the metric scale of a scene from multi-view images, while optionally incorporating auxiliary geometric priors when available. By combining global contextual reasoning with camera-aware feature representations, UniScale is able to recover the metric-scale of the scene. In robotic settings where camera intrinsics are known, they can be easily incorporated to improve performance, with additional gains obtained when camera poses are also available. This co-design enables robust, metric-aware 3D reconstruction within a single unified model. Importantly, UniScale does not require training from scratch, and leverages world priors exhibited in pre-existing models without geometric encoding strategies, making it particularly suitable for resource-constrained robotic teams. We evaluate UniScale on multiple benchmarks, demonstrating strong generalization and consistent performance across diverse environments. We will release our implementation upon acceptance.

Mohammad Mahdavian, Gordon Tan, Binbin Xu +3

Huawei Noah’s Ark Lab · University of Toronto

3D Reconstruction Multi-View 3D Reconstruction Feed-Forward 3D Reconstruction Robotic Perception Camera Pose Estimation

Feb 26, 2026

InCoM: Intent-Driven Perception and Structured Coordination for Mobile Manipulation

Mobile manipulation is a fundamental capability for general-purpose robotic agents, requiring both coordinated control of the mobile base and manipulator and robust perception under dynamically changing viewpoints. However, existing approaches face two key challenges: strong coupling between base and arm actions complicates control optimization, and perceptual attention is often poorly allocated as viewpoints shift during mobile manipulation. We propose InCoM, an intent-driven perception and structured coordination framework for mobile manipulation. InCoM infers latent motion intent to dynamically reweight multi-scale perceptual features, enabling stage-adaptive allocation of perceptual attention. To support robust cross-modal perception, InCoM further incorporates a geometric-semantic affinity refinement mechanism that enhances multimodal correspondence. On the control side, we design a decoupled coordinated flow matching action decoder that explicitly models bidirectional base-arm interactions, alleviating optimization difficulties caused by control coupling. Under matched non-privileged observations and task-specific training data, InCoM improves mean success over DSPv2 by 33.0, 26.1, and 23.6 percentage points on SetTable, TidyHouse, and PrepareGroceries, respectively. Furthermore, its effectiveness is consistently validated in real-world mobile manipulation tasks, where InCoM maintains a superior success rate over existing baselines.

Jiahao Liu, Cui Wenbo, Zhongpu Xia +3

SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Anyverse Dynamics

Active Perception Mobile Manipulation Robotic Perception

Feb 25, 2026

A Pairwise Human-Human Interaction Detection and Recognition Framework for Mobile Service Robots

Autonomous mobile service robots, such as lawnmowers or cleaning robots, operating in human-populated environments need to reason about human-human interactions to support safe and socially aware navigation. For such systems, interaction understanding is not primarily a fine-grained recognition problem, but a perception problem under limited sensing quality and computational resources. Many existing approaches focus on holistic group activity recognition, often relying on complex and computationally expensive models that are not well suited for mobile robotic platforms. In this work, we argue that pairwise human interactions constitute a minimal yet sufficient perceptual unit for robot-centric social understanding. We study the problem of identifying interacting person pairs and classifying coarse-grained interaction behaviors sufficient for downstream group-level reasoning and robot decision-making. To this end, we adopt a two-stage framework in which candidate interacting pairs are first identified using lightweight geometric and motion cues, and interaction types are subsequently classified using a relation network. We evaluate the proposed approach on the JRDB dataset, where it achieves competitive performance with reduced computational cost and model size compared to appearance-based methods. Additional experiments on the Collective Activity Dataset (CAD) and zero-shot evaluation on a lawnmower-collected dataset further demonstrate the generalizability of the proposed framework. These results suggest that simple geometric and motion cues provide a practical and efficient basis for interaction-aware perception in mobile service robots. Code is released.

Mengyu Liang, Iolanda Leite, Sarah Gillet

KTH Royal Institute of Technology · MIT Media Lab

Human Activity Recognition Mobile Robotics Robotic Perception

Feb 25, 2026

Multiview Progress Prediction of Robot Activities

For robots to operate effectively and safely alongside humans, they must be able to understand the progress of ongoing actions. This ability, known as action progress prediction, is critical for tasks ranging from timely assistance to autonomous decision-making. However, modeling action progression in robotics has often been overlooked. Moreover, a single camera may be insufficient for understanding robot's ego-actions, as self-occlusion can significantly hinder perception and model performance. In this paper, we propose a multi-view architecture for action progress prediction in robot manipulation tasks. Experiments on Mobile ALOHA demonstrate the effectiveness of the proposed approach.

Elena Zoppellari, Federico Becattini, Marco Fiorucci +1

University of Padova, Italy · University of Siena, Italy

Robotic Perception Robotic Manipulation

Feb 20, 2026

Ori-Sense: origami capacitive sensing for soft robotic applications

This work introduces Ori-Sense, a compliant capacitive sensor inspired by the inverted Kresling origami pattern. The device translates torsional deformation into measurable capacitance changes, enabling proprioceptive feedback for soft robotic systems. Using dissolvable-core molding, we fabricated a monolithic silicone structure with embedded conductive TPU electrodes, forming an integrated soft capacitor. Mechanical characterization revealed low stiffness and minimal impedance, with torque values below 0.01 N mm for axial displacements between -15 mm and 15 mm, and up to 0.03 N mm at 30 degrees twist under compression. Finite-element simulations confirmed localized stresses along fold lines and validated the measured torque-rotation response. Electrical tests showed consistent capacitance modulation up to 30%, directly correlated with the twist angle, and maximal sensitivity of S_theta ~ 0.0067 pF/deg at 5 mm of axial deformation.

Hugo de Souza Oliveira, Xin Li, Mohsen Jafarpour +1

Soft Robotics Robotic Perception

Feb 20, 2026

Zero-shot Interactive Perception

Interactive perception (IP) enables robots to extract hidden information in their workspace and execute manipulation plans by physically interacting with objects and altering the state of the environment -- crucial for resolving occlusions and ambiguity in complex, partially observable scenarios. We present Zero-Shot IP (ZS-IP), a novel framework that couples multi-strategy manipulation (pushing and grasping) with a memory-driven Vision Language Model (VLM) to guide robotic interactions and resolve semantic queries. ZS-IP integrates three key components: (1) an Enhanced Observation (EO) module that augments the VLM's visual perception with both conventional keypoints and our proposed pushlines -- a novel 2D visual augmentation tailored to pushing actions, (2) a memory-guided action module that reinforces semantic reasoning through context lookup, and (3) a robotic controller that executes pushing, pulling, or grasping based on VLM output. Unlike grid-based augmentations optimized for pick-and-place, pushlines capture affordances for contact-rich actions, substantially improving pushing performance. We evaluate ZS-IP on a 7-DOF Franka Panda arm across diverse scenes with varying occlusions and task complexities. Our experiments demonstrate that ZS-IP outperforms passive and viewpoint-based perception techniques such as Mark-Based Visual Prompting (MOKA), particularly in pushing tasks, while preserving the integrity of non-target elements.

Venkatesh Sripada, Frank Guerin, Amir Ghalamzan

University of Surrey · University of Sheffield

Robotic Perception Zero-Shot Robotic Manipulation Contact-Rich Robotic Manipulation VLMs for Robotics

Feb 20, 2026

GrandTour: A Legged Robotics Dataset in the Wild for Multi-Modal Perception and State Estimation

Accurate state estimation and multi-modal perception are prerequisites for autonomous legged robots in complex, large-scale environments. To date, no large-scale public legged-robot dataset captures the real-world conditions needed to develop and benchmark algorithms for legged-robot state estimation, perception, and navigation. To address this, we introduce the GrandTour dataset, a multi-modal legged-robotics dataset collected across challenging outdoor and indoor environments, featuring an ANYbotics ANYmal-D quadruped equipped with the Boxi multi-modal sensor payload. GrandTour spans a broad range of environments and operational scenarios across distinct test sites, ranging from alpine scenery and forests to demolished buildings and urban areas, and covers a wide variation in scale, complexity, illumination, and weather conditions. The dataset provides time-synchronized sensor data from spinning LiDARs, multiple RGB cameras with complementary characteristics, proprioceptive sensors, and stereo depth cameras. Moreover, it includes high-precision ground-truth trajectories from satellite-based RTK-GNSS and a Leica Geosystems total station. This dataset supports research in SLAM, high-precision state estimation, and multi-modal learning, enabling rigorous evaluation and development of new approaches to sensor fusion in legged robotic systems. With its extensive scope, GrandTour represents the largest open-access legged-robotics dataset to date. The dataset is available at https://grand-tour.leggedrobotics.com on HuggingFace (ROS-independent), and in ROS formats, along with tools and demo resources.

Turcan Tuna, Jonas Frey, Frank Fu +5

Robotic Systems Lab, ETH Zurich, Zurich 8092, Switzerland · 1ETH Zurich, Switzerland · 2Stanford University, USA +1

Legged Robotics Simultaneous Localization and Mapping Robotic Data Collection Robot State Estimation Robotic Perception

Feb 18, 2026

Reactive Motion Generation With Particle-Based Perception in Dynamic Environments

Reactive motion generation in dynamic and unstructured scenarios is typically subject to essentially static perception and system dynamics. Reliably modeling dynamic obstacles and optimizing collision-free trajectories under perceptive and control uncertainty are challenging. This article focuses on revealing tight connection between reactive planning and dynamic mapping for manipulators from a model-based perspective. To enable efficient particle-based perception with expressively dynamic property, we present a tensorized particle weight update scheme that explicitly maintains obstacle velocities and covariance meanwhile. Building upon this dynamic representation, we propose an obstacle-aware MPPI-based planning formulation that jointly propagates robot-obstacle dynamics, allowing future system motion to be predicted and evaluated under uncertainty. The model predictive method is shown to significantly improve safety and reactivity with dynamic surroundings. By applying our complete framework in simulated and noisy real-world environments, we demonstrate that explicit modeling of robot-obstacle dynamics consistently enhances performance over state-of-the-art MPPI-based perception-planning baselines avoiding multiple static and dynamic obstacles.

Xiyuan Zhao, Huijun Li, Lifeng Zhu +3

The authors are with the State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Robot Sensing and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China.

Robot Motion Generation Dynamic Obstacle Avoidance Model-Based Planning Robotic Perception MPPI Control

Feb 16, 2026

Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments

Endoluminal surgery offers a minimally invasive option for early-stage gastrointestinal and urinary tract cancers but is limited by surgical tools and a steep learning curve. Robotic systems, particularly continuum robots, provide flexible instruments that enable precise tissue resection, potentially improving outcomes. This paper presents a visual perception platform for a continuum robotic system in endoluminal surgery. Our goal is to utilize monocular endoscopic image-based perception algorithms to identify position and orientation of flexible instruments and measure their distances from tissues. We introduce 2D and 3D learning-based perception algorithms and develop a physically-realistic simulator that models flexible instruments dynamics. This simulator generates realistic endoluminal scenes, enabling control of flexible robots and substantial data collection. Using a continuum robot prototype, we conducted module and system-level evaluations. Results show that our algorithms improve control of flexible instruments, reducing manipulation time by over 70% for trajectory-following tasks and enhancing understanding of surgical scenarios, leading to robust endoluminal surgeries.

Ruofeng Wei, Kai Chen, Yui Lun Ng +7

Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Department of Mechanical Engineering, The University of Hong Kong, China. · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, and also with the Department of Mechanical Engineering, The University of Hong Kong.

Robotic Perception Surgical Robotics Continuum Robot Control Object Pose Estimation Monocular 3D Object Detection

Feb 2, 2026

PokeNet: Learning Kinematic Models of Articulated Objects from Human Observations

Articulation modeling enables robots to learn joint parameters of articulated objects for effective manipulation which can then be used downstream for skill learning or planning. Existing approaches often rely on prior knowledge about the objects, such as the number or type of joints. Some of these approaches also fail to recover occluded joints that are only revealed during interaction. Others require large numbers of multi-view images for every object, which is impractical in real-world settings. Furthermore, prior works neglect the order of manipulations, which is essential for many multi-DoF objects where one joint must be operated before another, such as a dishwasher. We introduce PokeNet, an end-to-end framework that estimates articulation models from a single human demonstration without prior object knowledge. Given a sequence of point cloud observations of a human manipulating an unknown object, PokeNet predicts joint parameters, infers manipulation order, and tracks joint states over time. PokeNet outperforms existing state-of-the-art methods, improving joint axis and state estimation accuracy by an average of over 27% across diverse objects, including novel and unseen categories. We demonstrate these gains in both simulation and real-world environments.

Anmol Gupta, Weiwei Gu, Omkar Patil +2

School of Computation and AI, ASU, Tempe · AI Institute, Seoul National University, Seoul, South Korea

Articulated Object Manipulation Robotic Perception

Jan 26, 2026

DeFM: Learning Foundation Representations from Depth for Robotics

Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/

Manthan Patel, Jonas Frey, Mayank Mittal +5

Robotic Systems Lab (RSL), ETH Zurich, Switzerland · Stanford University, Stanford, USA · UC Berkeley, Berkeley, USA +1

Representation Learning Robot Foundation Models Self-Supervised Visual Representation Learning Robotic Perception

Jan 22, 2026

PUMA: Perception-driven Unified Foothold Prior for Mobility Augmented Quadruped Parkour

Parkour tasks for quadrupeds have emerged as a promising benchmark for agile locomotion. While human athletes can effectively perceive environmental characteristics to select appropriate footholds for obstacle traversal, endowing legged robots with similar perceptual reasoning remains a significant challenge. Existing methods often rely on hierarchical controllers that follow pre-computed footholds, thereby constraining the robot's real-time adaptability and the exploratory potential of reinforcement learning. To overcome these challenges, we present PUMA, an end-to-end learning framework that integrates visual perception and foothold priors into a single-stage training process. This approach leverages terrain features to estimate egocentric polar foothold priors, composed of relative distance and heading, guiding the robot in active posture adaptation for parkour tasks. Extensive experiments conducted in simulation and real-world environments across various discrete complex terrains, demonstrate PUMA's exceptional agility and robustness in challenging scenarios.

Liang Wang, Kanzhong Yao, Yang Liu +4

Institute of Cyber-Systems and Control, Zhejiang University, 310027, China · Institute of Artificial Intelligence (TeleAI), China Telecom.

Legged Robotics Terrain-Aware Robot Locomotion RL for Legged Locomotion Robotic Perception Quadruped Locomotion

Jan 2, 2026

Learning to Segment Liquids in Real-world Images

Liquids like water, wine and medicine are everywhere. However, limited attention has been given to the task of segmenting liquids, hindering the ability of robots to safely avoid and interact with them. The segmentation of liquids is difficult because liquids come in diverse appearances and shapes; moreover, they can be both transparent or reflective, taking on arbitrary objects and scenes from their background and surroundings. To take on this challenge, we construct a liquid dataset, LQDS, consisting of 5000 real-world images annotated into 14 distinct classes, and design a novel liquid detection model, LQDM, which leverages cross-attention between a dedicated boundary branch and the main segmentation branch to enhance mask predictions. Extensive experiments demonstrate the effectiveness of LQDM on the testing set of LQDS, outperforming state-of-the-art methods to establish a strong baseline for the semantic segmentation of liquids. We believe that LQDS and LQDM will facilitate future research in liquid segmentation and enable practical applications in robotics. Our dataset and code is released at https://lonaslee.github.io/LQDM/.

Jonas Li, Michelle Li, Luke Liu +2

Department of Computer Science and Engineering, University of North Texas, Denton, TX 76203, USA

Image Segmentation Boundary-Aware Segmentation Robotic Perception Semantic Segmentation

Dec 29, 2025

Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation

Vision-Language-Action (VLA) models have shown remarkable generalization by mapping web-scale knowledge to robotic control, yet they remain blind to physical contact. Consequently, they struggle with contact-rich manipulation tasks that require reasoning about force, texture, and slip. While some approaches incorporate low-dimensional tactile signals, they fail to capture the high-resolution dynamics essential for such interactions. To address this limitation, we introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future. Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs coupled with wrist-camera local vision and third-person macro vision. To reconcile these multi-scale sensory streams, we first train a unified policy with a Hierarchical Spatial Alignment (HSA) loss that aligns tactile tokens with their spatial counterparts in the wrist and third-person views. To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals. To mitigate tactile data scarcity and the wear-prone nature of tactile sensors, we construct a hybrid large-scale dataset sourced from both high-fidelity digital twin and real-world experiments. By anticipating upcoming tactile states, DreamTacVLA acquires a rich model of contact physics and conditions its actions on both real observations and imagined consequences. Across contact-rich manipulation tasks, it outperforms state-of-the-art VLA baselines, achieving up to 95% success, highlighting the importance of understanding physical contact for robust, touch-aware robotic agents.

Guo Ye, Zexi Zhang, Xu Zhao +4

Northwestern University, Evanston, IL, USA

Contact-Rich Robotic Manipulation World Models Tactile Sensing for Robotic Manipulation VLMs for Robotics Robotic Perception Tactile World Models

Nov 21, 2025

MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots

Dense 3D semantic occupancy perception is critical for mobile robots operating in pedestrian-rich environments, yet it remains underexplored compared to its application in autonomous driving. To address this gap, we present MobileOcc, a semantic occupancy dataset for mobile robots operating in crowded human environments. Our dataset is built using an annotation pipeline that incorporates static object occupancy annotations and a novel mesh optimization framework explicitly designed for human occupancy modeling. It reconstructs deformable human geometry from 2D images, then refines and optimizes it using associated LiDAR point data. Using MobileOcc, we establish benchmarks for two tasks: i) Occupancy prediction and ii) Pedestrian velocity prediction, using different methods, including monocular, stereo, and panoptic occupancy, with metrics and baseline implementations for reproducible comparison. Beyond occupancy prediction, we further assess our annotation method on 3D human pose estimation datasets. Results demonstrate that our method exhibits robust performance across different datasets. Our code and dataset are released at https://autonomousrobots.nl/paper_websites/mobileocc

Junseo Kim, Guido Dumont, Xinyu Gao +3

Delft University of Technology, Mekelweg 5, 2628 CD Delft, Netherlands

Mobile Robotics Semantic Occupancy Mapping 3D Occupancy Prediction Robotic Perception

Oct 2, 2025

What Matters in RL-Based Methods for Object-Goal Navigation? An Empirical Study and A Unified Framework

Object-Goal Navigation (ObjectNav) is a key capability for deploying mobile robots in everyday environments such as homes, schools, and workplaces. In this task, an agent must locate an instance of a target object category in previously unseen environments using only onboard perception, requiring the integration of semantic understanding, spatial reasoning, and long-horizon planning. Reinforcement learning (RL) has become a dominant paradigm for ObjectNav, yet modern systems involve numerous design choices across perception modules, policy architectures, and inference-time strategies. The relative impact of these components, however, remains poorly understood. In this work, we present a large-scale empirical study of modular RL-based ObjectNav systems. We decompose the navigation pipeline into three key components: perception, policy, and test-time enhancement, and conduct extensive controlled experiments to analyze their individual contributions. Our results suggest that improvements in perception quality and test-time strategies often yield larger performance gains than policy improvements alone, highlighting the importance of understanding how different components interact within modular navigation systems. Motivated by these findings, we introduce a unified framework for systematically studying modular ObjectNav systems. Guided by our analysis, we build an enhanced system that achieves state-of-the-art performance on the Gibson benchmark, improving SPL by 6.6% and success rate by 2.7% over prior methods. We also introduce a human expert baseline, achieving 98% success, highlighting the significant gap between current RL agents and human-level navigation. Finally, we provide practical insights and design recommendations for each module to help guide future research. Project page: https://honwang0054.github.io/What-matters-in-RL-ObjNav-web/.

Hongze Wang, Boyang Sun, Jiaxu Xing +5

Computer Vision and Geometry Group, ETH Zurich, Switzerland · Robotics and Perception Group, University of Zurich, Switzerland · Robotic Systems Lab, ETH Zurich, Switzerland +2

Robotic RL Robotic Perception Object-Goal Navigation

Sep 11, 2025

Scensory: Real-Time Robotic Olfactory Perception for Joint Identification and Source Localization

Olfaction offers robots access to chemical information that is largely inaccessible to vision, touch, and audition, yet using airborne chemical signals for spatial perception remains challenging because local volatile organic compound (VOC) measurements are shaped by complex chemical transport and sensor dynamics. We introduce Scensory, a robotic olfaction framework that learns to jointly infer biological source identity and relative location from short temporal VOC measurements. Using a robot-automated data collection platform, we pair VOC dynamics from cross-sensitive gas sensor arrays with spatial supervision and train models to predict fungal identity, source direction, and distance. We show that a single sensor array can extract all three quantities from only 3 s of local measurements under ambient environmental conditions, achieving species classification accuracy of up to 80.13%, directional accuracy of up to 68.65%, and mean absolute distance errors of 0.110-0.131 m. Incorporating measurements from multiple spatial locations further reduces ambiguity, improving peak species and directional accuracies by 9.72 and 18.66 percentage points, respectively. We then embody this learned olfactory perception on a mobile robot, where successive local predictions acquired during motion are transformed into a world-frame evidence map, allowing observations from different positions and headings to reinforce persistent source hypotheses and guide closed-loop localization. Across eight selected indoor runs, the robot achieves a planar endpoint error of 0.606 +/- 0.294 m. Our results establish airborne chemical dynamics as a viable perceptual signal for robots to recognize biological sources, reason about their spatial origin, and autonomously navigate toward them under ambient environments.

Yanbaihui Liu, Erica Babusci, Claudia K. Gunsch +1

Department of Mechanical Engineering and Materials Science, Duke University, Durham, NC, USA. · Department of Civil and Environmental Engineering, Duke University, Durham, NC, USA. · Department of Electrical and Computer Engineering,Duke University, Durham, NC, USA. +1

Robot Navigation Robot Localization Robotic Perception

Sep 5, 2025

CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus

Fast and efficient photorealistic 3D reconstruction with (semantic) Gaussian Splatting (GS) is crucial for time-critical robotic perception and navigation, where robots may need to rapidly reconstruct and inspect specific points of interest (POIs), such as in search-and-rescue scenarios. Existing semantic GS methods prioritize reconstruction and segmentation quality while uniformly optimizing entire scenes, often overlooking runtime constraints crucial for real-world deployment. Moreover, object extraction often suffers from floaters and incomplete reconstructions, while post-optimization approaches often produce incomplete POIs. We propose CoRe-GS, a coarse-to-refined, task-driven approach that first creates a segmentation-ready GS representation through lightweight late-stage semantic refinement, then selectively refines only the views and Gaussians associated with the POI. A color-based filtering strategy further suppresses segmentation-induced floaters. On LERF-Mask, our representation achieves competitive mIoU with 6-8x fewer optimization steps. On NeRDS 360 and SCRREAM, CoRe-GS reduces time-to-POI by 91-95% over full-scene semantic GS while improving reconstruction quality and reducing floaters. Across all benchmarks, it is 44-76% faster than post-optimization object extraction, demonstrating that task-aware refinement can deliver efficient, high-quality POI reconstruction for time-critical robotic perception and operation.

Hannah Schieber, Dominik Frischmann, Victor Schaack +4

Technical University of Munich · Human-Centered Computing and Extended Reality Lab · TUM University Hospital, Clinic for Orthopedics and Sports Orthopedics +3

3D Gaussian Splatting 3D Reconstruction 3D Semantic Segmentation Robotic Perception

Aug 20, 2025

Decentralized Vision-Based Autonomous Aerial Wildlife Monitoring

Wildlife field operations demand efficient parallel deployment methods to identify and interact with specific individuals, enabling simultaneous collective behavioral analysis, and health and safety interventions. Previous robotics solutions approach the problem from the herd perspective, or are manually operated and limited in scale. We propose a decentralized vision-based multi-quadrotor system for wildlife monitoring that is scalable, low-bandwidth, and sensor-minimal (single onboard RGB camera). Our approach enables robust identification and tracking of large species in their natural habitat. We develop novel vision-based coordination and tracking algorithms designed for dynamic, unstructured environments without reliance on centralized communication or control. We validate our system through real-world experiments, demonstrating reliable deployment in diverse field conditions.

Makram Chahine, William Yang, Alaa Maalouf +6

Project CETI, New York 10003, NY, USA · CSAIL MIT, Cambridge MA 02139, USA · SEAS Harvard University, Boston MA 02134, USA

Multi-Robot Coordination Drone-Based Wildlife Monitoring Robotic Perception Wildlife Monitoring Multi-Object Tracking

Aug 15, 2025

Visual Perception Engine: Fast and Flexible Multi-Head Inference for Robotic Vision Tasks

Deploying multiple machine learning models on resource-constrained robotic platforms for different perception tasks often results in redundant computations, large memory footprints, and complex integration challenges. In response, this work presents Visual Perception Engine (VPEngine), a modular framework designed to enable efficient GPU usage for visual multitasking while maintaining extensibility and developer accessibility. Our framework architecture leverages a shared foundation model backbone that extracts image representations, which are efficiently shared, without any unnecessary GPU-CPU memory transfers, across multiple specialized task-specific model heads running in parallel. This design eliminates the computational redundancy inherent in feature extraction component when deploying traditional sequential models while enabling dynamic task prioritization based on application demands. We demonstrate our framework's capabilities through an example implementation using DINOv2 as the foundation model with multiple task (depth, object detection and semantic segmentation) heads, achieving up to 3x speedup compared to sequential execution. Building on CUDA Multi-Process Service (MPS), VPEngine offers efficient GPU utilization and maintains a constant memory footprint while allowing per-task inference frequencies to be adjusted dynamically during runtime. The framework is written in Python and is open source with ROS2 C++ (Humble) bindings for ease of use by the robotics community across diverse robotic platforms. Our example implementation demonstrates end-to-end real-time performance at

\geq

50 Hz on NVIDIA Jetson Orin AGX for TensorRT optimized models.

Jakub Łucki, Jonathan Becktor, Georgios Georgakis +2

Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA. · Swiss Federal Institute of Technology (ETH Zürich), Zürich, Switzerland.

Efficient Neural Network Inference Robotic Perception

Aug 9, 2025

$\mathcal{P}^3$ : Toward Versatile Embodied Agents

Embodied agents have shown promising generalization capabilities across diverse physical environments, making them essential for a wide range of real-world applications. However, building versatile embodied agents poses critical challenges due to three key issues: dynamic environment perception, open-ended tool usage, and complex multi-task planning. Most previous works rely solely on feedback from tool agents to perceive environmental changes and task status, which limits adaptability to real-time dynamics, causes error accumulation, and restricts tool flexibility. Furthermore, multi-task scheduling has received limited attention, primarily due to the inherent complexity of managing task dependencies and balancing competing priorities in dynamic and complex environments. To overcome these challenges, we introduce

\mathcal P^3

, a unified framework that integrates real-time perception and dynamic scheduling. Specifically,

\mathcal P^3

enables agents to perceive task-relevant information actively from the environment, plug and utilize tools without feedback requirements, and plan multi-task execution by prioritizing urgent tasks and dynamically adjusting task order based on dependencies. Extensive real-world experiments show that our approach bridges the gap between benchmarks and practical deployment, delivering highly transferable, general-purpose embodied agents. Code and data are available at https://github.com/fz-zsl/P3.

Shengli Zhou, Xiangchen Wang, Jinrui Zhang +4

Southern University of Science and Technology · Spatialtemporal AI · MBZUAI

Robot Task Planning Embodied AI Robotic Perception

Aug 8, 2025

VLEM: Real-Time 3D Vision-Language Embedding Mapping

Semantic scene understanding in robotics requires representations that are both metric-accurate and queryable via natural language in real-time. While recent Vision-Language Models enable powerful 2D image-text alignment, their integration into real-time 3D mapping systems remains challenging due to their requirements on ground truth poses, computational cost, and memory constraints. We present VLEM (Vision-Language Embedding Mapping), a real-time framework for integrating pixel-aligned 2D vision-language embeddings from various backends into a globally consistent, metric-accurate 3D representation, requiring only a raw RGB-D stream. Compared to ConceptFusion, Open-Fusion, and RayFronts, VLEM provides better open-set segmentation performance and a more compact representation. We further demonstrate VLEM's versatility in interactive real-time robotic manipulation tasks and mobile mapping scenarios.

Christian Rauch, Björn Ellensohn, Linus Nwankwo +2

Technical University of Leoben, Franz Josef-Straße 18, 8700 Leoben, Austria

Open-Vocabulary 3D Segmentation 3D Scene Representation Robotic Perception 3D VLMs

Aug 4, 2025

Multi-Class Human/Object Detection on Robot Manipulators using Proprioceptive Sensing

In physical human-robot collaboration (pHRC) settings, humans and robots collaborate directly in shared environments. Robots must analyze interactions with objects to ensure safety and facilitate meaningful workflows. One critical aspect is human/object detection, where the contacted object is identified. Past research introduced binary machine learning classifiers to distinguish between soft and hard objects. This study improves upon those results by evaluating three-class human/object detection models, offering more detailed contact analysis. A dataset was collected using the Franka Emika Panda robot manipulator, exploring preprocessing strategies for time-series analysis. Models including LSTM, GRU, and Transformers were trained on these datasets. The best-performing model achieved 91.11% accuracy during real-time testing, demonstrating the feasibility of multi-class detection models. Additionally, a comparison of preprocessing strategies suggests a sliding window approach is optimal for this task.

Justin Hehli, Marco Heiniger, Maryam Rezayati +1

Dep. of Informatics, University of Zurich, and MINDLab, Institute of Mechatronic Systems, ZHAW, Switzerland · Institute of Mechatronics system, ZHAW, Switzerland

Time Series Classification Human-Robot Collaboration Robotic Perception

Jul 21, 2025

Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers

Human vision is a highly active process driven by gaze, which directs attention to task-relevant regions through foveation, dramatically reducing visual processing. In contrast, robot learning systems typically rely on passive, uniform processing of raw camera images. In this work, we explore how incorporating human-like active gaze into robotic policies can enhance efficiency and robustness. We develop GIAVA (Gaze Integrated Active-Vision ALOHA), a robot vision system that emulates human head and neck movement, and gaze adjustment for foveated processing. Extending the AV-ALOHA robot platform, we introduce a framework for simultaneously collecting eye-tracking, perspective control, and robot manipulation demonstration data from a human operator. We also open-source a simulation benchmark and dataset for training robot policies that incorporate human gaze. Inspired by recent work in foveated image segmentation and given the widespread use of Vision Transformers (ViTs) in robot learning, we integrate gaze information into ViTs using a foveated patch tokenization scheme. Compared to uniform patch tokenization, this significantly reduces the number of tokens, and thus computation. Our results show that our method for foveated robot vision drastically reduces computational overhead, and enhances robustness to background distractors. Notably, on certain high-precision tasks, foveated vision also improves performance, as reflected in higher success rates. Together, these findings suggest that human-inspired foveated visual processing offers untapped potential and should be further considered as a useful inductive bias in robotic vision systems. https://soltanilara.github.io/giava/

Ian Chuang, Jinyu Zou, Andrew Lee +2

University of California, Berkeley · 2Tongji University · University of California, Davis

Visuomotor Policy Learning Efficient ViTs Active Visual Perception Robotic Perception