cs.ROMar 16, 2026

Coupled Particle Filters for Robust Affordance Estimation

Authors: Patrick Lowin, Vito Mengers, Oliver Brock

Organizations: Robotics and Biology Laboratory, Technische Universität Berlin · Robotics Institute Germany · Science of Intelligence (SCIoI), Cluster of Excellence, Berlin, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Robotic affordance estimation is challenging due to visual, geometric, and semantic ambiguities in sensory input. We propose a method that disambiguates these signals using two coupled recursive estimators for sub-aspects of affordances: graspable and movable regions. Each estimator encodes property-specific regularities to reduce uncertainty, while their coupling enables bidirectional information exchange that focuses attention on regions where both agree, i.e., affordances. Evaluated on a real-world dataset, our method outperforms three recent affordance estimators (Where2Act, Hands-as-Probes, and HRP) by 308%, 245%, and 257% in precision, and remains robust under challenging conditions such as low light or cluttered environments. Furthermore, our method achieves a 70% success rate in our real-world evaluation. These results demonstrate that coupling complementary estimators yields precise, robust, and embodiment-appropriate affordance predictions.

Figures & tables

Fig. 1: Robots can interact by grasping an object and then moving it. Consequently, affordances emerge in regions where these properties co-occur. By estimating these properties separately and leveraging their relationship, we can resolve ambiguities in both modalities and estimate robust affordances.
Fig. 2: Our approach estimates affordances as a combination of the graspable and movable regions of our environment. Since our measurement sources provide only uncertain measurements for these properties, we recursively estimate a belief for them and apply additional priors to resolve ambiguities. This aggregates a strong belief in regions with high measurements for their individual property. However, affordances need to be both graspable and movable. Therefore, we exchange information between our estimators, focusing each estimator’s attention on relevant parts of the scene that satisfy both properties, i.e., affordances.
Fig. 3: Our method can handle dynamic and complex scenes. While the non-recursive learning-based approaches detect new affordances directly, we rely on our injection step to introduce new affordances, such as the object inside the shelf, into our belief. Our estimates are more precise, focusing on manipulable affordances.
Fig. 4: Our approach outperforms recent learning-based approaches in both standard, i.e., well-lit and clutter-free (top), and more challenging, i.e., dark or cluttered (bottom), settings. The monolithic learned models struggle with real-world ambiguities, while coupling resolves them by aligning estimator attention and filtering noise.
Fig. 5: Our affordance estimates remain precise in dark environments. Hands-as-Probes [ 8 ] and HRP [ 26 ] suffer from visual ambiguities, and Where2Act [ 21 ] struggles with object complexity, while our coupled system yields reliable affordances.
Fig. 6: Due to the coupling, our affordance estimates remain focused on manipulable object regions even in clutter. Other approaches are less precise, spreading across distractor surfaces such as the table edge.
Fig. 7: Our cross-modal fusion aligns the estimators’ attention on regions that satisfy both properties, that is the affordances. The individual estimators suffer from their respective ambiguities, but by sharing information they can resolve these, focus on relevant regions and estimate robust affordances even under adverse conditions.
Fig. 8: Coupling particle filters with our cross-modal fusion outperforms other sensor fusion schemes, in particular, naive and late fusion. Early fusion combines signals before recursive estimation, which leads to good estimates, but does not leverage the shared structure of the two properties to further disambiguate the signals.
Fig. 9: Our approach estimates robust affordances that enable robots to interact with their environment. We evaluate this in real-world trials on cluttered tabletop and shelf scenes with four objects (cup, banana, toothpaste, and Cheez-it box). Our estimated affordances correspond to precise and actionable grasps, even in cluttered environments.
More in this topic

Robotic Affordance Inference

See all 20 papers

Explore similar work

CardsList

  1. RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation

    Mar 8, 2026Zhanqi Xiao, Ruiping Wang, Xilin ChenRobotic Affordance InferenceAffordance-Based Robotic Manipulation

    The authors are with the Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing, 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China.

  2. RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

    Mar 31, 2026Qiyuan Zhuang, He-Yang Xu, Yijun Wang +3Robotic Affordance InferenceAffordance-Based Robotic Manipulation

    School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 211189, China · Southeast University-Monash University Joint Graduate School, Southeast University, Suzhou 215123, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China

  3. Cross-Embodiment Affordance Transfer for Robotic Manipulation

    Oct 13, 2026Sawera Yaseen, Serge Thill, Rekha RajaAffordance-Based Robotic ManipulationRobotic Affordance Inference

    School of Artificial Intelligence, Donders Centre for Cognition, Department of Human-Centred Intelligent Systems, Radboud University, Nijmegen, The Netherlands.