OpenHEART: Opening Heterogeneous Articulated Objects with a Legged Manipulator
Authors: Seonghyeon Lim, Hyeonwoo Lee, Seunghyun Lee, I Made Aswin Nahrendra, Hyun Myung
Organizations: School of Electrical Engineering, KAIST (Korea Advanced Institute of Science and Technology), Daejeon, 34141, Republic of Korea. · KRAFTON, Seoul, 06142, Republic of Korea.
Legged manipulators offer high mobility and versatile manipulation. However, robust interaction with heterogeneous articulated objects, such as doors, drawers, and cabinets, remains challenging because of the diverse articulation types of the objects and the complex dynamics of the legged robot. Existing reinforcement learning (RL)-based approaches often rely on high-dimensional sensory inputs, leading to sample inefficiency. In this paper, we propose a robust and sample-efficient framework for opening heterogeneous articulated objects with a legged manipulator. In particular, we propose Sampling-based Abstracted Feature Extraction (SAFE), which encodes handle and panel geometry into a compact low-dimensional representation, improving cross-domain generalization. Additionally, Articulation Information Estimator (ArtIEst) is introduced to adaptively mix proprioception with exteroception to estimate opening direction and range of motion for each object. The proposed framework was deployed to manipulate various heterogeneous articulated objects in simulation and real-world robot systems. Videos can be found on the project website: https://openheart-icra.github.io/OpenHEART/
Figures & tables
Figure 1: We present a framework for opening heterogeneous articulated objects with a legged manipulator. The robot can open diverse articulated objects, such as (a) a revolute cabinet with a vertical handle and (b) a prismatic drawer with a horizontal handle without any object-specific models.
Figure 2: The framework is hierarchically structured, with a high-level planner and (a) a low-level controller. SAFE is introduced to efficiently represent the object shape, while mitigating overfitting. (b) ArtIEst estimates the articulation information by adaptively mixing exteroception and proprioception. The history encoder extracts features from proprioception history.
Figure 3: Defined articulation information (pink solid arrow) for objects with (a) a revolute joint and (b) a prismatic joint. The expected opening trajectory is shown as a blue dashed arrow.
Reward
Equation
Weight
Opening
ξtnorm
6.5
EE approach
{exp(−4∥ptEE−ptH∥2),1,if ξtnorm<0.2otherwise
1.9
EE alignment
cos2θt
0.75
Grasping
I{grasp}
1.0
Command rate
60λt⋅∥ct−ct−1∥2
-0.1
EE command rate
30λt⋅∥ctEE−ct−1∥2
-0.1
Table 1: Reward function elements. ptEE and ptH denote the position of the end-effector and handle, respectively. θt is the angle between the dorsal-palmar axis of EE and the longer side of the handle. I{⋅} is the indicator function, which returns one if the condition is satisfied, and zero otherwise.
Figure 4: Demonstration of a heterogeneous articulated object opening in the simulation. The objects differ in their handle shape, dimension, and opening direction.
Figure 5: (a) Learning curve of opening reward. Ours shows the highest performance compared with the baselines. Saliency maps are shown for the object with (b) revolute joint and (c) prismatic joint. Object appearances are depicted at the top line, the saliency maps from Point cloud-based policy , and Ours are shown in the middle and below each subfigure.
Figure 6: Estimation error transition of exteroception-based, proprioception-augmented, and mixed estimations during the manipulation. The progress of the opening motion is indicated by the object’s open angle normalized using the joint limits. (a) The appearance of the object can be inferred to be opened in either a rightward or upward direction, while the correct direction is upward, leading to visual ambiguity in exteroception-based estimation. (b) In contrast, no visual ambiguity exists in this case.
Method
Estimation direction error (rad) ↓
Before contact
During contact
Entire episode
Ours
0.2293
0.0687
0.1701
Ours w/o prop. est.
0.2623
0.2228
0.2482
Ours w/ mono. est.
0.2574
0.0606
0.1827
Table 2: ArtIEst evaluation. The estimation errors are separately measured before and during the contact.
Figure 7: Real-world demonstration of opening a drawer with a prismatic joint and a horizontally elongated handle. Although the first grasp pose (2.1 s) was not stable, the robot autonomously retrieved the handle (3.0 s) and successfully opened the drawer.
Method
Success rate (%) ↑
Test/Train (%) ↑
Train set
Test set
Ours
79.35
79.02
99.35
Ours w/o sampling
77.56
70.23
92.92
Center-based teacher [ 47 ]
62.43
50.60
81.05
Point cloud-based policy
56.58
41.39
73.15
Table 3: Success rate across training and test sets. The proposed method shows the highest performance retention.
The kinematics of an articulated object is often ambiguous from vision alone. Interaction resolves the ambiguity, and active perception methods exploit this by searching for the single action that most sharpens a belief over the kinematic parameters at each step. Such greedy search cannot be extended over a horizon without forward models of the contact and inertial dynamics, which are themselves unknown. We instead amortize action selection into training. We maintain a belief distribution over joint type and parameters, initialized from a generative prior and updated by Bayesian filtering on the observed part motion. To condition the policy on this belief, we render it as a per-point articulation flow field, the motion that the current posterior predicts for every point on the object. Carrying the inductive bias of articulated motion, this representation generalizes better than a latent encoding of the belief or flow tracked from observation. We train the policy with reinforcement learning, rewarding the entropy that each interaction removes from the posterior, so that informative exploration becomes learned behavior rather than a search at every step. Our method outperforms previous approaches across door and drawer manipulation on the PartManip benchmark, and reaches 61.7% success on ArticuRiddle, a new dataset of objects whose appearance implies the wrong articulation, against 44.4% for the best previous method. Project Website: https://hiddenkinematics.github.io/
Ruiyao Liu, Boshu Lei, Zhuoyang Pan +1
GRASP Lab, University of Pennsylvania, Philadelphia, PA 19104, USA
Generalizable manipulation involving cross-type object interactions is a critical yet challenging capability in robotics. To reliably accomplish such tasks, robots must address two fundamental challenges: "where to manipulate" (contact point localization) and "how to manipulate" (subsequent interaction trajectory planning). Existing foundation-model-based approaches often adopt end-to-end learning that obscures the distinction between these stages, exacerbating error accumulation in long-horizon tasks. Furthermore, they typically rely on a single uniform model, which fails to capture the diverse, category-specific features required for heterogeneous objects. To overcome these limitations, we propose HeteroGenManip, a task-conditioned, two-stage framework designed to decouple initial grasp from complex interaction execution. First, Foundation-Correspondence-Guided Grasp module leverages structural priors to align the initial contact state, thereby significantly reducing the pose uncertainty of grasping. Subsequently, Multi-Foundation-Model Diffusion Policy (MFMDP) routes objects to category-specialized foundation models, integrating fine-grained geometric information with highly-variable part features via a dual-stream cross-attention mechanism. Experimental evaluations demonstrate that HeteroGenManip achieves robust intra-category shape and pose generalization. The framework achieves an average 31% performance improvement in simulation tasks with broad type setting, alongside a 36.7% gain across four real-world tasks with different interaction types.
Human-like generalization in open-world remains a fundamental challenge for robotic manipulation. Existing learning-based methods, including reinforcement learning, imitation learning, and vision-language-action-models (VLAs), often struggle with novel tasks and unseen environments. Another promising direction is to explore generalizable representations that capture fine-grained spatial and geometric relations for open-world manipulation. While large-language-model (LLMs) and vision-language-model (VLMs) provide strong semantic reasoning based on language or annotated 2D representations, their limited 3D awareness restricts their applicability to fine-grained manipulation. To address this, we propose LAMP, which lifts image-editing as 3D priors to extract inter-object 3D transformations as continuous, geometry-aware representations. Our key insight is that image-editing inherently encodes rich 2D spatial cues, and lifting these implicit cues into 3D transformations provides fine-grained and accurate guidance for open-world manipulation. Extensive experiments demonstrate that \codename delivers precise 3D transformations and achieves strong zero-shot generalization in open-world manipulation. Project page: https://zju3dv.github.io/LAMP/.