The complexity of teaching humanoid robots new tasks is one of the major reasons hindering their widespread adoption in the industry. While Imitation Learning (IL), particularly Action Chunking with Transformers (ACT), enables rapid task acquisition, there is no consensus yet on the optimal sensory hardware required for manipulation tasks. This paper benchmarks 14 sensor combinations on the Unitree G1 humanoid robot equipped with three-finger hands for two manipulation tasks. We explicitly evaluate the integration of tactile and proprioceptive modalities alongside active vision. Our analysis demonstrates that strategic sensor selection can outperform complex configurations in data-limited regimes while reducing computational overhead. We develop an open-source Unified Ablation Framework that utilizes sensor masking on a comprehensive master dataset. Results indicate that additional modalities often degrade performance for IL with limited data. A minimal active stereo-camera setup outperformed complex multi-sensor configurations, achieving 87.5% success in a spatial generalization task and 94.4% in a structured manipulation task. Conversely, adding pressure sensors to this setup reduced success to 67.3% in the latter task due to a low signal-to-noise ratio. We conclude that in data-limited regimes, active vision offers a superior trade-off between robustness and complexity. While tactile modalities may require larger datasets to be effective, our findings validate that strategic sensor selection is critical for designing an efficient learning process.
Figures & tables
Fig. 1 : Custom ACT Architecture: Schematic representation integrating pressure sensors, joint velocities, and torques into the state observation vector. Building upon: [ 2 , 9 , 7 ] .
Fig. 2 : Teleoperation Setup: The operator controls the Unitree G1 (with three-finger hands) via VR. Head movements are synchronized, enabling the collection of active vision data.
Fig. 3 : Unified Ablation Framework: To prevent human variance from influencing results, we record a master dataset with all sensors. During training, a masking module creates specific configurations (e.g., A , WA−P ) from identical demonstration data.
Policy
Training
Trials
Execution
Success
Time (h)
Time (min)
Rate (%)
Task 1: Can Sorting (20 Episodes)
A
7.35
11
3.9
94.4
A-P
7.40
4
-
67.3
SW
-
6
4.2
76.4
SLWA
-
6
8.7
85.1
TABLE I : Performance Statistics per Policy and Task. All policies were trained for 80,000 optimization steps. Note: Evaluation counts (Trials) vary because failing policies (e.g., due to hovering) exceeded maximum trial durations, leading to truncated evaluations.
Fig. 4 : Execution time vs. success rate (Task 1): The maximalist setup ( WA−P ; marked with ① in the plot) achieves the best combination of execution time and success. However, using only a single active stereo-camera ( A ②) offers nearly identical results with a fraction of the state-vector and thereby computational complexity. Notably, simply adding pressure sensors ( A−P ③) causes a performance drop compared to the camera-only baseline. Execution time vs. success rate (Task 2): In generalization tasks, the minimal active setup ( A ④) dominates with the combination of the highest success (mean 87.5% ) and the fastest execution time with a mean of 0.4 min . Conversely, setups involving static cameras (e.g., S ⑤, SLWA-P ⑥) yield suboptimal performance and increased execution times.
Figure 6
Fig. 7 : Mode Averaging Failure in Task 1: The policy fails to commit to a single drop-off target (e.g., green vs. red box) and instead generates a path that interpolates between the two distinct modes. This results in a trajectory that is notably lower than the demonstration data, causing the end-effector to collide with the box rim (2.1–2.3) or to collide with previously dropped cans (1.1–1.3).
Touch sensing is beneficial for solving a wide variety of manipulation tasks. While there exists a wide range of tactile sensors with different properties, exploiting the fusion of multiple heterogeneous tactile sensors to improve manipulation learning remains underexplored. We present Multi-Resolution Tactile Sensing (MiTaS), a representation framework that leverages multiple tactile sensors operating at different temporal resolutions in order to solve complex contact-rich manipulation tasks. We propose a novel architecture using modality-specific convolutional stems and transformer-based fusion that effectively fuses information from an RGB camera stream, a vision-based GelSight Mini sensor and a high-frequency event-based Evetac sensor. This multi-sensor representation then conditions a flow-matching policy for solving downstream tasks. Experimental results across five contact-rich manipulation tasks demonstrate the effectiveness of multi-resolution tactile features in imitation learning. MiTaS achieves an average success rate of 80 %, while vision-only (31 %) and visual-tactile (54 %) baselines cannot solve the task reliably. Co-training a visuo-tactile model with multi-tactile data boosts performance by over 10 % in certain tasks, without having access to the Evetac sensor during policy evaluation. A detailed sensor-reading and attention analysis reveals the importance of different sensors throughout task execution, validating our multi-resolution tactile sensing approach. Project Page: http://mitas-touch.github.io.
Rickmer Krohn, Erik Helmut, Niklas Funk +3
Interactive Robot Perception & Learning, TU Darmstadt · Hessian AI · Robotics Institute Germany +1
Tactile feedback is important for contact-rich robotic manipulation, yet effective use of tactile observations remains challenging when tactile signals are image-like, hardware-dependent, and only weakly aligned with external visual observations. This study addresses this representation-learning problem by proposing a visuo-tactile contrastive learning framework for imitation-based manipulation. The method aligns external RGB observations and calibrated tactile images in a shared embedding space using a CLIP-style objective, and integrates the resulting representation into an Action Chunking Transformer (ACT) policy. A low-cost visuo-tactile gripper (LVTG) is proposed to provide a modular and durable sensing platform for reproducible data collection, supplying tactile observations that can be used by downstream manipulation algorithms. Experiments on contact-rich manipulation tasks show that tactile feedback improves the average task completion rate from 30% for a vision-only ACT baseline to 42%, and that the proposed contrastive pretraining further increases the completion rate to 54%. These results indicate that explicitly aligning visual and tactile observations provides more useful contact-aware features for downstream policy learning than directly adding tactile images without pretraining.
Yaohua Liu, Rong Fu, Amir H. Gandomi +2
Chinese Medicine Guangdong Laboratory, Hengqin, Zhuhai 519031, China · Institute of Collaborative Innovation, University of Macau, Macau 999078, China · Faculty of Engineering & IT, University of Technology Sydney, Ultimo, NSW 2007, Australia +3
Robotic imitation learning often relies on external cameras, yet local interaction cues such as object proximity, contact onset, and grasp state are difficult to observe near the fingertips because of occlusion and limited temporal resolution. We study how to effectively incorporate complementary fingertip sensing into imitation learning using pressure-sensitive tactile and reflective proximity sensors, along with pretrained sensor encoders. The two modalities provide information at different manipulation phases: proximity sensing is informative before contact, whereas tactile sensing becomes informative after contact. However, naively adding these signals to a policy does not consistently improve performance and can even underperform vision-only policies, suggesting that sparse, phase-dependent sensor signals are difficult to exploit from limited demonstrations. We therefore propose a proprioception-anchored pretraining method, PROprioceptive-and-PRoactive Anchoring (PROPRA), which independently aligns each fingertip sensor history with proprioceptive and action segments. This provides a continuously available sensorimotor reference, allowing each sensor to be aligned independently during its informative phases. Experiments on real-world manipulation tasks show that our pretraining method improves average success rates over vision-only policies and image-anchored pretraining baselines. Representation analysis further shows that it preserves richer information about pre-contact states, enabling more effective use of complementary fingertip sensing. Please refer to our project page: https://tomohiromotoda.github.io/nia.propra/
Embodied AI Research Team, AIRC and National Institute of Advanced Industrial Science and Technology (AIST) · CNRS-AIST JRL (Joint Robotics Laboratory), IRL and National Institute of Advanced Industrial Science and Technology (AIST)