Organizations: Jiuzhou Lei, Chang Liu, and Minghui Zheng are with the J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA. · Yu She is with the Edwardson School of Industrial Engineering, Purdue University, West Lafayette, Indiana, USA. · Xiao Liang is with the Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843, USA.
Vision-based policies have achieved a good performance in robotic manipulation due to the accessibility and richness of visual observations. However, purely visual sensing becomes insufficient in contact-rich and force-sensitive tasks where force/torque (F/T) signals provide critical information about contact dynamics, alignment, and interaction quality. Although various strategies have been proposed to integrate vision and F/T signals, including auxiliary prediction objectives, mixture-of-experts architectures, and contact-aware gating mechanisms, a comparison of these approaches remains lacking. In this work, we provide a controlled comparison of different F/T-vision integration strategies within diffusion-based manipulation policies. In addition, we propose an adaptive integration strategy that ignores F/T signals during non-contact phases while adaptively leveraging both vision and torque information during contact. Experimental results demonstrate that our method outperforms the strongest baseline by 14% in success rate, highlighting the importance of contact-aware multimodal fusion for robotic manipulation.
Figures & tables
Fig. 1 : Contact-Aware Manipulation Challenges. The top row shows that relying solely on visual input leaves the policy contact-unaware, causing failures in relevant tasks. The middle row shows that naively fusing F/T signals with visual features can degrade policy accuracy during free-space motion. The bottom-left panel shows that the predicted torque guidance weight varies interpretably over inference steps, rising during contact phases and falling during free-space motion. We evaluate across three contact-rich tasks (right), and our method achieves 82% average success rate, outperforming all baselines by a substantial margin (bottom).
Fig. 2 : Overview of the Proposed Method. RGB images are encoded using ResNet, with each camera view processed by a separate encoder. Torque signals are encoded using an MLP. The resulting torque features are then passed through a contact-gated module, which modulates them based on contact status to produce contact-gated torque features. A scale predictor takes the concatenated vision and torque features as input and outputs a scalar weight that determines the relative influence of torque information. The final denoising noise ϵ^ is obtained by blending these two predictions according to the predicted scale.
Fig. 3 : Visualization of the Three Tasks. First row: egg boiler lid opening, second row: weight-based bottle placement, and third row: twisty connector pull out.
Fig. 4 : Experiment Setup
Method
Bottle
Connector
Lid
Average
Vision-only
8/20
0/10
7/20
30.0%
Feature Concatenation
3/20
0/10
12/20
30.0%
Torque Gating
14/20
5/10
15/20
68.0%
Auxiliary Goals
6/20
1/10
7/20
28.0%
MoE
5/20
1/10
7/20
24.0%
MoE w/o torque encoding
15/20
1/10
11/20
54.0%
TABLE I : Controlled comparison of torque–vision fusion modules under a shared Robomimic Diffusion Policy backbone. The methods differ only in the fusion module. The best performance in terms of task success rate in each column is highlighted in bold.
Method
Empty
Full
Total
Vision-only
8/10
0/10
8/20
Feature Concatenation
1/10
2/10
3/20
Torque Gating
7/10
7/10
14/20
Auxiliary Goals
4/10
2/10
6/20
MoE
5/10
0/10
5/20
MoE w/o torque encoding
9/10
6/10
15/20
TABLE II : Success rates of placing empty and full water bottles.
Method
Success Rate (%)
Avg. Task Horizon
Torque Gating
20.0
110.7
Ours
60.0
87.1
Higher success and lower horizon indicate better performance.
TABLE III : Comparison of Single-Shot Success Rates
Fig. 5 : External Joint Torque vs. Time During a Task Execution. During the approaching and lifting stages, the torques measurements are patternless and fluctuate. Different colors represent the torque measurements at different joints.
Fig. 6 : Failure Cases Due to Distracting Torque Features
Fig. 7 : Ablation Study. A comparison between the predicted weights associated with force/torque by our approach (top) and by MoE (bottom) in the ablation study. The orange dots in the figures represent the values of the weights when there is contact.