Organizations: Jiuzhou Lei, Chang Liu, and Minghui Zheng are with the J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA. · Yu She is with the Edwardson School of Industrial Engineering, Purdue University, West Lafayette, Indiana, USA. · Xiao Liang is with the Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843, USA.
Vision-based policies have achieved a good performance in robotic manipulation due to the accessibility and richness of visual observations. However, purely visual sensing becomes insufficient in contact-rich and force-sensitive tasks where force/torque (F/T) signals provide critical information about contact dynamics, alignment, and interaction quality. Although various strategies have been proposed to integrate vision and F/T signals, including auxiliary prediction objectives, mixture-of-experts architectures, and contact-aware gating mechanisms, a comparison of these approaches remains lacking. In this work, we provide a controlled comparison of different F/T-vision integration strategies within diffusion-based manipulation policies. In addition, we propose an adaptive integration strategy that ignores F/T signals during non-contact phases while adaptively leveraging both vision and torque information during contact. Experimental results demonstrate that our method outperforms the strongest baseline by 14% in success rate, highlighting the importance of contact-aware multimodal fusion for robotic manipulation.
Figures & tables
Fig. 1 : Contact-Aware Manipulation Challenges. The top row shows that relying solely on visual input leaves the policy contact-unaware, causing failures in relevant tasks. The middle row shows that naively fusing F/T signals with visual features can degrade policy accuracy during free-space motion. The bottom-left panel shows that the predicted torque guidance weight varies interpretably over inference steps, rising during contact phases and falling during free-space motion. We evaluate across three contact-rich tasks (right), and our method achieves 82% average success rate, outperforming all baselines by a substantial margin (bottom).
Fig. 2 : Overview of the Proposed Method. RGB images are encoded using ResNet, with each camera view processed by a separate encoder. Torque signals are encoded using an MLP. The resulting torque features are then passed through a contact-gated module, which modulates them based on contact status to produce contact-gated torque features. A scale predictor takes the concatenated vision and torque features as input and outputs a scalar weight that determines the relative influence of torque information. The final denoising noise ϵ^ is obtained by blending these two predictions according to the predicted scale.
Fig. 3 : Visualization of the Three Tasks. First row: egg boiler lid opening, second row: weight-based bottle placement, and third row: twisty connector pull out.
Fig. 4 : Experiment Setup
Method
Bottle
Connector
Lid
Average
Vision-only
8/20
0/10
7/20
30.0%
Feature Concatenation
3/20
0/10
12/20
30.0%
Torque Gating
14/20
5/10
15/20
68.0%
Auxiliary Goals
6/20
1/10
7/20
28.0%
MoE
5/20
1/10
7/20
24.0%
MoE w/o torque encoding
15/20
1/10
11/20
54.0%
TABLE I : Controlled comparison of torque–vision fusion modules under a shared Robomimic Diffusion Policy backbone. The methods differ only in the fusion module. The best performance in terms of task success rate in each column is highlighted in bold.
Method
Empty
Full
Total
Vision-only
8/10
0/10
8/20
Feature Concatenation
1/10
2/10
3/20
Torque Gating
7/10
7/10
14/20
Auxiliary Goals
4/10
2/10
6/20
MoE
5/10
0/10
5/20
MoE w/o torque encoding
9/10
6/10
15/20
TABLE II : Success rates of placing empty and full water bottles.
Method
Success Rate (%)
Avg. Task Horizon
Torque Gating
20.0
110.7
Ours
60.0
87.1
Higher success and lower horizon indicate better performance.
TABLE III : Comparison of Single-Shot Success Rates
Fig. 5 : External Joint Torque vs. Time During a Task Execution. During the approaching and lifting stages, the torques measurements are patternless and fluctuate. Different colors represent the torque measurements at different joints.
Fig. 6 : Failure Cases Due to Distracting Torque Features
Fig. 7 : Ablation Study. A comparison between the predicted weights associated with force/torque by our approach (top) and by MoE (bottom) in the ablation study. The orange dots in the figures represent the values of the weights when there is contact.
Contact-rich manipulation requires not only vision-dominant task semantics but also closed-loop reactions to force/torque (F/T) transients. Yet, generative visuomotor policies are typically constrained to low-frequency updates due to inference latency and action chunking, underutilizing F/T for control-rate feedback. Furthermore, existing force-aware methods often inject force continuously and indiscriminately, lacking an explicit mechanism to schedule when / how much / where to apply force across different task phases. We propose PhaForce, a phase-scheduled visual--force policy that coordinates low-rate chunk-level planning and high-rate residual correction via a unified contact/phase schedule. PhaForce comprises (i) a contact-aware phase predictor (CAP) that estimates contact probability and phase belief, (ii) a Slow diffusion planner that performs dual-gated visual--force fusion with orthogonal residual injection to preserve vision semantics while conditioning on force, and (iii) a Fast corrector that applies control-rate phase-routed residuals in interpretable corrective subspaces for within-chunk micro-adjustments. Across multiple real-robot contact-rich tasks, PhaForce achieves an average success rate of 86% (+40 pp over baselines), while also substantially improving contact quality by regulating interaction forces and exhibiting robust adaptability to OOD geometric shifts.
Vision-Language-Action (VLA) models have demonstrated strong performance across diverse robotic manipulation tasks, yet their predominantly vision-centric perception and position-controlled execution remain insufficient for contact-rich manipulation. Visual observations alone often provide limited evidence of contact onset and interaction magnitude, while position-control policies cannot respond compliantly to rapidly changing contact dynamics. To bridge both the perception and control gaps, we propose TAO-Force, a force-conditioned VLA framework that combines force-aware policy learning with contact-regulated execution. For force-aware perception, TAO-Force introduces Force-conditioned Feature-wise Linear Modulation (F-FiLM) to inject encoded force feedback into the representations of a frozen pretrained visual-language backbone while preserving its semantic priors. For responsive control, it employs a contact-gated fast-slow architecture, with a slow position-control branch tracking nominal trajectories during non-contact phases and a fast admittance-control branch regulating physical interaction during contact phases. Detailed analyses on a force-perception task and real-world evaluations across four contact-rich manipulation tasks validate the effectiveness and robustness of TAO-Force.
Bohan Gan, Xuanzhang Wen, Yongsheng Zhao +8
China Mobile (Hangzhou) Information Technology Co., Ltd., Hangzhou 310023, China
Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirectly inferring tactile cues. Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge. Existing methods typically use either raw images or cumulative motion fields to represent tactile cues. However, both are prone to perception ambiguity. Raw tactile images mainly capture appearance changes, while cumulative motion fields only reflect the aggregate gel deformation. Consequently, distinct fine-grained contact states can exhibit highly similar patterns, making it difficult to explicitly distinguish subtle contact variations. To address this issue, we explore the dynamic priors of tactile motion and discover that the correlation between transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-aware tactile representation to facilitate contact-rich manipulation. Beyond tactile representation, effective fusion of tactile and visual modalities is also critical. Most existing fusion methods either directly concatenate features from each modality or train modality-specific networks separately and fuse their outputs. However, these strategies struggle to simultaneously model cross-modal interactions and preserve modality-specific characteristics. In this work, we take advantage of the Mixture-of-Transformers architecture and propose a unified modality-aware visuo-tactile policy that captures cross-modal complementarity while maintaining modality-specific properties.
Shengqi Xu, Guojin Zhong, Yang Liu +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · NeoteAI +1