Multimodal Interaction Estimation for Contact-Aware Adaptive Impedance Control

Mateusz Jaszczuk; Paris Perdikaris; Nadia Figueroa

cs.ROOct 16, 2026

Multimodal Interaction Estimation for Contact-Aware Adaptive Impedance Control

Authors: Mateusz Jaszczuk, Paris Perdikaris, Nadia Figueroa

Organizations: Department of Mechanical Engineering, University of Pennsylvania

Abstract

With advancements in learning-based policies, robots are increasingly being deployed in human-centric environments, such as manufacturing floors and homes, where safe and precise operation requires compensating for unknown loads while remaining compliant during human interactions. Reliably distinguishing between these interaction types is challenging with a single sensing modality, as force signals cannot separate payloads from human pushes, and vision is susceptible to occlusions. We propose a task-agnostic interaction estimation and adaptive control framework that addresses this challenge by fusing force history and 3D point cloud through a transformer-based classifier. Trained with modality dropout for occlusion resilience and fine-tuned from simulation to hardware, the classifier achieves 91.89% accuracy and maintains above 90% under complete occlusion of one modality. The classifier drives a system identification framework that selectively compensates for unknown payloads and a variable-impedance controller that switches between adaptive and reactive modes, while simultaneously estimating interaction forces for downstream tasks, such as human intent estimation. We validate the framework under external perturbations and unknown loads across single-arm manipulation, deployment with an imitation-learning-based policy, intent-driven target switching, and zero-shot bimanual deployment on two Franka arms lifting a box of unknown mass, demonstrating generalization to closed kinematic chains and unseen setups, enabling safe and precise deployment.

Figures & tables

Panel of Figure 1 — Fig. 1: Interaction Classification and Adaptive Control Framework. We utilize a transformer-based architecture to fuse latent force and point cloud representations, allowing the model to reliably distinguish applied payloads from perturbations (applied to any link), allowing the adaptive controller to adjust gravity compensation and compliance accordingly.

Panel of Figure 2 — Fig. 2: Multimodal Transformer architecture.

Fig. 3: IsaacLab dataset generation and domain randomization.

Panel of Figure 4 — Fig. 4: Confusion matrices for simulation-trained classifiers evaluated on a simulation for the force-only, point cloud-only, and multimodal models.

	w/o Modality Dropout	w/ Modality Dropout
Force Occlusion	43.10%	95.40%
PC Occlusion	38.53%	90.10%

TABLE I: Model classification accuracy under modality dropout ablations.

Experiment	Time (min)	Samples	Test Intervals
Load	8.87	6,537	7
Perturbation	28.35	20,864	21
Perturbation (z)	3.33	2,457	3
Load + Pert.	10.46	7,618	8
Load + Pert (z)	7.47	5,519	6
Contact Prox.	5.90	4,301	5

TABLE II: Real-Robot Evaluation Dataset

Experiment	Baseline [ 21 ]	MM-MLP	MM-Transformer
Load	100 $\pm$ 0.00%	99.74 $\pm$ 0.76%	99.48 $\pm$ 0.81%
Perturbation	82.18 $\pm$ 10.62%	87.96 $\pm$ 5.59%	87.87 $\pm$ 5.26%
Perturbation (z)	40.33 $\pm$ 6.92%	84.41 $\pm$ 9.62%	90.92 $\pm$ 5.94%
Load + Pert.	81.58 $\pm$ 6.53%	92.15 $\pm$ 4.27%	91.65 $\pm$ 3.44%
Load + Pert (z)	53.63 $\pm$ 8.96%	94.08 $\pm$ 2.89%	95.33 $\pm$ 1.48%
Contact Prox.	100 $\pm$ 0.00%	97.88 $\pm$ 1.85%	96.44 $\pm$ 3.59%

TABLE III: Classification accuracy across various real-robot experimental scenarios for RME-based Baseline [ 21 ] , Multimodal-MLP and -Transformer.

Panel of Figure 8 — Fig. 5: Classifier’s performance in continuous perturbation detection. Our model reliably flags contact ( red ) during the intervals when perturbations ( gray ) and loads ( orange ) are applied, including cases where a $z$ -axis perturbation is applied alongside loads. The red , black, and green plots represent $F_{\text{ext}}$ in the x-, y-, and z-directions, respectively.

Experiment	Contacts	Baseline	MM-MLP	MM-Trans.
Perturbation	9,461	91.45%	93.24%	94.95%
Perturbation (z)	1,722	25.09%	85.77%	93.84%
Load + Pert.	3,381	81.87%	94.20%	95.59%
Load + Pert. (z)	3,147	34.41%	95.07%	98.47%
Total	17,711	73.03%	93.02%	95.59%

TABLE IV: % of contact instances with controller in compliant mode .

Applied Load (g)	Avg. Tracking Error $(\times 10^{2}$ m)
Applied Load (g)	Soft	Stiff	Adaptive (Ours)
0	1.5 $\pm$ 0.15	0.7 $\pm$ 0.02	0.6 $\pm$ 0.16
500	3.1 $\pm$ 0.10	1.6 $\pm$ 0.02	0.8 $\pm$ 0.07
900	4.8 $\pm$ 0.07	2.2 $\pm$ 0.15	0.6 $\pm$ 0.09
Average	3.13	1.50	0.67

TABLE V: Tracking error after recovery from perturbation/payload.

Fig. 7: Intent estimation using interaction force estimation. As shown, the framework provides informative estimates of interactions, enabling natural target switching upon detecting human intent.

Fig. 8: Zero-shot framework deployment on bimanual setup.

Panel of Figure 13 — Fig. 9: EMP deployment on the soft , stiff, and adaptive controllers, compared against nominal desired EMP’s trajectory; $\times$ indicates task failure, and indicates target/pot position.

Controller	Soft	Stiff	Adaptive (ours)
Mean TE (m)	0.088 $\pm$ 0.015	0.028 $\pm$ 0.002	0.011 $\pm$ 0.002
Mean DTWD (m)	5.990 $\pm$ 0.563	0.585 $\pm$ 0.196	0.194 $\pm$ 0.030

TABLE VI: Tracking errors evaluated with Mean Tracking Error (the mean pointwise Euclidean distance between the current and the nominal trajectory), and the Mean Dynamic Time Warping Distance [ 47 ] (3 trials each).

Impedance Control

Impedance control lets robots adjust their stiffness and damping to manage forces during contact with people and objects.

See all 42 papers

Imp-ACT: Adaptive Impedance Control and Action Chunking with Transformers to Learn Contact-Rich Manipulation from Demonstrations
Sep 25, 2026Luca Zanetti, Doganay Sirintuna, Idil Ozdamar +3Contact-Rich Robotic Manipulation Impedance Control
Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, Genoa, Italy · Safe AI Lab, Carnegie Mellon University
IMPACT: Learning Internal-Model Predictive Control for Forceful Robotic Manipulation
Jun 9, 2026Jiawei Gao, Chaoqi Liu, Peilin Wu +2Robotic Control Contact-Rich Robotic Manipulation
Harvard University · Stanford University
FAME: Force-Adaptive RL for Expanding the Manipulation Envelope of a Full-Scale Humanoid
Mar 9, 2026Niraj Pudasaini, Yutong Zhang, Jensen Lavering +3Bimanual Robotic Manipulation Humanoid Robot Control
Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN 46556, USA. · Department of Computer Science, University of Colorado Boulder, Boulder, CO 80309, USA.

Sep 25, 2026J/K move · Enter open · S save

Imp-ACT: Adaptive Impedance Control and Action Chunking with Transformers to Learn Contact-Rich Manipulation from Demonstrations

Luca Zanetti, Doganay Sirintuna, Idil Ozdamar, Pietro Balatti+2

Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, Genoa, Italy · Safe AI Lab, Carnegie Mellon University

Contact-Rich Robotic Manipulation Impedance Control Robot Imitation Learning

Open paper AskPDF arXiv

Contact-rich manipulation requires robots to balance accurate motion tracking with compliant interaction, yet most visual-action policies leave compliance fixed at the controller level. We present Imp-ACT, a methodologically grounded and practical approach to incorporating direction-dependent Cartesian stiffness modulation directly into demonstration collection, without manual stiffness selection or offline target reconstruction. During teleoperation, a self-tuning impedance controller adapts stiffness along the instantaneous direction of motion while maintaining compliance in orthogonal directions. The adapted stiffness is applied and recorded alongside visual observations and motion commands, capturing motion and compliance under the same dynamics. We implement this pipeline using Action Chunking with Transformer (ACT) to predict end-effector pose, gripper action, and motion-direction stiffness from visual, proprioceptive, and wrench observations. The performance of Imp-ACT is evaluated on wiping and plug insertion using both success rate and quantitative measures of contact behavior. Compared with fixed low- and high-stiffness baselines, Imp-ACT achieves comparable or higher success while maintaining low interaction forces. In wiping, it reduces contact-force vibration by approximately

29\times

relative to the compliant baseline and

180\times

relative to the stiff baseline. In plug insertion, it reduces forces orthogonal to the insertion direction by

43\%

relative to the better fixed-stiffness baseline. These results highlight the benefit of maintaining sufficient stiffness along the direction needed for task execution while preserving compliance in other directions to limit contact forces and accommodate environmental constraints.

Abstract

Impedance Control

Explore similar work

Imp-ACT: Adaptive Impedance Control and Action Chunking with Transformers to Learn Contact-Rich Manipulation from Demonstrations

IMPACT: Learning Internal-Model Predictive Control for Forceful Robotic Manipulation

FAME: Force-Adaptive RL for Expanding the Manipulation Envelope of a Full-Scale Humanoid