Organizations: Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genoa, Italy. · Georgia Institute of Technology, Atlanta, USA.
We present OmniVIC, a universal variable impedance controller (VIC) enhanced by a vision language model (VLM), which improves safety and adaptation in any contact-rich robotic manipulation task to enhance safe physical interaction. Traditional VIC have shown advantages when the robot physically interacts with the environment, but lack generalization in unseen, complex, and unstructured safe interactions in universal task scenarios involving contact or uncertainty. To this end, the proposed OmniVIC interprets task context derived reasoning from images and natural language and generates adaptive impedance parameters for a VIC controller. Specifically, the core of OmniVIC is a self-improving Retrieval-Augmented Generation(RAG) and in-context learning (ICL), where RAG retrieves relevant prior experiences from a structured memory bank to inform the controller about similar past tasks, and ICL leverages these retrieved examples and the prompt of current task to query the VLM for generating context-aware and adaptive impedance parameters for the current manipulation scenario. Therefore, a self-improved RAG and ICL guarantee OmniVIC works in universal task scenarios. The impedance parameter regulation is further informed by real-time force/torque feedback to ensure interaction forces remain within safe thresholds. We demonstrate that our method outperforms baselines on a suite of complex contact-rich tasks, both in simulation and on real-world robotic tasks, with improved success rates and reduced force violations. OmniVIC takes a step towards bridging high-level semantic reasoning and low-level compliant control, enabling safer and more generalizable manipulation. Overall, the average success rate increases from 27% (baseline) to 61.4% (OmniVIC).
Figures & tables
Fig. 1: Overview of the proposed OmniVIC. The system integrates a VLM with a VIC to enable safe and adaptive manipulation in contact-rich tasks. The VLM processes multimodal inputs, including visual observations, natural language instructions, and real-time force/torque feedback, to generate context-aware impedance parameters (stiffness and damping) for the VIC.
Fig. 2: Tasks used in simulation experiments.
Task No.
Descriptions
Task 1
put the bowl on top of the cabinet
Task 2
pick up the salad dressing and place it in the basket
Task 3
pick up the tomato sauce and place it in the basket
Task 4
pick up the milk and place it in the basket
Task 5
pick up the orange juice and place it in the basket
Task 6
put the wine bottle on top of the cabinet
TABLE I: Tasks in simulation experiments
Fig. 3: Deep dive into force and impedance profiles during task execution (Task 7 in Table I ) in low-level controller. Left and middle: stiffness and damping profiles of OmniVIC. Right: force (only Fz , the main contact force) profiles. The impedance profiles show that OmniVIC dynamically adjusts stiffness and damping based on task phases, reducing them during contact (middle area) to enhance compliance. The blue dots in force profiles demonstrate that OmniVIC maintains interaction forces within safe limits, avoiding spikes that violate constraints, while it happens (yellow dots in the red box) in the baseline controller and terminates the episode as a failure.
Fig. 4: Simulation results: Task success rates of baseline ( π0 with position control) vs. OmniVIC (with VLM-informed variable impedance control).
Fig. 5: Real-world experiments: Top: Task 1: The robot gently closes the top drawer. Middle: Task 2: The robot moves along the negative Y-axis while maintaining the same height in Z-axis, where there is a ramp in the middle of the path to introduce contact. Bottom: Task 3: the same as Task 2, but with two different ramp shapes to test multiple phases of contact.
Fig. 6: Task-1: The robot gently closes the top drawer using OmniVIC variable impedance control. The system adapts stiffness and damping parameters based on visual and language context to ensure safe interaction and avoid excessive force.
Fig. 7: Task‑2 (top): The robot moves along the negative Y‑axis while keeping a fixed Z‑height, with OmniVIC adjusting stiffness and damping from visual–language cues to maintain safe, low‑force interaction. Task‑3 (bottom): The robot executes a sequence of tasks, continuously adapting impedance parameters from visual–language context to avoid excessive force. Shaded regions denote contact phases detected by OmniVIC.
Contact-rich manipulation, requiring robots to regulate not only motion but also how they yield to external forces, has emerged as the next frontier for Vision-Language-Action (VLA) models. However, existing VLAs output purely kinematic commands, degrading performance on real-world contact-rich tasks. In this paper, we introduce CompVLA, a unified VLA framework that jointly predicts motion and stiffness matrix from RGB and language inputs. Our approach augments the conventional architecture with a dedicated Compliance Expert, which outputs time-varying stiffness and virtual displacement profiles executed via geometric impedance control. We demonstrate that CompVLA achieves the highest average success rate across diverse contact-rich tasks, outperforming both vanilla and compliance-aware VLA baselines, with ablations confirming each component is essential.
Jongmin Kim, Junsu Ha, Che-Sang Park +5
Seoul National University · Microsoft Research Asia
Bimanual mobile manipulation requires a seamless integration between high-level semantic reasoning and safe, compliant physical interaction - a challenge that end-to-end models approach opaquely and classical controllers lack the context to address. This paper presents GenerativeMPC, a hierarchical cyber-physical framework that explicitly bridges semantic scene understanding with physical control parameters for bimanual mobile manipulators. The system utilizes a Vision-Language Model with Retrieval-Augmented Generation (VLM-RAG) to translate visual and linguistic context into grounded control constraints, specifically outputting dynamic velocity limits and safety margins for a Whole-Body Model Predictive Controller (MPC). Simultaneously, the VLM-RAG module modulates virtual stiffness and damping gains for a unified impedance-admittance controller, enabling context-aware compliance during human-robot interaction. Our framework leverages an experience-driven vector database to ensure consistent parameter grounding without retraining. Experimental results in MuJoCo, IsaacSim, and on a physical bimanual platform confirm a 60% speed reduction near humans and safe, socially-aware navigation and manipulation through semantic-to-physical parameter grounding. This work advances the field of human-centric cybernetics by grounding large-scale cognitive models into predictable, high-frequency physical control loops.
Marcelino Julio Fernando, Miguel Altamirano Cabrera, Jeffrin Sam +3
Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology, Moscow, Russia
Contact-rich manipulation requires robots to balance accurate motion tracking with compliant interaction, yet most visual-action policies leave compliance fixed at the controller level. We present Imp-ACT, a methodologically grounded and practical approach to incorporating direction-dependent Cartesian stiffness modulation directly into demonstration collection, without manual stiffness selection or offline target reconstruction. During teleoperation, a self-tuning impedance controller adapts stiffness along the instantaneous direction of motion while maintaining compliance in orthogonal directions. The adapted stiffness is applied and recorded alongside visual observations and motion commands, capturing motion and compliance under the same dynamics. We implement this pipeline using Action Chunking with Transformer (ACT) to predict end-effector pose, gripper action, and motion-direction stiffness from visual, proprioceptive, and wrench observations. The performance of Imp-ACT is evaluated on wiping and plug insertion using both success rate and quantitative measures of contact behavior. Compared with fixed low- and high-stiffness baselines, Imp-ACT achieves comparable or higher success while maintaining low interaction forces. In wiping, it reduces contact-force vibration by approximately 29× relative to the compliant baseline and 180× relative to the stiff baseline. In plug insertion, it reduces forces orthogonal to the insertion direction by 43% relative to the better fixed-stiffness baseline. These results highlight the benefit of maintaining sufficient stiffness along the direction needed for task execution while preserving compliance in other directions to limit contact forces and accommodate environmental constraints.
Luca Zanetti, Doganay Sirintuna, Idil Ozdamar +3
Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, Genoa, Italy · Safe AI Lab, Carnegie Mellon University