IROSA: Interactive Robot Skill Adaptation using Natural Language

Markus Knauer; Samuel Bustamante; Thomas Eiband; Alin Albu-Schäffer; Freek Stulp; João Silvério

cs.ROMar 4, 2026

IROSA: Interactive Robot Skill Adaptation using Natural Language

Authors: Markus Knauer, Samuel Bustamante, Thomas Eiband, Alin Albu-Schäffer, Freek Stulp, João Silvério

Organizations: All authors are with the German Aerospace Center (DLR), Institute of Robotics and Mechatronics (RMC), Münchener Str. 20, 82234 Weßling, Germany. · Markus Knauer, Alin Albu-Schäffer and Samuel Bustamante are also with the School of Computation, Information and Technology (CIT), Technical University of Munich (TUM), Arcisstr. 21, 80333 Munich, Germany.

Abstract

Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.

Figures & tables

Fig. 1 : Overview of our approach I nteractive RO bot S kill A daptation using natural language. Showing the interactive selection and parameterization of a tool by a LLM based on a user query leading to a skill adaptation via the used execution model. Some of the tools we are providing are shown. ”Respond to User” is a general tool, whereas ”Repulsion Point”, ”Via-Point Insertion” and ”Speed Modulation” are specific tools to adapt KMPs.

Method	Approach	Training- Free	Open Vocab.	Zero- Shot	Explain- able	Local LLM
OLAF [ 18 ]	Policy retraining	$\times$	$\sim$	$\times$	$\times$	✓
LaTTe [ 12 ]	End-to-end	$\times$	$\times$	✓	$\times$	✓
Merlo [ 20 ]	Action Replan.	✓	✓	$\times$	✓	$\sim$
Yu [ 21 ]	Reward Synth.	$\times$	✓	$\times$	$\sim$	$\times$
OVITA [ 22 ]	Code Gen.	✓	✓	$\times$	✓	$\times$
Ours	Tool-based	✓	✓	✓	✓	✓

TABLE I : Comparison of Language-Conditioned Trajectory Adaptation Approaches. Training-Free: no task-specific (re)training required. Open Vocabulary: handles arbitrary natural language instructions. Zero-shot: no iterative user feedback required to validate model output. Explainable: adaptation steps are traceable and interpretable. Local LLM: operates with locally-deployed language models.

Panel of Figure 3 — Fig. 2 : Demonstration and prediction analysis for the pick-and-insert task. (Top) Trajectory in robot frame. (Bottom) Demonstrations and KMP model predictions with uncertainty visualization.

Fig. 3 : Speed adaptation results showing temporal trajectory modification through natural language commands. The plot shows the adapted trajectories following command “slow down between box and station,” demonstrating precise temporal control while preserving spatial trajectory characteristics.

Panel of Figure 5 — Fig. 4 : Trajectory adaptation through natural language command showing (top) experimental setup with new object (camera) placement and (bottom) resulting KMP trajectory modification with via-point insertion to reach the new object while maintaining task completion.

Panel of Figure 6 — Fig. 5 : Obstacle avoidance through natural language command showing (top) experimental workspace with introduced obstacle and (bottom) KMP trajectory modification using repulsion points. The system generates collision-free paths while maintaining the overall task structure and successfully completing the pick-and-insert operation.

Method	Time (s)
OVITA - LLM: Gemini (cloud)	72.1
OVITA - LLM: Qwen2.5-VL-72B-Instruct (local)	27.1
IROSA - LLM: Qwen2.5-VL-72B-Instruct (local)	15.4

TABLE II : Comparison with OVITA baseline

Experiment	OVITA (cloud)			OVITA (local)			IROSA (local)
	CSR	ISR	TCR	CSR	ISR	TCR	CSR	ISR	TCR
Speed Adapt.	100.0	100.0	100.0	70.0	10.0	30.0	100.0	100.0	100.0
Traj. Correct.	90.0	40.0	30.0	40.0	10.0	0.0	100.0	80.0	100.0
Obst. Avoid.	80.0	40.0	30.0	60.0	60.0	0.0	100.0	100.0	100.0

TABLE III : Quantitative comparison across experiments (10 prompts

\times

5 runs each). Prompts collected from public demonstration visitors and colleagues. Full prompt set available in our repository.

Explore similar work

MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation
Apr 22, 2026Markus Knauer, Edoardo Fiorini, Maximilian Mühlbauer +10Robot Skill Learning Robot Imitation Learning
German Aerospace Center (DLR), Institute of Robotics and Mechatronics (RMC), Münchener Str. 20, 82234 Weßling, Germany · School of Computation, Information and Technology (CIT), Technical University of Munich (TUM), Arcisstr. 21, 80333 Munich, Germany
RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation
Aug 26, 2026Sanghwan Jang, Minjin Jeon, Minsoo Kim +3Test-Time Adaptation of VLA Models Vision-Language-Action Models
Department of Computer Science and Engineering, POSTECH, Pohang, South Korea · Graduate School of Artificial Intelligence, POSTECH, Pohang, South Korea
Towards Robot Skill Learning and Adaptation with Gaussian Processes
Mar 2, 2026A K M Nadimul Haque, Fouad Sukkar, Sheila Sujipto +3Robot Skill Learning Robot Learning

Apr 22, 2026J/K move · Enter open · S save

MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation

Markus Knauer, Edoardo Fiorini, Maximilian Mühlbauer, Stefan Schneyer+9

German Aerospace Center (DLR), Institute of Robotics and Mechatronics (RMC), Münchener Str. 20, 82234 Weßling, Germany · School of Computation, Information and Technology (CIT), Technical University of Munich (TUM), Arcisstr. 21, 80333 Munich, Germany

Robot Skill Learning Robot Imitation Learning Industrial Robotics Language-Conditioned Robot Control Human-Robot Interaction Robot Learning

Open paper AskPDF arXiv

Industrial robot applications require increasingly flexible systems that non-expert users can easily adapt for varying tasks and environments. However, different adaptations benefit from different interaction modalities. We present an interactive framework that enables robot skill adaptation through three complementary modalities: kinesthetic touch for precise spatial corrections, natural language for high-level semantic modifications, and a graphical web interface for visualizing geometric relations and trajectories, inspecting and adjusting parameters, and editing via-points by drag-and-drop. The framework integrates five components: energy-based human-intention detection, a tool-based LLM architecture (where the LLM selects and parameterizes predefined functions rather than generating code) for safe natural language adaptation, Kernelized Movement Primitives (KMPs) for motion encoding, probabilistic Virtual Fixtures for guided demonstration recording, and ergodic control for surface finishing. We demonstrate that this tool-based LLM architecture generalizes skill adaptation from KMPs to ergodic control, enabling voice-commanded surface finishing. Validation on a 7-DoF torque-controlled robot at the Automatica 2025 trade fair demonstrates the practical applicability of our approach in industrial settings.

Abstract

Explore similar work

MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation

RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation

Towards Robot Skill Learning and Adaptation with Gaussian Processes