Automating Manual Tasks through Intuitive Robot Programming and Cognitive Robotics

Bijan Kavousian; Petar Tesic; Oliver Petrovic; Christian Brecher

cs.ROApr 7, 2026

Automating Manual Tasks through Intuitive Robot Programming and Cognitive Robotics

Authors: Bijan Kavousian, Petar Tesic, Oliver Petrovic, Christian Brecher

Organizations: Laboratory for Machine Tools and Production Engineering (WZL) of RWTH Aachen University, Steinbachstraße 19, D-52074 Aachen

Abstract

This paper presents a novel concept for intuitive end-user programming of robots, inspired by natural interaction between humans. Natural language and supportive gestures are translated into robot programs using large language models (LLMs) and computer vision (CV). Through equally natural system feedback in the form of clarification questions and visual representations, the generated program can be reviewed and adjusted, thereby ensuring safety, transparency, and user acceptance.

Robotics

Robotics studies the design and control of machines that perceive, move, and act in physical environments.

See all 751 papers

APPROVE: Visual End-User-in-the-Loop Robot Programming with LLMs
Aug 19, 2026Bijan Kavousian, Miray Özakkas, Josefine Monnet +2Robotics LLM-Based Program Synthesis
Laboratory for Machine Tools and Production Engineering WZL of RWTH Aachen University, Steinbachstraße 19, 52074 Aachen, Germany · RWTH Aachen University, Templergraben 55, 52062 Aachen, Germany
RoboCritics: Enabling Reliable End-to-End LLM Robot Programming through Expert-Informed Critics
Mar 6, 2026Callie Y. Kim, Nathan Thomas White, Evan He +2Robot Safety Program Synthesis
Department of Computer Sciences, University of Wisconsin–Madison, Madison, Wisconsin, USA
SHRIMP: Iterative Refinement of Robot Task Plans
Aug 9, 2026Mya Schroder, Yuna Hwang, Callie Y. Kim +5Large Language Model-Based Robot Planning Robot Task Planning
Department of Computer Sciences, University of Wisconsin–Madison, Madison, Wisconsin, USA

Aug 19, 2026J/K move · Enter open · S save

APPROVE: Visual End-User-in-the-Loop Robot Programming with LLMs

Bijan Kavousian, Miray Özakkas, Josefine Monnet, Oliver Petrovic+1

Laboratory for Machine Tools and Production Engineering WZL of RWTH Aachen University, Steinbachstraße 19, 52074 Aachen, Germany · RWTH Aachen University, Templergraben 55, 52062 Aachen, Germany

Robotics LLM-Based Program Synthesis Human-in-the-Loop AI

Open paper AskPDF arXiv

Programming robots remains challenging for non-experts, as traditional methods require expert knowledge and even block-based interfaces often lack flexibility. Recent work has explored Large Language Models (LLMs) to automatically generate robot programs from natural language, but these systems remain limited by a lack of transparency, missing mechanisms to ensure alignment with user intent, and little support for reuse. We present APPROVE (AI-Powered Programming for Robots with Visual End-User Feedback), an LLM-based multi-modal end-user programming framework that integrates natural language input with a block-based interface and an explicit user confirmation step. Generated programs are visualized using a block-based interface in Blockly, allowing users to confirm, modify, or reject them before execution. Confirmed functions are stored in a library for reuse, gradually building a set of reliable program components. Our approach contributes a human-centered design for LLM-based robot programming that emphasizes user trust, intent alignment, and reusability.

Abstract

Robotics

Explore similar work

APPROVE: Visual End-User-in-the-Loop Robot Programming with LLMs

RoboCritics: Enabling Reliable End-to-End LLM Robot Programming through Expert-Informed Critics

SHRIMP: Iterative Refinement of Robot Task Plans