cs.ROOct 30, 2026
SaveTeach, Abstract, Apply: A Framework for Dialogue-Driven Task Learning with Collaborative Robots
Organizations: Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
Abstract
Collaborative robots (cobots) are designed to be safer and more interactive than traditional industrial robots, share the same workspace as humans, and assist them in a range of shop-floor tasks such as palletization, joint assembly, and part inspection. Many of these tasks involve repetitive workflows that may evolve with changes in production environments and task specifications. These requirements motivate the need for intuitive interfaces that allow workers to modify workflows on the shop floor and generalise these routines for future use. This work introduces the Teach-Abstract-Apply (TAA) framework, which enables human operators to (a) teach new workflows to cobots in natural language, (b) abstract the taught workflows into skills for later use, and (c) apply these skills in different settings. As a case study, we apply the TAA framework to a collaborative structure-building task and evaluate it with both open-weight (Qwen3-Coder) and closed (GPT5.2-Codex) models in self-play and cross-model configurations. Our results show that skill abstraction improves task success rates and reduces interaction cost. Beyond automated evaluation, a preliminary human-in-the-loop study suggests that the pipeline also works with human instructions.
Figures & tables
Figure 1: Overview of the Teach and Abstract steps represented through the structure building task in a simulated environment. Teach: the worker uses natural-language to teach the Cobot an assembly arrangement; Abstract: successful interaction traces are abstracted into a reusable skill.
Figure 2: Overview of the Apply step represented through the structure-building task in a simulated environment. Reuse: refers to using an existing skill at a new location; Repeat: refers to using an existing skill with new properties — the washer color is set to ‘green’ and the skill is repeated across a set of locations in a pattern.
Figure 3: An example interaction trace showing how the conversation proceeds. At Turn 1, the Programmer instructs the Cobot to place a red screw on top of a yellow washer at position [1,3]. However, the Cobot responds with an action that places only a washer and also uses the wrong color. The Programmer corrects the color and proceeds with the next step. At Turn 3, the Cobot asks for clarification about the screw color, although it was already specified in Turn 1. In the next turn, the Programmer repeats the details. The shared context shows how the reconstructed board is updated after each Cobot action. When the Programmer responds with DONE , the interaction is completed.
Figure 4: Overview of reusable function generation. Given the instruction-response pairs and a predefined function signature, the Code LLM abstracts the primitive code snippets into a reusable higher-order function. The generated function is then evaluated by comparing the structure it reconstructs with the structure produced by the input code snippets, evaluating each cell for component type, color, and stacking level.
| Model | Success Rate | Token Cost ($) | ||
| Reconstruction | Optimization | Overall | ||
| Qwen3-Coder | 0.69 | 0.93 | 0.64 | 1.41 |
| GPT5.2-Codex | 0.93 | 1.00 | 0.93 | 10.43 |
Table 1: Performance comparison between Qwen3-Coder and GPT5.2-Codex on reconstruction and optimization tasks, reporting success rates for each component and overall performance along with token cost; Overall = Reconstruction x Optimization success; Token cost is the total across all test instances.
Figure 5: Performance of Qwen3-Coder and GPT5.2-Codex as task complexity increases (measured by the number of elements per object). The plots show average number of clarification and correction turns, and overall success rate for reconstruction task with varying structural complexity.
Figure 6: Success rate of Qwen3-Coder and GPT5.2-Codex on code optimization objectives. Each point represents the proportion of test instances where the model correctly applied the required transformation, including using correct function names, using loop constructs, reducing put() calls, and minimizing output byte length; Vertical lines connect the two models scores to highlight the gap.
Figure 7: Cost–performance trade-off for the reuse task. Each point represents a model configuration, comparing success rate and token cost when responses are generated without skills, with Qwen skills, or with Codex skills; G-Q, Q-Q etc. Labels denote Programmer–Cobot pairs; G-Q: GPT as Programmer and Qwen as Cobot; Q-G: Qwen as Programmer and GPT as Cobot; The y-axis is truncated to the range 0.6-1.0 to highlight differences among performance scores.
Figure 8: Trade-off between token cost and repeat task success rate across model configurations. Results compare responses generated without skills, with Qwen skills, and with Codex skills, showing that skill usage significantly improves success rates while maintaining relatively low token cost; The y-axis is truncated to the range to highlight differences among performance scores.
| Evaluation Task | Skill-Set | Model | OTSR | SRSR (%) | FRSR (%) |
| Reuse | Qwen | Q-Q | 0.865 | 83.64 | 16.35 |
| G-Q | 0.994 | 93.90 | 6.10 | ||
| G-G | 0.970 | 86.67 | 13.33 | ||
| Q-G | 0.943 | 88.65 | 11.35 | ||
| Codex | Q-Q | 0.863 | 89.23 | 10.77 | |
| G-Q | 0.992 | 94.50 | 5.50 |
Table 2: Skill retrieval rate across model configurations for Reuse and Repeat tasks. The highlighted values indicate the higher-performing configuration when using the corresponding skill set. Labels denote Programmer–Cobot pairs; G-Q: GPT as Programmer and Qwen as Cobot; Q-G: Qwen as Programmer and GPT as Cobot; OTSR: Overall Task Success Rate; SRSR: Skill-retrieval Success Rate; FRSR: Fall-back Reconstruction Success Rate.
| Evaluation Task | Skill-Set | Model | Hallucinated (%) |
| Reuse | Qwen | Q-Q | 29.61 |
| G-Q | 34.41 | ||
| G-G | 24.76 | ||
| Q-G | 26.09 | ||
| Codex | Q-Q | 7.87 | |
| G-Q | 6.47 |
Table 3: Skill hallucination percentages across model configurations for Reuse and Repeat tasks computed as the ratio of hallucinated skills to the total retrieved skills. Labels denote Programmer–Cobot pairs; G-Q: GPT as Programmer and Qwen as Cobot; Q-G: Qwen as Programmer and GPT as Cobot;
Figure 9: Qualitative examples (Q-Q setup) showing clarification and correction during reconstruction, misinterpretation during skill reuse, fallback from reuse to reconstruction, and hallucinated skill use. These showcase how Programmer and Cobot interact across task settings, how errors arise from ambiguous or incorrect skill selection, and how the system either continues through correction, falls back to primitive reconstruction; Q-Q: Qwen3-Coder is used as both Programmer and Cobot.
Figure 10: Reuse Task: Clarification behavior under skill reuse settings as a function of task complexity. (a) Clarification episodes remain sparse across all model configurations, with occasional increases for G–Q at higher complexity, while Q–Q and H–Q exhibit minimal clarification needs overall. (b) Median dialogue turns for successful instances are nearly constant, with H–Q consistently requiring two turns, and both Q–Q and G–Q completing interactions in approximately a single turn. Model Configuration: Q–Q ( Qwen3-Coder – Qwen3-Coder ), G–Q ( GPT5.2-Codex – Qwen3-Coder ), and H–Q (Human– Qwen3-Coder ).
Figure 11: Repeat Task: Clarification behavior across model configuration as a function of task complexity. (a) Number of clarification episodes increases with the number of elements, with Q–Q and H–Q showing stronger sensitivity compared to G–Q. (b) Median dialogue turns for successful instances remain relatively stable, with H–Q requiring consistently more turns, while G–Q maintains minimal interaction length across all settings. Model Configuration: Q–Q ( Qwen3-Coder – Qwen3-Coder ), G–Q ( GPT5.2-Codex – Qwen3-Coder ), and H–Q (Human– Qwen3-Coder ).
| Model Configuration | Reuse | Repeat |
| Human-Q | 0.96 | 0.94 |
| Q-Q | 0.88 | 0.78 |
| G-Q | 1.00 | 0.94 |
Table 4: Success rates on reuse and repeat tasks across model configurations, including a human baseline for the evaluated subset using Qwen skill set. Results highlight how configuration choices impact performance, with best results per task shown in bold.
| Ablations | Abort Rate | Failure Rate | Success Rate |
| n =1 | 0.013 | 0.642 | 0.345 |
| n =3 | 0.006 | 0.558 | 0.436 |
| n =5 | 0.006 | 0.448 | 0.545 |
| n =10 | 0.012 | 0.424 | 0.564 |
| n =15 | 0.0 | 0.382 | 0.618 |
| n =20 | 0.0 | 0.380 | 0.620 |
Table 5: Overview of ablation results for the optimal number of turns required for the player interaction; A higher success rate indicates better performance.
| Ablations | Abort Rate | Failure Rate | Success Rate |
| n =1 | 0.006 | 0.552 | 0.442 |
| n =3 | 0.018 | 0.352 | 0.630 |
| n =5 | 0.000 | 0.376 | 0.624 |
Table 6: Overview of ablation results for the optimal number of turns required for the abstraction; A higher success rate indicates better performance.
| Ablations | Abort Rate | Failure Rate | Success Rate |
| n =1 | 0.182 | 0.224 | 0.594 |
| n =3 | 0.039 | 0.276 | 0.685 |
| n =5 | 0.000 | 0.309 | 0.691 |
Table 7: Overview of ablation results for the optimal number of turns required for the Robot to retry in case of execution errors; A higher success rate indicates better performance.
| Ablations | Abort Rate | Failure Rate | Success Rate |
| No difference grid | 0.182 | 0.224 | 0.594 |
| No error feedback | 0.594 | 0.055 | 0.352 |
| No clarification questions | 0.455 | 0.097 | 0.448 |
Table 8: Overview of ablation results for including various types of feedback in the interaction; A higher success rate indicates better performance.
Figure 12: Part-1 of prompt template used for the Programmer in the Teach step.
Figure 13: Part-2 of prompt template used for the Programmer in the Teach step.
Figure 14: An example goal board representation.
Figure 15: An example difference board representation.
Figure 16: Part-1 of prompt template used for the Robot in the Teach step.
Figure 17: Part-2 of prompt template used for the Robot in the Teach step.
Figure 18: Part-3 of prompt template used for the Robot in the Teach step.
Figure 19: Part-4 of prompt template used for the Robot in the Teach step.
Figure 20: Part-5 of prompt template used for the Robot in the Teach step.
Figure 21: Part-6 of prompt template used for the Robot in the Teach step.
Figure 22: Part-7 of prompt template used for the Robot in the Teach step.
Figure 23: Part-8 of prompt template used for the Robot in the Teach step.
Figure 24: Part-1 of prompt template used in the Abstract step.
Figure 25: Part-2 of prompt template used in the Abstract step.
Figure 26: Part-3 of prompt template used in the Abstract step.
Figure 27: Prompt template used for the Programmer in the Reuse step.
Figure 28: Prompt template used for the Robot in the Reuse step.
Figure 29: Prompt template used for the Programmer in the Repeat step.
Figure 30: Prompt template used for the Robot in the Repeat step.
More in this topic
Robot Learning
Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.
See all 329 papers