CAFE: Closed-Loop Autonomous Task Formulation and Execution without Instance-Specific Instructions
Organizations: School of Electrical and Automation Engineering, Nanjing Normal University, Nanjing, Jiangsu 210023, China · Department of Mechanical Engineering, Temple University, Philadelphia, PA 19122, USA
Abstract
Scene-driven robot autonomy requires selecting tasks from observations as well as executing them. We study scene-driven task formulation and execution under predefined cleaning, safety, and storage/organization categories, without instance-specific goals or action sequences. CAFE maintains online memory of objects, functional zones, and execution states. A rule-guided Trigger-LLM admits formulation from accumulated evidence; Draft, Complete, and Audit propose goals, resolve execution details, and check dependencies. Outcomes update memory for correction and subsequent discovery. Evaluation in 12 AI2-THOR scenes compares three backbones and task-complexity tiers using coverage, constraint, and execution metrics. Tri-LLM records higher GCR scores than the evaluated baselines across three backbones and task-complexity tiers. Controls examine completion, auditing, zones, feedback, triggering, and prompt examples. LimoCobot traces illustrate storage, cleanup, and recovery from a failed grasp. The study examines how shared scene state connects task discovery to execution and outcome-driven revision.
Figures & tables
| Method | Initiation | State and planning | Closed-loop operation | Hardware | ||||
| No instance instruction | Long-horizon execution | Online semantic memory | Explicit checks | Execution recovery | Feedback into new goals | On-demand reasoning | Real robot | |
| SayCan [ 6 ] | ✔ | ✔ | ||||||
| ProgPrompt [ 12 ] | ✔ | ✔ | ✔ | ✔ | ||||
| LLM-Planner [ 13 ] | ✔ | ✔ | ✔ | ✔ | ||||
| EMPOWER [ 14 ] | ✔ | ✔ | ||||||
| AutoRT [ 16 ] | ✔ | ✔ | ✔ | |||||
| Method | Simple | Compound | Complex | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UC | CUR | sUC | SR | GCR | UC | CUR | sUC | SR | GCR | UC | CUR | sUC | SR | GCR | |
| Tri-LLM (DeepSeek-V3) | 0.917 | 0.717 | 0.667 | 0.865 | 0.793 | 0.874 | 0.763 | 0.750 | 0.808 | 0.706 | 0.896 | 0.936 | 0.825 | 0.842 | 0.754 |
| Tri-LLM (Qwen3-Max) | 1.000 | 0.800 | 0.812 | 0.972 | 0.972 | 0.972 | 0.725 | 0.870 | 0.915 | 0.889 | 0.922 | 0.815 | 0.892 | 0.856 | 0.789 |
| Tri-LLM (Gemini-2.5-Pro) | 1.000 | 0.925 | 0.800 | 0.960 | 0.960 | 0.960 | 0.725 | 0.782 | 0.875 | 0.840 | 0.920 | 0.613 | 0.826 | 0.800 | 0.736 |
| Single-LLM (DeepSeek-V3) | 0.833 | 0.717 | 0.875 | 0.858 | 0.715 | 0.812 | 0.425 | 0.604 | 0.672 | 0.546 | 0.409 | 0.252 | 0.142 | 0.608 | 0.249 |
| Single-LLM (Qwen3-Max) | 0.875 | 0.550 | 0.812 | 0.761 | 0.666 | 0.883 | 0.388 | 0.745 | 0.586 | 0.517 | 0.738 | 0.417 | 0.784 | 0.604 | 0.446 |
| Full | w/o Complete | |||||
|---|---|---|---|---|---|---|
| Tier | UC | CUR | sUC | UC | CUR | sUC |
| Simple | 1.000 | .800 | .812 | .750 | .775 | .250 |
| Compound | .972 | .725 | .870 | .945 | .788 | .193 |
| Complex | .922 | .815 | .892 | .905 | .820 | .194 |
Robot Task Planning
Robot task planning turns goals and observations into executable actions while coordinating skills, motion, uncertainty, and safety.
See all 255 papers