CAFE: Closed-Loop Autonomous Task Formulation and Execution without Instance-Specific Instructions
Authors: Mingjia Chen, Xuewen Zhao, Philip Dames, Qi Mao, Fei Xie, Jun Chen
Organizations: School of Electrical and Automation Engineering, Nanjing Normal University, Nanjing, Jiangsu 210023, China · Department of Mechanical Engineering, Temple University, Philadelphia, PA 19122, USA
Scene-driven robot autonomy requires selecting tasks from observations as well as executing them. We study scene-driven task formulation and execution under predefined cleaning, safety, and storage/organization categories, without instance-specific goals or action sequences. CAFE maintains online memory of objects, functional zones, and execution states. A rule-guided Trigger-LLM admits formulation from accumulated evidence; Draft, Complete, and Audit propose goals, resolve execution details, and check dependencies. Outcomes update memory for correction and subsequent discovery. Evaluation in 12 AI2-THOR scenes compares three backbones and task-complexity tiers using coverage, constraint, and execution metrics. Tri-LLM records higher GCR scores than the evaluated baselines across three backbones and task-complexity tiers. Controls examine completion, auditing, zones, feedback, triggering, and prompt examples. LimoCobot traces illustrate storage, cleanup, and recovery from a failed grasp. The study examines how shared scene state connects task discovery to execution and outcome-driven revision.
Figures & tables
Fig. 1: Overview of CAFE. Online observations update semantic memory; Trigger-LLM admits a Draft–Complete–Audit reasoning pass to form ordered executable tasks; execution outcomes return to memory for correction and replanning.
Method
Initiation
State and planning
Closed-loop operation
Hardware
No instance instruction
Long-horizon execution
Online semantic memory
Explicit checks
Execution recovery
Feedback into new goals
On-demand reasoning
Real robot
SayCan [ 6 ]
×
✔
×
△
△
×
×
✔
ProgPrompt [ 12 ]
×
✔
△
✔
✔
×
△
✔
LLM-Planner [ 13 ]
×
✔
✔
×
✔
×
✔
×
EMPOWER [ 14 ]
×
✔
△
×
×
×
×
✔
AutoRT [ 16 ]
✔
×
△
✔
△
△
×
✔
TABLE I: Comparison of reported system capabilities
Fig. 2: Scene-grounded formulation. Nodes N1–N4 ground washing, storage, and preparation zones. Draft proposes bowl-washing and potato-storage tasks; Complete binds parameters, and Audit checks dependencies. The bottom strip connects the grounded tasks to selected execution actions.
Fig. 3: AI2-THOR trajectories: cabinet-closing failure and retry before washing/storage (kitchen); clutter storage, bowl relocation, and computer shutoff (living room); faucet shutoff and trash disposal (bathroom). Frames show pivotal actions.
Fig. 4: Five-metric profiles (0–1 scale): complexity tiers by column and backbones by row. Solid/dashed profiles denote Tri/Single; dotted ProgPrompt appears only with Qwen3-Max. Exact metric values: Table II .
Method
Simple
Compound
Complex
UC
CUR
sUC
SR
GCR
UC
CUR
sUC
SR
GCR
UC
CUR
sUC
SR
GCR
Tri-LLM (DeepSeek-V3)
0.917
0.717
0.667
0.865
0.793
0.874
0.763
0.750
0.808
0.706
0.896
0.936
0.825
0.842
0.754
Tri-LLM (Qwen3-Max)
1.000
0.800
0.812
0.972
0.972
0.972
0.725
0.870
0.915
0.889
0.922
0.815
0.892
0.856
0.789
Tri-LLM (Gemini-2.5-Pro)
1.000
0.925
0.800
0.960
0.960
0.960
0.725
0.782
0.875
0.840
0.920
0.613
0.826
0.800
0.736
Single-LLM (DeepSeek-V3)
0.833
0.717
0.875
0.858
0.715
0.812
0.425
0.604
0.672
0.546
0.409
0.252
0.142
0.608
0.249
Single-LLM (Qwen3-Max)
0.875
0.550
0.812
0.761
0.666
0.883
0.388
0.745
0.586
0.517
0.738
0.417
0.784
0.604
0.446
TABLE II: Task formulation and execution across three complexity tiers. Higher is better; bold values mark the best score per tier and metric.
Full
w/o Complete
Tier
UC
CUR
sUC
UC
CUR
sUC
Simple
1.000
.800
.812
.750
.775
.250
Compound
.972
.725
.870
.945
.788
.193
Complex
.922
.815
.892
.905
.820
.194
TABLE III: Component and policy comparisons using Qwen3-Max.
Fig. 5: LimoCobot validation: (a) key storage and trash disposal; (b) prioritized spill cleanup, cloth storage, two trash disposals, and tape storage; (c) failed grasp and feedback-driven retry. Routes and frames connect scene-level task ordering to selected physical movements and actions.
Autonomous robots operating in open environments need the ability to continuously handle tasks that are not covered by predefined local methods. However, existing approaches often rely on repeated large-language-model (LLM) interaction for uncovered tasks, and even successful executions or observed successful external behaviors are not always autonomously transformed into reusable local knowledge. In this paper, we propose an LLM-driven closed-loop autonomous learning framework for robots facing uncovered tasks in open environments. The proposed framework first retrieves the local method library to determine whether a reusable solution already exists for the current task or observed event. If no suitable method is found, it triggers an autonomous learning process in which the LLM serves as a high-level reasoning component for task analysis, candidate model selection, data collection planning, and execution or observation strategy organization. The robot then learns from both self-execution and active observation, performs quasi-real-time training and adjustment, and consolidates the validated result into the local method library for future reuse. Through this recurring closed-loop process, the robot gradually converts both execution-derived and observation-derived experience into reusable local capability while reducing future dependence on repeated external LLM interaction. Results show that the proposed framework reduces execution time and LLM dependence in both repeated-task self-execution and observation-driven settings, for example reducing the average total execution time from 7.7772s to 6.7779s and the average number of LLM calls per task from 1.0 to 0.2 in the repeated-task self-execution experiments.
Hong Su
School of Computer Science, Chengdu University of Information Technology, Chengdu, China.
A mobile manipulator asked to "put the toys away in the toy box" must decide which objects are toys and which container the user means, then find base poses, grasps, and placements it can execute. Because a scene representation is built before the instruction is known, it may lack the attributes, affordances, or concepts needed to resolve the task. We present FEIJOA, a language-model agent that grounds the instruction at test time. It queries a grounding tool over a posed RGB-D scan, inspects the returned candidates, and refines its queries until they resolve the instruction. An embodiment-aware harness checks each skill's preconditions, samples base poses from which the target is reachable, and verifies the skill's effect from observations before and after execution. On 103 open-ended grounding queries, FEIJOA resolves 94.2% exactly, compared with 68.0% for the best prebuilt representation. Over 50 real-world trials on Spot, it completes 35 tasks; 17 of these successes follow recovery from a failed tool call or verification check. Of the 15 failed trials, 13 were caused by the robot's physical skills, and only 2 by grounding or the agent's reasoning. Project website: https://williamshen-nz.github.io/feijoa/.
Aditya Agarwal, William Shen, Sahit Chintalapudi +6
Chain-of-Thought (CoT) prompting enables LLMs to perform explicit, step-by-step reasoning, creating opportunities for sophisticated autonomous robots. However, recent research reveals that reasoning models verbalize their actual decision processes only 25-39% of the time, with faithfulness degrading 44% on complex tasks. This paper presents CT-SAFR (Chain-of-Thought Safety and Faithfulness for Robotics), a multi-layered verification framework achieving 94.2% hallucination detection (n = 500, 95% CI: 91.8-95.9%) with sub-500ms latency. Through a warehouse robot case study, this work demonstrates 87% reduction in unsafe reasoning outputs (p < 0.001) and provides recommendations for responsible deployment of reasoning-capable autonomous robots.
Cagri Temel
Hezarfen LLC Seattle, WA, USA · Grand Canyon University Phoenix, AZ, USA