Recent advances in frontier models enable robot manipulation from only a few demonstrations, but high inference latency limits their use for real-time robot control. To bridge this gap, we study two complementary approaches that connect frontier reasoning with low-latency local execution. First, we use a frontier model to autonomously generate demonstrations that supplement human demonstrations for training a fast local policy. We augment its in-context examples with corrective demonstration segments that show how to recover from physical errors, improving generation reliability. Generation time and cost decrease as successful examples accumulate in context, suggesting a path toward more efficient data collection. At deployment, a harness combines frontier-generated instructions with a fast local policy, enabling efficient execution while preserving the frontier model's ability to guide and correct actions. With low-latency execution delegated to the local policy, the bottleneck shifts to its capacity to reliably follow the frontier model's diverse instructions. Our second bridge introduces dense language supervision across three nested granularities---primitive, atomic, and composite---with multi-aspect descriptions at each level. Across long-horizon tasks in RoboCasa 365 and BEHAVIOR-1K, where instructions change as execution progresses, the combined supervision achieves the highest performance under both oracle and frontier-model instructors, demonstrating more reliable instruction following through the policy's language interface. Finally, we evaluate both bridges together on a crossword task that combines semantic planning and manipulation within a fixed time budget. These results support autonomous demonstration generation and dense language supervision as complementary components for connecting frontier reasoning to low-latency local execution.
Figures & tables
Figure 1: Autonomous manipulation from five successful in-context demonstrations. Task views and success counts are shown for Fable 5 and GPT-6 Astra across four tasks, with ten trials per model and task. Each trial starts from a reset environment and has a one-hour timeout, with no subsequent human intervention.
Figure 2: Corrective segments for pickup. Arrows illustrate adjustment directions; they are not recorded actions or model inputs.
Figure 3: Effect of adding corrective demonstration segments to the in-context examples. Success counts compare the baseline and augmented conditions for Fable 5 and GPT-6 Astra across four manipulation tasks, with ten trials per condition.
Figure 4: Generation efficiency as successful demonstrations accumulate in context. Bars show time in minutes (left axis), and lines show API cost in USD (right axis) for GPT-6 Astra over ten successful episodes per task. Episode indices count successful rollouts; time and API cost from failed attempts are excluded.
Figure 5: Average elapsed task time and task success across four control configurations: GPT-6 Astra alone, Astra with corrective demonstration segments (CS), the local policy alone, and the harness with Astra. Each panel reports one manipulation task, with n=20 trials per configuration and task. Time is averaged over successful trials only and includes reasoning, planning, tool calls, and API latency, excluding hardware-fix intervals. Success rate uses all 20 trials.
(a) Temporal granularity
Composite (task-level)
Open the cabinet. Place the apple and banana from the cabinet into the bowl. Then close the cabinet.
Atomic (subtask-level)
Open the cabinet → transfer the apple into the bowl → transfer the banana into the bowl → close the cabinet.
Primitive (apple transfer only)
Reach for the apple → grasp the apple → lift the apple → carry it over the bowl → lower the apple → release it into the bowl → retract the arm.
(b) Instruction formats for apple transfer
Arm/gripper kinematics
Grasp the apple, lift it out of the cabinet, carry it over the bowl, and release it into the bowl.
Spatial destinations
Move the apple from inside the cabinet into the bowl on the counter.
Table 1: Illustrative language supervision for MakeFruitBowl , an example task from RoboCasa 365, with an apple and a banana in the cabinet and a bowl on the counter. (a) The complete task and subtask sequence, with primitives shown only for apple transfer. (b) Four descriptions of that same apple-transfer subtask. Object names and kinematic descriptions must match the demonstrated episode.
RoboCasa 365
BEHAVIOR-1K
Language Supervision
Oracle
Fable 5
Oracle
Fable 5
Task-level (fixed)
0.169
0.076
Subtask-level
0.406
0.381
0.143
0.084
Task + Subtask
0.417
0.402
0.154
0.091
Hierarchical only
0.456
0.440
0.143
0.084
Multi-aspect only
0.425
0.406
0.160
0.103
Table 2: Effect of language supervision on long-horizon task performance under oracle and frontier-model instructors. Values average seeds 42 and 142: success rate on RoboCasa 365 and mean subgoal progress on BEHAVIOR-1K. Table 10 gives both seed results.
Language Supervision
Opus 5
Gemini-3.1-Pro
Fable 5
Original
Rephrased
Original
Rephrased
Original
Rephrased
Subtask-level
.346
.333 (-.013)
.325
.329 (+.004)
.392
.350 (-.042)
Hierarchical only
.413
.408 (-.005)
.400
.388 (-.012)
.446
.400 (-.046)
Multi-aspect only
.371
.363 (-.008)
.350
.350 (.000)
.417
.408 (-.009)
Hierarchical + Multi-aspect
.488
.471 (-.017)
.467
.454 (-.013)
.492
.483 (-.009)
Table 3: Task success rates on RoboCasa 365 by source planner (seed 42). Rephrased instructions are generated by Fable 5, preserving the task goal while changing the wording. Parentheses report changes between the displayed original and rephrased rates.
Figure 6: The crossword task. (a) Example board states and the robot workspace. (b) Mean total score after one hour across five games for four control configurations. Per-game scores are reported in Table 13 .
Figure 7: First 20 minutes of crossword execution under direct control and the frontier harness, both using GPT-6 Astra. Colored intervals show the frontier model’s initial planning and subsequent observation and reasoning, as well as robot action execution. Ticks mark model-requested camera observations. Each timeline represents one run per configuration. Full 60-minute timelines appear in Figure 11 .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Five interfaces for robot control, distinguished by the frontier model’s output and the components that translate it into physical actions.
Task
Corrective segments
Operate a Claw Machine
8
Stack Cups
8
Fold a Cloth
16
Close a Lid
16
Physical Crossword Puzzle
32
Appendix
Table 4: Number of corrective demonstration segments used per task.
Figure 9: Illustrative corrective demonstrations for crossword cube pickup. Each adjacent pair shows the initial offset (left) and subsequent approach (right) in a recorded wrist-camera sequence. The top row shows B and L cube pickups; the bottom row shows a second B cube pickup and a W cube pickup. Cyan arrows indicate illustrative alignment directions toward the target, manually added for visualization; they are not recorded action vectors or overlays supplied to the model. Timestamps indicate elapsed time within each episode.
Figure 10: Policy success rates with mixed and human-only training corpora. “Human demo + Synthetic” uses 20 human and 20 autonomously generated demonstrations per task; “Human demo only” uses 40 human demonstrations per task. Each bar reports success over 20 evaluation trials. Both conditions achieve 41/80 successes across the four tasks.
Primitive
Segments
Physical interpretation
Transport
6,256
Held object translates approximately horizontally.
Grasp
4,086
Gripper closes on an object.
Reach
4,066
Empty arm approaches an object with the gripper open.
Release
3,408
Gripper opens to release an object.
Lower
3,356
Held object descends.
Retract
3,057
Empty arm withdraws after release.
Appendix
Table 5: Composition of the RoboCasa primitive annotation corpus.
Source
Segments
Share
Object motion within the segment
14,935
55.1%
Inheritance within the grasp-to-release span
10,236
37.8%
Expanded temporal window
1,147
4.2%
Unresolved; decomposition label retained
795
2.9%
Simulator-derived total
26,318
97.1%
Appendix
Table 6: Sources of primitive-segment object labels. Simulator-derived coverage includes inherited identities and is not an annotation-accuracy estimate.
Check
Result
Transport with more than 6 mm of gripper-finger motion
0.0%; median 0.7 mm
Grasp containing gripper closing
96.9%
Release containing gripper opening
93.3%
Retract starting with gripper at least 80% open
74.4%; median openness 0.87
Retract overlapping release
0 / 3,057
Appendix
Table 7: Physical consistency checks for primitive annotations. These checks measure agreement with motion signals, not downstream task success.
Task
Subgoals
BreadAndCheese
2
ThawInSink
2
StartElectricKettle
2
ClearCuttingBoard
3
GatherCuttingTools
3
RinseBowls
3
Appendix
Table 8: RoboCasa 365 evaluation tasks by execution-plan depth. Subgoals count distinct instruction phases, not evaluation predicates.
Condition
Annotation levels in prompt
Training format
Task-level
Episode-level goal only
{goal}
Subtask-level
Current subgoal
{subgoal}
Task + Subtask
Episode-level goal + current subgoal (no primitive supervision)
{goal}. Current step: {sampled hierarchical instruction}
Appendix
Table 9: Prompt construction for the six language supervision conditions. Both hierarchical conditions pair the episode-level goal with the instruction corresponding to the sampled segment’s hierarchy level, using “Current step:” as a fixed phrase.
RoboCasa 365
BEHAVIOR-1K
Language Supervision
Oracle
Fable 5
Oracle
Fable 5
Task-level (fixed)
0.169 (0.175 / 0.163)
0.076 (0.050 / 0.102)
Subtask-level
0.406 (0.408 / 0.404)
0.381 (0.392 / 0.371)
0.143 (0.130 / 0.156)
0.084 (0.070 / 0.098)
Task + Subtask
0.417 (0.425 / 0.408)
0.402 (0.408 / 0.396)
0.154 (0.140 / 0.168)
0.091 (0.080 / 0.102)
Hierarchical only
0.456 (0.463 / 0.450)
0.440 (0.446 / 0.433)
0.143 (0.130 / 0.156)
0.084 (0.070 / 0.098)
Multi-aspect only
0.425 (0.433 / 0.417)
0.406 (0.417 / 0.396)
0.160 (0.150 / 0.170)
0.103 (0.090 / 0.116)
Appendix
Table 10: Language supervision results: mean, followed by (seed 42 / seed 142). Metrics are success rate for RoboCasa 365 and mean subgoal progress for BEHAVIOR-1K. The task-level condition uses a fixed task instruction and is shown once per benchmark.
Metric
Opus 5
Gemini- 3.1-Pro
Fable 5
Step-count match
29/30
25/30
27/30
Mean semantic similarity
.886
.855
.895
Appendix
Table 11: Plan quality against oracle references: step-count match and mean semantic similarity from direct 0–1 judgments of individual steps by Fable 5.
Primitive Action
Subtask-level
Hierarchical only
threshold
Object up ("lift the object up")
0.250
0.833
16 cm
Object down ("lower the object down")
0.139
0.750
16 cm
Arm up ("move the arm up")
0.111
0.750
34 cm
Arm down ("move the arm down")
0.111
0.778
34 cm
Arm forward ("move the arm forward")
0.056
0.556
24 cm
Arm back ("move the arm back")
0.028
0.444
24 cm
Appendix
Table 12: Single-seed policy compliance rates for primitive kinematic instructions.
Configuration
Game 1
Game 2
Game 3
Game 4
Game 5
Mean score
Fable 5 (direct control)
6 / 0
4 / 0
4 / 0
6 / 0
4 / 0
4.8
GPT-6 Astra (direct control)
8 / 0
4 / 0
10 / 1
10 / 1
8 / 0
8.0
GPT-6 Astra harness
14 / 2
16 / 2
14 / 1
14 / 1
14 / 1
14.4
GPT-6 Astra harness + dense supervision
16 / 2
16 / 2
14 / 1
16 / 1
16 / 2
15.6
Cell IDs prohibited in instructions
GPT-6 Astra harness
4 / 0
4 / 0
2 / 0
2 / 0
2 / 0
2.8
Appendix
Table 13: Crossword results after one hour per game. Each game cell reports total score / completed words. Scores award +2 per correctly placed cube and +2 per completed valid word; the final column reports mean total score. The final two rows prohibit cell IDs in frontier-generated instructions.
Configuration
Successful trials
Accuracy
Fable 5 (direct control)
5/20
25%
GPT-6 Astra (direct control)
13/20
65%
Local policy only
10/20
50%
GPT-6 Astra harness
10/20
50%
Appendix
Table 14: Crossword pick-and-place accuracy over 20 trials per configuration, excluding retries.
Figure 11: Full 60-minute crossword activity timelines for GPT-6 Astra, one run per configuration. Ticks mark camera observations requested by the model. Cumulative action-execution time is 2 min 40 s under direct control and 9 min 8.55 s with the harness; these are individual runs, not averages across games.
Vision-Language-Action (VLA) models have emerged as a promising paradigm for general-purpose robot control. However, their performance remains fundamentally constrained by the availability of high-quality robot trajectory data. In current robot learning practice, such data are primarily collected through human teleoperation, which is labor-intensive, costly, and difficult to scale. In this paper, we propose RDGen, a sim-to-real reinforcement learning framework for generating high-quality robot demonstrations. Rather than employing reinforcement learning solely as the final control policy, RDGen leverages trained RL policies as a structured trajectory generator. The system consists of a VLM-based task parser that identifies task-relevant objects, a Grounding DINO-based object localizer, and an RL policy transferred from simulation to the real robot. Successful rollouts are then harvested as clean, high-quality demonstrations for downstream VLA training, while the simulation stage further provides a scalable source of additional trajectories at little marginal cost. Experiments on a pick-and-place task demonstrate that the transferred RL policy achieves a high task success rate. Compared with human teleoperation, RDGen produces significantly smoother trajectories and yields superior downstream VLA performance. These results indicate that RL-generated demonstrations can serve as more reliable and consistent supervisory signals for robot policy learning.
We use a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy on RoboMME, a benchmark of 16 memory-dependent manipulation tasks spanning temporal, spatial, object, and procedural memory. The model keeps the whole episode in context and outputs every end-effector waypoint and gripper command, with no RoboMME-specific training. On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%, which was obtained with an oracle skill interface, and far ahead of the best leaderboard entry (60.8%) and of a hierarchical system that pairs the same model with a RoboMME-trained policy (79.1%). It matches or exceeds the human score on 13 of the 16 tasks. Its 63 failures concentrate in two tasks and are mostly premature completions and mistimed or unintended button presses. These results blur the usual division between long-horizon planning, memory and end-effector control, and raise the question of how much of this stack a single general-purpose model can take over in embodied systems. Project website: https://embodied-astra.github.io. Code, prompts and all episode logs: https://github.com/jackbjed/embodied-astra-robomme.
Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large language model (LLM) agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for embodied manipulation. We introduce FAEA (Frontier Agent as Embodied Agent), which applies an LLM agent framework directly to embodied manipulation without modification. Using the same iterative reasoning that enables software agents to debug code, FAEA enables embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier agent, Claude Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in embodied learning. Our results indicate that general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim