CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoning
Organizations: School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30332, USA
Abstract
Robot learning has largely relied on human-teleoperated demonstrations to acquire effective learnable behaviors. However, human-operated data collection processes can be unintuitive, difficult to scale, and inherently asynchronous. We explore an alternative: distilling physical behavior from general-purpose multimodal foundation models into deployable robot policies by using the foundation model itself as an autonomous demonstrator. While sufficiently capable models can generate successful zero-shot manipulation trajectories, repeatedly invoking them during physical execution is slow and expensive, limiting their utility as scalable data generators. As a solution, we introduce CAPEX, an experience-conditioned demonstration collection framework that uses execution experience from previous attempts to adapt how frequently the foundation model must observe, reason, and replan. We evaluate across RoboCasa tasks and on physical Franka and bimanual YAM-arm platforms, measuring task success, model calls, token usage, collection time, and cost. We further train Diffusion Policy and ACT on matched sets of human-teleoperated and foundation-model-generated demonstrations to evaluate the downstream learning value of autonomously collected data. We find that CAPEX increases the number of successful demonstrations by 4.3x while reducing the cost per successful demonstration by 80%. Policies trained on CAPEX-generated data approach the performance of those trained on matched human demonstrations; with longer training, this gap largely closes for policies trained from scratch. These results suggest that foundation models can serve as scalable sources of reusable robot experience. Project page: https://capex-paper.github.io/
Figures & tables
| Method | Success (95% CI) | \downarrow$ | Requests / Success | Min. / Success |
| Astra (Base) | 45/360 12.5% (9.5–16.3) | 9.15 | 244 | 39.7 |
| Astra + CAPEX | 193/360 53.6% (48.4–58.7) | 1.83 | 37 | 6.6 |
| Qwen (Base) | 0/360 0.0% (0.0–1.1) | — | ||
| Qwen + CAPEX | 5/350 1.4% (0.6–3.3) | — | 4,819 | 2,332 |
| DP Fine-Tuned | DP From Scratch | ACT | ||||
| Task (Demos) | Ours | Human | Ours | Human | Ours | Human |
| Close Fridge (50) | 0.98 0.02 | 0.89 0.05 | 0.79 0.04 | 0.97 0.01 | 0.57 0.11 | 0.93 0.01 |
| Turn On Microwave (46) | 0.75 0.05 | 0.85 0.08 | 0.27 0.09 | 0.50 0.05 | 0.33 0.17 | 0.33 0.08 |
| PnP Counter Stove (50) | 0.25 0.06 | 0.29 0.04 | 0.07 0.01 | 0.01 0.01 | 0.29 0.05 | 0.09 0.03 |
| Close Toaster Oven (50) | 0.81 0.06 | 0.77 0.06 | 0.51 0.07 | 0.54 0.09 | 0.38 0.07 | 0.61 0.11 |
| PnP Toaster Counter (33) | 0.19 0.05 | 0.64 0.05 | 0.05 0.06 | 0.06 0.02 | 0.09 0.03 | 0.19 0.08 |
| YAM: Autonomous Collection | ||||
| Success | \downarrow$ | |||
| Task | Base | CAPEX | Base | CAPEX |
| Bread + Corn | 12/20 | 20/21 | 1.10 | 0.43 |
| Bowl Handover | 6/20 | 20/21 | 2.73 | 0.88 |
| Overall | 18/40 | 40/42 | 1.64 | 0.65 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Feature | ||
| 0 | bias | |
| 1 | stated log-odds | |
| 2 | gripper closed after waypoint | |
| 3 | gripper phase (gripper commands so far) | |
| 4 | leg length | |
| 5 | vertical leg |
| VLM | GPT-6 Astra (Responses API), reasoning effort medium, at most 8,000 output tokens per request |
| Open-weight VLM | Qwen3.8-27B with FP8 weights served by vLLM on one A40; temperature 1.0, top- 0.95, top- 20; 32k-token context |
| Plans | at most 24 waypoints; ; the first waypoint always runs |
| Episode limits | horizon (Table 7 ), 80 decisions, $25 of API cost |
| Memory | at most 8 skeleton phases and 2 references; all records used for the calibrator |
| Cost | list prices per million tokens: 1 cached input, 50 output (reasoning tokens are output). sums the cost of all attempts, failed ones included, and divides by the successes |
| Compute | one job per task and arm: 1 NVIDIA A40 (rendering, and serving Qwen3.8-27B), 8 CPU cores |
| Waypoint | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| Stated | 0.90 | 0.70 | 0.90 | 0.90 | 0.90 | 0.80 | 0.65 | 0.65 | 0.70 |
| Calibrated | 0.88 | 0.54 | 0.85 | 0.84 | 0.84 | 0.67 | 0.40 | 0.42 | 0.53 |
| Outcome | reached | reached | reached | reached | reached | stalled | deferred | ||
| GPT-6 Astra | Qwen3.8-27B | $/demo | ||||||
| Task | Success criterion | Base | CAPEX | Base | CAPEX | Base | CAPEX | |
| Pick and place | ||||||||
| CoffeeSetupMug | 600 | Mug within 4 cm (horizontal), 10 cm (vertical) of dispenser site; gripper 25 cm from object | 3 | 6 | 0 | 0 | 8.14 | 5.80 |
| PnPCounter Cabinet | 750 | Object bounding box inside cabinet interior; gripper 25 cm from object | 2 | 12 | 0 | 0 | 14.57 | 1.43 |
| PnPCounter Stove | 600 | Object touches pan, within 7 cm of its centre (horizontal); gripper 25 cm from object | 0 | 20 | 0 | 3 | — | 0.41 |
| PnPDrawer Counter | 750 | Object touches a counter; gripper 25 cm from object | 0 | 13 | 0 | 0 | — | 1.81 |
| Diffusion Policy (fine-tuned / scratch) | ACT | |
| Inputs | 3 RGB views at ; end-effector position, quaternion and gripper joints | |
| Actions | 12-D at 20 Hz: end-effector deltas, gripper, base, control mode | |
| Language | CLIP ViT-L/14 instruction embedding | none |
| Model | ResNet-18 encoders, transformer denoiser; DDPM, 100 steps; predicts 10 steps, executes 8 | ResNet-18, transformer, CVAE (latent 32); chunk 20, temporal ensembling |
| Initialisation | RoboCasa365 multitask checkpoint / random | ImageNet backbone |
| Optimiser | AdamW, lr , cosine, 500 warm-up steps, EMA | AdamW, lr , constant |
| Base GPT-6 Astra | CAPEX | |
| Prompt | interface only (the two-arm version of the simulation base prompt) | the simulation prompt plus YAM embodiment passages (grasp alignment, rim pinching) |
| Per decision | one waypoint | a plan of up to 24 waypoints, each with a stated confidence; committed up to the first later waypoint with stated confidence below 0.5 |
| Perception | RGB only | RGB, and locate_pixel on calibrated depth |
| Memory | none | reliability skeleton and two of the run’s own successes with their start images |
| Task | Teacher | Success | 95% CI | Min./success | Calls/success | $/success |
| Bread and corn | Base GPT-6 Astra | 12/20 | 39–78% | 3.33 | 25.7 | 1.10 |
| CAPEX | 20/21 | 77–99% | 1.49 | 7.5 | 0.43 | |
| Bowl handover | Base GPT-6 Astra | 6/20 | 15–52% | 7.58 | 61.5 | 2.73 |
| CAPEX | 20/21 | 77–99% | 2.26 | 15.8 | 0.88 | |
| Both | Base GPT-6 Astra | 18/40 | 4.75 | 37.6 | 1.64 | |
| CAPEX | 40/42 | 1.87 | 11.7 | 0.65 |