Encore: Few-Shot Agentic Discovery of Manipulation Strategies

Yifan Kang; Zihan Wang; Zhiwen Fan; Bangya Liu

cs.ROSep 29, 2026

Encore: Few-Shot Agentic Discovery of Manipulation Strategies

Authors: Yifan Kang, Zihan Wang, Zhiwen Fan, Bangya Liu

Organizations: Massachusetts Institute of Technology · AI Open Source Foundation · Texas A&M University · University of Wisconsin–Madison

Abstract

Coding agents can now write, run, and debug programs with little human help. Robot tasks, however, are usually specified by a sentence that leaves out how to grasp, in what order to make contact, and what the result should look like, and an agent given only the sentence must find these details by trial and error. We introduce ENCORE, which gives the agent a few demonstrations as evidence to read rather than as training data. A deterministic builder distills each demonstration into a pack of multi-view keyframes, gripper events, frame strips, and the full trajectory. A coding agent studies the pack, writes a policy program against a fixed perception and action API, refines it iteratively over a few development rollouts, and freezes it before a sealed evaluation that never reveals the success signal. On LIBERO-PRO, the agent's first program already succeeds in half of the perturbed tasks with demonstrations and in one task without them, and the frozen programs outperform the strongest prior agentic system run with the same language model (96.3% against 89.3%). On RoboDojo tasks whose instructions leave the goal unstated, no program succeeds without demonstrations. ENCORE also runs on a real bimanual robot, learning cube handover and cup inversion from five demonstrations each.

Figures & tables

Figure 1: Encore . A coding agent writes a policy program, runs it in simulation or on a bimanual robot, and refines it iteratively, with no success signal at run time. In the handover pack shown, the two grips overlap through the transfer, which the task sentence never mentions.

	call	returns
sense	capture(cam)	RGB-D frame with intrinsics and extrinsics
	deproject(u, v)	3-D point in the robot frame
	eef() , tool_rotation() , gripper()	pose, tool rotation, jaw width and effort
perceive	sam3(q) , ground(q) , vqa(q)	masks, boxes, or an answer for a text prompt
act	move(xyz, R) , move_path(pts, R)	Cartesian motion; returns the position residual
	grip(w)	sets the jaw width

Table 1: The API a policy program calls, the same in simulation and on hardware.

Figure 2: Development on LIBERO-PRO, Pos and Task cells pooled. (a) Mean development success of each cell’s current program at version

k

(60 cells; a frozen cell keeps its final version). (b) Paired per cell, how often the

K{=}3

agent needs less or more of each resource. The demonstrations reduce the number of attempts, while agent time and cost stay similar (0.51 against 0.55 hours per cell), because the

K{=}3

agent spends part of the saved iterations reading the pack. Same harness, budget, model, and sealed evaluation.

Figure 3: Recombination on the Task axis. Each cell gets two packs (left, centre) and a sentence asking for a third task (right) that neither demonstrates; red marks the object the sentence takes, teal the target, and which pack supplies which half flips between the two cells. Both cells score $50/50$ with the two packs.

Figure 4: LIBERO-PRO [ 12 ] , sealed success per suite and axis (10 tasks

\times

50 episodes).

\pi_{0.5}

, CaP-Agent0, and ASPIRE on opus-4.6 as reported by Lu et al. [8] ; ASPIRE on opus-5 is our rerun of its released code.

K{=}1

is in Appendix A .

Figure 5: RoboDojo tasks, one sealed $K{=}3$ success each (start, middle, end). The first row states its goal only through the demonstrations; the others need a delicate contact or a visual cue.

task	$K{=}3$	$K{=}1$	$K{=}0$
build tower	$\mathbf{36}$	$0$	$0$
make kong	$\mathbf{36}$	$0$	$0$
bottles to dustbin	$\mathbf{15}$	$2$	$0$
classify objects	$\mathbf{8}$	$7$	$0$
arrange by number	$0$	$\mathbf{31}$	$0$
10 tasks	$\mathbf{95}$	$40$	$0$

Table 2: RoboDojo, successes in 50 sealed episodes per task, scored by the benchmark’s judge. Left: the tasks not shown (organize table, pack objects, imitate sorting, fold clothes) score 0 in every arm, and one task has no demonstration data. Right: tasks with an images-only arm; the six contact-precise tasks score 0 in every arm (

K{=}3

, images and

K{=}0

on 20 episodes). Bold: best arm.

Figure 6: From five demonstrations to a frozen policy on the real robot: (a) a cup-inversion demonstration as the pack presents it; (b) the agent’s reading, quoted from its notes; (c) the frozen program, which verifies its own outcome by querying a VLM about the scene.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	libero-object		libero-goal		libero-spatial		Overall
Method	Pos	Task	Pos	Task	Pos	Task	Pos	Task
OpenVLA	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
$\pi_{0}$	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
$\pi_{0.5}$	0.17	0.01	0.38	0.00	0.20	0.01	0.25	0.01
CaP-Agent0	0.22	0.18	0.26	0.17	0.12	0.14	0.20	0.16
ASPIRE (opus-4.6)	0.98	0.95	0.81	0.45	0.51	0.60	0.77	0.67

Appendix

Table 3: LIBERO-PRO success under position ( Pos ) and instruction ( Task ) perturbation, macro-averaged over 10 tasks per suite.

libero-object
	Stock	Pos			Task
Task	$K{=}3$	$K{=}3$	$K{=}1$	$K{=}0$	$K{=}3$	$K{=}1$	$K{=}0$
alphabet soup	1.00	1.00	1.00	1.00	1.00	1.00	1.00
bbq sauce	1.00	1.00	1.00	1.00	1.00	1.00	1.00
butter	1.00	1.00	0.98	1.00	1.00	1.00	1.00
chocolate pudding	1.00	1.00	1.00	1.00	1.00	1.00	1.00

Appendix

Table 4: Per-task success on LIBERO-PRO, 50 sealed episodes per cell. Stock is the unperturbed task.

K{=}3

receives three demonstrations,

K{=}1

one,

K{=}0

none, with the framework otherwise identical.

verification ablation, Encore
Arm	Success
full system ( $K{=}3$ )	$440/450$
verification loop removed	$117/450$
ASPIRE’s loop
as released	$431/500$
plus three demonstrations as raw trajectories	$437/500$

Appendix

Table 5: Ablations. Top: verification loop excised with demonstrations kept (nine goal-suite tasks, 50 sealed episodes each; earlier brief with a note file shared across cells). Bottom: ASPIRE’s released loop on opus-5 on the goal-swap axis (10 tasks

\times

50 seeds), with and without the three demonstrations as raw trajectory files.

Explore similar work

RAPID: Robot Agentic Programming from Demonstrations
Sep 24, 2026Yuyao Liu, Jiayuan Mao, David Hsu +2Robot Systems Real-World Manipulation Tasks
Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1
ENPIRE: Agentic Robot Policy Self-Improvement in the Real World
Jun 18, 2026Wenli Xiao, Jia Xie, Tonghe Zhang +14Robot Policies Embodied Agents
1NVIDIA · 2CMU · 3UC Berkeley
CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
Sep 27, 2026Yiheng Lyu, Xueying Jiang, Wenhao Li +2
Nanyang Technological University, Singapore · Independent Researcher

Sep 24, 2026, cs.ROJ/K move · Enter open · S save

RAPID: Robot Agentic Programming from Demonstrations

Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling+1

Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania+1

Robot Systems Real-World Manipulation Tasks Action Expert Object-Centric Representations

Open paper AskPDF arXiv

Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.

Abstract

Explore similar work

RAPID: Robot Agentic Programming from Demonstrations

ENPIRE: Agentic Robot Policy Self-Improvement in the Real World

CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation