Agent Priors-guided Policy Learning

Puming Jiang; Tianrun Hu; Haozhe Du; Yibo Li; Zhiwei Xue; Xinhu Li; Harold Soh

cs.ROSep 28, 2026

Agent Priors-guided Policy Learning

Authors: Puming Jiang, Tianrun Hu, Haozhe Du, Yibo Li, Zhiwei Xue, Xinhu Li, Harold Soh

Organizations: National University of Singapore

Abstract

Robots that learn from a few demonstrations often require two forms of generalization. Compositional generalization recombines skills to solve new tasks, and skill generalization lets the learned policy behind each skill work in new situations. The two depend on each other, yet information is lost between composition and the skills it calls. Where a skill works is determined by the structure its policy is trained with, while composition sees the skill only through a separate description, such as a name, an instruction, or a symbolic operator, that omits this structure. Our key idea is to use each policy's structural prior as part of the interface between composition and the skill. A structural prior states what a behavior depends on, for example that a grasp depends only on the gripper's pose relative to the object. Built into training, it shapes where the policy generalizes; stated in language, it tells composition where the policy applies. We instantiate this idea in Agent Priors-guided Policy Learning (APPL). A construction agent segments complete demonstrations into reusable skills, proposes several structural priors for each skill, and trains and verifies one policy per prior. A runtime agent then selects among these prior-specific policies and composes them toward new task goals using their interfaces. Across MetaWorld and long-horizon ManiSkill tasks, APPL improves out-of-distribution skill generalization and enables previously unseen skill compositions; ablating the interface information substantially reduces performance. These results support the use of training-time structural assumptions as a bridge between skill learning and skill composition.

Figures & tables

Panel of Figure 1 — Figure 1: APPL builds prior-specific policies offline and composes them online. The construction agent segments demonstrations, proposes several priors per skill, trains one documented policy per prior, and verifies each policy from demonstrated entry states. The runtime agent reads these documents to choose a policy, its arguments, a duration, and stop conditions. Orange marks the prior’s two roles.

	$N=2$		$N=5$		$N=10$		$N=20$		Mean
Method	IID	OOD	IID	OOD	IID	OOD	IID	OOD	OOD
Diffusion Policy (B0)	73.33	28.96	82.50	37.29	97.50	38.54	100.00	46.67	37.86
Relational prior (B1)	85.83	37.92	93.33	42.71	100.00	52.29	100.00	56.25	47.29
APPL, first proposal ( $q_{1}$ )	84.17	56.04	95.83	63.54	100.00	73.33	100.00	78.75	67.92
APPL, best of three ( $q_{3}$ )	90.83	80.00	100.00	81.46	100.00	90.42	100.00	88.96	85.21
APPL ( $q_{4}$ )	96.67	89.58	100.00	93.54	100.00	93.33	100.00	93.13	92.40

Table 1: Exp. 1 hidden-test success (%), averaged equally over six tasks, with

N

demonstrations per task. IID uses fresh states within the demonstrated factor ranges; OOD averages recombined (C) and extrapolated (E) states equally. Mean averages OOD over the four values of

N

. Bold marks the best entry in each column.

q_{1}

is the first proposal (A1);

q_{3}

and

q_{4}

are selected by development OOD success from A1–A3 and from A1–A4.

Panel of Figure 2 — Figure 2: Five long-horizon tasks at their demonstrated start states. The robot exchanges blocks between a drawer and a pad, swaps two blocks through a buffer, stores a block from a low tunnel in a drawer, uncovers and inserts a peg, and pours particles into a bowl before returning the container.

	Drawer		Buffer		Retrieve		Peg		Pour		Total (%)
Method	M	T	M	T	M	T	M	T	M	T	M	T	Comp.
DP	0/8	–	0/8	–	1/8	–	3/8	–	0/8	–	10.0	–	–
SinglePrior	1/8	–	0/8	–	2/8	–	1/8	–	0/8	–	10.0	–	–
Agent+VLA	1/8	8/8	1/8	4/8	5/8	8/8	5/8	4/8	1/8	4/8	32.5	70.0	6/16
APPL w/o interface information	0/8	4/8	4/8	5/8	2/8	6/8	0/8	7/8	2/8	4/8	20.0	65.0	2/16
APPL w/o prior information	4/8	7/8	0/8	4/8	4/8	6/8	3/8	6/8	7/8	6/8	45.0	72.5	3/16

Table 2: Exp. 2 first-attainment success. Each task has eight motion-OOD (M) and eight task-level-OOD (T) cases, with one episode per case; totals average over five tasks. Comp. reports successes on 16 new compositions. DP and SinglePrior have no goal input (–). Bold marks the best entry per column.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task (native ID; $D$ )	Factor	L	H	E-L	E-H
pick-place-wall	object $x$	$[-.030,-.015]$	$[.015,.030]$	$[-.050,-.040]$	$[.040,.050]$
(pick-place-wall-v3; 45)	goal $x$	$[-.030,-.015]$	$[.015,.030]$	$[-.050,-.040]$	$[.040,.050]$
assembly	nut $x$	$[-.040,-.020]$	$[.020,.040]$	$[-.080,-.060]$	$[.060,.080]$
(assembly-v3; 42)	peg goal $x$	$[-.050,-.030]$	$[.030,.050]$	$[-.100,-.080]$	$[.080,.100]$
drawer	cabinet $x$	$[-.040,-.020]$	$[.020,.040]$	$[-.100,-.070]$	$[.070,.100]$
(drawer-open-v3; 41)	cabinet yaw	$[-10,-5]$	$[5,10]$	$[-37,-33]$	$[33,37]$

Appendix

Table 3: Layout factors. Translations are in metres and yaw in degrees;

D

is the dimension of one observation. L/H are the demonstrated (and IID) intervals of each factor, and E-L/E-H are the extrapolation intervals.

Figure 3: Demonstrated and evaluation factor layouts. Support points are the 20 demonstrations per task; test states were not available during design.

Task	Success condition
pick-place-wall	Object-to-goal Euclidean distance $\leq.07$ .
assembly	RoundNut site’s horizontal target error $<.02$ and target height minus that site’s height $>0$ .
drawer	$p\in[.75,1.01]$ for three consecutive control steps; joint validity requires $p\in[-.01,1.01]$ throughout the episode.
door	Same progress, hold, and validity rule as drawer, with the adapted rotating-root geometry.
peg-insert-side	$\\|(\mathrm{pegHead}-\mathrm{target})\odot(1,2,2)\\|_{2}\leq.07$ .
stick-push	Object2-to-target distance $\leq.12$ , and native grasp success: contact with the main object, positive aperture, and $z_{\rm stick}-.01>z_{\rm initial\ stick}$ .

Appendix

Table 4: Success conditions. Distances are in metres;

p

is the normalized joint travel from closed (0) to open (1).

Setting	Value
History / horizon / executed actions	2 observations / 16 / 4
U-Net	Down dimensions $[64,128,256]$ ; step embedding 128; kernel 5; 8 GroupNorm groups
Optimizer	AdamW; learning rate $10^{-4}$ (constant); $\beta=(.9,.999)$ ; weight decay $10^{-6}$
Updates / batch	20,000 / 128; gradient-norm clipping 1.0; EMA decay .995
Training diffusion	DDPM, 100 steps, epsilon prediction, squaredcos_cap_v2 schedule
Sampling	DDIM, 16 steps, $\eta=0$

Appendix

Table 5: Shared training and sampling settings.

System	$N=2$	$N=5$	$N=10$	$N=20$
B0	28.96 [26.88, 31.04]	37.29 [35.00, 39.58]	38.54 [36.25, 40.83]	46.67 [44.17, 49.17]
B1	37.92 [35.83, 40.00]	42.71 [41.04, 44.38]	52.29 [50.42, 54.17]	56.25 [55.42, 57.08]
$q_{1}$	56.04 [53.96, 58.13]	63.54 [61.66, 65.63]	73.33 [71.46, 75.21]	78.75 [77.29, 80.21]
$q_{3}$	80.00 [78.33, 81.67]	81.46 [80.00, 82.92]	90.42 [88.75, 91.88]	88.96 [87.71, 90.21]
$q_{4}$	89.58 [87.92, 91.25]	93.54 [91.88, 95.00]	93.33 [92.08, 94.38]	93.13 [91.46, 94.58]

Appendix

Table 6: Six-task macro OOD success (%) and 95% paired-state bootstrap intervals, conditional on the trained models and tasks. See Appendix A.4 for the sampling procedure.

Figure 4: Test OOD success for each task and system (C and E weighted equally). Designs can differ across

N

, so the curves show the design procedure across data sizes, not the scaling of one architecture.

Setting	Value
Data	12 complete demonstrations per task; one normalizer per task
History / prediction / execution	2 observations / 16 actions / 8 actions
Diffusion	1D U-Net, 100 DDPM training and sampling steps
Optimization	AdamW; learning rate $10^{-4}$ ; weight decay $10^{-6}$ ; batch 128
Schedule / gradient / EMA	Cosine, 500 warmup updates / norm limit 1 / decay 0.999; final EMA
Updates	60,000 per DP and SinglePrior policy; 20,000 per APPL policy

Appendix

Table 7: Shared training and execution settings of Exp. 2.

Task	Skill (or full-task policy)	h01	h02	h03	h04 (added)
Drawer exchange	open_drawer	3/12	6/12	12/12	2/12
	red_transfer	4/12	10/12	9/12	10/12
	blue_insert	10/12	12/12	10/12	11/12
	full-task policies	DP 5/12, SinglePrior 5/12
Buffer exchange	buffer_red	5/12	12/12	0/12	8/12
	place_blue	7/12	1/12	12/12	4/12

Appendix

Table 8: Verification successes from the twelve demonstrated entry states of each skill policy (h01–h03 original priors, h04 added after verification). The last row per task starts the full-task policies from the demonstrated initial states.

Comparator	Suite	Wins	Losses	$p$
DP	Motion OOD	18	2	$<0.001$
SinglePrior	Motion OOD	17	1	$<0.001$
w/o interface information	Motion OOD	17	5	0.017
w/o interface information	Task-level OOD	12	1	0.003
w/o interface information	Composition	6	0	0.031
w/o prior information	Motion OOD	6	4	0.754

Appendix

Table 9: Paired comparisons with full APPL on the same saved cases. Wins are cases only APPL solves; losses are cases only the comparator solves.

p

is the two-sided exact McNemar value.

Family	Shapes the policy	Informs the caller	Structure designed by
Structural priors	Yes	No; the prior remains in training	A human, per task family
Agents designing learning	Yes, one design per policy	No; the design serves training only	An agent
Generalist VLAs	Implicitly, through data	An instruction without stated scope	Data
Agents calling tools	No; tools are given	Names, feasibility estimates, or execution records	Designers or pretraining
Shared abstractions	Partly, through a fixed rule or recipe	Predicates, preconditions and effects, or skill specifications	A designer or a fixed rule
APPL	Yes, with alternative priors per skill	The same priors, with handoff overlap and training support	An agent, per skill

Appendix

Table 10: Two roles of task structure across related families.

Method	Low-level policies	Structure designed by	Alternatives per skill	Reads	Decision maker
LGA ( Peng et al., 2024 )	Imitation from few demonstrations	A language model, one state abstraction per task	No	Nothing; the abstraction is the policy input	None
KALM ( Fang et al., 2025 )	Keypoint-conditioned imitation	A VLM proposes keypoints checked on demonstrations	No	Nothing	None
SymSkill ( Shao et al., 2025 )	Dynamical-system skills in relative frames	Co-invented predicates; a VLM selects frames offline	No	Predicates and operators	Symbolic planner
Lorang et al. (2025)	Diffusion controllers	A fixed rule restricts inputs to operator-relevant objects	No	Learned operators	Symbolic planner
DR-LfD ( Chen et al., 2026 )	Diffusion policies and equivariant primitives	A fixed rule based on contact complexity	No	Learned initiation and termination keyposes	Task and motion planner
MaestroMotif ( Klissarov et al., 2025 )	Reinforcement-learned skills in NetHack	Human skill descriptions; LLM-derived rewards	No	Skill descriptions, initiation and termination code	LLM-written policy over skills

Appendix

Table 11: Design comparison with the closest methods. “Reads” is what the high-level decision maker sees about each low-level policy before or during invocation.

Explore similar work

May 20, 2026cs.RO

Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition

Learning from Demonstration (LfD) enables robots to learn complex behaviors from expert examples, yet existing approaches often fail to generalize to new compositions of known skills without retraining. Modern generative policies model distributions over action trajectories alone, thus are unable to reason about the symbolic outcomes required for robust composition. We propose that skills should jointly model action trajectories and the symbolic outcomes they induce. To address this gap, we introduce Predicate Action Skills (PACTS), a class of closed-loop visuomotor policies that model skills as a joint generative process over action and predicate belief trajectories, producing coherent action-outcome rollouts within a single model. Jointly generating actions and predicates enables PACTS to learn internal representations that improve both action generation and predicate classification. Furthermore, we demonstrate zero-shot composition of learned skills via planning by leveraging online predicate predictions from PACTS as a symbolic interface for sequencing and monitoring execution. Project website: https://planpacts.github.io/

Benedict Quartey, Sebastian Castro, Eric Rosen +3

Brown University · Robotics & AI Institute

Visuomotor Policy Action Generation Symbolic Predicates Generative Processes

Ask PDF

Jun 4, 2026cs.CL

SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization

Agent skills, which consist of reusable strategies that guide agent reasoning and action, have shown strong potential for improving model capability at inference time. However, current skill construction methods treat the problem as one-shot extraction, overlooking a fundamental tension: a skill tailored to the specific task fails to transfer, while the abstracted skill often provides insufficient guidance. We attribute this fragility to the absence of explicit mechanisms for skill specification and generalization. To address this gap, we introduce SkillComposer, a framework that decomposes skill construction into three learnable operations: create, improve, and merge. Trained via systematic rejection sampling recipe, SkillComposer enables language models to self-evolve skills at inference time and supports three deployment modes: offline for building generalized libraries, online for task-specific refinement, and hybrid for combining both. Comprehensive experiments on

τ^2

-Bench, LiveCodeBench v6, and AppWorld show that SkillComposer consistently outperforms baselines. Our SkillComposer-4B improves a 27B executor by up to +4.5 on agent tasks and +3.4 on code tasks, while generalizing across domains and task types unseen during training. Analysis reveals that merge and improve address orthogonal quality dimensions and that skill composition is a transferable meta-ability, providing a practical recipe for skill-augmented inference.

Qi Zhang, Zhaopeng Feng, Xiaonan Shi +8

1Zhejiang University · 2Tongyi Lab · 3National University of Singapore

Skills Agentic Inference

Ask PDF

May 2, 2026cs.RO

Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation

Cross-task generalization is a core challenge in open-world robotic manipulation, and the key lies in extracting transferable manipulation knowledge from seen tasks. Recent in-context learning approaches leverage seen task demonstrations to generate actions for unseen tasks without parameter updates. However, existing methods provide only low-level continuous action sequences as context, failing to capture composable skill knowledge and causing models to degenerate into superficial trajectory imitation. We propose Decompose and Recompose, a skill reasoning framework using atomic skill-action pairs as intermediate representations. Our approach decomposes seen demonstrations into interpretable skill--action alignments, enabling the model to recompose these skills for unseen tasks through compositional reasoning. Specifically, we construct a task-adaptive dynamic demonstration library via visual-semantic retrieval combined with skill sequences from a planning agent, complemented by a coverage-aware static library to fill missing skill patterns. Together, these yield skill-comprehensive demonstrations that explicitly elicit compositional reasoning for skill composition and execution ordering. Experiments on the AGNOSTOS benchmark and real-world environments validate our method's zero-shot cross-task generalization capability.

Xitie Zhang, Aming Wu, Yahong Han

School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China · School of Computer Science and Information Engineering, Hefei University of Technology, China

Human-To-Robot Transfer Robotic Manipulation Cross-Task Knowledge Transfer Skills Decomposition Recast+1

Ask PDF