MorphAct: Robot Policies Morphing with the Worlds They Act In

Ziyan Liu; Jiting Liu; Weihong Zhong; Wenda Yu; Yeqiu Chen; Jason Li; Xiaojiang Peng; Shuyuan Hu

cs.ROOct 14, 2026

MorphAct: Robot Policies Morphing with the Worlds They Act In

Authors: Ziyan Liu, Jiting Liu, Weihong Zhong, Wenda Yu, Yeqiu Chen, Jason Li, Xiaojiang Peng, Shuyuan Hu,

Organizations: University of Science and Technology of China · School of Artificial Intelligence, Shanghai Jiao Tong University · Tongji University · Nanyang Technological University ·

Abstract

Progress toward general intelligence in the physical world requires scaling robot learning across diverse environments and interactions. However, existing robot policies use shared parameters that remain fixed across interaction contexts. When different operations favor competing parameter adjustments, fitting their mixture can leave individual operations insufficiently adapted. Our atomic-task diagnostics reveal competing update directions in both vision-language and action parameters, with source-improving updates increasing loss on other tasks. This motivates MorphAct, a contextual parameterization framework that dynamically adjusts robot policy parameters as interaction contexts change. Two coupled generators produce context-dependent low-rank updates for the vision-language and action modules. Their coupling follows the policy's computational dependency: vision-language updates shape the representations that condition action-side parameter generation. Both generators are trained jointly through the native action objective, without operation labels or expert-adapter targets. MorphAct improves performance across three VLA backbones on LIBERO and Meta-World, with gains of up to 16.30%. In real-world evaluation, it achieves gains of up to 14% over $π_{0.5}$ . Ablations support the benefits of contextual parameter generation and representation coupling.

Figures & tables

Figure 1: Robot policies morph with changing interaction contexts. (a) A single manipulation task involves operations with different perceptual and control demands. (b) Existing robot policies use one fixed parameter configuration across contexts, requiring shared parameters to reconcile competing demands. (c) MorphAct generates context-dependent updates for the vision–language and action modules, dynamically adjusting policy parameters as interactions unfold.

Figure 2: Parameter conflicts across operations and within modules. (a) Negative-gradient-cosine frequency within and across tasks. (b) Number of other tasks with increased target-batch loss after a source update. All 260 updates reduce source-batch loss; dots represent updates and ticks mark round medians. (c) Fraction of absolute alignment contribution cancelled across parameter groups within each module. Measurements use one checkpoint with 16 sampling rounds for (a,c) and four for (b).

Figure 3: Representation-coupled parameter generation. $G_{v}$ generates vision–language updates from multimodal embeddings. The adapted VLM pass produces $h_{t}$ for action-side parameter generation and $C_{t}$ for native action conditioning. $G_{a}$ generates action updates from $h_{t}$ and robot state. Solid arrows carry data; dashed arrows inject updates. Both generators learn through the action objective, and their outputs are reused within each action chunk.

Backbone	Method	Spatial $\uparrow$	Object $\uparrow$	Goal $\uparrow$	Long $\uparrow$	Mean $\uparrow$	Var. (pp 2 )
	Baseline	$95.40$	$96.60$	$96.00$	$84.20$	$93.05$	–
X-VLA	MorphAct	98.67	99.87	97.07	93.20	97.20	$0.0300$
	Baseline	$96.80$	$98.80$	$95.80$	$85.20$	$94.15$	–
$\pi_{0}$	MorphAct	98.00	99.60	96.60	92.40	96.65	$0.0517$
	Baseline	98.80	$98.20$	$98.00$	$92.40$	$96.85$	–
$\pi_{0.5}$	MorphAct	$98.33$	99.67	98.87	97.53	98.60	$0.0525$

Table 1: LIBERO closed-loop success rates (%). Long denotes LIBERO-10; Mean averages the four suite scores. MorphAct scores are averaged across three random seeds, each covering 40 tasks with 50 rollouts per task and execution horizon H10. Var. is the sample variance of the three overall scores (pp 2 ); – denotes unreported baseline variance. Shaded rows denote MorphAct. Bold indicates the best result within each backbone and column, including ties.

Backbone	Method	Easy $\uparrow$	Medium $\uparrow$	Hard $\uparrow$	Very Hard $\uparrow$	Avg. $\uparrow$	Var. (pp 2 )
	Baseline	$72.50$	$46.36$	$48.30$	55.00	$55.54$	–
X-VLA	MorphAct	83.99	58.18	55.28	55.00	63.11	$1.8400$
	Baseline	$77.90$	$51.80$	$53.30$	$20.00$	$50.75$	–
$\pi_{0}$	MorphAct	82.44	71.36	56.39	58.00	67.05	$0.6342$
	Baseline	$70.50$	$45.00$	$38.30$	$41.00$	$48.70$	–
$\pi_{0.5}$	MorphAct	85.77	59.85	66.39	45.67	64.42	$0.3970$

Table 2: Meta-World closed-loop success rates (%). Avg. is the unweighted arithmetic mean of the Easy, Medium, Hard, and Very Hard success rates. MorphAct scores are averaged across three random seeds, each covering 50 tasks with 20 rollouts per task. Var. is the sample variance of the three overall scores (pp 2 ); – denotes unreported baseline variance. Shaded rows denote MorphAct. Bold indicates the best result within each backbone and column, including ties.

Panel of Figure 4 — Figure 4: Real-world manipulation results. Left: task examples, read row-wise in the order listed in the table. Right: success rates of the $\pi_{0.5}$ baseline and $\pi_{0.5}$ + MorphAct. Gains are absolute differences in percentage points (pp). Each method is evaluated over 50 physical rollouts per task.

Variant	Spatial $\uparrow$	Object $\uparrow$	Goal $\uparrow$	Long $\uparrow$	Mean $\uparrow$
MorphAct A,C	98.67	99.87	97.07	93.20	97.20
Constant A	$93.40$	$90.00$	$89.00$	$77.60$	$87.50$
Task-ID A	$90.00$	$90.80$	$91.20$	$81.00$	$88.25$
VLM-only B	$40.00$	$59.00$	$42.00$	$33.00$	$43.50$
Action-only B	$88.80$	$92.00$	$73.80$	$69.40$	$81.00$
Unadapted B,C	$91.00$	$97.80$	$83.60$	$71.60$	$86.00$

Table 3: X-VLA ablation success rates on LIBERO (%). Superscripts identify generation conditions (A, Section 5.1 ), adapted modules (B, Section 5.2 ), and feature coupling (C, Section 5.3 ). Mean averages the four suite scores. Shading denotes MorphAct; bold marks column maxima.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	A: alignment	B: one-step affinity	C: contributions
Batch size	32	16	32
Compared batches	$U_{i},U_{j}$ ; $U_{i},V_{i}$	Source $U_{i}$ , target $V_{j}$	$U_{i},U_{j}$ ; $U_{i},V_{i}$
Parameter precision	BF16	FP32 from BF16 checkpoint	BF16
Sampling rounds	16	4	16
Cross-task observations	33,280 unordered pair-rounds	16,640 directed pair-rounds	33,280 unordered pair-rounds
Within-task observations	1,040	260	1,040

Appendix

Table A1: Diagnostic settings. A/C resample batches over 16 rounds; B uses four complete source-to-target rounds at the same checkpoint.

Module	Comparison	Mean cosine ( $10^{-3}$ )	$p_{m}^{-}$ (%)	$b_{m}$ ( $10^{-3}$ )	Negative mean pairs
VLM	Cross-task	1.05	47.80	7.00	890/2,080
VLM	Within-task	10.24	38.94	12.01	11/65
Action	Cross-task	3.00	48.28	26.76	918/2,080
Action	Within-task	25.11	42.69	34.50	19/65

Appendix

Table A2: Gradient alignment at the fixed checkpoint. Within-task batches come from disjoint episode groups. Mean negative part includes zeros from nonnegative observations.

Comparison	Loss up	Loss down	Unchanged	Mean $Z$ ( $10^{-3}\%$ )	Mean increase ( $10^{-3}\%$ )
Cross-task	8,310	8,326	4	$-0.47$	44.36
Within-task	133	127	0	$+4.89$	96.62

Appendix

Table A3: One-step target effects across four sampling rounds. Affinity and loss changes are relative to the target’s initial loss. Mean increase is conditional on a loss-increase event. A positive

Z

means lower target loss.

Module	Comparison	Observations	Mixed signs (%)	Cancellation (%)
VLM	Cross-task	33,280	99.45	36.13
VLM	Within-task	1,040	94.13	23.51
Action	Cross-task	33,280	95.93	30.90
Action	Within-task	1,040	90.10	22.70

Appendix

Table A4: Opposing parameter-group contributions. Mixed-sign observations satisfy both

P>0

and

N>0

. Cancellation pools absolute contribution mass within observations.

Rank	VLM parameter group	Params (M)	Conflict (%)	Negative share (%)
1	Outside Transformer blocks	3.34	48.56	20.40
2	Language block 00	110.11	48.93	17.44
3	Vision block 00	15.24	48.15	7.83
4	Vision block 01	15.24	48.23	3.91
5	Language block 14	110.11	48.86	3.48
6	Language block 07	110.11	48.80	3.14

Appendix

Table A5: Complete parameter-group rankings. Groups are ranked within each module by negative cross-task contribution share. Conflict frequencies use group-local cosines. The remaining 14 VLM groups and all 19 action groups continue on the next page.

ID	Original RoboCasa source-task identifier	ID	Original RoboCasa source-task identifier
1	AdjustToasterOvenTemperature	34	PickPlaceCounterToCabinet
2	AdjustWaterTemperature	35	PickPlaceCounterToDrawer
3	CheesyBread	36	PickPlaceCounterToMicrowave
4	CloseBlenderLid	37	PickPlaceCounterToOven
5	CloseCabinet	38	PickPlaceCounterToSink
6	CloseDishwasher	39	PickPlaceCounterToStandMixer

Appendix

Table A6: Original identifiers retained in the diagnostic manifests. IDs follow the original alphabetical order. Each ID denotes one native RoboCasa365 atomic task and one diagnostic group.

Figure 5: Negative contributions are concentrated. The five largest groups are shown individually for each module; all others are explicitly combined. Parameters outside Transformer blocks remain separately identified. Percentages are shares of pooled negative cross-task contribution, not fractions of model parameters.

Figure 6: Complete Transformer depth profiles. Cross- and within-task comparisons report negative-cosine frequency and the mean negative part of group-local cosine. Block indices follow the actual 27-block vision, 18-block language, and 18-block action layouts.

Backbone	VLM-generator condition / targets	Action-generator condition / targets	Training prediction
$\pi_{0}$ / $\pi_{0.5}$	Multimodal prefix embeddings / selected VLM projections	Adapted final tokens and state / selected action-expert projections	Flow velocity
X-VLA	Visual/language embeddings / selected Florence encoder projections	Adapted features, auxiliary views, state / selected action-Transformer projections	Clean action

Appendix

Table A7: Architecture-specific interfaces. Both integrations preserve the base policy’s action-conditioning and loss definitions. Additional view and domain interfaces are specific to X-VLA.

Adaptation	Generator params (M)	Active factors (M)	Chunk latency (ms)	Memory (GiB)	Rate (Hz)
Backbone: $\pi_{0}$
Shared LoRA	–	49.99	120.00	7.43	28.09
MorphAct	300.09	49.99	145.00	9.10	23.20
Backbone: $\pi_{0.5}$
Shared LoRA	–	49.99	138.71	7.88	24.99
MorphAct	300.09	49.99	164.00	9.53	21.20

Appendix

Table A8: Adaptation capacity and execution cost. Generator counts include the two generators only; “–” indicates the absence of a generator. Active factors count scalar A/B entries used by one context, not the number of matrices. Execution measurements use matched settings within each backbone. Control rates are measured under the action-execution schedule.

Robot Policy Adaptation

Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.

See all 170 papers

In-Context Robot Learning with VLM Agents
Sep 16, 2026Dongzhou Cheng, Taoran Yi, Ye Fang +12Robot Policy Learning ICL for Robotics
Morphi Robot · Shanghai Innovation Institute · Huazhong University of Science and Technology +7
In-Context World Modeling for Robotic Control
Jun 24, 2026Siyin Wang, Junhao Shi, Senyu Fei +4ICL for Robotics World Model Learning
1Fudan University · 2Shanghai Innovation Institute · 3Tongji University
Fast-VLA: You Only Think Once for Robot Manipulation
Oct 15, 2026Kehong Gong, Guanli Hou, Xiaoxu Zheng +6Vision-Language-Action Models Closed-Loop Control of VLA Models
Huawei Technologies Co., Ltd · Huawei Central Research Institute

Sep 16, 2026J/K move · Enter open · S save

In-Context Robot Learning with VLM Agents

Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang+11

Morphi Robot · Shanghai Innovation Institute · Huazhong University of Science and Technology+7

Robot Policy Learning ICL for Robotics In-Context Learning Robot Imitation Learning VLMs for Robotics Robot Learning

Open paper AskPDF arXiv

Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.

Abstract

Robot Policy Adaptation

Explore similar work

In-Context Robot Learning with VLM Agents

In-Context World Modeling for Robotic Control

Fast-VLA: You Only Think Once for Robot Manipulation