HarnessPAI: An Evolving Harness for Physical AI

Xin Wang; Wenhao Wu; Menghao Zhang; Zhi Wang; Kun Shao; Jian Luan; Yang Li; Qing Li

cs.ROSep 24, 2026

HarnessPAI: An Evolving Harness for Physical AI

Authors: Xin Wang, Wenhao Wu, Menghao Zhang, Zhi Wang, Kun Shao, Jian Luan, Yang Li, Qing Li, +15 more

Abstract

Physical AI aims to build embodied agents that perceive the world, understand and reason about it, and decide how to act. Yet the field has focused primarily on the last component: the action model that maps observations to low-level controls. The prevailing training recipe can erode the perceptual and reasoning capabilities needed for robust behavior, leaving even strong action models vulnerable to scene perturbations and long-horizon tasks. We introduce HarnessPAI, a model- and embodiment-agnostic Harness framework for Physical AI that treats code as the executable and evolvable interface that organizes the underlying action primitive. The framework separates two timescales: within a rollout, it executes open-loop at the program level, with a fixed program guiding and checking execution; across rollouts, it evolves closed-loop, using execution feedback to revise the program and distill failures into reusable skills. Across desktop robot arms, household robots, a robot vacuum, and a legged walking agent, HarnessPAI improves on both pure action models and code-as-policy baselines without retraining the underlying model: a 61.6-point gain over $π_{0.5}$ on LIBERO-PRO and a 27.2-point gain over WorldDreamer on RoboCasa atomic tasks. Once a program is selected, rollout execution requires no online high-level LLM deliberation. Beyond execution, the converged program is also a cheap and reliable expert-data collector, and fine-tuning $π_{0.5}$ on collected expert data lifts success rate on LIBERO-PRO by 38.8 points. Our results suggest that the frontier of Physical AI depends not only on stronger action models, but also on executable harnesses that integrate perception, task understanding and reasoning, and action execution into a unified, verifiable, and feedback-driven system. Website: https://darwin-agent.github.io/HarnessPAI

Figures & tables

Figure 1 : Overview of HarnessPAI , a general physical harness framework.

Method family	Adaptation location	Coding agent during rollout	API-based control	Learned policy usage	Evolution across rollouts
VLA/WAM post-training	Model parameter	✗	✗	✓	Training only
Code-as-policy	Program	✗	✓	✗	Sometimes
VLA harness	Runtime/memory	✓	✓	✓	Usually
HarnessPAI	Program/memory	✗	✓	✓	Designed for persistent evolution

Table 1: Qualitative comparison of action-model and executable-harness paradigms, corresponding to Fig. 2 .

Figure 2 : Our paradigm versus prior ones. (1) Post-training : an action model is trained on expert trajectories by SFT or by RL on a reward; at inference, the frozen model takes an instruction and image and outputs an action chunk. (2) Code as policy : an online variant evolves one program per task and seed, where the coding agent reads the instruction and image, calls APIs and the skill library, runs the code, revises it from the output, and evolves again; an offline variant instead evolves a single program over several selected seeds and runs that same program directly on scenes unseen during evolution. (3) VLA harness : a coding agent serves as the planner, receiving the image, instruction, and memory and deciding whether to call an environment primitive or the action-model policy. Our HarnessPAI instead first builds a task graph from expert trajectories, derives the code and skill-library architecture from that graph, and uses a coding agent as the planner that invokes VLA/WAM actions through code , so code manages perception, understanding and reasoning, and action execution; for one or many tasks it evolves code over multiple environment seeds and tests a single program on many unseen test scenes.

Figure 3 : In-distribution success of a strong VLA ( $\pi_{0.5}$ ) on LIBERO. Given the in-distribution instruction “pick up the black bowl between the plate and the ramekin,” the VLA approaches, grasps the correct bowl, transports it, and places it on the plate.

Figure 4 : Language perturbation on LIBERO. Adding “not” before “between” flips the intended target to a different bowl, yet $\pi_{0.5}$ still grasps the original bowl, and externally masking the scene does not help. HarnessPAI re-grounds the instruction in code and grasps the correct bowl.

Figure 5 : Visual perturbation on LIBERO. After the positions of the plate and ramekin are swapped without changing the instruction, $\pi_{0.5}$ grasps the bowl but attempts to place it at the plate’s original location and fails. HarnessPAI re-locates the objects in code and succeeds.

Figure 6 : Fine-manipulation error on RoboCasa. The evaluated WAM reaches the microwave but fails to press the small start button in the illustrated trials. HarnessPAI successfully completes the button-pressing task.

Figure 7 : Irreversible mistake on RoboCasa. The robotic arm may knock over the pot, and this mistake cannot be undone, ending the rollout in an unrecoverable state. HarnessPAI grounds the grasp in code and succeeds on the first attempt.

Figure 8 : The code framework. Knowledge is organized into three memories, a Task Memory of instructions, a Structured Skill Memory of failure-to-repair skills, and an Executable Code Memory of evolved code, around a central planner. An instruction is matched to code that draws on a Primitive Library of perception primitives (SAM3), control primitives (raw control APIs, e.g., grasp, motion), and model primitives (VLA, WAM, PPO), and that verifies each step before committing an action to the environment.

Figure 9 : The overall pipeline of HarnessPAI . (1) Optional : a VLM reads the expert-trajectory video and generates a workflow graph for the task, which fixes the structure of the code and the skills. (2) Optional : the VLA/WAM is rolled out on the task several times and a VLM judges at which workflow nodes the model errs and needs code takeover or repair, producing a capability assessment report. (3) Optional : for perception-dependent tasks, a coding agent calls SAM/Molmo to evolve perception code, verified against the expert trajectories by a VLM, yielding the initial perception module. (4) The main evolution loop: given the task instruction, environment information, and the success contract, the coding agent builds the code package from the previously built code structure, runs it on multiple environments, and uses the diagnostics and rollout videos to decide the next edit; on errors it retrieves the matching skill from the skill library to repair the code, maintaining temporary in-progress skills, and on success it stores the evolved code in the code library and merges the evolved skills into the skill library.

Model	Role	Responsibility	Edits code
GPT-5.6-sol	writer + synthesizer	edits phase code; self-review; next instruction	✓
Gemini-3.5-Flash	diagnoser	watches rollout video; node-by-node verdict vs. graph	✗

Table 2 : Division of labor between the two models.

Phase	Executor
grounding	fixed code (segmentation + geometry)
approach	fixed code (Cartesian servo, Pyroki IK rescue)
grasp	VLA
transport	fixed code
place	VLA

Table 3 : The hybrid rollout. Deterministic phases run as fixed code; semantic phases are delegated to the VLA.

Figure 10 : How skills are stored and queried. (a) Storage : at the top a task memory holds the descriptions of the tasks already evolved; under each task sits its structured skill memory , and each task owns a workflow graph whose nodes each carry multiple skills describing how to repair the difficulties encountered at that node, with each skill pointing to the more detailed repair resources in the code memory. (b) Query : skills aggregate across tasks, so all skills under the same node are pooled together; when a new task hits a problem at an existing node (e.g., transport ), it directly queries the skills stored under that node.

Figure 11 : Flat versus graph-organized skill storage. Left: writing each “problem $\rightarrow$ repair” pair to its own markdown file yields a heap of flat files with a weak association to the task stage. Right: mounting each pair onto the graph node where it occurs (approach, grasp, transport, place) makes every skill retrievable by the stage of the task it belongs to.

Method	Spatial	Object	Goal	Overall
Pure-Model
DreamZero-LIBERO ( RLinf, 2026 )	81.6	90.2	41.2	71.0
TraceVLA ( Zheng et al., 2025 )	84.6	85.2	75.1	81.6
OpenVLA ( Kim et al., 2024 )	84.7	88.4	79.2	84.1
WorldVLA ( Cen et al., 2025 )	85.6	89.0	82.6	85.7
CoT-VLA ( Zhao et al., 2025 )	87.5	91.6	87.6	88.9

Table 4 : Comparison of selected methods on the LIBERO benchmark (success rate, %). We report success on the Spatial, Object, and Goal suites; Overall is the unweighted mean of the three displayed suite scores. Note that different papers adopt evaluation settings that are not entirely identical; our method uses the official

\pi_{0.5}

-LIBERO model (abbreviated as

\pi_{0.5}

), and the

\pi_{0.5}

and DreamZero-LIBERO results are our own evaluation, with each task in every suite run over 50 seeds.

Method	Type	Atomic-Seen
$\pi_{0}$ ( Black et al., 2024 )	VLA	34.6
$\pi_{0.5}$ ( Intelligence et al., 2025b )	VLA	39.6
RLDX-1 ( RLWRLD, 2026 )	VLA	60.0
WorldDreamer ( worldAgents-c, 2026 )	WAM	65.0
Harness VLA (Codex) ( Zhang et al., 2026b )	Harness	91.6
Harness VLA (CC) ( Zhang et al., 2026b )	Harness	79.4

Table 5 : Results on the Atomic-Seen subset of RoboCasa Target50 (18 tasks; success rate, %). Type distinguishes pure VLAs, World Action Models (WAM), and Harness methods. Note that different RoboCasa papers adopt different evaluation settings: for instance, Harness VLA runs 10 seeds per task on Atomic-Seen, whereas our method runs 20 seeds in a fixed-layout scene, with obstacles and distractors varying across seeds; to ensure a fair comparison, we evaluate both WorldDreamer and our method under a unified experimental setting.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	2A-Hand	2A-Lift	Insert	Lift	Stack	Restack	Wipe	Avg
CaP-Agent	20	74	0	97	98	89	100	68.3
Human	91	94	80	93	73	47	100	82.6
ASPIRE	92	71	9	97	99	100	99	81.0
HarnessPAI	100	91	83	100	100	100	100	96.3
$\Delta$ vs. ASPIRE	+8.0	+20.0	+74.0	+3.0	+1.0	0.0	+1.0	+15.3

Appendix

Table 6 : Per-task success rate on robosuite (%).

	Object		Goal		Spatial
Method	Swap	Task	Swap	Task	Swap	Task	Avg
OpenVLA	0	0	0	0	0	0	0.0
$\pi_{0}$	0	0	0	0	0	0	0.0
$\pi_{0.5}$	29.8	10.8	37.6	29.0	51.8	50.6	34.9
CaP-Agent	22	18	26	17	12	14	18.2
ASPIRE	98	95	81	45	51	60	71.7

Appendix

Table 7 : LIBERO-PRO success rate by suite and perturbation (%).

Task	Description	Success / Total	Success Rate
LIBERO-Object
Task 0	Pick up the alphabet soup and place it in the basket	50/50	100.0
Task 1	Pick up the cream cheese and place it in the basket	50/50	100.0
Task 2	Pick up the salad dressing and place it in the basket	50/50	100.0
Task 3	Pick up the BBQ sauce and place it in the basket	50/50	100.0
Task 4	Pick up the ketchup and place it in the basket	50/50	100.0

Appendix

Table 8 : Per-task results of HarnessPAI on the three LIBERO suites (success rate, %). All results are our own evaluation over 50 seeds per task.

Task	Description	Success / Total	Success Rate
LIBERO-Object
Task 0	Pick up the alphabet soup and place it in the basket	50/50	100.0
Task 1	Pick up the cream cheese and place it in the basket	50/50	100.0
Task 2	Pick up the salad dressing and place it in the basket	50/50	100.0
Task 3	Pick up the BBQ sauce and place it in the basket	50/50	100.0
Task 4	Pick up the ketchup and place it in the basket	50/50	100.0

Appendix

Table 9 : Per-task results of

\pi_{0.5}

on the three LIBERO suites (success rate, %). All results are our own evaluation over 50 seeds per task.

Task	Description	Success / Total	Success Rate
LIBERO-Object
Task 0	Pick up the alphabet soup and place it in the basket	45/50	90.0
Task 1	Pick up the cream cheese and place it in the basket	50/50	100.0
Task 2	Pick up the salad dressing and place it in the basket	50/50	100.0
Task 3	Pick up the BBQ sauce and place it in the basket	46/50	92.0
Task 4	Pick up the ketchup and place it in the basket	41/50	82.0

Appendix

Table 10 : Per-task results of DreamZero-LIBERO on the three LIBERO suites (success rate, %). All results are our own evaluation over 50 seeds per task.

	Swap			Task
Task	Succ./Total	Fail	Rate	Succ./Total	Fail	Rate
LIBERO-Object
Task 0	50/50	0	100.0	50/50	0	100.0
Task 1	50/50	0	100.0	50/50	0	100.0
Task 2	50/50	0	100.0	47/50	3	94.0
Task 3	50/50	0	100.0	50/50	0	100.0

Appendix

Table 11 : Per-task results of HarnessPAI on LIBERO-PRO under Swap and Task perturbations (success rate, %).

	Swap			Task
Task	Succ./Total	Fail	Rate	Succ./Total	Fail	Rate
LIBERO-Object
Task 0	41/50	9	82.0	0/50	50	0.0
Task 1	50/50	0	100.0	50/50	0	100.0
Task 2	0/50	50	0.0	0/50	50	0.0
Task 3	5/50	45	10.0	0/50	50	0.0

Appendix

Table 12 : Per-task results of

\pi_{0.5}

on LIBERO-PRO under Swap and Task perturbations (success rate, %).

Task	Name	Layout / Style	Success / Total	Success Rate
Task 0	CloseBlenderLid	8 / 8	19/20	95.0
Task 1	CloseFridge	2 / 2	20/20	100.0
Task 2	CloseToasterOvenDoor	7 / 7	19/20	95.0
Task 3	CoffeeSetupMug	4 / 4	16/20	80.0
Task 4	NavigateKitchen	5 / 5	20/20	100.0
Task 5	OpenCabinet	7 / 7	18/20	90.0

Appendix

Table 13 : Per-task results on RoboCasa atomic tasks (success rate, %).

Task	Name	Layout / Style	Success / Total	Success Rate
Task 0	CloseBlenderLid	8 / 8	6/20	30.0
Task 1	CloseFridge	2 / 2	20/20	100.0
Task 2	CloseToasterOvenDoor	7 / 7	8/20	40.0
Task 3	CoffeeSetupMug	4 / 4	8/20	40.0
Task 4	NavigateKitchen	5 / 5	8/20	40.0
Task 5	OpenCabinet	7 / 7	18/20	90.0

Appendix

Table 14 : Per-task results of the underlying WorldDreamer WAM on RoboCasa atomic tasks (success rate, %).

	DreamZero (WAM)		HarnessPAI ( zero-shot )
Task	Swap	Task	Swap	Task
task0 (alphabet soup)	0.0	0.0	100.0	94.0
task1 (cream cheese)	0.0	96.0	84.0	100.0
task2 (salad dressing)	0.0	0.0	100.0	100.0
task3 (BBQ sauce)	0.0	0.0	84.0	96.0
task4 (ketchup)	0.0	0.0	96.0	78.0

Appendix

Table 15 : Per-task results of DreamZero and DreamZero with HarnessPAI on LIBERO-PRO Object Suite under Swap and Task perturbation (success rate, %)

Explore similar work

PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud
Aug 4, 2026Chenghua Wang, Daliang Xu, Dongqi Cai +21Physics-Aware Models Artificial Intelligence Systems
1Beijing University of Posts and Telecommunications · 2Nanjing University · 6ModelBest +3
Kairos: A Scalable Serving System for Physical AI
May 12, 2026Yinwei Dai, Ganesh Ananthanarayanan, Landon Cox +3Artificial Intelligence Systems Scalability
Princeton University · Microsoft
Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI
Jun 15, 2026Kairos Team, Fei Wang, Shan You +21Efficient World-Action Model Embodied Artificial Intelligence
Kairos Team

Aug 4, 2026, cs.AIJ/K move · Enter open · S save

PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud

Chenghua Wang, Daliang Xu, Dongqi Cai, Duojin Sun+20

1Beijing University of Posts and Telecommunications · 2Nanjing University · 6ModelBest+3

Physics-Aware Models Artificial Intelligence Systems Autoregressive Rollout Cloud Edge Device

Open paper AskPDF arXiv

Physical AI policies require inference throughout their lifecycle, including model evaluation, cloud reinforcement learning rollout, edge GPU serving, and onboard deployment. Although these settings share the same checkpoint and action semantics, they often rely on separate inference programs. To unify them, we build PhyAI, a Physical AI inference engine with a single runtime that keeps architecture-specific conditioning, solver, cache, and output logic in model adapters while sharing graph execution, kernels, memory management, and parallel services. The same codebase runs vision-language-action (VLA) models and world-action models (WAMs) on single or multiple GPUs across onboard, edge, and cloud deployments. We used the adapter interface to add MiniCPM-Robot on the day of its release. PhyAI achieves 1.40x-4.65x speedups over the official implementations of pi0, pi0.5, GR00T N1.7, and MiniCPM-Robot. On Cosmos3-Nano-Policy-DROID it reduces latency from 2.46 to 1.18 s on eight H20 GPUs (CFG=2, TP=4), a 2.08x speedup. Specialized runtimes remain faster in several configurations, so our goal is one runtime with competitive latency rather than the fastest result in every case. Detailed profiles reveal why different models need different execution policies: on a Hopper-series GPU at batch size one, the pi0.5 action expert accounts for 8.8% of FLOPs but 57.2% of latency; at batch size 32 its share drops to 13.5% and throughput reaches about 100 samples/s. Cosmos3 remains generation-dominated and gains only 14.3% throughput as batch size increases from 1 to 16. We further introduce the control-time Roofline, which distinguishes inference-bound from environment-bound control; the measured pi0.5 points on four LIBERO suites are environment-bound while Cosmos3 stays inference-bound. Code and benchmarks: https://github.com/mingti-org/phyai.

Abstract

Explore similar work

PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud

Kairos: A Scalable Serving System for Physical AI

Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI