Continual Field-Adaptive Models (CFAMs) for Post-Deployment Physical AI

Amarjot Singh; Tanmay R. Pancholi; Jainam Kothari; Shrirang Mahajan; Ketan Bansal; Zackory Erickson; Giuseppe Loianno; Alexandre M. Bayen

cs.ROSep 3, 2026

Continual Field-Adaptive Models (CFAMs) for Post-Deployment Physical AI

Authors: Amarjot Singh, Tanmay R. Pancholi, Jainam Kothari, Shrirang Mahajan, Ketan Bansal, Zackory Erickson, Giuseppe Loianno, Alexandre M. Bayen, +2 more

Organizations: Skylark Labs · Carnegie Mellon University · University of California, Berkeley

Abstract

Unattended interactive autonomy - machines that step into danger in place of humans and complete tasks with human tools - remains a missing capability in mission-critical operations. These domains offer scarce training data and only onboard compute, yet deployed systems must face novelty without erasing prior competence. We introduce Continual Field-Adaptive Models (CFAMs), which learn efficiently in the lab and continue learning after deployment through autonomous, gradient-free, on-device updates. CFAM uses a complementary learning architecture with a frozen slow-learning component and a fast-learning Capsule Field. The slow component contains three cortices: Sensor, which maps multimodal input into 3D-grounded geometry; Reasoning, which decomposes tasks into skills and evaluates outcomes; and Action, which executes geometric skills. The Capsule Field stores field learning one-shot and gradient-free as Competence Capsules. Skill installation is few-shot in the lab and continual in the field; open-world novelty is outside scope. We evaluate CFAM across five embodiments: manipulator, quadruped, humanoid, quadrotor, and off-road vehicle. Baselines (pi0, CogACT, SpatialVLA) use the same in-house multi-embodiment dataset for physical-platform comparisons. CFAM reaches the operating point of a standard policy trained on the full prior-training dataset using 40% of the data, or 2.5x fewer trajectories. At test time, autonomous capture of verified near-OOD cases improves action success by 13.9 percentage points. In sequential simulation, backward transfer is -0.5 percentage points versus -11.4 for LoRA. CFAM therefore provides a bounded form of post-deployment physical intelligence: few-shot skill learning, autonomous field growth from verified near-OOD experience, and retention of prior competence.

Figures & tables

Figure 1 : The Continual Field-Adaptive Model (CFAM) lifecycle and architecture. (1) Few-shot Build: a handful of operator demonstrations install seed skills as Competence Capsules without retraining the frozen prior. (2) Autonomous Test-Time Growth: after deployment, confidence and retrieval distance identify verified successful near-OOD cases, stored by an on-device, one-shot write with no operator label; each capture expands the support envelope, bounded to known skill families. (3) Capsule Field Architecture: the Sensor, Reasoning, and Action modules retrieve, execute, and consolidate bounded capsules in a fixed-budget field.

Figure 2 : The brain’s learning architecture, transcribed. Left: the slow-learning neocortical structures (visual: perceive; prefrontal: decide; motor: act) and the fast-learning hippocampus, which encodes new episodes in one exposure. Right: CFAM realizes each structure (dashed mappings): the Sensor, Reasoning, and Action cortices are trained in the lab and frozen at deployment, and the Capsule Field $\mathcal{F}$ is the hippocampal memory, written one-shot and gradient-free by the new-learning layer ( cfur ), one Competence Capsule per stored competence element. In CFAM the division is strict in both directions: the slow part is frozen after the lab (a deliberate idealization of the cortical side, which in the biological account learns slowly rather than not at all), and the fast part never acts on the world directly.

Figure 3 : How a CFAM runs a task, and how its library grows. The Reasoning cortex decomposes the task into a chain of skills; for each skill the system perceives the scene in 3D, retrieves the matching capsule, and warps it onto that scene to act—no gradient step, no per-step decoding. An execution that succeeds at the edge of a skill’s envelope is stored as a new Competence Capsule ( Equation 4 ), so the library grows with use, while decay and consolidation ( Compress ) hold it within a fixed memory budget. Colors follow Figure 2 .

Figure 4 : The Geometric Residual Transform: one stored trajectory, any object pose. A capsule stores a seed trajectory $\tau_{i}$ with the 3D anchors of the object it was demonstrated on (left) ; in the field the object is translated and rotated (right) . GRT matches stored to current anchors under the task-weighted distance $D_{w}$ , selects the admissible transform $T^{*}_{i}$ , and carries the whole trajectory through it in one pass: the skill is re-aimed, not re-learned. The phase objective $\Phi_{i}$ verifies the emission; a failure receives a bounded local residual restricted to the failing phase. Figure 9 shows the operator on real hardware.

Figure 5 : Generalization beyond the extremes: the envelope grows with use. GRT re-aims one stored seed trajectory at whatever pose the object takes, so a single skill covers a band of situations around the demonstration (pink) . Each success at the edge of the envelope is captured as a new Competence Capsule ( $u_{i}^{\prime}$ , $u_{i}^{\prime\prime}$ ), so the covered region expands with use (dashed) , with no gradient step and no failure oracle. Section 6.1 formalizes the capture rule; Section 7.3 measures the resulting rise.

Panel of Figure 6 — Figure 6 : Building the library in the lab. Seed skills are acquired from human demonstration on the bimanual humanoid, then augmented in simulation (Isaac Sim, bottom right); each demonstration is encoded into one Competence Capsule by a single one-shot write ( Section 5.4 ).

Panel of Figure 7 — Figure 7 : The tactile modality of the Sensor module. Per-finger and palm tactile arrays during grasps of a bottle, a ball, and a hex key, with live per-taxel response maps; these sensor events ( $\delta_{\text{sensor}}$ : contact, slip, and force spikes) are inputs to failure detection.

Panel of Figure 8 — Figure 8 : One skill, executed phase by phase. A stored manipulation skill on the bimanual humanoid: (a) approach, (b) align, (c) grasp, with the detected object (yellow box) and live hand-pose keypoints overlaid; each phase objective $\Phi$ must pass before the next begins ( Section 5.3 ).

Panel of Figure 9 — Figure 9 : GRT on real hardware: one stored skill, any scene. Four deployments of the same stored grasp capsule: (a) nominal pose, (b) target displaced, (c) occluded among clutter, (d) raised to a different height, the latter two engaging the reasoning-mediated edit library (detour, raise) on top of the warp. Overlays: detected target, matched anchors, grasp point, bottleneck state, and end-effector trail. The capsule is stored once; per scene, GRT recomputes the transform and emits the warped trajectory in one shot, no retraining between panels.

Panel of Figure 10 — Figure 10 : Test-time execution in the field. Representative executions on the humanoid platform: IED recognition (the REC skill exercised here on the humanoid; the metal-detector IED-sweep mission of Table 3 runs on the quadruped) and protective-gear handling. Slightly out-of-distribution variants remain within the same skill family but change the physical arrangement—for example, a new combination of bag and box locations that requires a different maneuver to reach a shield, or a heavier shield that changes the verified grasp–carry execution.

Type	Dataset / platform	Embodiment	Tasks	$D_{\mathrm{train}}$	$D_{\mathrm{test}}$	$D_{\mathrm{var}}$ (axes / conditions)
Synthetic	Bridge	WidowX	4	60,096	96 trials, 3 seeds	4 / 13 conditions
	Fractal	Google Robot	5	$\sim$ 130K	540–1,500 trials, 3 seeds	5 / 17 conditions
	LIBERO	Franka Panda	40	2,000	800 trials, 3 seeds	4 / 11 conditions
Robotic	Arm	Franka Panda	10	1.1M $+$	200 trials, 3 seeds	5 / 19 conditions
	Dog	Unitree Go2	4	87K $+$	48 trials, 3 seeds	4 / 14 conditions
	Humanoid	Unitree H1	2	1.2M $+$	40 trials, 3 seeds	5 / 16 conditions

Table 1 : Training datasets and evaluation protocols across synthetic and robotic settings.

D_{\mathrm{train}}

is the trajectory count used to train the frozen prior (a public backbone in simulation, the in-house multimodal prior on the five physical platforms); it is not per-task CFAM supervision.

D_{\mathrm{test}}

is the held-out Stage-B test protocol, given as scored trials per seed and the number of seeds (the column sums to

1{,}784

scored trials per seed; rates are macro-averaged over task–condition cells and then over seeds, Section 7.1 ); the Stage-A learning curve is scored on a separate fixed held-out set

D_{\mathrm{curve}}

, disjoint from

D_{\mathrm{train}}

and from

D_{\mathrm{test}}

( Section 7 ).

D_{\mathrm{var}}

is the variation stream on which autonomous test-time growth is measured ( Figure 11 ); each row lists the number of controlled axes (viewpoint, object position, orientation, lighting, occlusion) and the number of held-out variation conditions (a base task under one variation setting, not a trial count);

|D_{\mathrm{var}}|

544

scored trials per seed across the eight rows, about

68

per row on average, which is

23.4\%

of the

2{,}328

scored trials per seed of

D_{\mathrm{test}}

and

D_{\mathrm{var}}

together. Each task contains two to four skills; the missions of Table 3 are longer chains (three to six skills) built from the same skill library.

Method	SE-Bridge	SE-Fractal	Real Arm	Real Dog	Real Humanoid	Real Drone	Real Vehicle
CogACT	51.3	68.1	61.5	54.5	52.5	55.2	56.3
SpatialVLA	42.7	75.1	60.4	50.5	49.6	48.9	45.6
$\pi_{0}$	68.4	71.4	59.2	52.3	55.0	58.5	55.6
CFAM (Ours): $\pi_{0}$ backbone (sim) / in-house prior (real)	76.4	78.9	68.6	59.7	60.8	71.5	68.5

Table 2 : Comparison of VLA models with CFAM on the held-out test split

D_{\mathrm{test}}

; CFAM after few-shot Build, before autonomous test-time growth. The CFAM row uses the public

\pi_{0}

backbone for the SE-Bridge and SE-Fractal columns and the in-house multimodal prior for the five robotic columns. In the five robotic columns the

\pi_{0}

, CogACT, and SpatialVLA rows are those policies trained on the same in-house dataset as matched-data baselines (not the released public policies); in the SE-Bridge and SE-Fractal columns they are the released public policies (trained on their own public data), and the

\pi_{0}

row there is the same frozen backbone the CFAM row sits on.

			Mission accuracy (%)
Asset	Mission	Ordered skill chain	Standard Build	CFAM Build	Standard Grown	CFAM Grown
Humanoid	Hazardous-object recovery (firearm)	REC $\to$ L-legged-flat $\to$ M-reach $\to$ M-align $\to$ M-grasp $\to$ M-carry	52.2	66.3	13.4	79.2
Humanoid	Protective-gear donning (shield) †	REC-gear $\to$ M-reach $\to$ M-grasp $\to$ M-don/wear $\to$ PA-posture-stance	48.0	62.1	10.0	75.8
	Metal-detector IED sweep	L-legged-rough $\to$ M-carry (payload) $\to$ REC-anomaly $\to$ PA-mark-target $\to$ L-slip-recovery	47.8	68.9	6.1	77.5
Robot Dog	Confined-space structural recon †	L-legged-rough $\to$ L-gap-cross $\to$ REC-hazard $\to$ PA-mark-target	50.0	71.1	10.0	81.4
Drone	Perimeter recon & track	L-aerial-hover $\to$ L-aerial-waypoint $\to$ REC-target $\to$ PA-loiter-track	56.5	75.0	21.1	83.7

Table 3 : Master mission table: ten mission-critical scenarios (two per asset), each an ordered skill chain over

\mathcal{F}

. Build = the supervised one-shot library at deployment; Grown = the pre-novelty endpoint after automatic capture of verified within-family extremes, i.e. after the whole Stage-C stream (Standard

=

the static standard policy under the same mission protocol;

\dagger

marks the mission plotted in Figure 11 , whose Build,

x{=}100

, and Grown,

x{=}160

, values are its anchors). Prefixes: L-

=

LOCO-, M-

=

MANIP-, REC

=

recognize.

Figure 11 : CFAM versus the standard policy across the deployment stream, per real-world embodiment and averaged. Solid = CFAM; dotted = the standard (static) policy. In the Training region ( $0\text{--}100$ , Stage A) CFAM learns faster, rising above the standard policy at every fraction of $D_{\mathrm{train}}$ . In the Testing region ( $100\text{--}120$ , Stage B) CFAM sustains that advantage on the held-out test. In the Growth region ( $120\text{--}160$ , Stage C) CFAM keeps growing as verified near-edge cases are captured, while the standard policy falls. The pattern holds on every embodiment and on the average; each panel shows one mission, marked $\dagger$ in Table 3 ; its Build ( $x{=}100$ ) and Grown ( $x{=}160$ ) values are the anchors, and the Training region is the same mission scored at training checkpoints (Average: $70.3\to 80.5$ ).

Skill family	Build (%)	Grown (%)	$\Delta$ (pp)
Legged locomotion	74.4	86.8	+12.4
Aerial	77.6	88.8	+11.2
Wheeled	73.5	89.0	+15.5
Manipulation	72.2	88.3	+16.1
Other perception–action	72.5	86.6	+14.1
Overall (unweighted mean)	74.0	87.9	+13.9

Table 4 : Robotic growth by skill family on

D_{\mathrm{var}}

. Build = the one-shot library scored on

D_{\mathrm{test}}

after Build, before growth; Grown = the same library scored on

D_{\mathrm{test}}

after the

D_{\mathrm{var}}

stream has been traversed with writes enabled (the verified pre-novelty endpoint).

\Delta

is Grown minus Build success rate. Overall row is the unweighted mean across the five family rows.

Adaptation and memory baselines on $D_{\mathrm{var}}$
Method	SE-Bridge	SE-Fractal	Real Arm	Real Dog	Real Humanoid	Real Drone	Real Vehicle
CronusVLA	51.4	67.8	54.5	39.8	42.9	48.7	41.8
MemoryVLA	64.9	67.7	57.3	48.4	45.6	51.8	48.4
$\pi_{0}$ + LoRA	65.3	63.6	52.6	46.7	49.2	53.1	50.8
CogACT + LoRA	61.2	64.4	53.3	49.3	49.9	50.1	50.4
CFAM (ours): 1-shot Build + test-time growth	77.9	79.7	70.8	61.9	63.2	73.9	70.3

Table 5 : Comparison of adapted policies with CFAM after adaptation on the variation stream (

D_{\mathrm{var}}

); the metric is per-platform task success (%) on

D_{\mathrm{test}}

, the same metric and split as Table 2 . Each adaptation method receives the same

D_{\mathrm{var}}

exposure under its own update mechanism (LoRA fine-tune, memory writes), and the comparison is head-to-head on the same split: CFAM leads the strongest adaptation baseline in every column, by

11.9

20.8

pp. The CFAM row is above its own Build values in Table 2 in every column, by

1.8

–

2.4

pp on the five real platforms and by

0.8

–

1.5

pp on the SimplerEnv splits, so the two tables are a before/after pair.

	CFAM		LoRA		MemVLA
Stage	FT	BT	FT	BT	FT	BT
Env 1	+14.2	—	+16.8	—	+8.3	—
Env 2	+12.8	$-$ 0.3	+14.1	$-$ 4.7	+7.1	$-$ 1.2
Env 3	+11.5	$-$ 0.5	+11.3	$-$ 8.9	+6.4	$-$ 2.8
Env 4	+13.1	$-$ 0.4	+9.2	$-$ 13.6	+5.8	$-$ 4.1
Env 5	+12.4	$-$ 0.6	+7.8	$-$ 18.2	+5.1	$-$ 5.7

Table 6 : Continual adaptation across five sequential simulated environments. SimplerEnv task families,

\pi_{0}

backbone; mean over

3

seeds. FT = forward transfer; BT = backward transfer; Forget = prior tasks with

>5\%

success drop.

Figure 12 : Retention of previously learned tasks across the sequential suite. The curve shows competence retained after each successive environment (dashed: perfect retention). CFAM remains within $0.6$ pp of perfect retention, while MemVLA drifts to $94.3\%$ and LoRA falls to $81.8\%$ .

Configuration	Mean	SD	$\Delta$
Base only	74.1/69.8/67.5/51.2	1.4/1.6/1.7/2.3	$-16.1$
corr only	81.7/76.5/74.2/57.8	1.2/1.4/1.5/1.9	$-9.2$
ext only	79.4/75.1/72.8/56.4	1.3/1.5/1.6/2.0	$-10.9$
Full CFAM	87.9/85.2/83.1/70.9	1.0/1.1/1.3/1.5	—
Dense memory	83.1/80.4/78.3/65.2	1.2/1.3/1.5/1.7	$-5.0$
No field controller	80.2/77.1/74.8/61.8	1.4/1.5/1.7/1.9	$-8.3$

Table 7 : Component ablations on LIBERO (SpatialVLA backbone, static-set evaluation). Mean and SD are over three seeds, per suite as Spatial/Object/Goal/Long success rate (%).

\Delta

is the change in the four-suite average against Full CFAM, the reference row; negative is worse. The blocks isolate the pre-novelty regimes ( corr , ext ), swap one architectural component at a time (storage, field controller, retrieval, GRT, capsule dynamics), and remove consolidation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Figure 13 : Field deployment imagery and concrete near-OOD cases. (a) Robot Dog: the legged platform provides the embodiment for extending a moving gait to an unseen impulsive or recoil-like payload disturbance while preserving stability. (b) Ground Vehicle: a route learned on open grass is extended when dense weeds leave only a narrow opening. These are new, slightly out-of-distribution conditions inside known skill families; the system identifies, verifies, and stores the successful extension autonomously on-device.

Explore similar work

Jun 15, 2026cs.AI

Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI

We introduce \textbf{Kairos}, a regret-aware native world-action model stack for Physical AI. Kairos is motivated by the view that a physical world model should not aim to fully simulate all future pixels, but should learn and maintain the information most relevant to embodiment control: object state, spatial relations, contact conditions, task progress, action consequences, failure boundaries, and deployment uncertainty. Kairos establishes three model-side prerequisites toward this goal. First, it \textbf{learns} control-relevant information through a \textbf{Cross-Embodiment Data Curriculum}, which organizes open-world videos, human behavioral data, and robot interactions into an intervention-strength progression from passive physical observation to intentional behavior and embodied action grounding. Second, it \textbf{maintains} control-sufficient states through a unified \textbf{understanding, generation, and prediction architecture} equipped with \textbf{Hybrid Linear Temporal Attention}, where local, mid-range, and global temporal pathways support multi-timescale state maintenance under efficient inference. Third, it \textbf{deploys} these states through a \textbf{Deployment-Aware System Co-Design}, treating latency, memory footprint, and hardware compatibility as first-order constraints for future observation, action, and feedback loops. Experiments on embodied world-model benchmarks, world-action benchmarks, long-horizon generation, and inference-efficiency evaluation show that Kairos achieves superior performance while offering a favorable efficiency to capability trade-off.

Kairos Team, Fei Wang, Shan You +21

Kairos Team

Efficient World-Action Model Embodied Artificial Intelligence Cross-Embodiment Transfer Artificial Intelligence Systems Contextual Grounding

Ask PDF

Aug 19, 2026cs.LG

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.

Borui Kang, Jinrui Gu, Junhan Lv +3

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Replay-Based Continual Learning Continual Learning Catastrophic Forgetting Plasticity Open-World

Ask PDF

Sep 24, 2026cs.RO

HarnessPAI: An Evolving Harness for Physical AI

Physical AI aims to build embodied agents that perceive the world, understand and reason about it, and decide how to act. Yet the field has focused primarily on the last component: the action model that maps observations to low-level controls. The prevailing training recipe can erode the perceptual and reasoning capabilities needed for robust behavior, leaving even strong action models vulnerable to scene perturbations and long-horizon tasks. We introduce HarnessPAI, a model- and embodiment-agnostic Harness framework for Physical AI that treats code as the executable and evolvable interface that organizes the underlying action primitive. The framework separates two timescales: within a rollout, it executes open-loop at the program level, with a fixed program guiding and checking execution; across rollouts, it evolves closed-loop, using execution feedback to revise the program and distill failures into reusable skills. Across desktop robot arms, household robots, a robot vacuum, and a legged walking agent, HarnessPAI improves on both pure action models and code-as-policy baselines without retraining the underlying model: a 61.6-point gain over

π_{0.5}

on LIBERO-PRO and a 27.2-point gain over WorldDreamer on RoboCasa atomic tasks. Once a program is selected, rollout execution requires no online high-level LLM deliberation. Beyond execution, the converged program is also a cheap and reliable expert-data collector, and fine-tuning

π_{0.5}

on collected expert data lifts success rate on LIBERO-PRO by 38.8 points. Our results suggest that the frontier of Physical AI depends not only on stronger action models, but also on executable harnesses that integrate perception, task understanding and reasoning, and action execution into a unified, verifiable, and feedback-driven system. Website: https://darwin-agent.github.io/HarnessPAI

Xin Wang, Wenhao Wu, Menghao Zhang +20

Embodied Artificial Intelligence Agent Harness Embodied Agents Perception Pipeline Artificial Intelligence Systems Evolving Harness+1

Ask PDF