Physical AI: Concepts, Techniques, and Challenges

Weinan Zhang; Chunhua Shen; Tianfan Xue; Bin Zhao; Bingqi Jiang; Zihan Guo; Zicai Cui; Yang Li

Figure 1.1: A conceptual expansion in the scope and organization of AI interaction with the physical world, from digital tools and simulated agents to real-world single-agent systems, multi-agent collaboration, and infrastructure-level physical AI. The real-world forms shown fall within the scope of physical AI adopted in this monograph

Figure 1.2: Physical AI as closed-loop interaction between artificial systems and the physical world. Systems observe physical states, maintain task-relevant representations of physical relationships and action consequences, and select and execute interventions through physical interfaces. Outcome feedback guides adaptation toward objectives under physical, operational, and computational constraints

Figure 1.3: Embodied AI and physical AI within this monograph’s framework. Embodied AI emphasizes how an agent’s body shapes sensing and action. Physical AI emphasizes closed-loop intervention in real physical processes. The two overlap in physically deployed embodied systems

Figure 1.4: Key elements of Physical AI and their relationships. Chapter labels in the diagram refer to Chapters 3 to 9

Figure 1.5: The application landscape of Physical AI. All applications instantiate the same closed loop. The four domains illustrate its operation under different conditions. Five system archetypes (three physical compositions and two cross-cutting modes) and four assessment dimensions apply across domains

Figure 2.1: Open-loop and closed-loop control. (a) In open-loop control, the controller maps the reference

r(t)

directly to the plant input

u(t)

, so an output disturbance

d(t)

passes to the output

y(t)

uncorrected. (b) In closed-loop control, the measured output is fed back and compared with the reference, and the controller acts on the error

e_{\mathrm{track}}(t)=r(t)-y(t)

. For a stable closed loop, the feedback attenuates the output-disturbance response over frequencies at which the loop-gain magnitude

|L(\mathrm{j}\omega)|

is sufficiently large.

C(s)

and

P(s)

denote the controller and plant transfer functions, respectively

Figure 2.2: Grounding and multimodal pretraining

Figure 3.1: Embodied perception from multimodal observations. A complex physical environment is sampled through multiple sensors, producing a compressed and incomplete digital representation. Perception and cognition integrate these observations to infer the environment state relevant to an embodied agent’s task

Figure 3.2: Schematic illustration of different 3D representations of the same object. The 3D visualizations are illustrative and are not ground-truth reconstructions or outputs computed from the source data. The scene image is from suresh2022shapemap

Figure 3.3: Illustration of object detection and instance segmentation. Bounding boxes indicate the locations and categories of detected objects, while colored masks delineate individual object instances at the pixel level

Evidence	Provides	Does not establish
Observation	Appearance, scene context, visible motion	Executable commands or contact forces
Action proxies	Reconstructed poses, trajectories, or latent actions	Feasibility on the target embodiment
Execution	Recorded commands and measured responses	Coverage of unfamiliar states or failures
Closed-loop experience	Policy-induced states, interventions, and outcomes	Correct behavior or reliable annotations
Synthetic evidence	Controlled variation, scalable trajectories, dense model-derived labels	Real-world dynamics or sensor behavior

Table 4.1: Evidence levels in physical data and their limits

Figure 4.1: Human-demonstration data collection paradigms, arranged by their distance from executable robot actions . (a) Teleoperation drives the target robot through leader-follower mechanisms and records robot observations, actions, and outcomes directly (Mobile ALOHA, photo from Stanford News). (b) Portable interfaces such as UMI record retargetable end-effector motion in natural environments using a hand-held gripper, a wrist-mounted camera, and visual-inertial tracking (photo from the UMI project). (c) Egocentric human data capture natural activity at scale with head-mounted devices, providing broad behavioral priors without native robot states (photo from ABC News)

Source	Main supervision	Executability	Main bias and suitable use
Target-embodiment teleoperation	Robot observations, commands, executed motions, contacts, and outcomes	High	Affected by interfaces, delays, and operator styles. Suitable for behavior cloning and skill fine-tuning
Portable human demonstration	Camera, tool, and body trajectories in natural scenes	Medium	Requires calibration, retargeting, and reachability filtering. Expands object and contact coverage
Egocentric or public video	Visual content, semantics, task steps, and behavioral timing	Low	Lacks native actions and physical feedback. Supports representations, latent actions, and high-level priors
Autonomous policy rollout	Policy-induced states, actions, events, and outcomes	High	May repeat behavior or amplify policy bias. Supports learning from boundaries, failures, and online experience
Human intervention and recovery	Warning signs, takeovers, corrections, recovery, and consequences	High	Selected by intervention triggers. Supports correction, recovery, and safety-oriented post-training

Table 4.2: Supervisory value and principal limitations of interaction data sources

Dataset / release	Scale and coverage	Overview and supervision	Official resources
A. Real-robot interaction datasets
Open X-Embodiment (OXE)	$>$ 1,000,000 trajectories, 22 embodiments (reported collection)	Aggregates heterogeneous robot datasets in a common format for learning across embodiments. Signals: images, robot states/actions, and language where available. Modalities vary by source.	Project Data / code Source-specific terms
BridgeData V2	60,096 trajectories, 24 environments, 13 skills	Combines teleoperated and scripted WidowX manipulation across varied household-style settings. Signals: RGB, robot states/actions, and language, with depth and additional views in subsets.	Project / code Data
DROID	76,000 trajectories, 350 h, 564 scenes	Uses a shared Franka platform for distributed collection of manipulation in varied real environments. Signals: wrist/external stereo video, robot states/actions, calibration, and language.	Project / code Data
RH20T	$>$ 110,000 sequences, 147 tasks	Pairs contact-rich robot manipulation with human demonstrations to study multimodal skill transfer. Signals: RGB-D, robot states/actions, force/torque, audio, and human video, with tactile data in one configuration.	Project / data / code
RoboMIND Original release	107,000 trajectories, 4 embodiments, 479 tasks	Applies a unified teleoperation protocol across single-arm, dual-arm, and humanoid platforms. Signals: multi-view images, robot states/actions, language, and failure demonstrations with causes.	Project / code Data

Table 4.3: Representative real-world datasets for Physical AI. Each entry identifies a release or collection, its reported scale, the available supervision, and official access points

Source	Role	Main benefit	Must be checked
Reconstructed scenes and digital cousins	The world: geometry and appearance taken from real sites, or variations of them	Deployment-like layouts and appearance	Collision geometry, articulation, physical parameters
Generated assets and tasks	The world and its goals	Object, scene, and task diversity	Scale, mass and inertia, joint axes, task feasibility
Physics engine	Consequences, from hand-written dynamics and contact	Privileged state, dense labels, counterfactual replay	Parameter values, missing mechanisms, solver artifacts
Learned simulator	Consequences, from a network trained on solver or real trajectories	Fast rollouts, materials that are hard to model	Extrapolation, conservation, long-horizon drift
Observation generator	Appearance only, with physics computed by an engine	Visual diversity over fixed interactions	Label preservation near contacts, thin parts, and occlusions
Video world model	Consequences and observations, learned from video	Broad visual and behavioral priors	Action grounding, physical consistency, exploitation by the policy

Table 4.4: Sources of synthetic experience and what must be checked before use

Aspect	Simulated data	Real data
Scale and cost	Millions of episodes, with cost dominated by building scenes and by compute	Limited by robots, operators, and resets, with cost growing with every episode
Perception and labels	Exact poses, contacts, masks, and depth, although rendering may look unrealistic	Sensor noise, calibration error, missing depth, with labels that must be estimated or annotated
Dynamics	Only as accurate as the contact, material, and actuator models	Real consequences, including effects that no model captures, though their causes may go unmeasured
Rare and dangerous states	Staged and replayed on demand	Rare, costly, and sometimes unsafe to collect
Main role	Breadth, privileged supervision, pretraining, stress tests	Training where dynamics matter, calibration of simulators and world models, validation

Table 4.5: Complementary strengths of simulated and real data

Gap	Typical symptom	Diagnostic	Main remedies
Dynamics ( $P_{\xi_{\mathrm{sim}}}^{\mathrm{tr}}$ )	Overshoot, sluggish tracking, unexpected slip or jamming, wrong deformation	Replay identical command sequences in simulation and on hardware and compare the responses	System identification, actuator or residual models, dynamics randomization
Observation ( $O_{\psi_{\mathrm{render}}}$ )	A correct plan executed at the wrong place or time, and failures under particular lighting or materials	Compare sensor streams for matched states and measure end-to-end latency	Calibration, latency modeling, sensor noise models, visual randomization
Distribution (initial states and tasks)	Failures concentrated on unseen objects, layouts, or initial states	Stratify evaluation by condition	Targeted real or synthetic collection, broader scene and task generation
Objective (success predicate)	Episodes counted as successes in simulation that fail in reality	Audit success predicates against physical outcomes	Redefine success and reward and relabel data

Table 4.6: Components of the reality gap and how they are diagnosed and addressed

Figure 4.2: The data pyramid for Physical AI. For physically collected data, relevance to the target embodiment and cost per sample rise together toward the apex, where the target embodiment acts in the physical world. Lower layers are cheaper, larger, and more diverse but supervise physical execution less directly. Simulated and generated data are drawn separately because they supply robot actions at low cost while computing their consequences with a model, so their effective position depends on calibration. Adapted from bjorck2025gr00t and ye2026pyramid . The order of the layers reflects relevance and cost, not dataset quality

Study	Varied	Setting	Measured	Finding and caveat
lin2025datascaling	Environments, objects, demonstrations	Single-task imitation with handheld-gripper data	Real success in new environments with new objects	Rough power law in environments and objects, demonstrations saturate, single tasks only
Ai et al. (2025)	Number of training embodiments	Locomotion on about 1,000 generated bodies	Performance on unseen bodies	Adding embodiments beats adding data on fixed ones, locomotion only
pearce2025agents	Model size, data, compute	Pretraining of imitation policies and world models	Pretraining loss	Language-like power laws, with coefficients that depend on tokenizer, task, and architecture
Baniodeh et al. (2025)	Compute, model size, data	Motion forecasting and planning on 500,000 hours of driving	Loss, open- and closed-loop metrics	Power law in compute, closed-loop metrics also improve, optimal model size grows 1.5 times as fast as data
Barreiros et al. (2026)	Pretraining data and tasks	Multitask dexterous manipulation	Blind real and simulated success after fine-tuning	3 to 5 times less task data needed, mixed results without fine-tuning
generalist2025gen0	Pretraining data, model size	Over 270,000 hours of real manipulation data	Validation loss and action-prediction error	Power law in data, small models stop improving, company blog, not peer reviewed

Table 4.7: Selected evidence on scaling in robot learning and related domains

Resource	Contents	Access	Link
Hubs and indexes of many datasets
LeRobot hub	Community robot datasets in the LeRobot format, numbering in the tens of thousands	Licenses set per dataset, with some gated	huggingface.co/datasets?other=LeRobot
Open X-Embodiment	60 datasets from 34 labs, 22 embodiments, over 1 million trajectories in RLDS format ( openx2024 )	Open	robotics-transformer-x.github.io
ARIO	About three million episodes from 258 series and 321,064 tasks in a unified format ( wang2024ario )	Open, with original licenses retained	imaei.github.io/project_pages/ario
TensorFlow Datasets (TFDS) catalog	Robotics section with RLDS versions of many robot datasets	Open	www.tensorflow.org/datasets/catalog/overview
OpenDataLab	Dataset platform hosting AgiBot World among other collections	Registration	opendatalab.com

Table 4.8: Selected hubs and large public datasets for robot learning, with access conditions as of October 2026

Figure 5.1: A world model as a compact, queryable description of how the environment can change. Solid arrows carry information available at deployment. The dashed arrow carries the observation correction that revises the internal state after each step. A particular model may expose only a subset of the four query interfaces

System	Primary object	Typical interface	Not established by the interface alone
Video generator	Observations and temporal appearance	Images or videos under conditions	Correct interventions or physical quantities
Physics engine	Explicit state and physical dynamics	State propagation, contact, and forces	State inference from open observations
Digital twin	A specified physical counterpart	Monitoring, updating, and prediction	Transferable modeling across entities
Policy model	Goal-conditioned behavior	Actions, skills, or control targets	Independently queryable consequences
World model	Internal state and conditional evolution	Inference, prediction, and queries	A complete agent or controller

Table 5.1: Neighboring concepts and the capabilities their interfaces do not establish. The rows describe functions and may overlap within one system

Figure 5.2: Two-dimensional taxonomy of representative world models. The vertical axis distinguishes functional roles. Renderers produce observations, simulators predict states, and planners produce actions. The horizontal axis distinguishes architectural routes, including observation-level generative models, 3D or structured world representations, and latent-space dynamics models. Spanning regions indicate model families or systems that support multiple roles or combine representational choices. World-action models (WAMs) couple predictive modeling with action generation. Placement is schematic rather than a performance ranking. Planning capability may depend on additional policy or search components

Route	Useful interface / strength	Main cost or limitation	Critical validation
Observation generation	Detailed, inspectable futures and synthetic sensor data	Large output budget, decoding and sampling cost, plausible but wrong detail	Action-conditioned change, identity, and long-horizon consistency
Latent dynamics	Compact rollout, inexpensive task readout, planning	Objective-dependent information loss, latent drift, limited new queries	Predictive sufficiency, open-loop stability, and calibrated readouts
3D / object-centric	Metric geometry, entities, and local interactions	Reconstruction and correspondence, structural and simulator assumptions	Contact, topology, identity, physical parameters, and interventions
Multimodal / unified	Shared evidence and priors, multiple query interfaces	Objective interference, alignment and missing-data issues, runtime coupling	Information boundaries, cross-query agreement, missing modalities, and total cost

Table 5.2: Architectural routes, practical trade-offs, and principal validation questions. The routes are composable. Multimodal unification is also a cross-cutting organization choice

Underlying question	Representative work	Mechanism to compare
How is history compressed for control?	World Models, PlaNet, Dreamer	State inference, recurrent dynamics, and imagination
Which predictions are sufficient for decisions?	MuZero, TD-MPC2	Reward/value and task-relative latent consistency
Can useful futures be predicted without pixels?	I-JEPA, V-JEPA, V-JEPA 2	Target embeddings, masking, and action grounding
How are interactive observations generated?	DIAMOND, Genie, UniSim, Cosmos	Generative targets, controllable inputs, and rollout memory
How are spatial persistence and reconstruction maintained?	Marble, RTFM, Atlas	Explicit scenes or posed-view memory Measured versus completed content
How is physical structure exploited?	HRN, graph/mesh simulators, PhysGaussian, neural operators	Entities, local interactions, physical priors, and fields

Table 5.3: Representative lineages organized by enduring modeling questions. Examples instantiate the mechanisms discussed in the text. They do not share one evaluation protocol

Figure 5.3: Training world models. Three entry points (top) lead into a staged training pipeline (center), fed by the data sources of Chapter 4 (left). Interaction-driven updates retrain on the states reached by the deployed system, closing the loop through new data. The right panel defines the world-action interfaces as attention masks over observed-context (

\mathrm{C}

), action (

\mathrm{A}

), and future-observation (

\mathrm{F}

) tokens

Branch or target	Training information and updates	Deployment role
Observed-context features	Present/past observations Backbone updated if trainable	Available context May be cached
Frozen semantic conditioning	Current observations and instruction Encoder weights fixed	Retained when the action head depends on it
Future video branch	Noised future target and allowed context Video loss updates its expert	Optional Omitted in observation-only predictive-feature execution
Predictive change tokens	Observed context only Future outcomes provide losses, not inputs	Retained as action conditioning
Detached future features	Future branch supplies targets This alignment loss cannot update the target path	Not required as realized future inputs
External geometry teacher and auxiliary readout	Recorded clip targets Student learns and teacher stays fixed	Both removable if used only for distillation

Table 5.4: Information and optimization roles in predictive-feature world-action training. This is a general implementation checklist, not a claim that all world-action models use identical branches

Choice	Potential benefit	Cost or failure to check
Observation vs. feature targets	Detail and query breadth vs. compact predictive abstraction	Expensive nuisance modeling vs. omitted physical information
Frozen vs. jointly updated modules	Prior preservation and cheaper optimization vs. task adaptation	Restricted adaptation vs. objective interference and forgetting
Short vs. long rollout training	Stable local learning vs. delayed-effect and drift supervision	Myopic accuracy vs. memory cost and difficult optimization
Deterministic vs. distributional models	Low overhead vs. multiple outcomes and probabilistic queries	Hidden ambiguity vs. calibration and sampling cost
Soft vs. structural physics	Flexible constraints vs. restricted admissible predictions	Residual violations vs. bias from incorrect assumptions
Offline vs. interaction-driven updates	Reproducible reuse vs. coverage of policy-induced failures	Unseen-action errors vs. physical cost and retention risk

Table 5.5: Training choices and their principal trade-offs. Each comparison requires matched data or compute where feasible, plus evaluation on the actual prediction or control interface

Object	Evidence or protocol	Example readout	Overclaim to avoid
Observation forecast	Held-out, time-matched outcomes Declared sample budget	Pixel or perceptual error, conditional and distribution checks	Overall realism establishes action correctness
State persistence	Occlusion, reappearance, and history changes	Identity errors Position, orientation, or attribute drift	Single-frame recognition establishes memory
Action response	Controlled changes and matched initial conditions	Response magnitude, direction, and event timing	Any action sensitivity establishes a correct intervention model
Physical consistency	Independent geometry, force, and process measurements	Penetration, contact consistency, justified balance residuals	Plausibility establishes quantitative accuracy or causality
Predictive probability	Repeated held-out outcomes and separate calibration data	Brier or CRPS, coverage and width, reliability by horizon	Diverse samples establish calibrated uncertainty
Decision utility	Matched-policy and resource comparisons on real outcomes	Task score, completion time, failures, and latency	Prediction gains establish better real decisions

Table 5.6: Validation objects, suitable evidence, and common overclaims. The evaluation protocol must preserve the information available at the forecast issue time

Figure 6.1: Foundation models for robot manipulation

Figure 6.2: Algorithm development framework

Lineage	Knowledge source	Main learning target	Decision interface
Generalist robot policy	Diverse robot trajectories, task labels	Shared behavioral priors across tasks and setups	Actions, chunks, or skills RT-1, RT-X, Octo
Vision-language-action model	Vision-language pretraining plus robot interaction	Semantic grounding and conditioned action generation	Action tokens or continuous trajectories RT-2, OpenVLA, $\pi_{0}$
World-action model	Video and interaction data with available action labels	Coupled action and future-state prediction	Actions with predicted outcomes, or prediction-assisted decisions Unified World Models

Table 6.1: Three model lineages compared by knowledge source, learning target, and decision interface. The categories are complementary rather than exclusive

Figure 6.3: Two mainstream VLA structures. (a) In a mixture-of-transformers structure, the expert has its own weights but the same depth as the backbone, and in every layer its tokens attend to the backbone’s keys and values through a shared self-attention. The backbone can additionally generate reasoning text

r_{t}^{\mathrm{reason}}

or FAST action tokens through an optional language-model head. (b) In a cross-attention structure, a separate diffusion transformer reads backbone token-feature matrix

\mathbf{H}_{t}^{\mathrm{VLM}}

through cross-attention, so its depth, size and rate are independent of the backbone

Figure 6.4: Overview of the embodied VLA training pipeline. Pre-training combines multi-source data under multi-task objectives to produce a VLA base model. Supervised fine-tuning (SFT) during supervised post-training behavior-clones demonstrations into an executable initial policy. And reinforcement fine-tuning (RFT) during reinforcement learning-based post-training refines the policy from rewards, safety costs, and vision-language evaluator feedback. The stages are discussed in Sections 6.3.1 to 6.3.3

Data source	Available supervision	Supported objectives	Primary module updated	Missing labels
Internet image-text	Object/scene semantics, instruction pairing	$\mathcal{L}_{\mathrm{language}}$ ( $\mathcal{L}_{\mathrm{state}}$ from captions)	Vision-language backbone	Action, progress masked
Human video	Temporal order, task stages, narration	$\mathcal{L}_{\mathrm{progress}}$ , $\mathcal{L}_{\mathrm{language}}$ , latent-action $\mathcal{L}_{\mathrm{action}}$	Backbone, latent-action or embodiment adapter	Executable action masked
Robot demonstrations, teleoperation	Executable actions, proprioception Stages and outcomes when annotated	$\mathcal{L}_{\mathrm{action}}$ , $\mathcal{L}_{\mathrm{state}}$ ( $\mathcal{L}_{\mathrm{progress}}$ , $\mathcal{L}_{\mathrm{outcome}}$ )	Action expert (and adapters)	Unannotated items masked
Simulation	Dense state, contact, action, outcome	All five	Backbone and action expert	—
Deployment logs (interventions, failures)	Outcomes, corrections, takeovers	$\mathcal{L}_{\mathrm{outcome}}$ Value/critic targets at later stages	Value head, policy via post-training	Per record type

Table 6.2: Correspondence between heterogeneous data sources, the supervision they provide, the objectives of Eq. ( 6.7 ) they support, and the model components primarily updated. Objectives whose labels a source does not provide are masked rather than assigned artificial targets. A dash means that missing labels are not specified

Figure 6.5: The continual learning loop for embodied agents in long-term deployment. Deployment interactions are logged as experience records (success, failure, takeover, recovery), and each iteration updates the policy version

v_{k}

to

v_{k+1}

through gap identification, targeted collection, offline update, validation, and gated deployment. The process is detailed in Section 6.3.4

Figure 6.6: Shared modeling, embodiment-specific adaptation, and adaptation evaluation for manipulation. (a) Observation interfaces, shared task representations, and executable outputs illustrate complementary token, prompt, adapter, action-head, and latent-decoder mechanisms. (b) Changes in camera mounting pose (with camera count unchanged), end-effector degrees of freedom (DoF), and interaction dynamics motivate corresponding adaptation targets. (c) Independent closed-loop trials assess target success and budget-matched transfer gain, retention on previously supported embodiments, adaptation data efficiency, and physical execution. For compactness, panel (a) depicts current-observation inputs and deterministic execution mappings. Eq. ( 6.12 ) gives the general history-conditioned policy formulation

Placement	Useful property	Main constraint	Typical allocation
Robot onboard	Short communication path, local autonomy	Memory, power, cooling	Compact policy and local feedback
Local server	Larger shared accelerator capacity	Network and queueing variability	VLA inference with local execution
Cloud	Centralized resources and fleet-level services	Wide-area delay and outages	Task reasoning, training, and non-urgent analysis
Split deployment	Different compute budgets for different time scales	Context consistency and interface design	Slow reasoning with fast local action updates

Table 6.3: Deployment placement and its principal performance constraints

Measurement	Interpretation	Useful test condition
Median and tail latency	Typical response and occasional late decisions	Concurrent requests, warm-up, sustained load
Observation age, deadline misses	Freshness of feedback used in execution	Moving targets, delayed frames, network jitter
Memory, power, task energy	Resource feasibility and operating cost	Long runs at the intended thermal limit
Success and completion time	Whether speed changes useful performance	Matched tasks and initial-state distributions
Contact error and recovery	Precision and response beyond nominal motion	Insertion, slippage, unexpected contact
Interventions and fault recovery	Dependence on operator support	Timeouts, sensor loss, server interruption

Table 6.4: Deployment measurements and the questions they answer

Figure 6.7: The learning chain on Sharpa Wave. The upper path follows EgoScale’s training stages, with human and robot observations connected through wrist motion and hand-action representations ( zheng2026egoscale ; nvidia2026egoscaleproject ) . The lower panels place separate studies of execution feedback and simulation-based adaptation, discussed in Sections 7.4.4 and 7.4.5 . The grouping is conceptual. The studies share no weights or training with EgoScale

Method	Task progress	Final success
$\pi_{0}$	38.0	35.0
UniDex-VLA without pretraining	32.5	23.0
UniDex-VLA	81.0	76.0

Table 6.5: UniDex real-robot results across 5 tasks using Wuji and Inspire hands, with 20 trials per task and method. Task progress averages stage outcomes. Final success measures task completion. Aggregate means reproduced from Fig. 11 of zhang2026unidex . Values are percentages

Figure 7.1: The whole-body motor controller in the robot stack. A task-level planning layer issues goals to a goal-conditioned sensorimotor closed loop, which reads proprioceptive and exteroceptive measurements and issues control commands through the actuator interface. Contacts couple the body to the environment, and measured execution returns to constrain planning. Section markers indicate where Sections 7.1 to 7.5 develop this loop

Approach	Execution mechanism	Acquisition and reuse	Representative examples
Analytical feedback	Task-space motion and force feedback	Explicit models and designed interaction objectives	Operational-space and impedance control ( khatib1987 ; Hogan1985 )
Predictive optimization	Online finite-horizon optimization with feedback	Dynamics and constraints used to recompute motion or force plans	Cheetah 3 convex MPC ( dicarlo2018 )
RL feedback policies	Observation- and command- or reference-conditioned actions	Task rewards or motion imitation History-based inference can support adaptation	DeepMimic, ANYmal, RMA ( peng2018deepmimic ; hwangbo2019 ; kumar2021 )
Motor priors and skills	Feedback policy, optionally conditioned on a skill latent	Demonstration distributions or distilled experts reused in downstream learning	NPMP, AMP, ASE ( merel2018npmp ; peng2021amp ; peng2022ase )
Multi-mode motor pretraining	Shared goal- or command-conditioned policy	Distillation or broad RL pretraining exposes a reusable repertoire	HOVER, BFM, ScaleBFM ( he2025hover ; zeng2025behavior ; zeng2026scaling )
Hybrid planning and tracking	Online optimization with a learned feedback tracker	Planned references executed by a policy trained for tracking	Deep Tracking Control ( jenelten2024 )

Table 7.1: Selected control approaches compared by execution mechanism and the acquisition and reuse of motor knowledge

Figure 7.2: Closed-loop architecture and interfaces of a whole-body motor controller. Goal processing supplies

g_{t}

, observation processing supplies

h_{t}

and

\hat{s}_{t}

, and the control model outputs

\mathbf{a}_{t}

, which the actuator interface realizes as joint torques

\boldsymbol{\tau}_{t}

through actuator dynamics and low-level loops. Measurements close the loop. Operation at multiple update rates requires explicit temporal alignment

Figure 7.3: Floating-base dynamics, contacts, and actuation of a legged robot. Joint torques act through the selection matrix on the joint rows only, so contact forces are central to movement. Contacts contribute forces subject to sticking, unilateral, and friction-cone constraints, and the delivered torque follows the commanded torque through a first-order lag with saturation

Figure 7.4: Explicit and implicit state estimation on a legged robot. Explicit estimation fuses inertial and kinematic measurements, joint encoders, and an estimated contact set into a body state estimate through a propagate-correct loop. Implicit estimation maps recent observation-action history to a task-relevant latent. Deployed systems combine the two routes

Figure 7.5: Local terrain estimation for perceptive locomotion. Depth measurements are fused, under an uncertain estimated pose, into a local elevation map in which each cell retains a height estimate and a variance. The map is anchored to the estimated state and updated below the control rate, so it must be aligned to the current estimate before use

Sensor / input	Estimator	Output	Controller / use	Failure / limit
IMU, encoders, contact cues	Contact-aided EKF / InEKF	Base pose and velocity, footholds, biases, covariance	Legged feedback or trajectory tracking	Slip and contact errors, global position and yaw drift
Orientation, encoders, planned foothold	Anchor point and forward kinematics	Base translation and CoM	HRP-4 walking stabilization	Incorrect foothold assumption or foot flexibility
Point clouds, odometry, gravity alignment	Local elevation mapping	Terrain heights and available uncertainty	Footstep planning or perceptive policy	Pose drift, occlusion, outdated geometry
Proprioceptive and action history	RMA adaptation module	Task-relevant latent $\mathbf{z}_{t}$	Adaptive joint-target policy	Ambiguous history, conditions outside training
History, terrain observations in PIM	HIM / PIM hybrid estimator	Velocity estimate and response latent	Command-conditioned locomotion	Unseen dynamics, erroneous terrain input

Table 7.2: Representative sensor-estimator-controller interfaces and their characteristic failure modes

Figure 7.6: Model-based humanoid motion control, illustrated with Unitree G1. Command paths connect reference generation, NMPC, WBC, and joint control. Feedback paths supply estimated state, robot-model quantities, and local joint measurements

Figure 7.7: Visualization of the Humanoid Transformer architecture applied in ScaleBFM ( zeng2026scaling )

Concept	Humanoid body	Dexterous hand
Unactuated body	Floating base, Eq. ( 7.5 )	Held object, Eq. ( 7.41 )
Contacts	Feet or hands on the environment Friction cone	Fingertips, phalanges, and palm on the object Same cone
Contact schedule	Gait phase, stance and swing (Section 7.3 )	Finger gaiting (release, reposition, re-contact)
Stability objective	Balance of the center of mass over the support	Grasp stability (resultant wrench and friction margins)
Hidden state	Base velocity, foot contact, terrain	Object pose, contact locations, incipient slip
Command interface	Velocity commands, masked link targets, Eq. ( 7.32 )	Fingertip targets, object-pose goals, functional finger coordinates

Table 7.3: Correspondence between humanoid whole-body control and dexterous in-hand control. References in the humanoid-body column point to the relevant formulations and sections of this chapter. The dexterous-hand column shows analogous quantities for a multi-fingered hand holding an object

Figure 7.8: Sources of sim-to-real mismatch and their propagation through a planner-tracker system. The latency sum depicts a sequential processing path. The dashed state-to-planning connection applies when execution feedback conditions the planner

Figure 8.1: Closed-loop interaction between a physical agent and the physical world. Perception and task-relevant representations inform decision-making, which may include optional prediction and scenario simulation. Decisions are realized through embodied physical actions, and these actions change the physical state of the agent and/or environment. The resulting state gives rise to new observations that inform subsequent decisions. Here

x_{t}

is the joint physical state at step

t

,

x_{t+1}

its next state, and

o_{t+1}

the resulting observation

Figure 8.2: Physical forms distinguished by the boundary of attribution, the organization of action, and the scope of the task. Dashed outlines delimit the selected agent. Solid dark arrows denote physical action or flow, green arrows commands, and dashed blue arrows information. The three views can be nested. Original analytical synthesis informed by pfeifer2007self ; yim2007 ; monostori2016

Figure 8.3: Morphology, action interface, and limiting conditions for six representative forms. Shaded reach and sensing regions are qualitative. Arrows indicate motion or interaction directions. The compliant gripper contrasts nominal straight fingers with deformed contact. These original schematics describe physical affordances, not the autonomy of a particular product

Figure 8.4: Two forms of inter-module dependence. (a) Base configuration

\mathbf{q}_{\mathrm{B}}

and arm configuration

\mathbf{q}_{\mathrm{A}}

jointly determine tool pose.

\mathrm{W}

,

\mathrm{B}

, and

\mathrm{E}

denote world, base, and end-effector frames. (b) Separate routes require compatible recovery times and sufficient flight energy. Symbols within the schematic are defined in the accompanying text. The reach boundary and waiting timeline are qualitative. Original schematics informed by mobile manipulation ( fu2025 ) and truck-drone routing ( murray2015 )

Figure 8.5: Physical and informational organization of a system-level physical agent. Coordination functions operate through shared interfaces, while individual resources retain local feedback and interlocks. Commands, observations, and material flows are distinguished explicitly. The layers describe functional responsibilities and do not prescribe a centralized network architecture. Original synthesis based on holonic and cyber-physical manufacturing ( vanbrussel1998 ; lee2015 ; monostori2016 )

Task requirement	Physical properties to examine	Evidence of a suitable match
Spatial access and motion	Dimensions, degrees of freedom, workspace, support and clearance	Required positions and orientations can be reached and the intended motion can be executed
Physical action and load	Tool or process interface, actuation, stiffness, friction and contact geometry	Required loads or process conditions are maintained within the specified tolerances
Observation and verification	Sensor placement, field of view, measurement access and calibration	Task-relevant states and outcomes can be measured with sufficient accuracy
Sustained operation	Energy supply, thermal limits, replenishment and available capacity	Performance can be maintained for the required duration or rate of work
Handover between operations	Physical connections, transfer paths, interface compatibility and shared resources	The output of one operation meets the input conditions of the next

Table 8.1: Connecting task requirements to physical form and evidence of capability. The dimensions provide a framework for comparison, not a single score of intelligence

Figure 8.6: Grounding an instruction in task-relevant objects, state conditions, constraints, and completion evidence. Observations and embodiment capabilities inform interpretation. Unresolved uncertainty may require further observation or clarification

Figure 8.7: Hierarchical action organization with downward task refinement and upward feasibility feedback. Geometric, dynamic, and resource constraints can revise higher-level choices within modular, learned, or hybrid implementations

Figure 8.8: Observed effects update state estimates and execution context. Verified progress supports continuation. Discrepancies trigger diagnosis and recovery. Insufficient evidence prompts further observation. Protective monitoring can interrupt the loop independently of task planning

Figure 8.9: Four organizational architectures for distributed Physical AI, comprising centralized, hierarchical, decentralized, and hybrid

Dimension	Centralized	Hierarchical	Decentralized	Hybrid
Global coordination	Joint decisions from reported state, with sensitivity to missing or stale information.	Cross-team coordination through upper levels, where abstraction can hide dependencies.	Global constraints require agreements across local views.	Coordination concentrated on selected global dependencies.
Communication dependence	Requires access to the center for team-level decisions.	Depends on links within teams and between levels.	Depends on the connectivity required by local protocols.	Varies by function and operating mode.
Scalability	Central computation and reporting can become bottlenecks.	Grouping reduces load, while cross-team coupling adds overhead.	Sparse interactions scale more readily than strongly coupled tasks.	Flexible grouping adds management complexity.
Response time	Team decisions incur center communication and processing delays.	Local decisions are fast, while escalation adds delay.	Local response is fast, while negotiation may delay agreement.	Depends on authority transfer and coordination mode.
Fault tolerance	Center failure requires redundancy or local fallback.	Fault isolation depends on backup paths and coordinators.	Avoids a sole center, though partitions can disrupt commitments.	Alternative modes help when transitions are reliable.

Table 8.2: Typical tradeoffs among organizational architectures for distributed Physical AI

Figure 8.10: Eight interdependent mechanisms of multi-agent collaboration. The mechanisms may operate concurrently, exchanging information and revising decisions as task conditions and agent capabilities change

Embodiment	Physical advantage	What the feedback must resolve
Fixed instruments	Fixtures and specialized tools support repeatable operations	Whether a process achieved its target volume, temperature, or other measured result
Fixed-base arms	Stable support for manipulation within reach	Tool position and contact loading, including errors hidden by visual occlusion
Mobile manipulators	Access to several workstations	Base location and support as well as arm and object state
Legged and humanoid robots	Movement across steps and uneven ground	Balance and changing support contacts while moving or manipulating
Dexterous hands	Multiple contacts permit adjustment within the grasp	Finger configuration, contact distribution, and incipient slip
Soft and continuum robots	Deformation accommodates confined spaces or delicate objects	Changing shape and loading despite material hysteresis and distributed contact

Table 9.1: Common embodiments and the feedback their tasks require. Categories overlap. A hand belongs to a carrier, and a laboratory may combine several kinds of machine

Figure 9.1: Models reach the world through different control arrangements. A modular hierarchy exposes intermediate estimates and plans. An end-to-end policy learns a more direct observation-to-action mapping. An agent calls skills and checks their outcomes. Local feedback supports timely motion and contact regulation in each arrangement

Figure 9.2: Embodiment selection develops through increasingly realistic evidence. Task analysis and specifications identify candidates. Operation-level tests, complete workflows, and continued use reveal whether the proposed body and controller meet the requirements

Configuration	Interaction to evaluate	Evidence needed
Torque-sensing dual arm with impedance control	Yielding during cap engagement and vial seating	Contact loads, alignment errors, damage, and completion under matched tuning
Position-controlled dual arm without added force sensing	Precise free motion followed by constrained contact	Contact-force peaks, effects of passive compliance, and recovery from misalignment
Low-reduction dual arm with motor-current feedback	Responsive motion and current-based loading estimates	Accuracy under load, effects of friction and acceleration on force estimates
Arms carrying dexterous hands	Reorientation and cap handling through multiple contacts	Slip, successful in-hand adjustments, and whether fewer regrasps offset coordination costs

Table 9.2: Questions for evaluating four candidate configurations in a MOF workflow. These are mechanisms to test, not measured rankings. The first three configurations use parallel grippers

Figure 9.3: Measurements connect learning to physical action. Calibrated, synchronized observations become training trajectories. During execution, the policy supplies targets, while local controllers use fresh body and contact measurements to drive the actuators. The commanded movement and the movement actually achieved need not be identical

Figure 9.4: Sources of experience for Physical AI. Human video offers broad coverage. Simulation and handheld devices address different parts of the gap between observation and execution. Robot recordings capture the target body’s behavior. The levels describe complementary evidence, not a universal ranking of dataset quality. EMG denotes electromyography. It records muscle activity rather than robot contact forces

Figure 9.5: Different measurements support different parts of an action. A camera observes the scene, an encoder reads shaft angle, and a shunt converts electrical current into a small voltage. The opened encoder shows its coded disk. These examples illustrate principles rather than a prescribed robot sensor suite. Photos: (a) Marc Auledas , CC BY-SA 4.0 ; (b) Mike1024/Dicklyon , public domain; (c) Marek bydg , CC BY-SA 3.0

Figure 9.6: Optical touch converts contact into an image of a deformable surface. The reflective coating and illumination allow surface shape to be recovered independently of the contacted object’s optical appearance. Reproduced from Figure 2 of yuan2017gelsight , under CC BY 4.0 . The illustration describes the sensing principle, not the packaging of every modern fingertip

Figure 9.7: Motor construction and feedback integration. (a) A lightweight winding removed from a coreless DC motor. (b) Stationary windings of a brushless motor with its rotor removed. (c) A stepper motor, whose motion follows commanded magnetic alignments. (d) A hobby servo showing the motor, reduction gears, and output-position sensor. The servo is an integrated feedback system, not a separate electromagnetic motor family. Photos: (a) Avsararas , (b) DnaX , and (d) Jstapko , CC BY-SA 3.0 ; (c) Dolly1010 , CC BY 3.0

Figure 9.8: Functional materials become useful to physical intelligence through measured interaction. (a) In a conceptual HASEL cross-section, electrostatic attraction redistributes liquid and deforms the pouch. (b) Self-sensing contributes to feedback, while task measurements establish whether the interaction achieved its purpose ( Acome et al., 2018 ; ly2021hasel ) . Shapes are illustrative, and the diagram does not imply that capacitance alone measures contact force

Figure 9.9: Product hardware and the feedback needed for Physical AI. (a) RobotEra XHAND 1 links visual and optional tactile observations to commanded finger motion and renewed contact measurements. (b) Gallant on Unitree G1 links 3D terrain perception to learned joint targets and local control ( ben2025gallant ) . Official product renders from RobotEra and Unitree Robotics . The G1 image shows the base platform, without Gallant’s additional LiDARs. The control diagrams summarize information flow, and the products are not shown to scale

Figure 9.10: The challenge of physical AI and device-edge-cloud collaborative Architecture

Figure 9.11: Nested feedback loops with distinct control authority. Task proposals pass through the Harness and a skill interface before reaching local execution. Sensor feedback closes the control loop directly on the device. A separate protection path can inhibit actuation. Dashed paths carry perception evidence and executor status to the Harness. Protection-event reporting to the supervisor is omitted for clarity. Intervention uses the local path

Figure 9.12: Asynchronous inference coupled to local execution. Variable-duration policy computation produces candidate action chunks, while the local loop consumes a validated buffer at its own period. Admission and a controlled buffer switch connect the two. Expired results are discarded. Loss of valid command authority activates a local fallback. The control ticks are schematic. The gate checks continuity as well as validity before the atomic buffer switch

Contract field	Concrete interface and obligation
Input	request_id , vial identity, source and destination fixture IDs, pose and frame, calibration version, payload class, and maximum task duration $T_{\mathrm{task}}$ . Duplicate IDs query the recorded execution state rather than starting another transfer.
Preconditions	Identity and source occupancy confirmed, destination empty, pose age satisfying Eq. ( 9.5 ), compatible gripper and payload, local controller ready, required interlocks valid. Unknown conditions trigger inspection or rejection.
Resources	Exclusive arm and gripper authority, reservation of the destination slot and shared motion region. Acquisition is bounded. Partial acquisition is released on rejection.
Timing	Task duration $T_{\mathrm{task}}$ starts at local admission. Each phase has its own timeout. Task commands expire at the earlier of the local lease deadline and their observation-limited expiry. Fallback begins no later than that point, independently of Harness cancellation.
Progress	Timestamped phase, controller status, grasp/contact evidence, observation age, and remaining authority. “Goal accepted” and “target pose reached” are intermediate states.
Completion evidence	After release and retreat, destination occupancy and vial identity are observed, the source is empty, and the gripper no longer holds the vial. Return evidence timestamps, confidence or validity flags, and the relevant log references.

Table 9.3: An illustrative transfer_vial skill contract. Symbols denote configured bounds, not universal thresholds. The local executor enforces command validity and fallback independently of the task supervisor

Figure 9.13: Validation stages answer different questions. SIL tests execution logic against a modeled plant. HIL adds target electronics and timing. Bounded physical trials expose sensing, contact, and actuation effects absent from the model. Release decisions apply to a versioned combination of model, firmware, controller, action adapter, calibration, and interfaces. Failure evidence drives component revisions and regression tests

Level	Example deviation	Evidence needed	Scope of correction
Component	Gripper response changes after warm-up	Current, temperature, measured motion and load	Reduce duty and inspect the drive or transmission
Skill	Cap slips during opening	Contact change, cap motion, applied torque	Re-establish support and revise the grasp
Workflow	Vial contents disagree with the task record	Sample identity, measured quantity, verified checkpoints	Reconcile progress and recover or reject the sample
Fleet	Failures rise after a policy release	Outcomes by hardware, calibration and software version	Isolate affected configurations and validate restoration

Table 9.4: Reliability depends on matching physical evidence to the scope of correction. A failure observed at one level may originate at another

Figure 9.14: Detection and recovery change the probability of completing a 50-stage task. In Eq. ( 9.7 ),

p=0.98

,

d^{\mathrm{detect}}=0.9

, and

r^{\mathrm{rec}}=0.8

give an effective stage success of 0.9944 and about 76% task success, compared with 36% for

p=0.98

and 61% for

p=0.99

without recovery. These are illustrative calculations under the stated assumptions, not measured robot performance

Measure	Definition and interpretation
Task success	Valid completed tasks divided by all attempted tasks, with assistance reported separately
Effective throughput	Valid outputs per scheduled operating hour. A faster motion is useful only if it increases accepted output
Human intervention	Interventions per scheduled operating hour, together with operator minutes consumed and the reason for assistance
Downtime	Time unavailable for assigned work divided by scheduled time. Separate planned maintenance from unplanned stops
Recovery time	Elapsed time from fault detection to verified return to service. Include waiting for assistance and physical rechecks

Table 9.5: Operational measures for comparing releases under the same workload. Scheduled time includes stops and recovery. Aborted attempts remain in the task record. Report trial counts and uncertainty with success rates, and distributions for intervention and recovery times

Figure 9.15: Hand geometry and local rotational coordinates. The surface-smoothed MANO template and imported XHAND, Wuji, and Sharpa meshes are shown at a common orthographic scale after hand-frame alignment ( romero2017mano ; spider2026xhandassets ; wuji2026urdf ; sharpa2026urdf ) . Axis marks locate model joints. Adjacent numerals give the local rotational dimension, with coincident robot axes grouped. MANO uses three pose parameters at each finger joint. Robot labels count active Unified Robot Description Format (URDF) coordinates. Wrist and arm motion are excluded. Correspondence therefore requires retargeting through the robot’s kinematics, rather than copying the human pose vector

	XHAND 1	Wuji Hand (first gen.)	Sharpa Wave W01
Active DoF	12 (thumb 3, index 3, other fingers 2 each)	20 (four per finger)	22 (thumb 5, fingers 4 each, plus little-finger carpometacarpal (CMC) joint)
Drive (as described)	Geared motor at each driven joint	Self-locking rotary direct-drive joints, field-oriented control (FOC) drive	—; zero-force mode is backdrivable
Joint control	Joint targets with host-set gains and torque limits	Position control	Position, admittance, impedance (MIT), zero-force
Mass and size (mm)	1.1 kg, $190\times 94\times 47$	0.58 kg excl. cables, $201\times 75\times 50$	1.3 kg, $208\times 90\times 50$
Max. fingertip force	15 N (whole hand 80 N)	15 N	20 N (active)
Grasp-diameter limit	Minimum 16 mm	Maximum 100 mm	Minimum 10 mm

Table 9.6: Manufacturer specifications of the three dexterous hands used in the case studies of Sections 6.6 , 7.4.4 , and 9.6 ( robotera2026xhandmanual ; robotera2024xhandprotocol ; wuji2026v1manual ; sharpa2026manual ) . Values are as stated by each manufacturer for the cited version and were not independently verified. A dash means that the cited document does not state the value. Ratings are not directly comparable, because test conditions are not specified uniformly

Figure 9.16: Three routes from demonstration to learning data. A tracker and glove can record direct human manipulation or drive a robot through retargeting. A mechanically coupled passive hand supplies another form of direct demonstration. Solid arrows show motion or control. Dashed arrows show recording. The exoskeleton row illustrates passive collection, as in DexEXO. An actuated or remotely controlling exoskeleton requires a different feedback path. These are collection routes, not a fixed assignment of devices to training stages ( manus2026tracking ; wuji2026teleop ; zhu2026dexexo )

Figure 9.17: Two G1 integration routes. HOMIE’s teleoperation interface combines arm and hand targets with velocity, turning, and torso-height commands to a learned lower-body controller. Its autonomous visual policy predicts arm, hand, and height targets. In SONIC’s whole-body VLA route, a shared motion token conditions the body controller, while separate joint targets command the hands. In both systems, local proprioceptive feedback supports body control, and visual observations inform task commands. VR motions in SONIC are encoded into the shared motion representation ( ben2025homie ; luo2026sonic )

System question	Reported evidence	Further deployment question
Body and task compatibility	AdaPT couples the serving motion to end-effector choice and a specified observation setup.	Do payload, mounting, visibility, and sensing delay remain within the tested envelope at the intended site?
Feedback and execution	ASAP adapts the simulated response from robot measurements, while BeyondMimic separates estimation and policy updates.	Does the complete feedback path remain timely under the intended load, and what happens when an observation or command expires?
Task composition	HOMIE and SONIC report trials connecting hand actions to body motion through different command interfaces.	Which physical evidence permits each transition, and how are failed grasps, interrupted tasks, and operator interventions handled?
Repeated operation	ASAP reports overheating and robot damage during high-dynamic-motion data collection.	What duty cycle, maintenance, and recovery procedures preserve valid output over the scheduled operating period?

Table 9.7: From humanoid integration evidence to deployment questions. The middle column summarizes the cases discussed in this section. The final column identifies additional evidence needed for the intended operating conditions, following Sections 9.4 and 9.5

Figure 10.1: Overview of physical artificial general intelligence (Physical AGI). Building on Physical AI, Physical AGI is described by the four capability dimensions of breadth, depth, plasticity, and continuity. It is pursued through four pathways, which are systematic generalization, collaboration between humans and robots, collective Physical AI, and self-evolution. Its ultimate goal is to advance scientific discovery and to enable engineering beyond the reach of human capability

Figure 11.1: A prospective framework for evaluating and governing physical agents connects interaction evidence, capability assessment, action alignment, and continuous system safety assurance. Law, ethics, standards, and global coordination provide overarching obligations and oversight. Solid arrows show evidence and constraints, while dashed paths feed operational experience back into evaluation and governance

Stage	Evaluation question	Evidence and possible output
Perception	Does the physical agent identify and locate the target?	Reference images or tracking, selection accuracy and pose error.
Decision	Are its chosen steps feasible and responsive to feedback?	Task rules and decision logs, satisfied preconditions and appropriate replanning.
Action	Does its embodiment execute commands within constraints?	Joint, force, and tactile records, together with tracking error, slipping, and interventions.
Outcome	Is the required physical state reached and maintained?	Post-action observations, placement, stability, damage, and spillage.

Table 11.1: Evaluation evidence across a physical agent’s interaction episode

Figure 11.2: A proposed layered framework connects benchmark coverage with real-robot evaluation settings. Each benchmark is shown once at its highest represented layer, indicating partial and overlapping coverage rather than a universal difficulty ranking. Illustrative laboratory, industrial, and agricultural settings show where actions, feedback, and physical consequences must be measured. Claims of sustained deployment require closed-loop evidence from the intended working environments

Capability	Controlled test	Observable evidence
Perception	Occlude a handle, allow a viewpoint change	State accuracy, response to ambiguity, information-gathering cost
Reasoning	Change a prerequisite or block a planned step	Feasible alternatives, consequence-sensitive decisions, replanning
Manipulation	Vary mass, friction, tolerance, or compliance	Adaptive contact, force excursions, placement quality and damage
Mobility	Change access, floor conditions, or payload	Reachability, balance, usable final pose, route revision
Long-horizon tasks	Interrupt execution or revise an intermediate goal	Workflow success, memory, progress retention, intervention frequency
Generalization	Hold out objects, sites, properties, or embodiments	Transfer by shift type, adaptation cost, retained competence

Table 11.2: A proposed capability-to-test matrix for physical agents. Generalization and adaptation cross all capability layers

Benchmark	Task scope	Setting	Main metrics	Training conditions
CALVIN ( mees2022calvin )	34 skills, five-instruction chains.	PyBullet simulation, Panda arm.	Chain success, mean completed chain length.	Play demonstrations, D $\to$ D, ABCD $\to$ D, or ABC $\to$ D, declared sensors.
LIBERO ( liu2023libero )	130 tasks across four manipulation suites.	Tabletop simulation.	Success, transfer and retention during lifelong learning.	Demonstrations, sequential or joint training, optional LIBERO-90 pretraining.
LIBERO-Plus ( fei2026liberoplus )	LIBERO tasks with seven perturbation families.	LIBERO simulation, controlled variations.	Success by perturbation type and severity.	Existing checkpoints, distinguish additional robustness fine-tuning.
SIMPLER ( li2024simpler )	Manipulation with controlled visual and scene variations.	Simulation aligned to Google Robot and WidowX.	Success, sim-real ranking consistency in validated settings.	Real-world policy checkpoints, declare simulation adaptation.
RoboCasa365 ( nasiriany2026robocasa365 )	365-task catalog, varied kitchen scenes and task compositions.	Simulated household mobile manipulation.	Task success by split and learning protocol.	Specified demonstration mixtures, pretraining, adaptation, or lifelong learning.
RoboLab-120 ( yang2026robolab )	120 tasks Visual, relational, procedural competencies.	Isaac Lab, robot-agnostic task interface.	Success, graded completion, errors, trajectory quality, sensitivity.	Reported real-data-trained policies. Declare any simulation training or adaptation.

Table 11.3: Simulation benchmarks for physical agents. Task scope, setting, metrics, and training conditions follow the cited protocols

Benchmark	Task scope	Setting	Main metrics	Training conditions
RoboArena ( Atreya et al., 2025 )	Evaluator-chosen tasks across institutions.	Distributed real robots, DROID platform.	Double-blind preferences, relative ranking.	Submitter-selected training, compatible policy interface.
AutoEval ( zhou2025autoeval )	4 tasks in original prepared scenes.	Real WidowX, automated scoring and resets.	Task success from a validated detector.	Policy training varies, evaluator/reset training is separate.
Table30 ( yakefu2025robochallenge )	30 multi-stage manipulation tasks.	Shared remote real robots, fixed-table setups.	Success, stage progress with retry penalties.	Provided demonstrations, task or generalist adaptation.
Table30 V2 ( ma2026table30v2 )	30 tasks, revised coverage and held-out tracks.	Shared real robots, revised protocols.	Success by track, progress at timeout.	Multi-task training, object/environment holdouts, task training may be allowed.
RoboDojo ( chen2026robodojo )	42 simulated and 18 physical manipulation tasks.	Isaac Sim, three real robot embodiments.	Success, partial-progress score, capability-wise averages.	Provided demonstrations, Open tasks lack task-specific simulation demonstrations.
NIST assembly ( kimble2020assembly )	Insertion, fastening, connectors and flexible parts.	Physical task boards, declared hardware.	Completion, time and assembly quality.	No common training corpus, declare programming and tuning.

Table 11.4: Real, hybrid, and application-oriented physical agent benchmarks. Comparison dimensions match Table 11.3 , scores are protocol-specific

Figure 11.3: Risk-driven framework for system safety and risk governance. (a) Faults in sensing, software, communication, power, and actuation can propagate through perception, control, and execution into physical hazards. (b) The corresponding safety assessment maps perception degradation to sensor-validity and estimator checks. It maps timing and computation faults to latency, jitter, and command-age tests. It maps actuation or power hazards to current, thermal, and power-loss verification. (c) Safety evidence supports pre-deployment verification and post-deployment monitoring, user warnings, protective intervention, updates, maintenance, and reassessment. Solid arrows indicate the progression from identified risks to assessment and safeguards. The dashed arrow denotes field feedback

System layer	Representative failure modes	Potential safety consequences	Monitoring / evaluation dimensions
Models and planning	Unsafe decisions Infeasible joint targets	Hazardous motions Unintended contact	Constraint violations Safety-filter interventions
Real-time software	Missed deadlines Scheduling blocks, stale commands	Delayed corrective action Loss of stability	Deadline-miss rate Control jitter, command age
Computing hardware	Thermal throttling Processor faults	Increased control latency Interrupted execution	Chip temperature Clock rate, execution latency
Sensors	Image saturation IMU drift, frame loss	Missed obstacles Incorrect measurements	Saturated-pixel ratio Bias, frame-drop rate
Estimation and control	Estimator divergence Feedback delay, oscillation	Unstable motion Excessive contact	Estimation residuals Feedback age, oscillation amplitude
Actuators and transmissions	Motor stall, overcurrent Overheating, transmission wear	Joint overload or damage Loss of support	Motor current, temperature Torque-limit events, tracking error

Table 11.5: Representative failure modes, safety consequences, and evaluation dimensions across robotic system layers. Indicators are illustrative and platform-dependent

Tier	Illustrative conditions and harm	Governance focus
Minimal	Supervised, low-energy desktop sorting without sharp, hot, or heavy tooling. Contact harm is at most trivial and is reversible.	Basic product safety, use limits, instructions, and controlled updates.
Low	Industrial operation behind validated guarding and interlocks, without routine human access during automatic operation. Exposure is mainly during setup or maintenance, but contact remains potentially severe.	Machinery assessment, protective-function validation, access control, and safe maintenance with lockout procedures.
Moderate	Speed-limited goods transport in worker-accessible areas, repeated encounters with reversible injury or localized damage foreseeable.	Route and speed limits, detection and stopping tests, monitoring, and incident analysis.
High	Close-contact collaboration, hazardous household manipulation, or elder-care transfer assistance. Human exposure is frequent, and crushing, burns, cuts, falls, or incorrect assistance are foreseeable.	Independent evaluation, meaningful oversight, runtime safeguards, incident reporting, and privacy protection.
Very high	Traffic-speed autonomous driving, invasive surgical motion, or critical-infrastructure control. Death, multiple casualties, or cascading disruption are possible.	Rigorous assurance, independent review, continued supervision, appropriate fail-safe or fail-operational design, and sector-specific compliance.

Table 11.6: Illustrative risk tiers proposed in this monograph. Placements depend on task, exposure, and validated safeguards

Figure 12.1: Five development priorities for Physical AI, including foundation-model convergence, self-evolving agents, hardware-software co-evolution, the Internet of Physical Agents, and a supporting scientific and technological ecosystem. Operational evidence informs all five priorities. The research horizons are indicative and conditional, rather than fixed delivery commitments

Figure 12.2: The learning-action-discovery-relearning cycle. Physical interaction generates observations that reveal capability gaps and candidate discoveries. Qualified experience informs updates to memory, skills, or models, subject to evaluation and controlled deployment. Adaptation within a task is distinguished from persistent learning across tasks

Figure 12.3: Conceptual architecture of the Internet of Physical Agents. Shared capability services, federated coordination, and locally autonomous physical nodes support multidirectional exchange of requests, observations, results, and validated updates. Nodes also cooperate with peers and interact with their local environments. Logical services may be distributed across sites, while time-critical control remains local

Term	Meaning in this monograph	Chapters
2D	Two-Dimensional . Representation in a plane or another two-dimensional space.	3 , 5 , 6
3D	Three-Dimensional . Representation of spatial position, shape, and orientation in three dimensions.	1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 11
3DGS	3D Gaussian Splatting . Scene representation and rendering using anisotropic three-dimensional Gaussian primitives.	3 , 4
4D	Four-Dimensional . Three-dimensional spatial structure evolving over time in the perception context used here.	3 , 5
Action alignment	Consistency of intentions, plans, and execution with human authorization, constraints, and acceptable outcomes.	11
Action chunk	A contiguous sequence of actions generated or processed together.	4 , 5 , 6 , 7 , 9

Table 102

Symbol	Meaning	Chapters
$A_{t}^{\mathrm{seq}},\allowbreak\mathbf{A}_{t}^{\mathrm{seq}}$	Action sequence or candidate plan. In Chapter 6 , the explicitly matrix-valued action chunk has shape $H\times d$ , with one $d$ -dimensional action per row.	5 , 6
$A^{\mathrm{adv},\pi}(h,a),\allowbreak\widehat{A}_{t}^{\mathrm{adv}}$	Policy advantage conditioned on a history (or a state in the fully observed case), and its estimate.	2
$\mathbf{A}$	Linear state-transition matrix.	2
$A_{i},\allowbreak\mathcal{A}_{i}$	Taxel area and sensitive region, with $A_{i}=\int_{\mathcal{A}_{i}}\mathrm{d}A$ .	3
$\mathbf{A}_{\mathrm{G}}$	Centroidal momentum matrix mapping generalized velocity to momentum.	7
$a_{t},\allowbreak\mathbf{a}_{t}$	Action. Bold type denotes an explicitly vector-valued action. Superscripts $\mathrm{cmd}$ and $\mathrm{exec}$ distinguish commands and execution.	2 , 4 , 5 , 6 , 7 , 8 , 11

Table 103

Physical AI: Concepts, Techniques, and Challenges

Abstract

Figures & tables

Robotics

Towards Trustworthy Physical AI: From Theory to Practice Across Life Cycle

A Comprehensive Review of Generative Physical Artificial Intelligence

HarnessPAI: An Evolving Harness for Physical AI

Abstract

Robotics

Explore similar work

Towards Trustworthy Physical AI: From Theory to Practice Across Life Cycle

A Comprehensive Review of Generative Physical Artificial Intelligence

HarnessPAI: An Evolving Harness for Physical AI