Envisioning Agentic Robotics: A New Era of Robotic Systems
Abstract
The release of GPT-6 Astra marks the emergence of a new paradigm in robotics: agentic robotics. Within weeks, frontier models with agentic harnesses matched or exceeded task-specific policies on many embodied benchmarks. Instead of training a dedicated policy for each family of tasks, an agentic robotic system applies a foundation model to open-world embodied tasks. It operates in a closed loop: observing the environment, interpreting user intent, reasoning about the task, invoking skills and tools, verifying outcomes, and adapting its strategy. Despite these advances, the community has yet to converge on a shared definition, a reliable evaluation framework, or a research roadmap. We argue that Embodied System 2, realized by a foundation model together with an embodied harness, forms the decision-making core of agentic robotics. Its central challenge is to translate general-purpose reasoning into effective and verifiable actions under physical constraints such as limited reaction time, mechanical limits, unpredictable environment changes, and irreversible consequences. Motivated by this perspective, we propose an evaluation framework that measures both model capabilities and closed-loop system performance. Furthermore, we identify the principles that carry over from digital to physical agents and the challenges that must be addressed to achieve reliable physical autonomy. On top of the Embodied System 2 and the evaluation framework, we envision a research roadmap toward sustained autonomy: the ability to perform work safely and reliably with progressively broader responsibility, from executing individual primitives to fulfilling standing roles.
Figures & tables
| Component | Problem addressed |
|---|---|
| Core components | |
| Context manager | Context from the world, the user, and the body arrives continuously and in large volume; it must be summarized and well managed. |
| Tool interface | Skills and tools must be exposed to the model at a suitable level of abstraction. |
| Planner | A long task cannot be planned once and executed blindly. |
| Memory | State must persist beyond the context window, and be updated. |
| Verifier | A completed call does not show that a step achieved its goal. |
| Ability | Evaluation target | Candidate benchmark |
|---|---|---|
| Multimodal perception and semantics | Recognize objects, attributes, and task-relevant entities from multimodal observations. | RoboBench ( Luo et al., 2026b ) |
| Spatial relations and reference frames | Infer spatial relationships, relative positions, and geometry across viewpoints and frames. | MMSI-Bench ( Yang et al., 2026f ) , VSI-Bench ( Yang et al., 2025b ) |
| Physical states and mechanisms | Identify physical properties, object states, support and contact relationships, and mechanisms relevant to interaction. | PhysBench ( Chow et al., 2025 ) |
| Affordances and interaction feasibility | Identify physically feasible interactions and action regions given the observed scene and available capabilities. | RoboBench ( Luo et al., 2026b ) |
| Events and temporal structure | Identify events, their order, and temporal dependencies from observation sequences. | EgoTaskQA ( Jia et al., 2022 ) |
| Tasks, goals, and stated constraints | Interpret expressed goals, desired states, and task constraints. | Embodied Agent Interface ( Li et al., 2024b ) |
| Ability | Evaluation target | Candidate benchmark |
|---|---|---|
| Intent inference and disambiguation | Infer underspecified goals and decide whether clarification is needed before acting. | Ask-to-Act ( Ramrakhya et al., 2025 ) |
| Causal, predictive, and counterfactual reasoning | Predict the consequences of actions, explain observed outcomes, and compare alternatives under hypothetical changes. | EgoTaskQA ( Jia et al., 2022 ) |
| Hierarchical, long-horizon, and constrained planning | Decompose goals into feasible subgoals, order actions, and satisfy dependencies and constraints. | Embodied Agent Interface ( Li et al., 2024b ) , LADDER ( Petkov et al., 2026 ) |
| Spatial search and exploration planning | Select search locations, exploration strategies, and navigation subgoals to reach a target or reduce uncertainty. | GOAT-Bench ( Khanna et al., 2024 ) |
| Skill selection and orchestration | Select, sequence, and coordinate the available skills and tools according to task dependencies, capability constraints, and expected effects. | RobotWorld ( Yang et al., 2026j ) |
| Executable action inference | Generate executable action targets directly, such as end-effector poses, waypoints, trajectories, joint targets, or skill parameters, that satisfy embodiment, geometric, and environmental constraints. | RoboDojo ( Chen et al., 2026b ; Zhang et al., 2026c ) , Robocurve ( Menon et al., 2026 ) , EmbodiedBench ( Yang et al., 2025c ) |
| Benchmark | Evaluation target | Scale |
|---|---|---|
| CaP-Bench ( Fu et al., 2026 ) | Writing and running programs that compose perception and control primitives, with various abstraction, perception, and execution-feedback settings | 7 core manipulation tasks; 187 tasks in the full CaP-Gym release, from Robosuite, LIBERO-PRO ( Zhou et al., 2025c ) and BEHAVIOR ( Li et al., 2024a ) |
| EmbodiedBench ( Yang et al., 2025c ) | From high-level household planning to low-level navigation and manipulation | 1,128 tasks in four environments |
| EmbodiedSkillBench ( Han et al., 2026 ) | Whether packaged skills help general agents: native control, skill-only execution, and agent-directed reuse | 250 skills; 100 perturbed LIBERO-PRO tasks |
| RobotWorld ( Yang et al., 2026j ) | Robot use across embodiments, with interaction budgets and executable success checks | 84 tasks from 20 projects in five domains |
| Benchmark | Evaluation target | Scale |
|---|---|---|
| MIKASA-Robo ( Cherepanov et al., 2026b ) | Object, spatial, sequential, and capacity memory | 90 tasks in three horizon splits of 25 to 2,160 steps |
| RoboMME ( Dai et al., 2026 ) | Temporal (counting), spatial (permanence), object (reference), and procedural (imitation) memory | 16 tasks 50 test episodes; 1,600 training demonstrations |
| RoboMemArena ( Lei et al., 2026 ) | Multi-object transfer, occlusion, counting, and sequence, in long multi-stage manipulation | 26 simulated and 5 real-world tasks |
| EMem-Bench ( Liang et al., 2026a ) | Passive observation, dynamic tracking, interaction failure, and experience generalization | 2,554 episodes in 1,118 scenes |
| OpenEQA ( Majumdar et al., 2024 ) | Questions about a recorded episode and questions answered by active exploration | More than 1,600 questions from 180 real-world environments |
| Benchmark | Focus | Coverage |
|---|---|---|
| RobotWorld ( Yang et al., 2026j ) | Multi-body execution | Execution on many bodies; cannot isolate morphology, because objectives are not paired across bodies |
| Habitat-MAS ( Chen et al., 2025b ) | Heterogeneous capabilities and planning | Planning with robot capability descriptions; tests cooperation, not transfer |
| OmniNavBench ( Sun et al., 2026c ) | Navigation across bodies | The same instructions on wheeled, quadruped, and humanoid agents; navigation only |
| ZETA ( Yan et al., 2026 ) | Tabletop manipulation transfer | Transfer to held-out target bodies; target-data exposure must be disclosed |
| DexJoCo-X ( Jiang et al., 2026 ) | Dexterous-hand manipulation | Joint training across seven hands; transfer to an unseen hand is left to future work |
| Ability | Candidate benchmark | Coverage |
|---|---|---|
| Risk recognition: identify risks involving objects, environments, and people | IS-Bench ( Lu et al., 2025 ) ; HomeSafeBench ( Yao et al., 2025 ) | Risks that emerge during household tasks; hazards reported during free exploration of a home |
| Consequence prediction : anticipate immediate, delayed, and counterfactual consequences | SafeMindBench ( Chen et al., 2025c ) ; RoboHarm ( Sun et al., 2026a ) ; ASIMOV-2.0-Injury ( Jindal et al., 2025 ) | Factual, causal, and temporal constraints in planning; refusal on real arms; risk type and severity in text scenarios from injury reports, the safety effect of a candidate action, and the severity after it |
| Prudent decisions within capability limits: choose execution, verification, assistance, or refusal | ASIMOV-Agentic ( Gemini Robotics Team, 2026a ) ; SafeAgentBench ( Yin et al., 2024 ) | Decisions under payload and reach limits and under uncertainty; refusal of hazardous tasks in planning |
| Prevention before action : check preconditions and use a safe action order | IS-Bench, mitigation before risky steps ( Lu et al., 2025 ) ; DESPITE ( Zhang et al., 2026b ) | Mitigation performed in the correct order; dangerous plans among valid ones in symbolic planning |
| Monitoring and replanning : observe execution feedback and revise the plan after a deviation | SENTINEL ( Zhan et al., 2025 ) ; RT-Safe ( Xu et al., 2026d ) | Formal checks of plans and trajectories; collisions in a world that keeps running. Timed faults and dynamic disturbances are missing |
| Post-action handling and recovery: remove residual hazards, recover, or terminate safely | IS-Bench, mitigation after risky steps ( Lu et al., 2025 ) | Postconditions; protocols for incidents and recovery from failure are missing |
| Benchmark | S2 foundation model | S1 components | Comparison |
| Competitive with trained policies on narrower simulated suites | |||
| RoboDojo-Sim ( Chen et al., 2026b ; Zhang et al., 2026c ) | GPT-6 Astra | None | 22.48%; above all 40 public policies at the time |
| RoboDojo-Sim, official evaluation ( Gao et al., 2026b ) | GPT-6 Astra | Agent-written tools, mostly task-specific | 46.9%; first on the leaderboard when posted |
| RoboLab-120 ( Yang et al., 2026c ; Yang et al., 2026d ) | GPT-6 Astra, direct / agentic session | None | 66.3% / 74.8%; best leaderboard policy: 51.0% |
| LIBERO-PRO ( Zhou et al., 2025c ; Si et al., 2026 ) | GPT-6 Astra | Motion primitives, segmentation, and , composed in code | 94.0%; : 53.0% (different evaluation setup) |
| RoboMME ( Dai et al., 2026 ; Jedlicki et al., 2026 ) | GPT-6 Astra | None | 92.1%; best leaderboard entry: 60.8% |
| Benchmark | Spatial scope | Time span |
| LIBERO ( Liu et al., 2023b ) | Workcell | Single episodes |
| LIBERO-PRO ( Zhou et al., 2025c ) | Workcell | Single episodes |
| CALVIN ( Mees et al., 2022 ) | Workcell | Single episodes a |
| RoboDojo ( Chen et al., 2026b ) | Workcell | Single episodes b |
| RoboTwin 2.0 ( Chen et al., 2025d ) | Workcell (dual-arm) | Single episodes |
| RoboLab ( Yang et al., 2026c ) | Workcell | Single episodes |
| Coding agent | Physical agent | |
| Environment | A designed artifact: files, code, APIs | Objects, people, and spaces |
| Native interface | ||
| Input and state | Directly readable files, logs, and tool outputs | Partial, noisy, occluded, and delayed sensing; contact and force are invisible to cameras; state must be constructed |
| Action interface | Tool calls, file edits, commands, tests | Skills, trajectories, and actuators |
| Verifiable outcomes | ||
| Verification | Fast, reproducible automated tests | Costly, slow, hard to reproduce |
| Scope | Extent | What it adds for S2 |
|---|---|---|
| Workcell | Within reach and view | Grounding objects and actions in view; contact and precision |
| Room | Within view, but the base must move | Where to place the base; reachability that changes as the robot moves |
| Building | Beyond what the robot can observe at once | Maps and memory of places and objects; search and exploration |
| Site | Several buildings and outdoor routes | Long routes; other people and robots; handover between areas |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Work | What it says | What we adopt | What we add or change |
|---|---|---|---|
| RoboBench ( Luo et al., 2026b ) | “We refer to System 2 as the embodied brain, the cognitive core for decision-making in manipulation.” | S2 as the decision-making core | Embodied S2 includes the harness, not only a multimodal language model; criterion for when a component acts as S2 |
| From WAMs to Embodied Brains ( Liang et al., 2026b ) | The embodied brain is “a model-level target”, while “the harness manages memory, routing, logging, verification, and capability calls”. | Separation of model and harness | Harness counted inside Embodied S2; S1 and S0 are tools that S2 calls |
| Thea ( Wang et al., 2026e ) | “The world, however, withholds two abilities that software grants for free: reading the state of the world, and judging the outcome of an action.” | World, user, and body state built and maintained by the harness | User context as a separate category; the harness, including its outcome judgments, counted as part of Embodied S2 |
| SCOPER ( Wei et al., 2026a ) | “An embodied brain, or System 2, must turn an abstract goal into actions while discovering relevant scene conditions and checking whether executed skills achieved their intended effects.” | Choosing the next skill and judging the last one | Harness treated as part of Embodied S2 rather than an add-on; persistence across episodes |
| Embodied agentic harnesses survey ( Huang et al., 2026c ) | Harness as the “runtime control plane” that grounds observations, coordinates capabilities, monitors outcomes, and manages intervention. | Harness responsibilities and contracts | Each ability divided between the model and the harness, each evaluated with the other held fixed |
| Agentic robotics survey ( Wang et al., 2026b ) | “The distinguishing feature is this decision-making role of the FM, rather than any particular model or interface.” (FM: foundation model) | A decision-based criterion | Adjudication of vendor “S2” labels; two evaluation levels; ladder of delegable work |
| Domain | Earlier approaches and where they stop | What Embodied S2 could add | Reported reach and evidence; level needed |
|---|---|---|---|
| Home | Single-purpose appliances for fixed chores; learned policies complete single tasks in unseen homes; remote experts bridge longer chores | Interpreting open requests; sequencing chores; verification and recovery; user and home memory; interruption; privacy | Task; company-run evaluation in unseen homes; remote experts disclosed for some products. Needed: L3 to L4 |
| Manufacturing | Programmed cells for fixed products, with every change an engineering project; humanoid pilots on short tasks | Per-cycle verification; exception handling; re-tasking from work instructions | Primitive to task, within shifts of up to ten hours; company reports; intervention counts not disclosed. Needed: L3 to L4 |
| Logistics | Large fleets in structured warehouses; items that the picking robot cannot handle go to people | Exception handling; fleet orchestration | Primitive level per robot; fleet operation in structured warehouses; company reports; human takeover disclosed for picking. Needed: L3 to L4 |
| Inspection | Preplanned routes; recorded data interpreted by people | Verified reading of physical state; follow-up on anomalies; persistent site model | Task to workflow per inspection round, in structured sites; company reports. Needed: L4 |
| Healthcare and eldercare | Delivery between fixed points; eldercare only in trials | Varied requests; timing of intervention; long-term memory; safety | Task for hospital delivery (company report); eldercare only in trials. Needed: L4 for eldercare |
| Laboratories | Fixed protocols on automated equipment; autonomous runs in restricted domains | Choosing the next experiment; outcome verification; provenance | Sustained operation for 8 to 17 days in structured laboratories (peer-reviewed). Needed: L4 |