When the World Is the Test Suite: Agentic Intelligence for the Physical World
Organizations: National University of Singapore · Hong Kong Baptist University · University of Waterloo · The Australian National University ·
Abstract
Coding agents built to repair software now write, test, and repair programs for robots, laboratories, and industrial controllers. Unlike in software engineering, no test suite comes with the code. Whether a program works must be established by running it on a robot, in a physics simulator, or through the policies it helps to train, and such evidence is costly to obtain and can itself be wrong. As these agents take on more physical tasks, how their programs are judged matters as much as how they are written. This paper examines this emerging area, agentic intelligence for the physical world, by treating the world as the program's test suite. We organize the discussion around three questions: how agents write programs for the physical world, how these programs are tested, and how agents learn from the results. We first trace how language models for robots and software engineering agents came together, ask which software testing practices carry over to physical tests, and describe each program by where it sits relative to the actuator and by what judges it. For writing, we cover programs that act as robot policies and programs that support policy learning by producing rewards, tasks, and training worlds. For testing, we examine what physical trials cost, what traces, learned judges, simulators, and formal checks can establish in their place, and what safety requires beyond task success. For learning, we discuss how tested programs are retained as experience, in skill libraries and in model weights. Finally, we turn to benchmarks and evaluation practice and outline open challenges. Writing programs is becoming cheap, while trustworthy evidence about what they do remains costly. Progress will therefore depend on a physical counterpart of continuous integration: verification cheap enough to run often, reliable enough to trust under search, and maintained alongside the experience it validates.
Figures & tables
| Review | Physical world | Programs as the object | Judges analyzed | Lab and industrial code | Transfer from software |
| Foundation models and VLAs in robotics ( Zeng et al., 2023 ; Hu et al., 2023 ; Firoozi et al., 2024 ; Yao et al., 2026a ; Kawaharazuka et al., 2024 ; Ma et al., 2026b ; Zhong et al., 2025 ) | |||||
| Code generation and software-engineering agents ( Jiang et al., 2026b ; Liu et al., 2026d ) | |||||
| Code as the medium of agents ( Yang et al., 2024c ; Ning et al., 2026 ) | |||||
| Weights or Skills? ( Jena et al., 2026 ) | |||||
| No Free Checker ( Wan et al., 2026 ) | |||||
| Safety of language-model agents ( Dantas et al., 2026b ) |
| In software | Physical counterpart | Examples |
| Agent harness | Harness that sets the robot interface, the feedback, and access to privileged state | ASPIRE, RHO, OpenRUA ( Lu et al., 2026 ; Elmaaroufi et al., 2026 ; Chu et al., 2026 ) |
| Stack trace and logs | Per-primitive traces with keyframes and overlays | ASPIRE ( Lu et al., 2026 ) |
| Repository and refactoring | Multi-file policy repositories and rebinding skills to a new body | RHO, RECENT ( Elmaaroufi et al., 2026 ; Choi et al., 2026b ) |
| Executable behavioral tests | Symbolic trace properties and controller test suites in simulation | RoboEval, test-driven framework ( Hu et al., 2024 ; Tripathi et al., 2026 ) |
| Regression gate | Practice suites, validation against the current version, and regression tests before accepting a skill or harness edit | RPG, Skill2Real, HexaAnything ( Wang et al., 2026o ; He et al., 2026 ; Gao et al., 2026b ) |
| Contracts | Skill schemas, allowed operators, and encoded safety constraints | Contract-grounded BTs, SafeGate ( Salfity et al., 2026 ; Obi et al., 2026 ) |
| Position | The program is | Representative work | Typical evaluation | Evaluation cost |
| ACT Action | A policy over perception, planning, and control primitives | Code as Policies , ProgPrompt , VoxPoser , CaP-X , ASPIRE , RHO | Privileged predicates, learned judges, outcome checks | One episode, seconds to minutes |
| ORCH Orchestration | Logic that sequences learned policies and checks their results | HyMeS , Zetta , Harness VLA ∗ , HarnessPAI , Pigey ∗ | Learned judges, generated checks, outcome checks | One episode, plus calls to the learned policy |
| OBJ Objective | A reward, cost, or temporal-logic specification | Language to Rewards , Eureka , Text2Reward , ReKep , BIRD | Checks of controller or policy behavior, human ratings | Controller rollouts or policy training, then evaluation |
| WORLD World | Scenes, tasks, curricula, demonstrations, robot bodies | GenSim , RoboGen , Eurekaverse , RoboTwin 2.0 , DrEureka , Articulate-Anything | Solvability checks, learned judges, transfer of trained policies | Scene checks or trial episodes, plus training and transfer tests for downstream claims |
| Setting | Program or hybrid | End-to-end policy |
| Change in task structure Programs | ||
| LIBERO-PRO, file-name instruction ( Zhou et al., 2025b ) | – | OpenVLA, , 0.00–0.01 |
| Matching instruction ( Wang et al., 2026e ) | – | 11.2–60.6 |
| Task split over three suites ( Lu et al., 2026 ) | CaP-Agent0 0.16, below matching ASPIRE 0.67 | – |
| Suite averages ( Elmaaroufi et al., 2026 ) | RHO 0.37–0.62 | – |
| Task split, same model for both, demonstrations of the original tasks only ( Kang et al., 2026b ) | Encore 0.96 ASPIRE rerun 0.89 | – |
| Judge or source | Inputs and cost | Limitation or failure | Reported measurement |
| Simulator-state predicate | Simulator state Very low | Bugs, lenient thresholds, mismatch with the instruction, readable by the agent | 22 bugs and 4 design limitations in 7 benchmarks, including permissive success checkers ( Chen et al., 2026d ) 98 of 413 failures of one agent traced to criteria the instruction left unstated or contradicted ( Chu et al., 2026 ) |
| Model-written check | Runtime values, states, or observations Low per check | Shares mistaken task assumptions with the generator | Generated test cases for control function blocks often contain wrong assertions ( Koziolek et al., 2024 ) |
| Learned judge VLM, reward model | Camera images or video Low to medium | Can miss contact failures, and optimization can amplify its errors | Best of 13 VLMs: 0.77 macro-averaged balanced accuracy on FailBench ( Navasardyan et al., 2026 ) Optimizing a policy against a reward model adds 10.9 points of wrong-object failures ( Luo et al., 2026a ) |
| Formal check | Specification and system model Varies | Incorrect specification or state/action assumptions | Planetarium, GPT-4o zero-shot: 94.4% of generated PDDL problems solvable, 24.8% semantically correct ( Zuo et al., 2025 ) |
| Human | Scene observations or video Review time | Judgments vary across people, and throughput is limited | Three users passing 80%, 63%, and 41% of 30 generated simulated tasks ( Wang et al., 2024b ) |
| Physical trial Observation source | Observations from hardware Trial and reset time | Limited coverage, outcome still needs a judge, safety constraints | About 850 autonomous episodes in 24 hours with three interventions ( Zhou et al., 2025c ) |
| Approach | What it checks | Examples |
| Live controller runtime | Controller execution without a connected physical plant | SemaPLC |
| Plant simulation or digital twin | Closed-loop behavior against a model of the plant | PLCWorld , PRISM , EnvTrace , ABB RAPID programs in the vendor simulator |
| Simulation-gated authorization | Release only after a simulation check within per-device bounds or operator approval | Trapped-ion ARTIQ agent |
| Formal verification and rule-based gates | Conformance to encoded requirements and model assumptions, or to fixed release rules | LLM4PLC , Agents4PLC , external acceptance layer for commissioning |
| Compilation and syntax | That the program is well formed | Vendor-aware PLC generation , verifier errors for a chemistry language |
| Learned or imagined execution | Outcomes predicted without running the program | Language model as a static simulator , code world model trained on execution traces |
| Benchmark | Judge | Judge access | Judge error | Cost | Uncertainty | Progress | HW |
| CaP-Bench (CaP-X) ( Fu et al., 2026 ) | Simulator success (its S1 API exposes privileged state) | NR | NR | Time per tier (appendix) | NR | average dense reward | Franka, 25 trials per task |
| CodeActionBench ( Lyu et al., 2026 ) | Hidden verifier of physical outcomes | NR | API-equivalent dollars, median wall time | Three attempts per fixed scene | checkpoint coverage (diagnostic) | ||
| RoboEval ( Hu et al., 2024 ) | Temporal logic over symbolic traces | NR | NR | NR | Ranges over paraphrases | ||
| RoboTwin 2.0 code generation ( Chen et al., 2026f ) | Simulator success over ten runs per program, and a VLM observer for repair feedback | as feedback | observer only, accuracy 0.43 on 130 labeled runs | Iterations, code length, VLM tokens | NR | NR | trained policies |
| VLABench ( Zhang et al., 2025d ) | Simulator progress, skill and parameter recall | NR | NR | NR | NR | progress score | |
| ResponsibleRobotBench ( Zhang et al., 2025c ) | Simulator success and safety violations | NR | NR | Weighted execution cost, no tokens or dollars | Wilson 95% intervals (watering tasks) |
| Open problem | Findings so far | First experiment |
| Physical continuous integration | Existing trace-and-library loops, judge audits, and simulator studies motivate joint evaluation of reliability and cost ( Lu et al., 2026 ; Navasardyan et al., 2026 ; Verma et al., 2026 ) | Compare verification and acceptance procedures at matched search budgets, then audit accepted episodes and measure held-out success, retention, and cost |
| Judges robust to search | Optimizing against a learned reward model can amplify wrong-object failures ( Luo et al., 2026a ) | Audit accepted programs from matched searches with and without verifier feedback, using independent outcome checks |
| Code versus VLA on all splits | The code–VLA comparisons in Figure 7 cover only LIBERO-PRO position and task splits | Run program-based methods on the object, paraphrase, and environment splits with re-run VLA baselines |
| Staleness and retention | Library growth gives non-monotonic per-task transfer. SEAR re-runs frozen code policies under single hidden hardware faults, and Repo2Skill-Evo measures staleness for software skills, but no robot study we reviewed re-tests the entries of a stored skill library after a controlled one-factor change ( Lu et al., 2026 ; Wang et al., 2026n ; Duan et al., 2026 ) | Hold the library fixed and change one world or API factor before adaptation, then measure surviving success separately from retention after library updates |
| Interface for orchestration | Most harnesses adopt one interface. ALRM reports model-dependent preferences in action-sequence quality, judged without physical execution. PyRUA-Lean favors code cells at equal call budgets in simulation, with one run per instance ( dos Santos et al., 2026 ; Si et al., 2026 ) | Compare code and typed calls on the same composition tasks and budgets, and repeat runs to measure uncertainty |
| Postconditions for learned skills | VLA skills competent from clean starts stall when chained (pilot traces) ( Rui et al., 2026 ) | Learn or check postconditions per skill, then measure chained success |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| System | Artifact / interface | Role | Evaluation source | HW |
|---|---|---|---|---|
| Action programs | ||||
| Code as Policies ( Liang et al., 2023 ) | Python over perception and control functions | ACT | Simulator success and real demonstrations | |
| ProgPrompt ( Singh et al., 2023 ) | Python-style plans with assertions | ACT | Simulator state graph, and per-task real-arm outcomes the authors present as qualitative | |
| VoxPoser ( Huang et al., 2023b ) | Code composing 3D value maps for a planner | ACT | Simulator and real outcomes | |
| CodeBotler / RoboEval ( Hu et al., 2024 ) | Programs in a Python-embedded language | ACT | Temporal logic over symbolic traces | |
| Gemini Robotics-ER ( Gemini Robotics Team, 2025a ) | Python over a robot API | ACT | Task success in simulation and on ALOHA 2 | |