Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while the embodied layers on which deployment actually turns remain fragmented across benchmark-specific simulators, embodiments, and interfaces. The first DeepInsight report (v1) unified evaluation across this stack behind three abstractions---task, resource, and result---but its quantitative evidence centered on the foundation-model layer; navigation and manipulation (System 1) and whole-body control (System 0) remained simulation case studies, and physical execution was outside its empirical scope. DeepInsight II keeps that substrate fixed and quantifies the embodied half. First, it reproduces released-checkpoint references across two navigation and four manipulation benchmarks under their native protocols. Second, MotionBench places four released whole-body controllers under one workload and metric contract, then carries a qualified within-family cohort from parallel simulation to matched real-robot trials in which simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific records, making the sim-to-real gap a native reduction rather than a reconciliation across toolchains. Third, a composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels, each mapped to a concrete repair action, with a measured repairability criterion and physical episodes testing the same attribution under hardware-observable state. The contribution is therefore not a new evaluation architecture, but empirical continuity from benchmark execution to matched robot evidence and repair-oriented diagnosis.
Figures & tables
Figure 1: The embodied evidence of this report, on one runtime: System 1 checkpoints under native benchmark protocols, a System 0 motion matched between simulation and the physical robot, and a composed System 2–1–0 episode diagnosed down to a concrete repair and re-tested in simulation and on the physical robot. Representative H2 frames show the state before and after the orientation repair. Every rollout writes into one identity-addressed trace, so all three kinds of evidence share a single result representation.
Dimension
DeepInsight v1 established
DeepInsight II adds
Core substrate
Task, resource, and result: one reset / step driver, one acquire / release protocol, one identity-addressed trace.
No new abstraction ; embodied benchmarks and a physical robot bind to the same substrate.
System 2
Quantitative fidelity, throughput, ablations, and multi-node scaling.
Inherited; not re-benchmarked here.
System 1
Simulation-only coverage case studies.
Native-protocol fidelity: released checkpoints on two navigation and four manipulation benchmarks.
System 0
Trace-backed screening of internal policies, in simulation only.
MotionBench: four released WBC models under one frozen contract.
Simulation to hardware
Outside the empirical scope.
Matched sim and real-robot trials under one parent trace identity, with domain-specific records.
Full-system diagnosis
Task-, subgoal-, and subsystem-level failure localization, in simulation.
Five handoff labels, each mapped to a repair; measured repairability; tested on physical episodes.
Table 1: Scope delta from DeepInsight v1 [ 6 ] to DeepInsight II. The core substrate is inherited unchanged; DeepInsight II is an empirical and execution-domain extension of v1, not a replacement of its architecture.
Figure 2: The embodied evaluation substrate. Task contracts define the episode and benchmark semantics; simulators, policies under test, and the real robot attach through resource bindings; and every rollout writes into an identity-addressed result trace. Matched simulated and physical episodes share a parent trace identity while retaining execution-domain-specific records.
Domain
Benchmark
Simulator
Models (released checkpoints)
Reported metrics
Navigation
R2R-CE
Habitat-Sim
StreamVLN, RynnBrain-Nav, JanusVLN
SR, SPL, OSR, NE
RxR-CE
Habitat-Sim
StreamVLN, RynnBrain-Nav, JanusVLN
NE, SR, SPL, nDTW
Manipulation
LIBERO
MuJoCo/robosuite
π0 , π0.5 , π0 -FAST, OpenVLA, OpenVLA-OFT
per-suite SR
LIBERO-Plus
MuJoCo/robosuite
same checkpoints as LIBERO
per-perturbation SR
SimplerEnv (WidowX/Bridge)
SAPIEN/ManiSkill
StarVLA (3 heads)
SR
RoboCasa365
MuJoCo/robosuite
Diffusion Policy, π0 , π0.5 , GR00T N1.5
split-level SR
Table 2: System 1 coverage: six benchmarks, two domains, one substrate. Models are released checkpoints; reported metrics combine benchmark scores and trace-derived execution diagnostics and are reported in Table 3 –Table 7 .
Figure 3: Example R2R-CE episode from the unified navigation evaluation. Six semantic keyframes show the egocentric monocular observations used for policy inference alongside the trace-derived trajectory visualization, instruction, and terminal metrics. The top-down map is used only for visualization and is not exposed to the policy.
Model
SR ↑
SPL ↑
OSR ↑
NE ↓
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
StreamVLN [ 31 ]
56.40
55.85
50.20
49.97
63.60
63.13
4.90
5.08
RynnBrain-Nav [ 32 ]
58.60
57.31
49.60
48.01
71.60
69.55
4.92
5.02
JanusVLN [ 33 ]
60.50
57.26
56.80
52.99
65.20
62.91
4.78
4.92
Table 3: Navigation results on val-unseen splits. All runs use egocentric monocular input, and all DI values use a maximum of 500 steps. For each metric, Ref. (gray) is the value reported by the corresponding source publication and DI is our unified execution. Bounded metrics are percentages; NE is in meters.
Model
Spatial ↑
Object ↑
Goal ↑
LIBERO-10 ↑
Avg. ↑
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
π0
96.80
97.80
98.80
97.40
95.80
92.80
85.20
81.40
94.15
92.35
π0.5
98.80
98.80
98.20
98.40
98.00
97.20
92.40
91.80
96.85
96.55
π0 -FAST
96.40
97.00
96.80
97.20
88.60
87.20
60.20
62.20
85.50
85.90
OpenVLA
84.70
86.20
88.40
86.80
79.20
77.80
53.70
54.00
76.50
76.20
OpenVLA-OFT (comb.)
96.80
98.00
97.00
97.80
95.40
96.40
98.00
95.40
96.80
96.90
Table 4: LIBERO success rates across the four standard suites. For each suite, Ref. (gray) is the published value and DI is our execution.
Model
Camera
Robot init.
Lang.
Light
Backgr.
Noise
Layout
Total ↑
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
π0
13.8
17.0
6.0
7.5
58.8
64.9
85.0
86.9
81.4
83.3
79.0 †
18.7
68.9
73.3
53.6
46.8
π0.5
75.4
71.9
77.5
74.0
85.6
85.6
96.9
96.7
94.6
95.0
89.7
86.8
85.7
86.7
85.7
84.3
π0 -FAST
65.1
63.1
21.6
22.8
61.0
69.5
73.2
75.6
73.2
74.5
74.4
72.0
68.8
73.8
61.6
63.6
OpenVLA
0.8
0.5
3.5
3.7
23.0
37.5
8.1
13.7
34.8
36.3
15.2
5.9
28.5
42.2
15.6
19.2
OpenVLA-OFT (comb.)
55.6
55.9
21.7
24.1
81.0
85.2
92.7
93.3
91.0
87.4
78.6
73.8
68.7
74.0
67.9
68.7
Table 5: LIBERO-Plus success rates by perturbation family. Total is episode-weighted; Ref. (gray) is the published value and DI is our execution. † : the published π0 Sensor-Noise reference is an unresolved benchmark-reference issue (see text).
Model
Per-task success (DI)
Avg. ↑
Stack cube
Carrot/Plate
Spoon/Towel
Eggplant/Basket
Ref.
DI
StarVLA-FAST
41.7
45.8
75.0
87.5
58.6
62.5
StarVLA- π
34.4
69.8
82.3
76.0
62.5
65.6
StarVLA-GR00T
40.6
55.2
81.3
77.1
63.6
63.5
Table 6: SimplerEnv WidowX/Bridge Visual-Matching success rates. The model zoo publishes only the four-task average, so per-task values are DI-only; for the average, Ref. (gray) is the published value and DI is our execution.
Model
Atomic seen ↑
Comp. seen ↑
Comp. unseen ↑
Overall ↑
Ref.
DI
Ref.
DI
Ref.
DI
Ref.
DI
Diffusion Policy
15.70
16.56
0.20
0.13
1.25
0.13
6.10
6.04
π0
34.60
34.67
6.10
5.88
1.10
1.63
14.80
14.88
π0.5
39.60
41.67
7.10
6.13
1.20
2.13
16.90
17.64
GR00T N1.5
50.70
49.00
14.80
16.25
2.70
4.13
23.90
24.16
Table 7: RoboCasa365 multi-task-learning success rates by task split. For each split, Ref. (gray) is the official leaderboard value and DI is our execution.
Figure 4: Representative successful closed-loop rollouts from LIBERO, LIBERO-Plus, RoboCasa365, and SimplerEnv (top to bottom). Each row identifies the evaluated checkpoint and task and shows the reset observation ( t=0 ), two intermediate observations, and the final sampled observation at the indicated saved-rollout frame indices. Intervening observation–action steps are omitted; all four episodes satisfy their benchmark’s native success predicate.
Figure 5: System 0 evaluation overview. MotionBench fixes the versioned workload and measurement semantics; heterogeneous WBC models are compared in a common simulation space; and trace-qualified WBC–motion pairs enter matched real-robot trials for simulation–reality analysis. SR is used only in simulation; hardware correspondence uses matched root-relative and joint-space measurements.
Robot-profile retargeting, quality assessment, manual annotation, and difficulty/type screening
Controlled motion workload
Coverage
Upper-body, locomotion, in-place, dynamic, and low-posture motion across multiple durations
Capability and duration decomposition
WBC binding
Versioned checkpoint and declared WBC interface under the common evaluation contract
Comparable execution across WBC models
Core outputs
SR, MPJPE-l, and progress-soft
Executability, local fidelity, and usable completion
Table 8: MotionBench specification. The benchmark fixes the workload and evaluation semantics before model-specific execution.
Figure 6: Simulation SR–MPJPE-l plane on MotionBench. Each point is one released WBC model evaluated under the same task and metric contract.
WBC
Architecture
SR ↑
Progress-soft ↑
MPJPE-l (cm) ↓
HoloMotion 1.3.2
Sparse-MoE causal Transformer
96.7
93.1
4.53
HoloMotion 1.4.0
Sparse-MoE causal Transformer
91.9
77.9
5.15
SONIC release
MLP encoder–decoder + FSQ
90.8
72.7
5.65
SONIC low-latency
MLP encoder–decoder + FSQ
89.2
67.4
5.73
Table 9: MotionBench simulation results. SR and progress-soft are percentages; MPJPE-l is the macro average over motions.
WBC
Upper body
Locomotion
In-place
Dynamic
Low posture
HoloMotion 1.3.2
97.7
95.1
91.4
90.4
82.5
HoloMotion 1.4.0
95.2
71.0
83.1
54.3
68.0
SONIC release
93.8
63.4
74.7
55.7
55.6
SONIC low-latency
93.8
52.5
70.5
48.4
50.8
Table 10: MotionBench progress-soft (%) by capability. The decomposition localizes losses in usable tracking behind aggregate SR.
Metric
SONIC release
SONIC low-latency
Diagnostic role
Sim.
Real
Gap (%)
Sim.
Real
Gap (%)
MPJPE-l (cm)
3.08
3.41
+10.83
3.28
3.89
+18.65
Root-local whole-body tracking
Joint-position RMSE (rad)
0.13
0.16
+22.15
0.15
0.18
+22.53
Joint pose tracking and steady bias
Joint-velocity RMSE (rad/s)
0.74
0.75
+1.58
0.80
0.88
+10.46
Dynamic tracking, latency, and damping
Joint-acceleration RMS (rad/s 2 )
18.40
14.16
− 23.05
22.32
20.39
− 8.66
Sustained high-frequency activity
Joint-jerk RMS (rad/s 3 )
1181.51
941.36
− 20.33
1442.24
1375.07
− 4.66
Sustained command discontinuity
Table 11: Matched simulation–real kinematic and dynamic profile for the qualified SONIC cohort over 25 matched motion clips. For each variant, Sim. (gray) and Real give the paired aggregates and Gap is (Real−Sim)/Sim×100% . Hardware values average three trials per WBC–motion pair before aggregation across motions.
Evidence pattern
Label
Repair
Source reports success; its verified postcondition fails
Termination error (TE)
change the termination criterion
Source postcondition holds; a verified target precondition fails (readiness error if the target also signaled ready )
Handoff-condition violation (HCV)
add the missing precondition or alignment criterion
Both modules report success and all registered conditions pass; the composed step still fails
Contract gap (CG)
register the omitted condition, unless the geometry check shows the entry region is execution-limited
Entry handoff verified, preconditions hold throughout; failure inside the skill’s execution interval
Skill-internal execution (SIE)
improve the invoked skill; interface untouched
A condition that held at entry lapses mid-execution (user walks away, occlusion appears)
Execution-time invariant lapse (EIL)
promote the condition to continuous monitoring
Table 12: The five attribution labels: each verified evidence pattern maps to one repair action. The first three apply at the handoff boundary, the last two inside the downstream skill’s execution; abbreviations are used in Table 14 .
Figure 7: The two causes at the end of navigation. At H2 the heading error is never registered; at H5 the required start pose is declared but sits 150∘ away, in a band one step wide. Angles and offsets are representative; band widths and the step are measured (Table 13 ).
Handoff
Axis
Region (m)
Step (m)
Steps wide
Ready at budget
0
1
5
H2 navigation → greeting
approach
0.40
0.20
2.0
12/20
16/20
17/20
H5 navigation → presentation
approach
0.30
0.20
1.5
6/15
11/15
13/15
Table 13: Repairability criterion and readiness under a local adjustment budget. The entry region (rollout success surface) and the minimum step (actuator) are measured, and hence so is their ratio; Ready counts episodes whose handoff state lands inside the calibrated entry region, not those whose registered preconditions merely pass—at a contract gap the two differ.
Handoff
Reached
Failed
Both OK
Attributed label
TE
HCV
CG
SIE
EIL
Abst
H1 retrieval → nav. (person)
20
—
—
—
—
—
—
—
—
H2 nav. → greeting
20
4
2
1
3
—
—
—
—
H3 greeting → request
16
1
0
—
—
—
—
1
—
H4 resolution → nav. (viewpoint)
15
—
—
—
—
—
—
—
—
H5 nav. → presentation
15
4
3
—
—
3
1
—
—
Table 14: Per-handoff breakdown of the composed episodes: episodes arriving at each handoff (Reached), breaking there (Failed), and failures with both adjacent modules self-reporting success (Both OK). Label abbreviations follow Table 12 ; Abst is abstention; dashes denote zero.
Figure 8: H2 on hardware at the end of navigation. The dashed line marks the heading the greeting requires. Frames are representative; the entry region and step of Table 13 are calibrated in simulation, not re-measured on hardware.
Evaluating a Physical AI stack spans operators that differ by more than three orders of magnitude -- from a single foundation-model decoding step to thousands of physics ticks of whole-body control -- varying orthogonally in modality, reward semantics, and resource profile. No existing framework spans this range, so the stack is evaluated today by stitching together separate harnesses that share neither runtime nor scoring, preserving each segment's local validity but losing the shared identity needed to diagnose cross-layer regressions. We present DeepInsight, an evaluation infrastructure that serves this full spectrum on a single runtime. Rather than homogenize the regimes, it preserves their heterogeneity behind three narrow abstractions -- task, resource, and result -- each realized as one invariant shared by every subsystem: one episode driver, one resource-handle protocol implemented by every expensive backend (LLM inference and sandboxed runtimes alike), and one trace identity scheme under which every event is written. Deployed in production across all three layers of an embodied humanoid stack, this single set of invariants onboards new benchmarks largely by configuration. Where mature peer orchestrators exist -- at the foundation-model end -- it reproduces published references and peer-framework readings within their own spread, runs the same suites faster on a single node, and scales near-linearly across nodes. Its distinctive return is diagnostic: because every layer writes into one shared trace, a regression that begins in one layer and surfaces in another stays localizable on that trace -- a cross-layer payoff no federation of per-segment harnesses can reproduce.
General purpose language models are increasingly able to control robotic hardware. Understanding the capabilities and safety of these models when embodied is therefore increasingly important for understanding their societal impact and risks. To this end, we introduce Inspect Robots, a modular, open-source framework for developing and running evaluations of embodied agents. Inspect Robots pairs customizable, reusable abstractions for specifying physical evaluations and analyzing their results with infrastructure that automates evaluation setup, execution and termination. We demonstrate Inspect Robots by using it to evaluate the capabilities and safety of six policies based on frontier language models. Inspect Robots has seen significant early uptake, receiving nearly 100,000 downloads in the three months since its release.
Christopher Leet, Achu Menon, Sravanthi Machcha +12
Robocurve · University of Southern California · San Jose State University +7
Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.