Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena

Haojian Huang; Pukun Zhao; Zexi Li; Yehang Zhang; Yangkai Wei; Wenqian Li; Han Yang; Kaiwen Zhou

cs.ROOct 1, 2026

Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena

Authors: Haojian Huang, Pukun Zhao, Zexi Li, Yehang Zhang, Yangkai Wei, Wenqian Li, Han Yang, Kaiwen Zhou, +2 more

Organizations: HKUST (Guangzhou) · Knowin AI · The Chinese University of Hong Kong

Abstract

Frontier vision-language models (VLMs) combine scene estimation, interaction grounding, and executable actions. Understanding how these abilities support complete robotic tasks is central to evaluating their readiness as robot generalists. We introduce Embodied Agent Arena to examine where local competence supports, or falls short of, complete task success across Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation. The arena contains 1,000 cases drawn from 32 established sources and GeoProbe, our new benchmark for geometric estimation on Blender renders and real-scene images. A minimal harness preserves source observations and operations while separating metric precision, functional grounding, and native goal completion. We evaluate seven VLMs, analyze Astra's task-specific advantages, and compare richer-observation execution protocols and multi-round review. Across the arena, Astra's advantage is strongest in precise estimation and usable-contact localization; completing coordinated, goal-directed actions remains the key gap to robot generalism.

Figures & tables

Figure 1: Overview of Embodied Agent Arena. Each row illustrates six tasks in one capability domain, spanning camera and scene estimation, spatial relations, functional grounding, goal-directed planning, and robot manipulation. Fixed-input probes test estimates and judgments; interactive episodes test goal completion. Spatial-tracing overlays show reference paths and a 3D target region.

	Evaluation coverage					Diagnostic analysis
Study / benchmark	Fixed probes	Region probes	Online nav.	Robot manip.	State memory	Calib./ pose	Task forms
BEAR ( 2025 )	✓	✓	✗	✓ ∗	✗	✗	✓
EmbodiedBench ( 2025a )	✗	✗	✓	✓	✗	✗	✓
EmbodiedEval ( 2025 )	✗	✗	✓	✗	✗	✗	✓
Embodied Arena ( 2025 )	✓	✓	✓	✗	✗	✗	✓
Code-based control ( 2026 )	✗	✗	✓	✓	✗	✗	✓

Table 1: Evaluation coverage and diagnostic analysis. Reported evaluations in the cited papers. Region probes score image grounding; robot manipulation evaluates arm/gripper control; memory spans sessions. Calib./pose tests camera calibration or relative pose; task forms denote type/condition breakdowns.

Figure 2: Construction of the 1,000-case arena. (a) Representative task vocabulary across sources. (b) Five-domain composition, including 112 spatial-tracing cases in Geometry. (c) GeoProbe estimation targets. (d) Source queries, scenes, and targets connect model interaction to termination and scoring. Colors identify domains; logos denote source institutions.

	Geometry		Spatial Reasoning		Affordance		Task Planning	Manipulation
Agent	Rot. $\downarrow$	Trans. $\downarrow$	Pass@1 $\uparrow$	AbsRel $\downarrow$	Mask IoU $\uparrow$	Point $\uparrow$	Pass@1 $\uparrow$	Task SR $\uparrow$
	( ∘ )	(cm)	(%)			(%)	(%)	(%)
Astra	17.1	106.6	$\mathbf{69.4}\!\pm\!1.0$	0.213	0.286	$\mathbf{60.3}\!\pm\!2.6$	80.3	41.8
Sol	27.0	151.9	$60.7\!\pm\!2.5$	0.273	0.264	$52.0\!\pm\!3.6$	44.6	19.8
Fable	25.2	121.7	$62.0\!\pm\!4.6$	0.186	0.284	$57.1\!\pm\!2.5$	63.7	24.7
Qwen-Max	20.5	138.1	$68.9\!\pm\!1.1$	0.205	0.272	$52.9\!\pm\!4.2$	35.7	9.3

Table 2: Performance across five capability domains. Repeated rates show mean

\pm

SD; continuous errors use valid observations. Appendices B – C give full metrics.

Figure 3: Capability breadth across seven agents. (a) Astra (red) and peers (gray). (b) Mean of five domain indices. (c) Indices by domain, derived from Table 2 using the fixed scales in Appendix C.3 . Manipulation uses baseline results; individual profiles appear in Figure A1 .

Figure 4: Geometric precision and task completion. (a) Error reduction relative to each target’s lowest-error peer; the first five targets are GeoProbe. (b) Displacement under fixed, small-shift, and large-shift Blender cameras, plus separate real photos; points show mean

\pm

run SD. (c) Single-run 3D tracing error versus joint success. Appendix D.1 gives absolute metrics and scene definitions.

Figure 5: Astra’s spatial strengths and reference-frame limits. (a) Relation and viewpoint accuracy compared with each task’s best peer. (b) Camera-relative depth (Near/Far/Cam.) and inter-object distance (Obj./Rel.). (c) Astra’s accuracy advantage over the best peer on one task per source; C–R denotes camera–region relations. Panels a–b average available runs; c compares shared run indices. Whiskers show run-to-run SD. Appendix D.2 maps labels to all six sources and reports every task.

Figure 6: Astra’s native affordance localization. (a) Contact validity over all six UMD actions. (b) Astra’s boxes evaluated against reference boxes and masks. (c) Contact validity for four corresponding interactions in ReasonAff/RAGNet; 1H/2H means one/two hands. Scores show mean

\pm

run SD; IoUs are multiplied by 100. Appendix D.3 gives all actions and models.

Figure 7: Astra’s planning advantage across sources and goals. (a) Overall success for all six sources. (b) ALFRED/ALFWorld success on seven household goal families. (c) All five nonzero Astra–Fable gaps in ScienceWorld; thirteen families tie. K/U means known/unknown conditions. Single-run task outcomes; Appendix D.4 includes all seven models.

Figure 8: Manipulation across suites and execution settings. (a) Task success on thirteen suites; the last three rows share current results. (b) Average CALVIN goals completed consecutively before the first failure, out of five. (c) ManiSkill goal-condition satisfaction on five diagnostic tasks. Filled/open markers show Baseline/RGB-expanded settings (Section 4.4 ). Each task uses one run; Appendix D.5 gives metrics and coverage.

(a)		Geometry	Spatial Reasoning	Affordance
Model	Condition	I/T/W	Acc. (%) $\uparrow$	Box IoU $\uparrow$	SR 50 (%) $\uparrow$
Astra	Initial	–	72.2	0.2903	22.2
	Self-review	3/20/1	66.7	0.2914	22.2
	Crop review	4/19/1	66.7	0.2912	22.2
	Tool-assisted	5/15/4	72.2	0.3174	33.3
Qwen-27B	Initial	–	55.6	0.0336	0.0

Table 3: Multi-round review (a) and RGB-expanded Manipulation (b). Baseline and RGB-expanded share budgets and execution-step limits; RGB-expanded supplies per-round images. I/T/W counts improved/tied/worsened entries; SR 50 requires box IoU

\geq 0.5

. Goals gives average consecutive completions (0–5); Common pools nine suites (153 tasks). Details: Appendix E .

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Cases	Share (%)	Task coverage
Geometry	370	37.0	Calibration, pose, scale, tracing
Spatial Reasoning	220	22.0	Relations, views, video
Affordance	70	7.0	Functional parts, contact
Task Planning	157	15.7	Household, navigation, science
Manipulation	183	18.3	Robot operations, sequences
Total	1,000	100.0	660 fixed-input; 340 interactive

Appendix

Table A1: The unified 1,000-case arena, drawn from 32 established sources and our GeoProbe benchmark. Detailed source allocations follow below.

Diagnostic	Cases	Diagnostic	Cases
Object displacement, fixed camera	12	Full camera intrinsics	24
Object displacement, small view change	12	Focal length, centered principal point	12
Object displacement, large view change	12	Relative camera pose	24
Real tabletop displacement	4	Sparse depth	12
Object dimensions	12	Reference-based scale	44

Appendix

Table A2: GeoProbe allocation across ten diagnostic protocols. Several protocols share images and scenes.

Benchmark	Cases	Task coverage
CALVIN ( Mees et al., 2021 )	10	Ten sequences, each with five ordered subgoals
CLIPort ( Shridhar et al., 2021 )	18	Eighteen task and color-generalization configurations
RLBench ( James et al., 2019 )	10	Ten task configurations
RoboCasa ( Nasiriany et al., 2024 )	10	Seven atomic and three composite tasks
RoboCasa365	10	Three atomic and seven composite tasks
RoboWits	10	Ten official task configurations

Appendix

Table A3: Manipulation task allocation across 13 source benchmarks. All agents use the same assigned task set.

Domain	Observation support	Actions and feedback
Geometry	Image or image pair; intrinsics for Map-free	One final numerical submission
Spatial Reasoning	Images, crops, and video-frame queries	Evidence inspection and answer submission
Affordance	Image and interaction instruction	One point or bounding-region submission
Task Planning	Images, text, or symbolic state	Navigation, interaction, programs, public progress
Manipulation	Protocol-specific scene/state observations and native grounding helpers	Motion or skill calls; execution and task checks

Appendix

Table A4: Model-visible observations, actions, and feedback. Private scoring annotations are unavailable to the agent.

Model label	Identifier
GPT-6 Astra	gpt-6-astra
GPT-5.6 Sol	gpt-5.6-sol
Claude Fable 5.1	claude-fable-5-1
Gemini 3.8 Flash	gemini-3.8-flash
Qwen 3.8 Max	qwen3.8-max
Qwen 3.5 397B-A17B	qwen3.5-397b-a17b

Appendix

Table A5: Evaluated models and configured identifiers. All seven agents are represented across the five domains.

Entry	Code rounds	Tokens	Env. steps	Primitives	Episode seconds	Task seconds
Geometry	12	120k	–	–	–	960
Spatial: non-VSI	6	96k	–	–	900	960
Spatial: VSI	8	240k	–	–	1,200	1,500
ALFRED, ALFWorld	8	320k	120	540	1,200	1,500
DiscoveryWorld	8	320k	100	400	1,200	1,500
HumanCLAW	101	2,000k	100	400	10,800	11,000

Appendix

Table A6: Baseline episode budgets. Tokens accumulate within each case; rounds are code interactions, not repetitions. Dashes denote unspecified fields. Appendix B.4 lists additional fixed-input configurations.

Domain	Astra	Sol	Fable	Q-Max	Gemini	Q-397B	Q-27B
Geometry	1382/1402	1382/1402	819/886	1382/1402	1314/1402	1382/1402	1376/1402
Spatial Reasoning	1060/1100	1100/1100	586/660	1097/1100	1077/1100	1033/1100	982/1100
Affordance	350/350	350/350	208/210	350/350	335/350	349/350	349/350
Task Planning	157/157	157/157	157/157	157/157	157/157	157/157	157/157

Appendix

Table A7: Valid submission coverage for fixed-input domains and recorded terminal outcomes for Planning. Denominators include repetitions; Manipulation uses the fixed allocation in Table A3 .

Metric	$N$	Astra	Sol	Fable	Q-Max	Gemini	Q-397B	Q-27B
Geometry (370 cases)
InFlux focal MAPE (%) $\downarrow$	26	53.02	47.79	52.65	55.50	52.36	67.82	65.86
InFlux principal error (px) $\downarrow$	26	3.82	3.82	3.82	3.82	3.88	3.82	3.82
Angle relative L2 $\downarrow$	12	0.760	0.915	1.32	1.41	0.962	1.30	2.04
Distance relative L2 $\downarrow$	10	0.294	0.418	0.785	0.560	0.419	0.411	0.331
Vector relative L2 $\downarrow$	8	0.425	0.591	0.625	0.454	0.647	1.02	1.03

Appendix

Table A8: Complementary native metrics across four domains. Repeated binary rates show mean

\pm

run SD; single-run rates have no SD. Continuous values are pooled valid-observation means.

N

counts eligible cases. GP denotes GeoProbe; Manipulation source results appear in Table A9 .

Source	$N$	Astra	Sol	Fable	Q-Max	Gemini	Q-397B	Q-27B
Spatial Reasoning: categorical accuracy (%)
3DSRBench	50	73.6 $\pm$ 2.6	71.2 $\pm$ 1.8	72.7 $\pm$ 3.1	74.0 $\pm$ 3.7	70.0 $\pm$ 2.4	69.2 $\pm$ 3.6	57.6 $\pm$ 2.6
BOP-ASK	20	96.0 $\pm$ 4.2	87.0 $\pm$ 4.5	88.3 $\pm$ 2.9	92.0 $\pm$ 2.7	97.0 $\pm$ 2.7	85.0 $\pm$ 3.5	84.0 $\pm$ 4.2
MindCube	50	90.8 $\pm$ 1.1	68.0 $\pm$ 7.3	79.3 $\pm$ 8.1	90.8 $\pm$ 2.7	89.2 $\pm$ 3.0	46.0 $\pm$ 2.8	50.4 $\pm$ 4.1
MMSI-Bench	20	77.0 $\pm$ 5.7	55.0 $\pm$ 11.7	66.7 $\pm$ 7.6	55.0 $\pm$ 8.7	56.0 $\pm$ 9.6	35.0 $\pm$ 9.4	29.0 $\pm$ 7.4
RoboSpatial	20	85.0 $\pm$ 3.5	81.0 $\pm$ 8.9	88.3 $\pm$ 2.9	88.0 $\pm$ 5.7	76.0 $\pm$ 4.2	90.0 $\pm$ 3.5	92.0 $\pm$ 5.7

Appendix

Table A9: Source results. Spatial/Affordance: mean

\pm

run SD (%); Planning: solved/assigned cases; Manipulation: success (%).

N

counts eligible cases.

Agent	Spatial without VSI	Planning without text/symbolic	Affordance without UMD
Astra	83.6	66.7	16.5
Sol	71.4	20.6	10.5
Fable	77.9	33.3	15.0
Qwen-Max	80.9	12.7	10.0
Gemini	78.4	22.2	5.5
Qwen-397B	62.2	7.9	2.0

Appendix

Table A10: Task-composition analysis (success %). Spatial uses only the 183 categorical queries; Planning removes ScienceWorld and VirtualHome; Affordance removes UMD and uses strict box success. All columns retain the assigned-case denominator of the indicated subset.

Figure A1: Individual capability profiles with the same axes and fixed scales as Figure 3 a. G, S, A, P, and M denote Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation. All axes derive from the metrics in Table 2 .

Task	Astra	Sol	Fable	Q-Max	Gemini	Q-397B	Q-27B
GP focal-only	$6.2\!\pm\!2.1$	$11.2\!\pm\!5.5$	$29.5\!\pm\!1.8$	$13.8\!\pm\!3.1$	$23.7\!\pm\!12.9$	$38.6\!\pm\!3.2$	$38.4\!\pm\!5.0$
InFlux focal	$53.0\!\pm\!2.1$	$47.8\!\pm\!2.0$	$52.6\!\pm\!2.2$	$55.5\!\pm\!1.0$	$52.4\!\pm\!3.4$	$67.8\!\pm\!1.8$	$65.9\!\pm\!2.1$
GP depth	$10.4\!\pm\!2.9$	$22.9\!\pm\!2.7$	$14.2\!\pm\!2.7$	$16.3\!\pm\!1.3$	$19.2\!\pm\!2.1$	$31.2\!\pm\!2.8$	$29.2\!\pm\!1.3$
GP object size	$6.3\!\pm\!1.0$	$8.5\!\pm\!0.7$	$7.6\!\pm\!0.2$	$9.6\!\pm\!0.8$	$8.3\!\pm\!0.3$	$11.4\!\pm\!0.6$	$9.4\!\pm\!0.3$
GP ref. height	$0.31\!\pm\!0.11$	$0.90\!\pm\!0.36$	$2.31\!\pm\!0.08$	$1.69\!\pm\!0.72$	$2.73\!\pm\!0.75$	$17.55\!\pm\!1.22$	$13.69\!\pm\!0.86$
GP height ratio	$0.32\!\pm\!0.15$	$0.95\!\pm\!0.41$	$2.23\!\pm\!0.17$	$2.17\!\pm\!0.22$	$2.67\!\pm\!0.28$	$20.35\!\pm\!0.87$	$17.08\!\pm\!2.70$

Appendix

Table A11: Geometric errors (%) across seven agents, mean

\pm

run SD. GP denotes GeoProbe; lower is better.

Task	Cases	Rounds	Peer	Astra	Peer
GP focal-only	12	5	Sol	$6.21\!\pm\!2.10$	$11.16\!\pm\!5.47$
InFlux focal	26	5	Sol	$53.02\!\pm\!2.15$	$47.79\!\pm\!2.00$
GP depth	9	3	Fable	$10.16\!\pm\!3.49$	$14.22\!\pm\!2.88$
GP object size	12	3	Fable	$6.44\!\pm\!1.36$	$7.63\!\pm\!0.18$
GP ref. height	12	5	Sol	$0.31\!\pm\!0.11$	$0.90\!\pm\!0.36$
GP height ratio	12	5	Sol	$0.32\!\pm\!0.15$	$0.95\!\pm\!0.41$

Appendix

Table A12: Errors (%) on the common-valid cohort. Both agents use the same cases and runs within each row.

Condition	Cases	Astra	Sol
Blender: fixed camera	12	$0.240\!\pm\!0.109$	$0.467\!\pm\!0.267$
Blender: fixed, moving-only	10	$0.288\!\pm\!0.131$	$0.560\!\pm\!0.321$
Blender: small view change	12	$0.567\!\pm\!0.158$	$0.678\!\pm\!0.152$
Blender: large view change	12	$1.125\!\pm\!0.144$	$1.958\!\pm\!0.295$
Real photographs	4	$0.050\!\pm\!0.068$	$0.415\!\pm\!0.236$

Appendix

Table A13: Displacement MAE (cm), mean

\pm

run SD. Sol is the strongest non-Astra comparator under each listed condition. Moving-only excludes stationary cases.

Motion	Cases	Rotation ( ∘ )	Translation (cm)	Joint success (%)
Stationary	4	0.00	0.00	$100.0\!\pm\!0.0$
Rotation only	6	0.35	1.00	$96.7\!\pm\!7.5$
Translation only	6	0.00	1.55	$76.7\!\pm\!25.3$
Rotation + translation	8	3.26	14.16	$0.0\!\pm\!0.0$

Appendix

Table A14: GeoProbe camera-pose modes. Errors are pooled valid-observation means; joint success uses

2^{\circ}

/2 cm tolerances and shows mean

\pm

run SD.

Task	Peer	Rounds	Astra	Peer	Gap $\pm$ SD
MMSI camera–region	Fable	3	100.0	77.8	$+22.2\!\pm\!19.2$
VSI medium direction	Qwen-Max	5	100.0	83.3	$+16.7\!\pm\!11.8$
BOP-ASK left/right	Gemini	5	95.0	96.7	$-1.7\!\pm\!7.0$
MindCube camera shift	Qwen-Max	5	78.5	89.2	$-10.8\!\pm\!14.0$
3DSR object facing	Fable	3	50.0	72.2	$-22.2\!\pm\!9.6$
RoboSpatial configuration	Q-397B	5	78.0	100.0	$-22.0\!\pm\!4.5$

Appendix

Table A15: Source-specific Astra–peer comparisons for Figure 5 c. Accuracy and gaps use shared available runs and are in percentages and percentage points. Run-index alignment follows Appendix B.3 .

3DSRBench 50 cases Task

N

Astra Sol Fable Q-Max Gemini Q-397B Q-27B World height 4

60.0\!\pm\!13.7

50.0\!\pm\!0.0

\mathbf{66.7}\!\pm\!14.4

55.0\!\pm\!11.2

25.0\!\pm\!0.0

50.0\!\pm\!0.0

50.0\!\pm\!0.0

Above / below 7

\mathbf{77.1}\!\pm\!7.8

60.0\!\pm\!12.0

71.4\!\pm\!0.0

\mathbf{77.1}\!\pm\!12.8

74.3\!\pm\!12.0

74.3\!\pm\!6.4

68.6\!\pm\!12.0

Camera depth 10

90.0\!\pm\!7.1

98.0\!\pm\!4.5

93.3\!\pm\!5.8

90.0\!\pm\!0.0

92.0\!\pm\!4.5

96.0\!\pm\!5.5

\mathbf{100.0}\!\pm\!0.0

Next to 1

0.0\!\pm\!0.0

0.0\!\pm\!0.0

0.0\!\pm\!0.0

0.0\!\pm\!0.0

0.0\!\pm\!0.0

\mathbf{60.0}\!\pm\!54.8

0.0\!\pm\!0.0

Object distance 2

70.0\!\pm\!27.4

80.0\!\pm\!27.4

83.3\!\pm\!28.9

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

50.0\!\pm\!0.0

80.0\!\pm\!27.4

Parallel / perpendicular 1

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

20.0\!\pm\!44.7

\mathbf{100.0}\!\pm\!0.0

80.0\!\pm\!44.7

Same/different facing 4

55.0\!\pm\!11.2

50.0\!\pm\!0.0

50.0\!\pm\!0.0

50.0\!\pm\!0.0

50.0\!\pm\!0.0

\mathbf{75.0}\!\pm\!17.7

35.0\!\pm\!13.7

Object-facing side 6

56.7\!\pm\!9.1

53.3\!\pm\!13.9

\mathbf{72.2}\!\pm\!9.6

66.7\!\pm\!0.0

66.7\!\pm\!0.0

36.7\!\pm\!18.3

23.3\!\pm\!9.1

In front of actor 4

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

95.0\!\pm\!11.2

Actor left/right 7

\mathbf{74.3}\!\pm\!6.4

71.4\!\pm\!14.3

61.9\!\pm\!8.2

71.4\!\pm\!10.1

65.7\!\pm\!7.8

45.7\!\pm\!12.0

5.7\!\pm\!7.8

Object viewpoint 4

\mathbf{70.0}\!\pm\!11.2

\mathbf{70.0}\!\pm\!11.2

50.0\!\pm\!0.0

60.0\!\pm\!13.7

\mathbf{70.0}\!\pm\!11.2

\mathbf{70.0}\!\pm\!11.2

65.0\!\pm\!28.5

BOP-ASK 20 cases Task

N

Astra Sol Fable Q-Max Gemini Q-397B Q-27B Closer to camera 5

96.0\!\pm\!8.9

96.0\!\pm\!8.9

86.7\!\pm\!11.5

\mathbf{100.0}\!\pm\!0.0

96.0\!\pm\!8.9

80.0\!\pm\!14.1

84.0\!\pm\!8.9

Farther from camera 3

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

66.7\!\pm\!0.0

40.0\!\pm\!14.9

Left / right 12

95.0\!\pm\!4.6

80.0\!\pm\!4.6

86.1\!\pm\!4.8

86.7\!\pm\!4.6

\mathbf{96.7}\!\pm\!4.6

91.7\!\pm\!0.0

95.0\!\pm\!4.6

MindCube 50 cases Task

N

Astra Sol Fable Q-Max Gemini Q-397B Q-27B Behind observer 7

97.1\!\pm\!6.4

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

88.6\!\pm\!12.0

80.0\!\pm\!16.3

65.7\!\pm\!16.3

Camera translation 13

78.5\!\pm\!10.0

64.6\!\pm\!8.8

56.4\!\pm\!16.0

\mathbf{89.2}\!\pm\!6.9

84.6\!\pm\!5.4

20.0\!\pm\!6.9

35.4\!\pm\!12.9

Imagined observer 2

\mathbf{90.0}\!\pm\!22.4

70.0\!\pm\!27.4

50.0\!\pm\!0.0

80.0\!\pm\!27.4

70.0\!\pm\!27.4

30.0\!\pm\!27.4

30.0\!\pm\!27.4

Object relations 12

\mathbf{96.7}\!\pm\!4.6

63.3\!\pm\!12.6

91.7\!\pm\!8.3

85.0\!\pm\!3.7

95.0\!\pm\!7.5

31.7\!\pm\!7.0

35.0\!\pm\!3.7

Composed rotation 5

\mathbf{100.0}\!\pm\!0.0

96.0\!\pm\!8.9

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

96.0\!\pm\!8.9

Turn then translate 11

\mathbf{90.9}\!\pm\!6.4

43.6\!\pm\!7.6

75.8\!\pm\!10.5

\mathbf{90.9}\!\pm\!6.4

87.3\!\pm\!10.4

49.1\!\pm\!13.8

58.2\!\pm\!10.4

MMSI 20 cases Task

N

Astra Sol Fable Q-Max Gemini Q-397B Q-27B Attribute (Appr.) 2

\mathbf{90.0}\!\pm\!22.4

70.0\!\pm\!27.4

50.0\!\pm\!0.0

60.0\!\pm\!22.4

50.0\!\pm\!0.0

70.0\!\pm\!27.4

40.0\!\pm\!22.4

Attribute (Meas.) 2

\mathbf{100.0}\!\pm\!0.0

50.0\!\pm\!0.0

66.7\!\pm\!57.7

40.0\!\pm\!22.4

50.0\!\pm\!0.0

0.0\!\pm\!0.0

40.0\!\pm\!22.4

MSR 2

40.0\!\pm\!41.8

30.0\!\pm\!27.4

\mathbf{50.0}\!\pm\!0.0

40.0\!\pm\!22.4

30.0\!\pm\!27.4

30.0\!\pm\!44.7

40.0\!\pm\!22.4

Motion (Cam.) 1

80.0\!\pm\!44.7

80.0\!\pm\!44.7

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

60.0\!\pm\!54.8

0.0\!\pm\!0.0

0.0\!\pm\!0.0

Motion (Obj.) 1

\mathbf{100.0}\!\pm\!0.0

80.0\!\pm\!44.7

\mathbf{100.0}\!\pm\!0.0

80.0\!\pm\!44.7

80.0\!\pm\!44.7

80.0\!\pm\!44.7

20.0\!\pm\!44.7

Camera–camera relation 3

\mathbf{100.0}\!\pm\!0.0

73.3\!\pm\!27.9

\mathbf{100.0}\!\pm\!0.0

66.7\!\pm\!0.0

73.3\!\pm\!14.9

26.7\!\pm\!14.9

33.3\!\pm\!0.0

Camera–object relation 1

0.0\!\pm\!0.0

40.0\!\pm\!54.8

\mathbf{66.7}\!\pm\!57.7

0.0\!\pm\!0.0

60.0\!\pm\!54.8

40.0\!\pm\!54.8

0.0\!\pm\!0.0

Camera–region relation 3

\mathbf{100.0}\!\pm\!0.0

60.0\!\pm\!14.9

77.8\!\pm\!19.2

66.7\!\pm\!0.0

73.3\!\pm\!14.9

66.7\!\pm\!23.6

53.3\!\pm\!18.3

Object–object relation 1

\mathbf{100.0}\!\pm\!0.0

80.0\!\pm\!44.7

\mathbf{100.0}\!\pm\!0.0

60.0\!\pm\!54.8

20.0\!\pm\!44.7

80.0\!\pm\!44.7

60.0\!\pm\!54.8

Object–region relation 4

\mathbf{50.0}\!\pm\!0.0

30.0\!\pm\!11.2

25.0\!\pm\!25.0

45.0\!\pm\!20.9

\mathbf{50.0}\!\pm\!30.6

5.0\!\pm\!11.2

0.0\!\pm\!0.0

RoboSpatial 30 cases Task

N

Astra Sol Fable Q-Max Gemini Q-397B Q-27B Placement compatibility 10

92.0\!\pm\!4.5

92.0\!\pm\!8.4

\mathbf{96.7}\!\pm\!5.8

90.0\!\pm\!0.0

64.0\!\pm\!8.9

80.0\!\pm\!7.1

90.0\!\pm\!7.1

Configuration 10

78.0\!\pm\!4.5

70.0\!\pm\!10.0

80.0\!\pm\!0.0

86.0\!\pm\!11.4

88.0\!\pm\!4.5

\mathbf{100.0}\!\pm\!0.0

94.0\!\pm\!8.9

Free-space point set 10

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

VSI-Bench 50 cases Task

N

Astra Sol Fable Q-Max Gemini Q-397B Q-27B Appearance order 2

40.0\!\pm\!22.4

80.0\!\pm\!27.4

50.0\!\pm\!50.0

\mathbf{100.0}\!\pm\!0.0

40.0\!\pm\!41.8

40.0\!\pm\!22.4

80.0\!\pm\!27.4

Absolute distance 9

2.2\!\pm\!5.0

2.2\!\pm\!5.0

3.7\!\pm\!6.4

0.0\!\pm\!0.0

\mathbf{6.7}\!\pm\!6.1

0.0\!\pm\!0.0

2.2\!\pm\!5.0

Object count 5

32.0\!\pm\!11.0

20.0\!\pm\!14.1

\mathbf{40.0}\!\pm\!20.0

36.0\!\pm\!16.7

36.0\!\pm\!16.7

28.0\!\pm\!11.0

12.0\!\pm\!11.0

Direction: easy 4

\mathbf{100.0}\!\pm\!0.0

85.0\!\pm\!22.4

33.3\!\pm\!28.9

\mathbf{100.0}\!\pm\!0.0

85.0\!\pm\!13.7

30.0\!\pm\!20.9

45.0\!\pm\!11.2

Direction: hard 4

45.0\!\pm\!11.2

30.0\!\pm\!27.4

25.0\!\pm\!43.3

\mathbf{65.0}\!\pm\!28.5

60.0\!\pm\!22.4

30.0\!\pm\!32.6

5.0\!\pm\!11.2

Direction: medium 6

\mathbf{100.0}\!\pm\!0.0

73.3\!\pm\!9.1

38.9\!\pm\!38.5

83.3\!\pm\!11.8

66.7\!\pm\!11.8

40.0\!\pm\!14.9

20.0\!\pm\!13.9

Relative distance 6

50.0\!\pm\!20.4

\mathbf{76.7}\!\pm\!22.4

38.9\!\pm\!9.6

\mathbf{76.7}\!\pm\!9.1

56.7\!\pm\!14.9

60.0\!\pm\!19.0

53.3\!\pm\!13.9

Object size 9

4.4\!\pm\!6.1

\mathbf{22.2}\!\pm\!7.9

11.1\!\pm\!0.0

13.3\!\pm\!5.0

11.1\!\pm\!7.9

2.2\!\pm\!5.0

8.9\!\pm\!5.0

Room area 4

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

\mathbf{0.0}\!\pm\!0.0

Route planning 1

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

33.3\!\pm\!57.7

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

\mathbf{100.0}\!\pm\!0.0

Appendix

Table A16: All Spatial task groups, separated by benchmark. Entries are mean

\pm

run SD (%); bold means mark the highest score in each row, including ties. Astra’s column is shaded.

N

counts distinct cases; each assigned case remains in the success denominator.

Task	Astra	Sol	Fable	Q-Max	Gemini	Q-397B	Q-27B
Object count	$0.35\!\pm\!0.05$	$0.40\!\pm\!0.11$	$0.20\!\pm\!0.11$	$0.23\!\pm\!0.11$	$0.31\!\pm\!0.08$	$0.41\!\pm\!0.06$	$0.42\!\pm\!0.04$
Absolute distance	$0.28\!\pm\!0.11$	$0.42\!\pm\!0.05$	$0.28\!\pm\!0.07$	$0.26\!\pm\!0.02$	$0.12\!\pm\!0.03$	$0.29\!\pm\!0.09$	$0.31\!\pm\!0.08$
Object size	$0.09\!\pm\!0.01$	$0.09\!\pm\!0.02$	$0.10\!\pm\!0.01$	$0.12\!\pm\!0.02$	$0.05\!\pm\!0.01$	$0.13\!\pm\!0.02$	$0.19\!\pm\!0.02$
Room area	$0.24\!\pm\!0.03$	$0.19\!\pm\!0.03$	$0.21\!\pm\!0.04$	$0.26\!\pm\!0.07$	$0.27\!\pm\!0.04$	$0.22\!\pm\!0.05$	$0.26\!\pm\!0.10$

Appendix

Table A17: VSI-Bench AbsRel, equal-run mean

\pm

SD on valid predictions; lower is better. Each model uses its valid-prediction cohort; coverage is reported below.

Task	Cases	Valid / assigned	Common cases	Common AbsRel
Object count	5	25/25	5	$0.350\!\pm\!0.053$
Absolute distance	9	25/45	2	$0.279\!\pm\!0.083$
Object size	9	42/45	8	$0.091\!\pm\!0.017$
Room area	4	20/20	4	$0.240\!\pm\!0.029$

Appendix

Table A18: Astra numerical coverage and a fixed-cohort sensitivity check. Common cases have a valid numerical prediction in every recorded round.

Action / hands	Cases	Mask IoU	Point (%)	Box (%)
RAGNet-3DOI
free movement / two hands	2	$0.046\!\pm\!0.063$	$20.0\!\pm\!27.4$	$20.0\!\pm\!27.4$
free / one hand	1	$0.384\!\pm\!0.351$	$60.0\!\pm\!54.8$	$60.0\!\pm\!54.8$
free / two hands	5	$0.249\!\pm\!0.070$	$60.0\!\pm\!14.1$	$36.0\!\pm\!16.7$
pull / one hand	12	$0.008\!\pm\!0.002$	$25.0\!\pm\!5.9$	$0.0\!\pm\!0.0$
ReasonAff-style

Appendix

Table A19: Astra Affordance metrics, mean

\pm

run SD. IoU compares the native box to the GT mask; strict box success compares it to the GT box at IoU

\geq 0.5

. Point and box success are percentages.

Task / criterion	Astra	Best peer	Model
UMD cutting: mask IoU	$0.361\!\pm\!0.009$	$0.341\!\pm\!0.022$	Fable
UMD cutting: strict box (%)	$96.0\!\pm\!8.9$	$100.0\!\pm\!0.0$	Fable
UMD containment: mask IoU	$0.472\!\pm\!0.067$	$0.673\!\pm\!0.069$	Sol
UMD containment: strict box (%)	$44.0\!\pm\!16.7$	$96.0\!\pm\!8.9$	Qwen-Max
ReasonAff pulling: point (%)	$60.0\!\pm\!13.7$	$50.0\!\pm\!0.0$	Fable
RAGNet pulling: point (%)	$25.0\!\pm\!5.9$	$31.7\!\pm\!7.0$	Qwen-Max

Appendix

Table A20: Metric-specific Affordance peers. Values are mean

\pm

run SD, using the repetition counts in Appendix B.4 . Each peer has the highest non-Astra mean for the indicated criterion; comparisons use available runs.

Goal family	Astra	Best peer	Model
ALFRED
Illumination	1/2	1/2	Fable ∗
Placement	4/4	2/4	Sol ∗
Container transport	2/2	1/2	Fable ∗
Cleaning	2/2	1/2	Sol ∗
Cooling	2/4	2/4	Fable

Appendix

Table A21: Task Planning by native goal family, with one episode per assigned case. Entries are solved/assigned cases. One highest-success peer is shown per row; ∗ denotes a tie with another peer. Full model results are in the accompanying CSV.

Model	Process coverage	Target found	Within 20 cm	Near + sit issued	Success
Astra	8/8	6	5	5	0
Sol	8/8	3	1	1	0
Fable	8/8	0	0	0	0
Qwen-Max	0/8	–	–	–	0
Gemini	6/8	3 †	3 †	3 †	0
Q-397B	0/8	–	–	–	0

Appendix

Table A22: HumanCLAW sitting diagnostics for eight assigned cases per model. Counts pool independently recorded conditions. † denotes lower bounds because process fields are missing for two Gemini cases; dashes denote unavailable process metrics. Terminal success retains all eight assignments.

Benchmark	$N$	Astra	Sol	Fable	Q-Max	Gemini	Q-397B	Q-27B
Baseline
CALVIN	10	30.0	0.0	0.0	0.0	0.0	0.0	0.0
CLIPort	18	61.1	27.8	33.3	22.2	27.8	22.2	16.7
RLBench	10	60.0	20.0	10.0	0.0	20.0	0.0	10.0
RoboCasa	10	10.0	0.0	0.0	0.0	0.0	0.0	0.0
RoboCasa365	10	0.0	0.0	10.0	0.0	0.0	0.0	0.0

Appendix

Table A23: Complete Manipulation success by protocol (%). Baseline totals use 182 assignments; – marks missing records. The last three sources share current results. The common nine-source subset isolates sources with records under both protocols.

Benchmark	$N$	Astra	Sol	Fable	Q-Max	Gemini	Q-397B	Q-27B
Baseline
CALVIN goals (0–5)	10	3.10	1.40	0.90	0.80	1.80	0.00	0.10
RoboCasa	4	–	–	–	–	–	–	–
RoboCasa365	7	–	–	–	–	–	–	–
ManiSkill conditions	5	60.00	0.00	–	0.00	0.00	0.00	0.00
LIBERO-PRO	10	20.00	20.00	30.00	5.00	20.00	0.00	5.00

Appendix

Table A24: Manipulation auxiliary metrics under both protocols. CALVIN reports average consecutive goals completed (0–5); other rows average task-normalized criteria or control evidence (%). Definitions and time windows appear in Table A25 . – marks unavailable records; the final three rows share current results.

Source	$N$	Scoring items	Evidence window
RoboCasa	4	Approach, grasp and lift, stable release	Any verified event during execution
RoboCasa365	7	Same three transport milestones	Any verified event during execution
ManiSkill	5	Two or three native conditions per task	Final verification state
LIBERO-PRO	10	Two task-specific conditions	Final state and limited process records
robosuite	5	Two lifting, three stacking, or four wiping items	Native success and terminal reward-derived evidence
BEHAVIOR-1K	5	Six control-evidence items	Before the first episode-end signal

Appendix

Table A25: The six auxiliary evidence metrics, averaged over task-normalized items within each fixed subset.

Task	$K_{i}$	Scored conditions
PickCube	2	Object placed at its goal; placed and robot static. The suspended goal requires no release.
PlaceSphere	3	Sphere on its support; on support and released; both with the sphere static.
StackCube	3	Cube A on B; on B and released; both with cube A static.
PlugCharger	2	Goal distance $\leq 0.005$ m; distance satisfied and angular error $\leq 0.2$ rad.
PegInsertionSide	2	Peg-head $x\geq-0.015$ m in the hole frame; official insertion success.

Appendix

Table A26: ManiSkill’s five-task diagnostic. Items are evaluated at final verification; grasp history is separate. The other four binary tasks remain in complete-task success.

Item	Positive evidence
Localization	Finite coordinates or pose from public RGB-D localization or grasp sampling.
Candidate	A nonempty pregrasp or grasp candidate set.
Arrival	Measured arrival from approach or recovery feedback for the same arm.
Holding	An explicit in-hand check or holding confirmation inside checked placement.
Held motion	Associated measured motion of at least 5 cm while holding, or movement with a passed holding check inside checked placement.
Release	Holding, active gripper opening, then no longer holding; or confirmed release inside checked placement.

Appendix

Table A27: BEHAVIOR-1K control evidence. Each item records one type of control event and scores once per case; terminal BDDL goal success is evaluated separately.

Source	Rounds	Tools	Checks	Tokens	Trial	Code	Native
CALVIN	40	7,000	41	1,280	120	10	10
CLIPort	24	256	25	960	60	10	10
RLBench	16	256	17	640	60	10	10
RoboCasa	24	256	25	1,120	60	6	10
RoboCasa365	24	256	25	1,120	60	8	10
RoboTwin 2.0	16	256	17	640	60	10	20

Appendix

Table A28: Shared execution ceilings for Baseline and RGB-expanded. Times are minutes; token budgets are thousands. A round is a model decision, Tools counts interface calls, and Checks counts verifier calls. The limits apply to all seven agents within each source.

Source	Views	Resolution
CALVIN	Static; gripper	$200\!\times\!200$ ; $84\!\times\!84$
CLIPort	Front, left, right; no Oracle camera	$640\!\times\!480$
RLBench	Front, wrist, overhead	$128\!\times\!128$
RoboCasa/365	Left/right agent view, eye-in-hand	$512\!\times\!512$
RoboTwin 2.0	Head, left, right	$320\!\times\!240$
RoboWits	Ego, left/right wrist	$848\!\times\!480$

Appendix

Table A29: Expanded camera presets, width

\times

height in pixels. The preset inventory covers all 183 assigned cases, including AnymalC-Spin. Multiple views enter as separate images.

Source	Base $\to$ effective	Counter and termination semantics
CLIPort	$5\to 10$	Pick-and-place actions
RoboCasa/365	$1000\to 2000$	Control steps; field does not truncate episodes in the current wrapper
RoboTwin 2.0	$400\to 800$	Native action calls, not all simulator steps
RoboWits	$800\to 1600$	Control steps for the checked case
ManiSkill	$50\to 100$	Control steps; PickCube termination/boundary checks verify the effective limit
LIBERO-PRO	$4000\to 8000$	Wrapper/control counter; field does not truncate episodes

Appendix

Table A30: Native execution-step limits shared by both conditions. Values show environment defaults

\to

effective evaluation limits; control frequency is unchanged.

Condition	Evidence available during review	Additional response budget
Initial	Original task observations	None
Self-review	Original observations and initial answer	Two responses, up to 1,800 tokens each
Crop review	Self-review evidence plus image crops	Two responses, up to 1,800 tokens each
Tool-assisted	Original RGB, crops, and specialist measurements	Six responses, up to 4,096 tokens each; up to 16 specialist requests

Appendix

Table A31: Review conditions for the same cases and native scoring interfaces. Budgets are additional to native execution; response limits count output tokens.

Panel of Figure 44 — Figure A2: Spatial judgments from the first evaluation run (top: 3DSRBench, MindCube, RoboSpatial) and affordance localization (bottom: UMD, ReasonAff, RAGNet). Top questions concern the dog’s facing direction, camera motion, and whether the plant fits beside the drawer; green answers match GT. Below, green points lie inside the GT mask and green boxes reach box IoU $\geq 0.5$ ; red marks fail their respective criteria.

Figure A3: Controller construction and task outcomes. (a) CALVIN dots mark controller construction rounds; filled tiles count final consecutive goals. (b) PickCube shows Fable’s ordered calibration, feedback-control, and success events, plus all three agents’ physical-step totals and outcomes under RGB-expanded execution. R denotes model rounds; physical steps count environment updates.

Figure A4: Archived execution contrasts under matched task settings and budgets. CloseLaptopLid uses RLBench’s front camera; PullCube uses ManiSkill’s base camera. Native records determine success. These earlier episodes are separate from the protocol comparison.

Explore similar work

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning
Jun 28, 2026Taozhao Chen, Ian Manchester, Huaming ChenSemantic Representations Position
School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia
VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
Dec 27, 2025Borong Zhang, Jiahao Li, Jiachen Shen +7Diffusion-Based Vision-Language-Actions Robot Policies
Institute for Artificial Intelligence, Peking University. · Zhongguancun Academy. · PKU-PsiBot Joint Lab. +2
Towards Generalizable Visually Grounded Exploration of Household Devices
Sep 1, 2026Linhao Zheng, Zeming Liu, Wangke Chen +4Embodied Agents Weak Visual Grounding
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Beihang University · Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology

Jun 28, 2026, cs.ROJ/K move · Enter open · S save

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

Taozhao Chen, Ian Manchester, Huaming Chen

School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia

Semantic Representations Position Evaluation Metrics

Open paper AskPDF arXiv

Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulation benchmarks. These gains are commonly interpreted as evidence that semantic representations learned from internet-scale data transfer to physical execution generalization. This position paper argues that the assumption underlying this interpretation -- that semantic generalization is sufficient to support physical action decisions -- has not been independently verified and cannot be tested under current evaluation protocols. We support this claim by decomposing VLA policies into semantic mapping and physical action decision, and showing that task success rate -- the dominant evaluation metric -- cannot distinguish between these two sources of capability. As a result, improvements in benchmark performance are consistent with multiple competing explanations, including semantic matching, distributional overlap, and genuine physical generalization. We further argue that this identifiability gap has been reinforced through narrative drift, whereby successive systems inherit and strengthen prior interpretations of performance gains without isolating the underlying causal mechanism. To address this limitation, we propose a research direction based on evaluation designs that introduce controlled variation to separately measure semantic and physical generalization. Such designs make it possible to causally attribute performance without requiring access to model internals, and to empirically assess the role of VLM backbones as semantic interfaces rather than implicit sources of physical competence. Our goal is not to refute the role of VLMs in robotics, but to clarify the conditions under which claims of physical generalization can be meaningfully evaluated.

Abstract

Explore similar work

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

Towards Generalizable Visually Grounded Exploration of Household Devices