A Statistical Audit of Physical AI Benchmark Redundancy

Zaruhi Navasardyan; Hrant Davtyan

cs.ROAug 26, 2026

A Statistical Audit of Physical AI Benchmark Redundancy

Authors: Zaruhi Navasardyan, Hrant Davtyan

Organizations: Metric AI Lab

Abstract

Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse and the relationship between benchmarks unmeasured. We construct a matrix of 51 models on 12 physical AI benchmarks, selected from a registry of 51 benchmarks and 152 models by reporting density, combining scores from model cards and benchmark papers with our own evaluation runs under each benchmark's official protocol. We measure how much information the benchmarks share and show quantitative evidence of Redundancy. Redundancy affects reported rankings: collapsing the two substitute pairs into single columns moves 22 of 51 models by three or more places under an equally weighted average. We then select benchmarks greedily under a utility combining score dispersion with variance not explained by the already-selected set, and obtain a four-benchmark subset retaining 78.5% of the utility of all 12, on which we fit a Bradley--Terry ranking. The procedure requires only benchmark-level scores with sufficient overlap and is not specific to physical AI.

Figures & tables

Benchmark	Claimed ability	Items	$n$	Mean	SD	Min–Max	$g$
VSI-Bench	Visual-spatial intelligence (video)	5,130	51	46.9	12.8	12.6–69.5	0.153
EmbSpatial	Egocentric spatial relations	3,640	51	73.2	8.0	43.2–84.1	0.057
RefSpatial-Bench	Spatial referring (pointing)	200	51	29.9	18.1	0.3–72.2	0.343
Where2Place	Affordance pointing / free space	100	50	42.3	19.8	7.6–76.0	0.265
ERQA	Embodied reasoning, planning	400	49	44.5	8.5	25.7–65.0	0.104
CV-Bench	Classical CV as VQA (depth, count)	2,638	49	81.7	6.1	61.0–89.2	0.039

Table 1: The 12 physical-AI benchmarks. Items = number of samples in the benchmark;

n

= models with a score in our matrix, of 51.

g

= Gini coefficient.

Panel of Figure 1 — Figure 1: Left: pairwise Spearman $\rho$ across the 12 benchmarks, computed on pairwise-complete rows. Right: average-linkage hierarchical clustering under $D=1-\rho$ .

Benchmark	$n$	LOO $R^{2}$	Uniqueness $1-R^{2}$	Top 3 predictors
Where2Place	50	0.727	0.273	RefSpatial-Bench, MindCube, BLINK
RefSpatial-Bench	51	0.721	0.279	Where2Place, ERQA, MindCube
ERQA	49	0.706	0.294	MindCube, RefSpatial-Bench, RealWorldQA
CV-Bench	49	0.614	0.386	EmbSpatial, ERQA, RefSpatial-Bench
MindCube	42	0.599	0.401	ERQA, RefSpatial-Bench, SAT
SAT	49	0.468	0.532	OmniSpatial, MindCube, BLINK

Table 2: Leave-one-model-out predictability of each benchmark from its eleven physical peers.

1-\text{LOO }R^{2}

is the benchmark’s uniqueness. Top 3 predictors are the benchmarks which contribute most to explaining current benchmark’s variance.

Figure 2: Forward-selection path. Bars are the per-step marginal utility $U(b\mid S)=g(b)\,(1-R^{2}(b\sim S))$ ; the line is cumulative utility as a percentage of the 12-benchmark total. Four benchmarks reach 78.5%.

Rank	Model	Elo	Core
1	HY-Embodied-0.5 MoE-407B-A32B	2251	3
2	Qwen3.5-397B-A17B	2032	3
3	Qwen3-VL-235B-A22B-Instruct	1828	3
4	Seed 2.0	1790	3
5	Kimi K2.5	1750	4
6	HY-Embodied-0.5 MoT-4B-A2B	1744	4

Table 3: The compact leaderboard: top 10 of 51 models under a benchmark-as-judge Bradley–Terry fit on the four selected benchmarks. Core is how many of the four the model has a score on.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Provider	Size	Released	Base model	Pub.	Own
Claude-Sonnet-4-2025-05-14	Anthropic	—	2025-05	—	10	0
RoboBrain-32B-2.0	BAAI	32B	2025-07	Qwen2.5-VL-32B-Instruct	8	3
RoboBrain-7B-2.0	BAAI	7B	2025-07	Qwen2.5-VL-7B-Instruct	9	3
RoboBrain-7B-1.0	BAAI	7B	2025-02	LLaVA-OneVision-7B	10	2
RoboBrain-2.5-4B	BAAI	4B	2026-01	Qwen3-VL-4B-Instruct	8	3
Seed 2.0 (ByteDance)	ByteDance	—	2026-01	—	8	0

Appendix

Table 4: The 51 models in the matrix: provider, parameter count (open-weight only), release date, base model where the checkpoint is a post-train of a named model, and how many of its 12 benchmark scores are grouped as published versus produced by our own runs.

General anchor	$n$	$\rho$	95% CI
MMStar	29	0.950	$[0.85,\ 0.98]$
Video-MME	26	0.942	$[0.81,\ 0.99]$
MMLU-Pro	16	0.871	$[0.59,\ 0.98]$
MMMU	37	0.820	$[0.58,\ 0.95]$
OCRBench	27	0.774	$[0.46,\ 0.94]$
DocVQA	22	0.754	$[0.42,\ 0.93]$

Appendix

Table 5: Spearman correlation between the physical suite’s PC1 — estimated from the 12 physical benchmarks alone — and each general anchor, over the models reporting both.

n

= overlapping models; CI = bootstrap interval with the PCA refit inside each resample. General PC1 is a separate PC1-physical-vs-PC1-general comparison.

Benchmark A	Benchmark B	Raw $\rho$	Residual $\rho$	Dense block
EmbSpatial	CV-Bench	0.876	0.784	0.836
Where2Place	RefSpatial-Bench	0.860	0.801	0.773
ERQA	MindCube	0.656	0.613	0.658
VSI-Bench	BLINK	0.002	$-0.363$	$-0.425$
ERQA	EmbSpatial	0.729	0.197	$-0.095$
ERQA	CV-Bench	0.725	0.243	$-0.055$

Appendix

Table 6: Benchmark-pair correlations before and after removing an externally estimated general axis (the first PC of the 9 anchors). Residual = correlation of the two benchmarks’ residuals, all 51 models; Dense block = the same using only the 32 models with

\geq 4

observed anchors and no imputation. Top: the three bonds that survive. Middle: the VSI / BLINK suppression, which appears only after conditioning. Bottom: two representative collapses.

Step	Benchmark	$g$	$R^{2}$	$U$	Cum. %	$n$
1	RefSpatial-Bench	0.343	0.000	0.3431	38.8	51
2	MindCube	0.153	0.087	0.1396	54.6	42
3	VSI-Bench	0.153	0.254	0.1144	67.5	42
4	BLINK	0.106	0.087	0.0972	78.5	33
5	Where2Place	0.265	0.783	0.0576	85.0	32
6	RealWorldQA	0.069	0.442	0.0383	89.4	28

Appendix

Table 7: Full greedy selection path.

g

is the Gini coefficient of the raw score spread,

R^{2}

is the fit of the candidate on the already-selected set,

U

their product, and Cum. the running total as a percentage of the all-12 sum.

n

is the number of models on which

R^{2}

is estimated.

Forced first	First four selected	Kept	Cum. % @4
(none)	RefSpatial-Bench, MindCube, VSI-Bench, BLINK	4/4	78.5
VSI-Bench	VSI-Bench, RefSpatial-Bench, MindCube, BLINK	4/4	77.8
BLINK	BLINK, RefSpatial-Bench, MindCube, VSI-Bench	4/4	78.3
Where2Place	Where2Place, MindCube, VSI-Bench, RefSpatial-Bench	3/4	74.5
RoboSpatial	RoboSpatial, RefSpatial-Bench, MindCube, VSI-Bench	3/4	69.2
SAT	SAT, RefSpatial-Bench, VSI-Bench, MindCube	3/4	71.6

Appendix

Table 8: Seed robustness of the selection path. Kept counts how many of the four unseeded core benchmarks survive in the first four slots; Cum. % @4 is the utility captured at four benchmarks.

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

PhysicsBench: A Unified Leaderboard for Generative and Predictive Models in Engineering Design and Simulation
Aug 25, 2026Sang Won Lee, Hyogu Jeong, Namwoo KangGenerative AI Evaluation Benchmark Design
Narnia Labs, Daejeon, Republic of Korea · Cho Chun Shik Graduate School of Mobility, KAIST, Daejeon, Republic of Korea
Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths
Aug 19, 2026Andrej Tschalzev, Stefan Lüdtke, Heiner Stuckenschmidt +1Benchmark Design Benchmark Validity
University of Mannheim · University of Rostock · Technical University of Clausthal
The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains
May 11, 2026Jung Min KangBenchmark Design Benchmark Validity
Independent Researcher · Seoul, South Korea

Aug 25, 2026J/K move · Enter open · S save

PhysicsBench: A Unified Leaderboard for Generative and Predictive Models in Engineering Design and Simulation

Sang Won Lee, Hyogu Jeong, Namwoo Kang

Narnia Labs, Daejeon, Republic of Korea · Cho Chun Shik Graduate School of Mobility, KAIST, Daejeon, Republic of Korea

Generative AI Evaluation Benchmark Design Physics-Based Simulation

Open paper AskPDF arXiv

Generative and predictive artificial intelligence models are increasingly used to generate geometry and to predict physical fields and scalar quantities in engineering design and simulation. Yet these models are typically evaluated in isolation, on academic datasets at unconstrained scales, with inconsistent metrics and procedures. We present PhysicsBench, a unified benchmark and leaderboard that evaluates generative and predictive models under one standardized procedure. PhysicsBench spans seven generation and prediction tasks across 1D, 2D, and 3D domains and ranks 66 models on nine datasets, comprising industrial-scale CAD/CFD/FEA simulations and public references, expanded into 28 configurations. One procedure and ranking apply to both families, each ranked within its own tasks. Evaluation spans realistic, limited data scales from S to XL rather than the unlimited training sets common in academic benchmarks. A common metric suite captures geometric fidelity with distributional distances, physical-field and scalar accuracy, and engineering-specific field- and shape-validity. BenchRank debiases correlated metrics and ranks by PageRank over a head-to-head dominance graph, so every reported quality metric is also ranked, with computational cost in a separate efficiency view. Across tasks, an architecture's large-scale academic standing weakly predicts its small-data ranking. The top model changes with data scale in six of the seven tasks, and no model leads more than one task. PhysicsBench turns "state-of-the-art" from a self-reported claim into an openly published foundation for model selection.

Abstract

Benchmark Auditing

Explore similar work

PhysicsBench: A Unified Leaderboard for Generative and Predictive Models in Engineering Design and Simulation

Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths

The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains