RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

Andrew Perrault; Anmol Takiar; Dominik Winecki; Arnab Nandi

cs.CVOct 9, 2026

RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

Authors: Andrew Perrault, Anmol Takiar, Dominik Winecki, Arnab Nandi

Organizations: Relantic, Inc., Columbus, OH, USA

Abstract

Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation. Despite advances in vision language models, their capabilities in these settings remain uncertain and are challenging to benchmark because most physical work data is proprietary, and involves too much variability to study in a principled manner. Unlike structured settings such as kitchens and warehouses that have predictable lighting, geometry, and objects, field-facing tasks such as rooftop solar installation and vehicle repair are unconstrained and high variance. Here, we model real-world physical work in Blender across eleven environments, incorporating variants in both image acceptability and documentation. The acceptability task asks whether a particular view would be suitable proof-of-work and documentation asks whether an agent can understand the task-relevant state of the environment from that view. Across 26 vision-language models, we find that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations, such as whether a tape measure leans. Acceptability is harder to pin down because human raters disagree with each other. Measured against a scorer that agrees with the raters' majority about as often as a single rater does, the best models cannot be distinguished from a single non-author rater, at a cost within the range of a human pass. In the three environments where we can trace acceptability errors, most wrong rejections come from misreading the scene rather than the requirement. On the three documentation questions for which we have real images, model rankings on rendered frames transfer to real images.

Figures & tables

Figure 1: The eleven environments of RealityBench with one scorer-acceptable frame from each. Because each scene is rendered in Blender, each frame can be scored programmatically. Under each frame: a short form of the environment’s documentation question and the ground truth.

Panel of Figure 2 — Figure 2: Top: acceptability F1 against scorer-acceptability (left) and documentation score on scorer-acceptable frames (right) against the cost of the whole set. Color marks the developer and shade the generation (darker is newer; Gemma 4 26B-A4B is a lighter shade only to tell it from Gemma 4 31B) and shape is a rough size tier ( $\bigstar$ flagship, $\blacksquare$ large, $\bullet$ mid, $\blacktriangle$ small, $\blacklozenge$ 3–4B). The dotted line is the “always accept” baseline; the hollow markers at the left edge are the small open-weight models we served ourselves. routes each environment to the model that scored best on it (chosen on the other half of the frames) and takes the majority verdict and answer of all models. marks a single non-author rater’s F1 (95% CI 0.50–0.67) at the cost of one human pass over the set ($11–165). Bottom: documentation score on synthetic frames (scorer-acceptable; for severity, frames where the damage is readable) against real photographs on the three matched questions.

Figure 3: Documentation score by field (columns) and model (rows) on scorer-acceptable frames (tape leaning: all tape_gap frames). Columns are grouped by skill and sorted by the mean over all 26 models (top row). Rows are the models on a cost frontier of Figure 2 , the other models in the documentation top group and the three models we served, sorted by documentation score.

Environment	Scene	Documentation question	Acceptability terms	Acceptable
rail_attachment	roof	At this roof attachment, is a lag bolt present on the L-foot, and is flashing installed under it?	attachment in frame ( $\geq$ 90% of it); attachment unoccluded; bolt location unoccluded ( $\geq$ 50% of the spot where the lag bolt sits, or would sit, visible); fills the frame (bbox $\geq$ 2%); not a wide shot ( $\leq$ 1.8 m); not scored when the lag bolt or flashing is missing	0.12
wire_adequacy	roof	Is any wire or cable touching the roof surface?	a lead wire in frame ( $\geq$ 50% of it); that wire’s lowest point visible; that wire unoccluded ( $\geq$ 30% of it); wire visible to the eye: wire pixels $\geq$ 0.05% of the frame; wire stands out: edge contrast $\geq$ 0.03; close enough to judge contact (cam within 2.5 m of that wire’s low point)	0.39
tape_gap	roof	What number is on the tape where the module frame crosses it, what is your estimate of the true perpendicular module-to-roof gap in inches, and is the tape leaning rather than standing flush?	crossing in frame; crossing unoccluded; close enough to read 1/16in ticks ( $\leq$ 0.65 m); tape seated flush (tilt $\leq$ 5 ∘ ); photographed from the front of the blade; gray: a lean over 5 ∘ that shows as at most 5 ∘ in the image	0.17
slope_gauge	roof	What does the angle gauge read, in degrees, and is it sitting on the module (glass or frame) or on bare roof?	display in frame; display unoccluded; digits legible: $\geq$ 14 px digit height; display readable angle (view angle $\leq$ 65 ∘ ); gauge on the module, not bare roof; display not blown out ( $\leq$ 30% saturated); gauge aligned with the slope (\|yaw\| $\leq$ 15 ∘ , else it under-reads); array context: module glass visible in frame; module edge in view to judge alignment ( $\geq$ 300 px of frame edge visible); text stands out in the render (Weber contrast $\geq$ 0.2); gray: \|yaw\| 15–45 ∘	0.25
equipment_label	roof	What does it say on the equipment label (brand, model number and serial number) and what kind of equipment is it on?	label fully in frame; label unoccluded ( $\geq$ 95% of it); legible: $\geq$ 10 px per text row; barcode resolvable: $\geq$ 1 px per module; square-on enough to OCR (view angle $\leq$ 60 ∘ ); label not blown out ( $\leq$ 30% saturated); site context: some of the equipment/mounting in view; label lit: paper luminance $\geq$ 0.05; text stands out in the render (Weber contrast $\geq$ 0.3)	0.06
horizon_peak	roof	Is the horizon visible in this frame, and is it roughly level?	horizon in frame, in the middle 80% of the image; taken from the peak: the house’s own roof hides $\leq$ 10% of the horizon arc; skyline inside the frame: sky above it and the surroundings below it across $\geq$ 65% of the width; level (roll $\leq$ 10 ∘ ); own roof visible along the bottom of the frame ( $\geq$ 4% of the bottom quarter): proof it was taken from the peak; phone at or above the ridge line (camera height above the ridge $\geq$ 0 m)	0.19

Table 1: Environment, task and scorer summaries. Every capture task additionally requires sharp at target (defocus and motion blur at the target

\leq 5

px;

6

px for car_plate , car_view , ro_board , damage_view ). Acceptable is the fraction of the random field-pose frames of each environment that the scorer accepts. Documentation question is a summary of the task—not the prompt given to the model.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Primitive	Definition
in-frame / unoccluded fraction	Sample points on the target’s surface (vertices and face interiors—corner-only sampling under-reports thin objects); project each; cast a ray from the camera; the point is visible if the first hit is the target.
bbox fraction	Area of the target’s projected bounding box within the frame.
view angle	Angle between the viewing ray to the target and the target’s surface normal ( $0^{\circ}$ = square-on).
px per unit	Pixels spanned by a unit length at the target (text row height, character cell, inch, centimeter): the legibility proxy.
clearance	Signed distance from a wire’s surface (beveled mesh) to the roof plane; $\leq 0$ is contact.
apparent reading	For a tape, the blade point that projects onto the module’s front-bottom edge from the camera (§ 3 ).

Appendix

Table 2: Scoring primitives. The first group is computed in the scene after the camera is placed; the second reads the rendered frame (the target’s pixels are located by an object-index mask rendered from the same camera).

Environment	Requirement	Question	Keys
rail_attachment	The L-foot, its flashing and the lag bolt, close enough to judge the attachment. Reject wide array shots where the foot is a speck.	At this roof attachment, is a lag bolt present on the L-foot, and is flashing installed under it?	lag_present : bool; flashing_present : bool
tape_gap	Tape measure hooked at the roof surface and standing up to the module frame; the number where the frame crosses the tape must be legible, and the tape should be seated flush (perpendicular to the roof).	What number (inches) is on the tape where the module frame crosses it, what is your estimate of the true perpendicular module-to-roof gap in inches, and is the tape leaning rather than standing flush?	reading_in : number; gap_in : number; tilted : bool
wire_adequacy	Photograph the underside of each array close enough to judge the wiring: lead wires and trunk cable visible, so a reviewer can tell whether any wire touches the roof plane or has a hard bend. Reject photos that do not clearly show the wiring under the modules.	Is any wire or cable touching the roof surface (the shingles)?	touching : bool
slope_gauge	A digital angle gauge placed directly on the module glass or frame, aligned with the slope, showing its degree reading; the display must be legible with the module it sits on visible in the photo. Reject a gauge placed on bare roof instead of the module, and a shot where the display is unreadable or washed out.	What does the angle gauge read, in degrees, and is it sitting on the module (glass or frame) or on bare roof?	reading_deg : number; on_module : bool
equipment_label	Close-up of the equipment data label — on the back of an installed module, on a microinverter under the array, or on the inverter — with the model number, the serial number and its barcode all legible, square-on enough to read. Some of the equipment or mounting should be in view so it is clearly a label on site, not a stock image. Reject a shot where the small text is a blur, the barcode is cut off or washed out by glare.	Read the equipment data label: brand, model number and serial number, and say what the label is on.	brand : string; model : string; serial : string; on : module/microinverter/inverter
horizon_peak	A set of photos taken from the roof peak for the shading analysis: turning in place, each photo shows the skyline in that direction, with sky above it and the surroundings below it, and a little of this house’s own roof along the bottom as proof of where it was taken. Together the photos should go most of the way around: four or more directions with no large gap. Reject photos taken from below the peak where the roof blocks the view, photos with no roof in them at all, and sets that leave a large gap.	Is the horizon visible in this frame and roughly level, and does this house’s own roof hide part of the far view?	horizon_visible : bool; level : bool; own_roof_blocks : bool

Appendix

Table 3: The per-environment parts of the prompt.

Model	first	last	alone	last	alone ( $p$ )	rerun	first	alone ( $p$ )
	defect claims (of 150)			paired change in claims vs. first			rejects
Gemini 3.8 Flash ‡	82	85	83	4/7	7/8 (1.00)	8/5	96	92 (.60)
Gemini 3.5 Flash	94	97	88	7/10	12/6 (.24)	7/6	104	108 (.50)
Gemini 3.1 Flash Lite	125	103	66	27/5	60/1 ( $<$ .001)	10/6	125	133 (.13)
Gemma 4 31B ‡	92	96	78	5/9	18/4 (.004)	8/7	101	107 (.36)
GPT-6 Astra ‡	58	53	56	8/3	9/7 (.80)	9/6	76	73 (.65)

Appendix

Table 4: Verdict-order ablation: 150 items from tape_gap , slope_gauge and rail_attachment (25 per environment from the random frames and 25 from the positive stratum), re-asked under four prompts: the benchmark’s ( first : the verdict is the first key), verdict last , documentation alone (no verdict asked) and the verdict alone. A defect claim is the documentation boolean in the direction the requirement rejects (tape tilted; gauge not on the module; lag bolt or flashing absent). Paired columns count items whose claim is present under first only / under the other prompt only, with the exact McNemar

p

; rerun compares first with the same prompt’s record in the main run (the noise floor). Rejects : frames rejected under first and when the verdict is asked alone. ‡ on a cost frontier of Fig. 2 . One sample per prompt; provider-default thinking.

Model	FA (%)	miss (%)	P	R	F1 [95% CI]	doc. [95% CI]	gram. (%)	cost ($)	think
Gemini 3.8 Flash	11	38	0.48	0.62	0.54 [0.48, 0.59]	0.84 [0.82, 0.86]	100	7.16	1083
Gemini 3.5 Flash	13	34	0.49	0.66	0.56 [0.51, 0.62]	0.80 [0.78, 0.83]	97	7.36	1114
Gemini 3.1 Flash Lite	16	42	0.39	0.58	0.47 [0.41, 0.52]	0.73 [0.70, 0.75]	98	0.73	0
Gemma 4 31B	15	37	0.44	0.63	0.52 [0.46, 0.57]	0.79 [0.77, 0.81]	100	0.70	677
Gemma 4 26B-A4B	17	36	0.39	0.64	0.48 [0.42, 0.53]	0.77 [0.75, 0.79]	99	1.24	1325
GPT-6 Astra	12	32	0.55	0.68	0.61 [0.56, 0.66]	0.83 [0.81, 0.85]	100	30.05	165

Appendix

Table 5: Detailed statistics. False-accept rate (random frames the scorer rejects that the model accepts) and miss rate (scorer-acceptable frames, random and positive strata, that the model rejects); acceptability precision, recall and F1 against the scorer (precision on the 934 random frames, recall including the positive strata), with a 95% interval on F1; the documentation score on scorer-acceptable frames (mean per-field score) with its 95% interval; the fraction of answers in grammar; the cost of the set at the price in Table 7 (thinking at the output rate); mean thinking tokens per item. Below the rule: Gemma 4 E4B, Qwen3.5-4B and LFM2.5-VL-3B, which we served ourselves; their developers list no per-token price, so they have no cost. They sit below the same rule in the per-environment tables and are in every comparison except the cost frontier, which needs a price. Intervals: stratified bootstrap over items, 1,000 resamples, the random frames and each positive stratum resampled separately. Replies cut by the output cap: Qwen 3.8 27B 6, Qwen 3.5 397B-A17B 10, Kimi K3 6, DeepSeek V4.1 Flash 71, GLM-5.3 Flash 14.

GPT-6 Astra

Claude Opus 5.5

GPT-5.6 Sol

GLM-5.3 Flash

Gemini 3.5 Flash

Claude Fable 5.1

Claude Opus 5

Qwen 3.8 Max

GPT-6 Sol

Muse Spark 1.3

Qwen 3.5 397B-A17B

GPT-5.6 Terra

Qwen 3.8 27B

Grok 4.6

Gemini 3.8 Flash

Kimi K3

GPT-5.6 Luna

Gemma 4 31B

DeepSeek V4.1 Flash

Gemma 4 26B-A4B

Claude Sonnet 5

Gemini 3.1 Flash Lite

LFM2.5-VL-3B §

GPT-6 Luna

Gemma 4 E4B §

GPT-6 Astra

0.61

Claude Opus 5.5

0.58

GPT-5.6 Sol

0.57

GLM-5.3 Flash

0.57

Gemini 3.5 Flash

0.56

Claude Fable 5.1

0.56

Appendix

Table 6: Paired differences between models on the same 1,378 frames, acceptability F1 (top) and documentation score (bottom). Models are sorted by the metric (its value beside the name); a cell is the column model’s value minus the row model’s, both as printed, in hundredths. Bold : the paired 95% interval of the unrounded difference excludes zero; gray : it does not (160 of 325 pairs for F1, 76 for documentation). § no developer list price (served by us). Intervals: a stratified bootstrap that resamples the items once and scores both models on the same resample (2000 resamples).

Model	input	output	source	read
Gemini 3.8 Flash	0.75	3.75	Google a	2026-09-13
Gemini 3.5 Flash	0.75	3.75	Google a	2026-09-13
Gemini 3.1 Flash Lite	0.30	2.50	Google a	2026-09-13
Gemma 4 31B †	0.15	0.60	Google Cloud (Vertex AI) b	2026-09-20
Gemma 4 26B-A4B	0.15	0.60	Google Cloud (Vertex AI) b	2026-09-20
GPT-6 Astra	10.00	50.00	OpenAI c	2026-09-15

Appendix

Table 7: Prices behind the costs (USD per 1M tokens; thinking priced at the output rate), where the price was read and on which day. Every price is the list price of the model’s developer on its own API, whichever host served our calls (Alibaba Cloud prices differ by deployment region; we use International, the only one outside China offered for all three Qwen models). DeepSeek is the only developer that prices by time of day, off-peak at half of peak; we quote the peak rate. † Google lists a per-token price for Gemma 4 26B only; we use it for Gemma 4 31B as well.

Model	breaker	plate	car view	damage	equip.	horizon	rail	RO board	slope	tape	wire
Gemini 3.8 Flash	0.45	–	0.38	0.14	0.41	0.54	0.45	0.73	0.62	0.40	0.24
Gemini 3.5 Flash	0.41	–	0.53	0.13	0.39	0.65	0.60	0.59	0.71	0.21	0.47
Gemini 3.1 Flash Lite	0.34	–	0.49	0.14	0.28	0.55	0.31	0.62	0.37	0.37	0.43
Gemma 4 31B	0.46	–	0.39	0.08	0.37	0.60	0.36	0.66	0.57	0.16	0.48
Gemma 4 26B-A4B	0.38	–	0.25	0.25	0.35	0.54	0.55	0.59	0.50	0.20	0.34
GPT-6 Astra	0.66	–	0.61	0.26	0.62	0.63	0.48	0.73	0.63	0.72	0.55

Appendix

Table 8: Acceptability F1 per environment: positives are the frames the scorer accepts; precision on the random frames, recall on the random frames plus the positive stratum.

Model	breaker	plate	car view	damage	equip.	horizon	rail	RO board	slope	tape	wire
scorer-acceptable (%)	5	21	26	28	6	19	12	18	27	18	39
Gemini 3.8 Flash	9	–	5	5	18	11	5	15	33	3	7
Gemini 3.5 Flash	15	–	8	6	20	12	3	29	20	1	11
Gemini 3.1 Flash Lite	20	–	12	3	32	21	11	26	8	4	20
Gemma 4 31B	9	–	16	2	21	22	3	22	33	3	20
Gemma 4 26B-A4B	17	–	8	18	23	26	8	29	33	1	8

Appendix

Table 9: False-accept rate (%): frames the scorer rejects that the model accepts, on the 100 random field-pose frames per environment.

Model	in $/M	out $/M	prompt	output	thinking	prompt share	set ($)	rank by price	rank by set
GLM-5.3 Flash	0.15	0.50	1,289	37	1,460	21%	1.30	1	6
GPT-6 Luna	0.10	0.50	1,020	41	751	20%	0.69	2	1
Gemma 4 26B-A4B	0.15	0.60	527	43	1,325	9%	1.24	3	5
Gemma 4 31B	0.15	0.60	527	43	677	15%	0.70	4	2
DeepSeek V4.1 Flash	0.30	1.20	786	143	4,903	4%	8.67	5	12
GPT-5.6 Luna	0.20	1.20	1,170	39	482	27%	1.18	6	4

Appendix

Table 11: Token price against set cost, models with a list price, sorted by output price. Prompt, visible output and thinking tokens are per item (means over the 1,378 items; Anthropic reports one output count, so thinking is that count minus the tokens of the reply we received, counted with the provider’s tokenizer). Prompt share is the fraction of the set’s cost that is prompt tokens; the rest is output and thinking at the output rate. The last two columns rank the models by output price and by the cost of the set.

Environment	Defect reported	answers	reports defect	rejects if reported	rejected	of those, defect reported
tape_gap	tape is tilted	1,688	75%	99.6%	1,296	97%
slope_gauge	gauge is not on the module	1,950	19%	96.0%	488	74%
rail_attachment	lag bolt or flashing missing	1,273	35%	88.8%	626	63%

Appendix

Table 12: Misreading and rejection on scorer-acceptable frames, where the property the requirement checks always holds, so the right verdict is accept and reporting the defect is a misreading. Pooled over the 26 models: model answers, how often the defect is reported, how often a report comes with a rejection in the same reply, how many frames are rejected, and what share of those rejections come with the defect reported. rail_attachment frames with a missing part are not scored for acceptability.

Set	Frames	Raters	Ratings/frame	Agree	$\alpha$	Scorer (held-out)
breaker_label	30	5	4.1	0.94	0.79	0.91 (11)
equipment_label	29	5	4.3	0.90	0.58	0.91 (11)
ro_board	30	5	4.1	0.98	0.86	1.00 (8)
rail_attachment	30	5	4.1	0.88	0.53	0.80 (5)
car_plate †	28	5	4.2	0.80	0.54	–
wire_adequacy	30	5	4.3	0.78	0.56	0.91 (11)

Appendix

Table 13: Human acceptability ratings. Frames : random frames with at least two ratings (the frames used to adjust the scorer and the held-out frames); agree : raw agreement over all rater pairs on a frame;

\alpha

: Krippendorff’s alpha (nominal), which corrects for chance and allows raters to skip frames. Scorer : the frozen scorer’s agreement with the raters’ majority on the held-out frames with at least three ratings, ties excluded (all environments: 82% of 95). † acceptability not scored (the raters split on the requirement); frames the frozen scorer leaves unscored are not in the scorer column.

Model	tape: synthetic	tape: real	face: synthetic	face: real	severity: synthetic	severity: real
Claude Opus 5.5	0.36 [0.24, 0.48]	0.35 [0.26, 0.47]	1.00 [1.00, 1.00]	0.96 [0.91, 0.99]	0.63 [0.51, 0.73]	0.56 [0.44, 0.68]
GPT-6 Sol	0.36 [0.24, 0.48]	0.33 [0.25, 0.41]	0.87 [0.74, 1.00]	0.89 [0.83, 0.94]	0.51 [0.39, 0.63]	0.47 [0.36, 0.61]
GPT-6 Astra	0.30 [0.19, 0.40]	0.44 [0.34, 0.53]	0.96 [0.87, 1.00]	0.95 [0.91, 0.98]	0.42 [0.30, 0.54]	0.42 [0.31, 0.54]
GPT-5.6 Sol	0.27 [0.16, 0.37]	0.33 [0.25, 0.41]	0.74 [0.57, 0.91]	0.79 [0.72, 0.86]	0.46 [0.34, 0.58]	0.42 [0.31, 0.54]
Gemini 3.8 Flash	0.18 [0.09, 0.27]	0.21 [0.13, 0.29]	0.96 [0.87, 1.00]	0.91 [0.85, 0.96]	0.37 [0.27, 0.49]	0.56 [0.42, 0.68]
GPT-5.6 Terra	0.18 [0.09, 0.27]	0.10 [0.05, 0.16]	0.87 [0.70, 1.00]	0.78 [0.70, 0.85]	0.54 [0.42, 0.66]	0.54 [0.41, 0.68]

Appendix

Table 14: Synthetic against real, per model, three matched questions. Synthetic: the share of scorer-acceptable synthetic frames whose field is right ( tape_gap reading within

\nicefrac{{1}}{{8}}

in; car_view face; damage_view severity on the frames where the scorer finds the damage readable: sharp, visible and resolved, framing not required). Real: our tape photos read within

\nicefrac{{1}}{{8}}

in (133 readings on 84 photos of 35 items); the face named is one the labels list as visible (117 side-determinable frames); severity on the 59 real damaged-panel photos an author rated on the prompt’s area scale after the model runs, blind to the models’ answers (4 rated can’t tell are left out). Brackets: 95% bootstrap intervals over frames, or over items for the tape photos. Across the 22 models, synthetic against real (Spearman, Pearson): tape

\rho=0.68

(

p=0.001

r=0.82

(

p<0.001

); face

\rho=0.79

(

p<0.001

r=0.85

(

p<0.001

); severity

\rho=0.63

(

p=0.002

r=0.58

(

p=0.005

). Not run on the real photos: Qwen 3.8 Max, Muse Spark 1.3, Kimi K3, DeepSeek V4.1 Flash. Tape replies cut by the output cap, left out: GLM-5.3 Flash 54, Qwen 3.8 27B 15, Qwen 3.5 397B-A17B 8.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 999 papers

Mar 4, 2026

Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild

While Vision-Language Models (VLMs) achieve near-perfect scores on digital document benchmarks like OmniDocBench, their performance in the unpredictable physical world remains largely unknown due to the lack of controlled yet realistic evaluations. We introduce Real5-OmniDocBench, the first benchmark that performs a full-scale, one-to-one physical reconstruction of the entire OmniDocBench v1.5 (1,355 images) across five critical real-world scenarios: Scanning, Warping, Screen-Photography, Illumination, and Skew. Unlike prior benchmark that either lack digital correspondence or employ partial sampling, our complete ground-truth mapping enables, for the first time, rigorous factor-wise attribution of performance degradation-allowing us to pinpoint whether failures stem from geometric distortions, optical artifacts, or model limitations. Our benchmark establishes a challenging new standard for the community, demonstrating that the 'reality gap' in document parsing is far from closed, and provides a diagnostic tool to guide the development of truly resilient document intelligence.

Changda Zhou, Ziyue Gao, Xueqing Wang +4

PaddlePaddle Team, Baidu Inc. · The Hong Kong University of Science and Technology (Guangzhou)

VLM Evaluation Document Parsing Benchmarks Document Parsing Document Image Understanding

Ask PDF

Mar 25, 2026

How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning

The physical world is not merely visual; it is governed by rigorous structural and procedural constraints. Yet, the evaluation of vision-language models (VLMs) remains heavily skewed toward perceptual realism, prioritizing the generation of visually plausible 3D layouts, shapes, and appearances. Current benchmarks rarely test whether models grasp the step-by-step processes and physical dependencies required to actually build these artifacts, a capability essential for automating design-to-construction pipelines. To address this, we introduce DreamHouse, a novel benchmark for physical generative reasoning: the capacity to synthesize artifacts that concurrently satisfy geometric, structural, constructability, and code-compliance constraints. We ground this benchmark in residential timber-frame construction, a domain with fully codified engineering standards and objectively verifiable correctness. We curate over 26,000 structures spanning 13 architectural styles, ach verified to construction-document standards (LOD 350) and develop a deterministic 10-test structural validation framework. Unlike static benchmarks that assess only final outputs, DreamHouse supports iterative agentic interaction. Models observe intermediate build states, generate construction actions, and receive structured environmental feedback, enabling a fine-grained evaluation of planning, structural reasoning, and self-correction. Extensive experiments with state-of-the-art VLMs reveal substantial capability gaps that are largely invisible on existing leaderboards. These findings establish physical validity as a critical evaluation axis orthogonal to visual realism, highlighting physical generative reasoning as a distinct and underdeveloped frontier in multimodal intelligence. Available at https://luluyuyuyang.github.io/dreamhouse

Luyu Yang, Yutong Dai, An Yan +3

Salesforce AI Research

VLM Evaluation

Ask PDF

Jun 28, 2026

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulation benchmarks. These gains are commonly interpreted as evidence that semantic representations learned from internet-scale data transfer to physical execution generalization. This position paper argues that the assumption underlying this interpretation -- that semantic generalization is sufficient to support physical action decisions -- has not been independently verified and cannot be tested under current evaluation protocols. We support this claim by decomposing VLA policies into semantic mapping and physical action decision, and showing that task success rate -- the dominant evaluation metric -- cannot distinguish between these two sources of capability. As a result, improvements in benchmark performance are consistent with multiple competing explanations, including semantic matching, distributional overlap, and genuine physical generalization. We further argue that this identifiability gap has been reinforced through narrative drift, whereby successive systems inherit and strengthen prior interpretations of performance gains without isolating the underlying causal mechanism. To address this limitation, we propose a research direction based on evaluation designs that introduce controlled variation to separately measure semantic and physical generalization. Such designs make it possible to causally attribute performance without requiring access to model internals, and to empirically assess the role of VLM backbones as semantic interfaces rather than implicit sources of physical competence. Our goal is not to refute the role of VLMs in robotics, but to clarify the conditions under which claims of physical generalization can be meaningfully evaluated.

Taozhao Chen, Ian Manchester, Huaming Chen

School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia

VLM Evaluation Physical Reasoning Generalization in Robotic Manipulation Robotics Vision-Language-Action Models Robot Manipulation Benchmarks+1

Ask PDF

Abstract

VLM Evaluation

Explore similar work

Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild

How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning