PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image

Sneha Paul; Guile Wu; Bingbing Liu; Dongfeng Bai

cs.CVSep 16, 2026

PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image

Authors: Sneha Paul, Guile Wu, Bingbing Liu, Dongfeng Bai

Organizations: Huawei Noah’s Ark Lab · Concordia University · Foundation Model Department, Huawei

Abstract

Physical properties, such as friction, hardness, stiffness, and density, govern how robots should grasp, manipulate and interact with objects, yet estimating these properties from RGB images remains challenging. Existing methods typically employ per-object reconstruction augmented with physical properties or directly query vision-language models at test time, which results in substantial computational overhead that limits their applicability. In this work, we present PhysVGGT, a feed-forward model that predicts dense maps of friction coefficient, Shore hardness, Young's modulus, and density, together with object-level mass, from a single RGB image in one forward pass. The key idea of PhysVGGT is to formulate physical property estimation as a dense per-pixel prediction problem and employ a visual geometry transformer to extract geometry-aware tokens from the input image followed by a dense prediction branch for estimating local physical properties and a global prediction branch for estimating object-level mass. In addition, we introduce a scalable pseudo-label generation pipeline that enables large-scale weakly supervised training for dense physical property prediction, substantially reducing the need for expensive direct physical measurements. Extensive experiments show that PhysVGGT achieves state-of-the-art performance on the ABO-500 dataset and generalizes effectively to the out-of-distribution NeRF2Physics dataset. Moreover, PhysVGGT eliminates the need for per-object reconstruction and test-time optimization, achieving an inference latency of only 0.13s per image, making it $27\times$ faster than the previous state of the art.

Figures & tables

Figure 2: The overall framework of PhysVGGT.

Figure 3: The overview of the proposed pseudo-label generation pipeline.

Method	ADE (kg) $\downarrow$	ALDE $\downarrow$	MnRE $\uparrow$	Paradigm
NeRF2Physics [ 29 ]	12.73	0.736	0.564	per-scene NeRF + VLM
PUGS [ 21 ]	9.46	0.661	0.576	per-scene 3DGS
Phys4DGen ⋆ [ 12 ]	9.96	0.664	0.566	3DGS + MLLM
VoMP [ 5 ]	8.43	0.631	0.576	feed-forward
PGVME [ 11 ]	8.32	0.998	–	feed-forward
AURA [ 9 ]	8.33	0.582	0.634	feed-forward + VLM

Table 1: Mass estimation on ABO-500 (100 test objects). Baselines quoted from the VoMP protocol [ 5 ] and the respective papers.

\downarrow

\uparrow

denote lower/higher is better. Best in bold . ⋆ Taken from existing work for reference purpose.

	Friction $\mu$				Shore hardness
Method	ADE $\downarrow$	ALDE $\downarrow$	APE $\downarrow$	MnRE $\uparrow$	ADE $\downarrow$	ALDE $\downarrow$	APE $\downarrow$	MnRE $\uparrow$
NeRF2Physics [ 29 ]	0.217	0.459	0.446	0.661	66.21	0.672	1.105	0.546
PhysGS [ 3 ]	0.218	0.462	0.438	0.647	34.82	0.456	0.532	0.671
GPT-4V [ 1 ]	0.209	0.430	0.549	0.692	32.75	0.330	0.304	0.758
CLIP [ 18 ]	0.222	0.455	0.602	0.654	32.86	0.294	0.266	0.774
PhysVGGT (ours)	0.102	0.206	0.225	0.831	7.05	0.103	0.111	0.908

Table 2: Friction and Shore hardness on the NeRF2Physics 13-scene real benchmark. Reproduced baselines under the protocol of Zhai et al. [29] .

Method	Latency (s) $\downarrow$	Test-time dependency
NeRF2Physics [ 29 ]	1454.6	NeRF optimization + VLM
PUGS [ 21 ]	1058.3	3DGS optimization
Pixie [ 10 ]	201.6	Scene optimization
Phys4DGen [ 12 ]	51.7	3DGS + MLLM
AURA [ 9 ]	4.7	CLIP + VLM
VoMP [ 5 ]	3.6	DINOv2 + voxelization

Table 3: Inference efficiency. Per-item inference latency. Baseline latencies are reported from [ 5 , 9 ] and the corresponding papers. Lower is better.

Figure 4: Dense physical property predictions. From the single input view (left), PhysVGGT predicts per-pixel friction $\mu$ , Shore hardness, Young’s modulus $E$ and density in one forward pass.

Configuration	Friction MnRE $\uparrow$	Hardness MnRE $\uparrow$
Full (trunk + coupling + shared dec. + combined loss)	0.831	0.908
$-$ w/o cross-property coupling	0.805	0.869
$-$ w/o shared decoder ( $\to$ 4 independent)	0.810	0.860
$-$ w/o combined loss ( $\to$ plain L1)	0.750	0.784

Table 4: Component-wise ablation. N2P MnRE on a fixed

2000

-object subset using identical training settings, averaged over four epochs.

Coupling	Friction	Hardness	Mass
	MnRE $\uparrow$	ADE $\downarrow$	MnRE $\uparrow$
w/o CPC	0.802	0 9.09	0.474
Attention, 1-layer	0.822	10.79	0.468
Attention, 2-layer (default)	0.831	7.05	0.655
Cross-stitch	0.801	10.16	0.488
FiLM	0.790	13.50	0.469

Table 5: Cross-property coupling ablation. Comparison of different feature interaction mechanisms for modeling dependencies among physical properties.

Figure 5: Out-of-distribution scenes on RUGD (upper two examples) and RELLIS-3D (lower two examples). Ground-truth semantic regions coloured by our predicted friction and Young’s modulus. The model differentiates unseen materials.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
backbone (frozen)	StreamVGGT, 1.26 B params
backbone token dim $C$	2048
DPT feature width	128
UNet head width $h$	64 ( $h,2h,4h,8h$ stages)
dense properties	4 ( $\mu$ , $H$ , $E$ , $\rho$ )
coupling	attention, 2 layers

Appendix

Table 6: Hyperparameters of the deployed PhysVGGT. The frozen StreamVGGT backbone (1.26 B parameters) receives no gradient; all listed settings apply to the 32.4 M-parameter physics heads.

Figure 6: Detailed architectural design of PhysVGGT’s cross-property coupling (CPC) module, shared decoder, and property specific head.

	Friction $\mu$		Shore hardness
Head	MnRE $\uparrow$	ADE $\downarrow$	MnRE $\uparrow$	ADE $\downarrow$
Pooled-token	0.742	0.148	0.844	12.52
Dense trunk (ours)	0.831	0.102	0.908	7.05

Appendix

Table 7: Spatial dense trunk vs. pooled-token regression. The per-pixel trunk is essential for the spatial properties.

	Friction $\mu$		Shore hardness
Backbone	MnRE $\uparrow$	ALDE $\downarrow$	MnRE $\uparrow$	ALDE $\downarrow$
StreamVGGT (ours)	0.831	0.206	0.908	0.103
VGGT-1B [ 23 ]	0.819	0.292	0.875	0.159
DINOv2-L [ 16 ]	0.802	0.252	0.834	0.173

Appendix

Table 8: Choice of geometry backbone under identical head and training protocol.

StreamVGGT	Train.	Friction	Hardness	Mass
freezing	params	MnRE $\uparrow$	MnRE $\uparrow$	MnRE $\uparrow$
Fully frozen (ours)	32 M	0.831	0.908	0.655
Unfreeze global-attn	334 M	0.695	0.814	0.517

Appendix

Table 9: Backbone freezing strategy. Freezing the entire StreamVGGT backbone vs. fine-tuning its global-attention blocks.

Decoder	Dec.	Total	Fric.	Hard.	Hard.
	params	train.	MnRE $\uparrow$	ADE $\downarrow$	MnRE $\uparrow$
original (4 decoders)	12.1 M	40.9 M	0.808	8.62	0.887
slim (4 half-width)	4.3 M	33.2 M	0.808	7.48	0.899
shared + heads (ours)	3.0 M	31.9 M	0.831	7.05	0.908

Appendix

Table 10: Decoder ablation. Decoder architectures for multi-task physical property prediction.

Loss	Friction	Hardness	Mass
	MnRE $\uparrow$	MnRE $\uparrow$	ADE $\downarrow$
L1 baseline	0.684	0.826	12.39
+ foreground mask	0.759	0.856	10.72
+ pairwise ranking	0.811	0.851	11.49
+ all (mask, Charb., rank)	0.831	0.908	4.73

Appendix

Table 11: Loss-component ablation.

	Mass (ABO-500)			OOD (N2P)
Train. obj.	ADE (kg) $\downarrow$	ALDE $\downarrow$	MnRE $\uparrow$	$\mu$ MnRE $\uparrow$	$H$ MnRE $\uparrow$
500	5.62	0.633	0.651	0.787	0.824
2000	5.26	0.658	0.641	0.794	0.860
7225	4.73	0.574	0.655	0.831	0.908

Appendix

Table 12: Data scaling. Mass prediction on ABO-500 and OOD friction/hardness on N2P. Performance improves consistently with training-set size.

Mass branch	ADE (kg) $\downarrow$	ALDE $\downarrow$	APE $\downarrow$	MnRE $\uparrow$
dense, $\bar{\rho}\!\times\!V_{\text{shell}}\!\times\!\kappa$	10.3	0.896	1.043	0.517
global, direct mass head	4.73	0.574	0.787	0.655

Appendix

Table 13: Mass branch: dense (

\rho\!\times\!V

) versus global (direct head) on ABO-500. Both geometry tokens come from the same frozen geometry backbone and the same trained model; only the path from features to mass differs.

Supervision	Friction	Hardness	Mass (dense)
Raw	0.813	0.856	0.450
Refined	0.831	0.908	0.655

Appendix

Table 14: Refined vs. raw supervision comparison.

Hyperparameter	Setting	$\mu$ MnRE $\uparrow$	$H$ MnRE $\uparrow$
head width $h$	32	0.792	0.855
	48	0.839	0.877
	64 $\star$	0.829	0.877
	96	0.822	0.859
$\lambda_{\mathrm{friction}}$	1	0.807	0.864
	2	0.770	0.868

Appendix

Table 15: Hyperparameter sensitivity. N2P MnRE (

\uparrow

) under one-at-a-time sweeps from the final configuration (

\star

Method	Zero-shot	ALDE $\downarrow$	MnRE $\uparrow$
Xception $k$ -NN	No	1.454	0.341
Xception $k$ -NN (SIM)	No	0.781	0.534
Pure CNN	No	0.680	0.575
CNN with ALDE	No	0.666	0.573
No Geometry	No	0.520	0.638
Shape-aware	No	0.465	0.675

Appendix

Table 16: Zero-shot mass prediction on image2mass. All baselines are trained on image2mass [ 22 ] , whereas PhysVGGT is trained only on ABO and evaluated without fine-tuning.

\dagger

: human estimates provided with the dataset.

Panel of Figure 7 — Figure 7: Language-guided physical reasoning. Top: multi-round question answering about object- and region-level physical properties. Bottom: single-shot physical-composition captioning. PhysVGGT supplies dense friction, Shore hardness, Young’s modulus, density, and object-level mass predictions; SAM2 supplies spatial regions; and Qwen3-VL interprets the original and segmented images and generates the responses. Material names are estimated by matching each predicted property tuple $(\mu,H,E,\rho)$ to a reference material database. All numerical physical claims are derived from PhysVGGT predictions.

Figure 8: Grasp-force planning from predicted physics. From one image PhysVGGT predicts per-pixel friction $\mu$ (middle) and object mass, yielding a dense minimum grip-force map $F_{n}=mg/(2\mu)$ (right; green $=$ secure/low force, red $=$ slippery/high force) and a best contact region (marker). On both objects, the model marks the grippy top surface as the grasp region while the slick legs (wooden; thin metal) turn red ( $>\!250$ N).

Materials Property Prediction

Materials property prediction uses data and physical models to estimate how composition, structure, and processing determine a material’s behavior.

See all 173 papers

SiPhy: Single-Image Physical Property Reasoning
Jul 24, 2026Hoang Le, Joonwoo Kwon, Elkhan Ismayilzada +2
Michigan State University, East Lansing MI 48864, USA · Independent Researcher
SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents
Mar 25, 2026Rocktim Jyoti Das, Dinesh ManochaMaterials Property Prediction
University of Maryland, College Park
PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models
Mar 17, 2026Hisayuki Yokomizo, Taiki Miyanishi, Yan Gang +3VLM Reasoning VLMs for Robotics
The University of Tokyo · Institute of Science Tokyo · National Institute of Informatics

Jul 24, 2026J/K move · Enter open · S save

SiPhy: Single-Image Physical Property Reasoning

Hoang Le, Joonwoo Kwon, Elkhan Ismayilzada, Yufei Zhang+1

Michigan State University, East Lansing MI 48864, USA · Independent Researcher

Open paper AskPDF arXiv

Inferring physical properties such as mass, stiffness, and elasticity from a single image is essential for simulation and embodied AI, yet most existing approaches rely on multi-view reconstruction or physics-based supervision. We introduce SiPhy, a unified framework for single-image physical property reasoning that aligns 3D-aware visual cues, depth with language-based material knowledge. From one RGB image, SiPhy samples pseudo-voxel points, extracts CLIP features, and grounds them to material candidates proposed by a VLM. A part-based contrastive aggregator enforces region consistency, while a heaviness-aware refinement improves thickness and volume estimation for dense objects. Across ABO-500, MVImgNet-100, and PhysXNet-100, SiPhy achieves state-of-the-art single-image performance, surpassing multi-view reconstruction methods by improving mass MnRE by up to 93% (vs. PUGS), reducing density MAE by 35.5% (vs. NeRF2Physics), and lowering Young's modulus error by 23.5%. We further validate SiPhy on real hand-object interaction datasets, demonstrating its potential as a data annotation engine for physical understanding from single-view imagery.

Abstract

Materials Property Prediction

Explore similar work

SiPhy: Single-Image Physical Property Reasoning

SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents

PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models