PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image

Sneha Paul; Guile Wu; Bingbing Liu; Dongfeng Bai

cs.CVSep 16, 2026

PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image

Authors: Sneha Paul, Guile Wu, Bingbing Liu, Dongfeng Bai

Organizations: Huawei Noah’s Ark Lab · Concordia University · Foundation Model Department, Huawei

Abstract

Physical properties, such as friction, hardness, stiffness, and density, govern how robots should grasp, manipulate and interact with objects, yet estimating these properties from RGB images remains challenging. Existing methods typically employ per-object reconstruction augmented with physical properties or directly query vision-language models at test time, which results in substantial computational overhead that limits their applicability. In this work, we present PhysVGGT, a feed-forward model that predicts dense maps of friction coefficient, Shore hardness, Young's modulus, and density, together with object-level mass, from a single RGB image in one forward pass. The key idea of PhysVGGT is to formulate physical property estimation as a dense per-pixel prediction problem and employ a visual geometry transformer to extract geometry-aware tokens from the input image followed by a dense prediction branch for estimating local physical properties and a global prediction branch for estimating object-level mass. In addition, we introduce a scalable pseudo-label generation pipeline that enables large-scale weakly supervised training for dense physical property prediction, substantially reducing the need for expensive direct physical measurements. Extensive experiments show that PhysVGGT achieves state-of-the-art performance on the ABO-500 dataset and generalizes effectively to the out-of-distribution NeRF2Physics dataset. Moreover, PhysVGGT eliminates the need for per-object reconstruction and test-time optimization, achieving an inference latency of only 0.13s per image, making it $27\times$ faster than the previous state of the art.

Figures & tables

Figure 2: The overall framework of PhysVGGT.

Figure 3: The overview of the proposed pseudo-label generation pipeline.

Method	ADE (kg) $\downarrow$	ALDE $\downarrow$	MnRE $\uparrow$	Paradigm
NeRF2Physics [ 29 ]	12.73	0.736	0.564	per-scene NeRF + VLM
PUGS [ 21 ]	9.46	0.661	0.576	per-scene 3DGS
Phys4DGen ⋆ [ 12 ]	9.96	0.664	0.566	3DGS + MLLM
VoMP [ 5 ]	8.43	0.631	0.576	feed-forward
PGVME [ 11 ]	8.32	0.998	–	feed-forward
AURA [ 9 ]	8.33	0.582	0.634	feed-forward + VLM

Table 1: Mass estimation on ABO-500 (100 test objects). Baselines quoted from the VoMP protocol [ 5 ] and the respective papers.

\downarrow

\uparrow

denote lower/higher is better. Best in bold . ⋆ Taken from existing work for reference purpose.

	Friction $\mu$				Shore hardness
Method	ADE $\downarrow$	ALDE $\downarrow$	APE $\downarrow$	MnRE $\uparrow$	ADE $\downarrow$	ALDE $\downarrow$	APE $\downarrow$	MnRE $\uparrow$
NeRF2Physics [ 29 ]	0.217	0.459	0.446	0.661	66.21	0.672	1.105	0.546
PhysGS [ 3 ]	0.218	0.462	0.438	0.647	34.82	0.456	0.532	0.671
GPT-4V [ 1 ]	0.209	0.430	0.549	0.692	32.75	0.330	0.304	0.758
CLIP [ 18 ]	0.222	0.455	0.602	0.654	32.86	0.294	0.266	0.774
PhysVGGT (ours)	0.102	0.206	0.225	0.831	7.05	0.103	0.111	0.908

Table 2: Friction and Shore hardness on the NeRF2Physics 13-scene real benchmark. Reproduced baselines under the protocol of Zhai et al. [29] .

Method	Latency (s) $\downarrow$	Test-time dependency
NeRF2Physics [ 29 ]	1454.6	NeRF optimization + VLM
PUGS [ 21 ]	1058.3	3DGS optimization
Pixie [ 10 ]	201.6	Scene optimization
Phys4DGen [ 12 ]	51.7	3DGS + MLLM
AURA [ 9 ]	4.7	CLIP + VLM
VoMP [ 5 ]	3.6	DINOv2 + voxelization

Table 3: Inference efficiency. Per-item inference latency. Baseline latencies are reported from [ 5 , 9 ] and the corresponding papers. Lower is better.

Figure 4: Dense physical property predictions. From the single input view (left), PhysVGGT predicts per-pixel friction $\mu$ , Shore hardness, Young’s modulus $E$ and density in one forward pass.

Configuration	Friction MnRE $\uparrow$	Hardness MnRE $\uparrow$
Full (trunk + coupling + shared dec. + combined loss)	0.831	0.908
$-$ w/o cross-property coupling	0.805	0.869
$-$ w/o shared decoder ( $\to$ 4 independent)	0.810	0.860
$-$ w/o combined loss ( $\to$ plain L1)	0.750	0.784

Table 4: Component-wise ablation. N2P MnRE on a fixed

2000

-object subset using identical training settings, averaged over four epochs.

Coupling	Friction	Hardness	Mass
	MnRE $\uparrow$	ADE $\downarrow$	MnRE $\uparrow$
w/o CPC	0.802	0 9.09	0.474
Attention, 1-layer	0.822	10.79	0.468
Attention, 2-layer (default)	0.831	7.05	0.655
Cross-stitch	0.801	10.16	0.488
FiLM	0.790	13.50	0.469

Table 5: Cross-property coupling ablation. Comparison of different feature interaction mechanisms for modeling dependencies among physical properties.

Figure 5: Out-of-distribution scenes on RUGD (upper two examples) and RELLIS-3D (lower two examples). Ground-truth semantic regions coloured by our predicted friction and Young’s modulus. The model differentiates unseen materials.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
backbone (frozen)	StreamVGGT, 1.26 B params
backbone token dim $C$	2048
DPT feature width	128
UNet head width $h$	64 ( $h,2h,4h,8h$ stages)
dense properties	4 ( $\mu$ , $H$ , $E$ , $\rho$ )
coupling	attention, 2 layers

Appendix

Table 6: Hyperparameters of the deployed PhysVGGT. The frozen StreamVGGT backbone (1.26 B parameters) receives no gradient; all listed settings apply to the 32.4 M-parameter physics heads.

Figure 6: Detailed architectural design of PhysVGGT’s cross-property coupling (CPC) module, shared decoder, and property specific head.

	Friction $\mu$		Shore hardness
Head	MnRE $\uparrow$	ADE $\downarrow$	MnRE $\uparrow$	ADE $\downarrow$
Pooled-token	0.742	0.148	0.844	12.52
Dense trunk (ours)	0.831	0.102	0.908	7.05

Appendix

Table 7: Spatial dense trunk vs. pooled-token regression. The per-pixel trunk is essential for the spatial properties.

	Friction $\mu$		Shore hardness
Backbone	MnRE $\uparrow$	ALDE $\downarrow$	MnRE $\uparrow$	ALDE $\downarrow$
StreamVGGT (ours)	0.831	0.206	0.908	0.103
VGGT-1B [ 23 ]	0.819	0.292	0.875	0.159
DINOv2-L [ 16 ]	0.802	0.252	0.834	0.173

Appendix

Table 8: Choice of geometry backbone under identical head and training protocol.

StreamVGGT	Train.	Friction	Hardness	Mass
freezing	params	MnRE $\uparrow$	MnRE $\uparrow$	MnRE $\uparrow$
Fully frozen (ours)	32 M	0.831	0.908	0.655
Unfreeze global-attn	334 M	0.695	0.814	0.517

Appendix

Table 9: Backbone freezing strategy. Freezing the entire StreamVGGT backbone vs. fine-tuning its global-attention blocks.

Decoder	Dec.	Total	Fric.	Hard.	Hard.
	params	train.	MnRE $\uparrow$	ADE $\downarrow$	MnRE $\uparrow$
original (4 decoders)	12.1 M	40.9 M	0.808	8.62	0.887
slim (4 half-width)	4.3 M	33.2 M	0.808	7.48	0.899
shared + heads (ours)	3.0 M	31.9 M	0.831	7.05	0.908

Appendix

Table 10: Decoder ablation. Decoder architectures for multi-task physical property prediction.

Loss	Friction	Hardness	Mass
	MnRE $\uparrow$	MnRE $\uparrow$	ADE $\downarrow$
L1 baseline	0.684	0.826	12.39
+ foreground mask	0.759	0.856	10.72
+ pairwise ranking	0.811	0.851	11.49
+ all (mask, Charb., rank)	0.831	0.908	4.73

Appendix

Table 11: Loss-component ablation.

	Mass (ABO-500)			OOD (N2P)
Train. obj.	ADE (kg) $\downarrow$	ALDE $\downarrow$	MnRE $\uparrow$	$\mu$ MnRE $\uparrow$	$H$ MnRE $\uparrow$
500	5.62	0.633	0.651	0.787	0.824
2000	5.26	0.658	0.641	0.794	0.860
7225	4.73	0.574	0.655	0.831	0.908

Appendix

Table 12: Data scaling. Mass prediction on ABO-500 and OOD friction/hardness on N2P. Performance improves consistently with training-set size.

Mass branch	ADE (kg) $\downarrow$	ALDE $\downarrow$	APE $\downarrow$	MnRE $\uparrow$
dense, $\bar{\rho}\!\times\!V_{\text{shell}}\!\times\!\kappa$	10.3	0.896	1.043	0.517
global, direct mass head	4.73	0.574	0.787	0.655

Appendix

Table 13: Mass branch: dense (

\rho\!\times\!V

) versus global (direct head) on ABO-500. Both geometry tokens come from the same frozen geometry backbone and the same trained model; only the path from features to mass differs.

Supervision	Friction	Hardness	Mass (dense)
Raw	0.813	0.856	0.450
Refined	0.831	0.908	0.655

Appendix

Table 14: Refined vs. raw supervision comparison.

Hyperparameter	Setting	$\mu$ MnRE $\uparrow$	$H$ MnRE $\uparrow$
head width $h$	32	0.792	0.855
	48	0.839	0.877
	64 $\star$	0.829	0.877
	96	0.822	0.859
$\lambda_{\mathrm{friction}}$	1	0.807	0.864
	2	0.770	0.868

Appendix

Table 15: Hyperparameter sensitivity. N2P MnRE (

\uparrow

) under one-at-a-time sweeps from the final configuration (

\star

Method	Zero-shot	ALDE $\downarrow$	MnRE $\uparrow$
Xception $k$ -NN	No	1.454	0.341
Xception $k$ -NN (SIM)	No	0.781	0.534
Pure CNN	No	0.680	0.575
CNN with ALDE	No	0.666	0.573
No Geometry	No	0.520	0.638
Shape-aware	No	0.465	0.675

Appendix

Table 16: Zero-shot mass prediction on image2mass. All baselines are trained on image2mass [ 22 ] , whereas PhysVGGT is trained only on ABO and evaluated without fine-tuning.

\dagger

: human estimates provided with the dataset.

Panel of Figure 7 — Figure 7: Language-guided physical reasoning. Top: multi-round question answering about object- and region-level physical properties. Bottom: single-shot physical-composition captioning. PhysVGGT supplies dense friction, Shore hardness, Young’s modulus, density, and object-level mass predictions; SAM2 supplies spatial regions; and Qwen3-VL interprets the original and segmented images and generates the responses. Material names are estimated by matching each predicted property tuple $(\mu,H,E,\rho)$ to a reference material database. All numerical physical claims are derived from PhysVGGT predictions.

Figure 8: Grasp-force planning from predicted physics. From one image PhysVGGT predicts per-pixel friction $\mu$ (middle) and object mass, yielding a dense minimum grip-force map $F_{n}=mg/(2\mu)$ (right; green $=$ secure/low force, red $=$ slippery/high force) and a best contact region (marker). On both objects, the model marks the grippy top surface as the grasp region while the slick legs (wooden; thin metal) turn red ( $>\!250$ N).

Materials Property Prediction

Materials property prediction uses data and physical models to estimate how composition, structure, and processing determine a material’s behavior.

See all 173 papers

Jul 24, 2026

SiPhy: Single-Image Physical Property Reasoning

Inferring physical properties such as mass, stiffness, and elasticity from a single image is essential for simulation and embodied AI, yet most existing approaches rely on multi-view reconstruction or physics-based supervision. We introduce SiPhy, a unified framework for single-image physical property reasoning that aligns 3D-aware visual cues, depth with language-based material knowledge. From one RGB image, SiPhy samples pseudo-voxel points, extracts CLIP features, and grounds them to material candidates proposed by a VLM. A part-based contrastive aggregator enforces region consistency, while a heaviness-aware refinement improves thickness and volume estimation for dense objects. Across ABO-500, MVImgNet-100, and PhysXNet-100, SiPhy achieves state-of-the-art single-image performance, surpassing multi-view reconstruction methods by improving mass MnRE by up to 93% (vs. PUGS), reducing density MAE by 35.5% (vs. NeRF2Physics), and lowering Young's modulus error by 23.5%. We further validate SiPhy on real hand-object interaction datasets, demonstrating its potential as a data annotation engine for physical understanding from single-view imagery.

Hoang Le, Joonwoo Kwon, Elkhan Ismayilzada +2

Michigan State University, East Lansing MI 48864, USA · Independent Researcher

Topic pending

Ask PDF

Mar 25, 2026

SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents

Estimating the material property field of 3D assets is critical for physics-based simulation, robotics, and digital twin generation. Existing vision-based approaches are either too expensive and slow or rely on 3D information. We present SLAT-Phys, an end-to-end method that predicts spatially varying material property fields of 3D assets directly from a single RGB image without explicit 3D reconstruction. Our approach leverages spatially organised latent features from a pretrained 3D asset generation model that encodes rich geometry and semantic prior, and trains a lightweight neural decoder to estimate Young's modulus, density, and Poisson's ratio. The coarse volumetric layout and semantic cues of the latent representation about object geometry and appearance enable accurate material estimation. Our experiments demonstrate that our method provides competitive accuracy in predicting continuous material parameters when compared against prior approaches, while significantly reducing computation time. In particular, SLAT-Phys requires only 9.9 seconds per object on an NVIDIA RTXA5000 GPU and avoids reconstruction and voxelization preprocessing. This results in 120x speedup compared to prior methods and enables faster material property estimation from a single image.

Rocktim Jyoti Das, Dinesh Manocha

University of Maryland, College Park

Materials Property Prediction

Ask PDF

Mar 17, 2026

PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models

Vision-Language Models (VLMs) are increasingly applied to robotic perception and manipulation, yet their ability to infer physical properties required for manipulation remains limited. In particular, estimating the mass of real-world objects is essential for determining appropriate grasp force and ensuring safe interaction. However, current VLMs lack reliable mass reasoning capabilities, and most existing benchmarks do not explicitly evaluate physical quantity estimation under realistic sensing conditions. In this work, we propose PhysQuantAgent, a framework for real-world object mass estimation using VLMs, together with VisPhysQuant, a new benchmark dataset for evaluation. VisPhysQuant consists of RGB-D videos of real objects captured from multiple viewpoints, annotated with precise mass measurements. To improve estimation accuracy, we introduce three visual prompting methods that enhance the input image with object detection, scale estimation, and cross-sectional image generation to help the model comprehend the size and internal structure of the target object. Experiments show that visual prompting significantly improves mass estimation accuracy on real-world data, suggesting the efficacy of integrating spatial reasoning with VLM knowledge for physical inference.

Hisayuki Yokomizo, Taiki Miyanishi, Yan Gang +3

The University of Tokyo · Institute of Science Tokyo · National Institute of Informatics

VLM Reasoning VLMs for Robotics

Ask PDF

Abstract

Materials Property Prediction

Explore similar work

SiPhy: Single-Image Physical Property Reasoning

SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents

PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models