PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models

Hisayuki Yokomizo; Taiki Miyanishi; Yan Gang; Shuhei Kurita; Nakamasa Inoue; Yusuke Iwasawa

cs.CVMar 17, 2026

PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Models

Authors: Hisayuki Yokomizo, Taiki Miyanishi, Yan Gang, Shuhei Kurita, Nakamasa Inoue, Yusuke Iwasawa

Organizations: The University of Tokyo · Institute of Science Tokyo · National Institute of Informatics

Abstract

Vision-Language Models (VLMs) are increasingly applied to robotic perception and manipulation, yet their ability to infer physical properties required for manipulation remains limited. In particular, estimating the mass of real-world objects is essential for determining appropriate grasp force and ensuring safe interaction. However, current VLMs lack reliable mass reasoning capabilities, and most existing benchmarks do not explicitly evaluate physical quantity estimation under realistic sensing conditions. In this work, we propose PhysQuantAgent, a framework for real-world object mass estimation using VLMs, together with VisPhysQuant, a new benchmark dataset for evaluation. VisPhysQuant consists of RGB-D videos of real objects captured from multiple viewpoints, annotated with precise mass measurements. To improve estimation accuracy, we introduce three visual prompting methods that enhance the input image with object detection, scale estimation, and cross-sectional image generation to help the model comprehend the size and internal structure of the target object. Experiments show that visual prompting significantly improves mass estimation accuracy on real-world data, suggesting the efficacy of integrating spatial reasoning with VLM knowledge for physical inference.

Figures & tables

Fig. 1: Comparison of previous mass estimation methods and our proposed PhysQuantAgent. While prior approaches rely on computationally expensive 3D reconstruction from RGB-D images, PhysQuantAgent directly infers object mass with a VLM, enabling fast estimation for grasp force control.

Dataset	Video	View	Category	Real	Scale	Mass
OmniObject3D [ 31 ]	RGB	Full	190	✓	✗	✗
CO3D [ 26 ]	RGB	Full	50	✓	✗	✗
WildRGB-D [ 32 ]	RGBD	Full	44	✓	✗	✗
Objectron [ 1 ]	RGB	Limited	9	✓	✓	✗
HANDAL [ 18 ]	RGBD	Limited	17	✓	✓	✗
ABO [ 11 ]	✗ (3D)	-	63	✗	✓	✓

TABLE I: Comparison of existing datasets.

Fig. 2: PhysQuantAgent framework: First, the VLM selects the tool to use (Stage1), then estimates the mass of the object from the original image and the image with the tool applied (Stage2).

Panel of Figure 4 — Fig. 3: Example of the instruction prompt in PhysQuantAgent.

Fig. 6: VisPhysQuant contains approximately 300 RGBD videos of objects with mass annotations. To ensure diversity, the data were collected in diverse background settings.

Fig. 7: A comparison of category distribution of ABO [ 11 ] (left) and VisPhysQuant (right)

Fig. 8: A comparison of weight distribution of ABO [ 11 ] and VisPhysQuant.

Fig. 9: The mass estimation performance of NeRF2Physics and VLMs with PhysQuantAgent.

Base Model	Visual Prompt	MnRE $\uparrow$
Gemini-2.5-pro	Baseline	0.721
	+ Object Detection	0.747
	+ Scale Estimation	0.754
	+ Cross-sectional Image	0.738
Gemini-3.1-pro	Baseline	0.778
	+ Object Detection	0.806

TABLE II: Ablation of Visual Prompts.

Fig. 11: Qualitative example of PhysQuantAgent.

Fig. 12: Error cases in Visual Prompting. Scale estimation can yield large errors for transparent objects (left), while Cross-sectional Image generation is prone to hallucinations, such as the synthesis of non-existent objects (right).

Fig. 13: Example of a robotic manipulation task.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image
Sep 16, 2026Sneha Paul, Guile Wu, Bingbing Liu +1Materials Property Prediction Robotic Perception
Huawei Noah’s Ark Lab · Concordia University · Foundation Model Department, Huawei
PhyPush: One Push is All You Need for Sensorless Physical Property Estimation with Physics-Guided Transformers
May 25, 2026Koyo Fujii, Luis Figueredo, Praminda Caleb-Solly +4Physical Reasoning Robotic Manipulation
School of Computer Science, University of Nottingham, Nottingham, United Kingdom · Dept. of Industrial Engineering, University of Bologna, Bologna, Italy
PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop
Sep 30, 2026Xinge Peng, Yiting Lu, Tianwu Zhi +4VLM Evaluation Physical Consistency in Video Generation
University of Science and Technology of China · ByteDance · City University of Hong Kong

Sep 16, 2026J/K move · Enter open · S save

PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image

Sneha Paul, Guile Wu, Bingbing Liu, Dongfeng Bai

Huawei Noah’s Ark Lab · Concordia University · Foundation Model Department, Huawei

Materials Property Prediction Robotic Perception

Open paper AskPDF arXiv

Physical properties, such as friction, hardness, stiffness, and density, govern how robots should grasp, manipulate and interact with objects, yet estimating these properties from RGB images remains challenging. Existing methods typically employ per-object reconstruction augmented with physical properties or directly query vision-language models at test time, which results in substantial computational overhead that limits their applicability. In this work, we present PhysVGGT, a feed-forward model that predicts dense maps of friction coefficient, Shore hardness, Young's modulus, and density, together with object-level mass, from a single RGB image in one forward pass. The key idea of PhysVGGT is to formulate physical property estimation as a dense per-pixel prediction problem and employ a visual geometry transformer to extract geometry-aware tokens from the input image followed by a dense prediction branch for estimating local physical properties and a global prediction branch for estimating object-level mass. In addition, we introduce a scalable pseudo-label generation pipeline that enables large-scale weakly supervised training for dense physical property prediction, substantially reducing the need for expensive direct physical measurements. Extensive experiments show that PhysVGGT achieves state-of-the-art performance on the ABO-500 dataset and generalizes effectively to the out-of-distribution NeRF2Physics dataset. Moreover, PhysVGGT eliminates the need for per-object reconstruction and test-time optimization, achieving an inference latency of only 0.13s per image, making it

27\times

faster than the previous state of the art.

Abstract

VLM Reasoning

Explore similar work

PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Image

PhyPush: One Push is All You Need for Sensorless Physical Property Estimation with Physics-Guided Transformers

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop