Which Reconstruction Model Should a Robot Use? Routing Image-to-3D Models for Cost-Aware Robotic Manipulation

Akash Anand; Aditya Agarwal; Leslie Pack Kaelbling

cs.ROMar 29, 2026

Which Reconstruction Model Should a Robot Use? Routing Image-to-3D Models for Cost-Aware Robotic Manipulation

Authors: Akash Anand, Aditya Agarwal, Leslie Pack Kaelbling

Organizations: Massachusetts Institute of Technology

Abstract

Robotic manipulation tasks require 3D mesh reconstructions of varying quality: dexterous manipulation demands fine-grained surface detail, while collision-free planning tolerates coarser representations. Multiple reconstruction methods offer different cost-quality tradeoffs, from Image-to-3D models - whose output quality depends heavily on the input viewpoint - to view-invariant methods such as structured light scanning. Querying all models is computationally prohibitive, motivating per-input model selection. We propose SCOUT, a novel routing framework that decouples reconstruction scores into two components: (1) the relative performance of viewpoint-dependent models, captured by a learned probability distribution, and (2) the overall image difficulty, captured by a scalar partition function estimate. As the learned network operates only over the viewpoint-dependent models, view-invariant pipelines can be added, removed, or reconfigured without retraining. SCOUT also supports arbitrary cost constraints at inference time, accommodating the multi-dimensional cost constraints common in robotics. We evaluate on the Google Scanned Objects, BigBIRD, and YCB datasets under multiple mesh quality metrics, demonstrating consistent improvements over routing baselines adapted from the LLM literature across various cost constraints. We further validate the framework through robotic grasping and dexterous manipulation experiments. We release the code and additional results on our website.

Figures & tables

Figure 1: Overview of SCOUT. Given an input image, (a) SCOUT routes to the most suitable reconstruction model and generates the selected 3D reconstruction (compared against other candidate models). (b) Grasp proposals generated on the reconstructed mesh, evaluated on the ground-truth mesh; colliding grasps are shown in red. (c) Utility of SCOUT’s reconstruction-aware routing in downstream robust robot grasping and dexterous manipulation.

Figure 2: Effect of the number of view-invariant methods on routing performance when evaluated on viewpoint-dependent methods. With decoupling, regret over viewpoint-dependent models remains constant regardless of how many view-invariant methods are included during training. Without decoupling, regret increases as the number of view-invariant methods grows.

Method	$\{\mathbf{c}\in\mathcal{C}_{0}\}$	$\{\mathbf{c}\in\mathcal{C}\}$	$\{\mathbf{c}\in\mathcal{C}\cap\,\mathbf{c}_{\mathrm{inv}}=\infty\}$
ZOOTER	$0.6489\pm 0.0038$	N/A	N/A
RouterDC	$0.8013\pm 0.0050$	N/A	N/A
MLP	$0.6716\pm 0.0038$	$0.4653\pm 0.0026$	$0.5004\pm 0.0021$
MF	$0.7557\pm 0.0066$	$0.5163\pm 0.0036$	$0.5814\pm 0.0040$
kNN	$0.6489\pm 0.0033$	$0.4630\pm 0.0022$	$0.4992\pm 0.0019$
LR	$0.6497\pm 0.0033$	$0.4519\pm 0.0021$	$0.5047\pm 0.0019$

Table 1: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret across different cost subspaces on the GSO dataset.

Panel of Figure 3 — Figure 3: Deferral curves for the latency $\odot$ memory (a) and latency (b) cost coefficient vectors, showing the normalized utility achieved by each method as a function of the allocated cost.

Method	Latency $\odot$ Memory	Latency	Memory
MLP	$0.7365\pm 0.0008$	$\mathbf{0.5815\pm 0.0003}$	$0.7911\pm 0.0007$
MF	$0.7233\pm 0.0011$	$0.5795\pm 0.0003$	$0.7736\pm 0.0011$
kNN	$0.7382\pm 0.0007$	$0.5798\pm 0.0002$	$0.7942\pm 0.0006$
LR	$0.7364\pm 0.0007$	$0.5787\pm 0.0002$	$0.7913\pm 0.0006$
Ours (no dc)	$0.7408\pm 0.0007$	$0.5796\pm 0.0003$	$0.7958\pm 0.0006$
Ours	$\mathbf{0.7416\pm 0.0007}$	$0.5814\pm 0.0002$	$\mathbf{0.7976\pm 0.0006}$

Table 2: AIQ (

\uparrow

) for one-dimensional cost families on the GSO dataset, evaluated on novel objects.

Method	DCD	Chamfer L2	Chamfer L1	IoU	MMD-EMD	Eval3D-geo	Eval3D-struct
MLP	$1.1628\pm 0.0147$	$1.1677\pm 0.0252$	$1.0268\pm 0.0156$	$0.7135\pm 0.0100$	$0.9354\pm 0.0147$	$1.1520\pm 0.0431$	$1.2257\pm 0.0209$
MF	$1.3116\pm 0.0152$	$1.3869\pm 0.0247$	$1.2151\pm 0.0169$	$0.8972\pm 0.0172$	$1.3003\pm 0.0222$	$1.6351\pm 0.0791$	$1.3365\pm 0.0218$
kNN	$1.0385\pm 0.0118$	$0.9109\pm 0.0134$	$0.8818\pm 0.0093$	$0.7411\pm 0.0089$	$0.9604\pm 0.0158$	$0.9957\pm 0.0531$	$1.1563\pm 0.0158$
LR	$0.9983\pm 0.0120$	$0.9174\pm 0.0147$	$0.8506\pm 0.0105$	$\mathbf{0.6421\pm 0.0090}$	$0.8554\pm 0.0139$	$\mathbf{0.8246\pm 0.0274}$	$\mathbf{1.0815\pm 0.0159}$
Ours (no dc)	$1.0171\pm 0.0129$	$0.8806\pm 0.0152$	$0.8395\pm 0.0121$	$0.6849\pm 0.0088$	$0.8447\pm 0.0143$	$0.8674\pm 0.0412$	$1.1197\pm 0.0167$
Ours	$\mathbf{0.9767\pm 0.0113}$	$\mathbf{0.8730\pm 0.0146}$	$\mathbf{0.8222\pm 0.0106}$	$0.6609\pm 0.0091$	$\mathbf{0.8136\pm 0.0144}$	$0.8588\pm 0.0413$	$1.1527\pm 0.0188$

Table 3: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret for

\mathcal{C}_{0}

across multiple quality metrics on BigBIRD + YCB. Ours (no dc) denotes our method without decoupling, equivalent to ZOOTER extended to support arbitrary cost vectors.

Method	DCD	Chamfer L2	Chamfer L1	IoU	MMD-EMD	Eval3D-geo	Eval3D-struct
MLP	$0.9944\pm 0.0165$	$1.1708\pm 0.0234$	$0.8983\pm 0.0149$	$0.7054\pm 0.0115$	$0.8153\pm 0.0128$	$3.1921\pm 0.1267$	$0.8975\pm 0.0117$
MF	$1.2518\pm 0.0158$	$2.7886\pm 0.0524$	$1.7604\pm 0.0281$	$0.8378\pm 0.0119$	$1.8527\pm 0.0301$	$5.9481\pm 0.2481$	$1.3899\pm 0.0197$
kNN	$0.9845\pm 0.0187$	$0.7939\pm 0.0096$	$0.8121\pm 0.0125$	$0.7951\pm 0.0121$	$0.7699\pm 0.0157$	$\mathbf{0.8413\pm 0.0122}$	$0.7794\pm 0.0094$
LR	$0.8925\pm 0.0170$	$0.8659\pm 0.0090$	$0.7776\pm 0.0116$	$0.7208\pm 0.0099$	$0.7080\pm 0.0128$	$1.0219\pm 0.0161$	$0.7366\pm 0.0095$
Ours (no dc)	$0.9451\pm 0.0164$	$\mathbf{0.7878\pm 0.0095}$	$0.7450\pm 0.0109$	$\mathbf{0.6767\pm 0.0094}$	$0.7147\pm 0.0119$	$0.9922\pm 0.0227$	$\mathbf{0.7336\pm 0.0094}$
Ours	$\mathbf{0.8880\pm 0.0160}$	$0.7991\pm 0.0107$	$\mathbf{0.7433\pm 0.0105}$	$0.6808\pm 0.0101$	$\mathbf{0.6741\pm 0.0107}$	$0.9253\pm 0.0166$	$0.7564\pm 0.0136$

Table 4: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret for

\mathbf{c}\in\mathcal{C}

across multiple quality metrics on BigBIRD + YCB. Ours (no dc) denotes our method without decoupling.

Object	Hunyuan3D	InstantMesh	TRELLIS	TripoSR	Ours (novel obj.)	Ours (novel view)
Tomato Soup Can	$0.7174$ / $0.0685$	$0.0227$ / $0.9164$	$0.3864$ / $0.2071$	$0.6585$ / $0.1807$	InstantMesh	InstantMesh
Cup (vC)	$0.0$ / $0.4019$	$0.2273$ / $0.1381$	$0.0$ / $0.4372$	$0.1500$ / $0.1487$	Hunyuan3D	Hunyuan3D
Marble	$0.0$ / $0.9580$	$0.0$ / $0.7423$	$0.0$ / $0.1464$	$0.0$ / $0.9537$	TripoSR	TripoSR
Chef Can	$0.0$ / $0.8543$	$0.0$ / $0.8763$	$0.0250$ / $0.9406$	$0.1190$ / $0.7730$	InstantMesh	TRELLIS
Potted Meat	$0.0$ / $0.6589$	$0.6098$ / $0.4604$	$0.0682$ / $0.7573$	$0.3571$ / $0.4632$	InstantMesh	TRELLIS
Cup (vB)	$0.0$ / $0.2960$	$0.0238$ / $0.1423$	$0.1087$ / $0.0338$	$0.3659$ / $0.1080$	Hunyuan3D	Hunyuan3D

Table 5: Grasp collision rate (

\downarrow

) / mesh-IoU (

\uparrow

) for each reconstruction model across 10 YCB objects. Grasps are generated on the reconstructed mesh using a two-fingered gripper [ 23 ] and evaluated against the ground-truth mesh. SCOUT is evaluated in two settings: routing over novel views of known objects, and routing over both novel views and novel objects.

Table 9

Method	$\{\mathbf{c}\in\mathcal{C}_{0}\}$	$\{\mathbf{c}\in\mathcal{C}\}$	$\{\mathbf{c}\in\mathcal{C}\cap\,\mathbf{c}_{\mathrm{inv}}=\infty\}$
Ours w/ smoothing	$0.6386\pm 0.0033$	$0.4319\pm 0.0021$	$0.4831\pm 0.0019$
Ours w/o smoothing	$0.6588\pm 0.0035$	$0.4371\pm 0.0021$	$0.4878\pm 0.0020$

Table 8: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret across different cost subspaces on the GSO dataset.

Method	$\{\mathbf{c}\in\mathcal{C}_{0}\}$	$\{\mathbf{c}\in\mathcal{C}\}$	$\{\mathbf{c}\in\mathcal{C}\cap\,\mathbf{c}_{\mathrm{inv}}=\infty\}$
LR w/ smoothing	$0.6515\pm 0.0034$	$0.4723\pm 0.0022$	$0.5064\pm 0.0019$
LR w/o smoothing	$0.6497\pm 0.0033$	$0.4519\pm 0.0021$	$0.5047\pm 0.0019$
kNN w/ smoothing	$0.6515\pm 0.0034$	$0.4723\pm 0.0022$	$0.5064\pm 0.0019$
kNN w/o smoothing	$0.6489\pm 0.0033$	$0.4630\pm 0.0022$	$0.4992\pm 0.0019$

Table 9: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret across different cost subspaces on the GSO dataset.

Metric	DCD	Chamfer L2	Chamfer L1	IoU	MMD-EMD	Eval3D-geo	Eval3D-struct
$T$	$0.5$	$0.1$	$0.2$	$2.0$	$0.15$	$0.1$	$0.2$

Table 10: Softmax temperature

T

per metric for BigBIRD + YCB.

Panel of Figure 4 — Figure 4: Degenerate and nondegenerate views of a cracker box.

Panel of Figure 5 — Figure 5: Different lighting conditions for a fixed viewpoint.

Panel of Figure 6 — Figure 6: Sensitivity of hyperparameters for SCOUT evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Figure 7: Sensitivity of alpha for SCOUT evaluated on novel objects in the GSO dataset over cost coefficient vectors sampled from $\mathcal{C}$ .

Panel of Figure 8 — Figure 8: Sensitivity of hyperparameters for SCOUT without decoupling evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Figure 9: Sensitivity of the number of nearest neighbors for the kNN baseline evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Figure 10: Sensitivity of alpha for the linear regression baseline evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Metric	True $z$	Equiweighted	One-hot
DCD	${3.24\text{e-}}4$	$5.44\text{e-}1$	${2.38\text{e-}24}$
Chamfer L2	${4.01\text{e-}35}$	${5.74\text{e-}35}$	$8.11\text{e-}1$
Chamfer L1	${2.03\text{e-}4}$	${3.92\text{e-}2}$	${7.71\text{e-}16}$
IoU	$4.35\text{e-}1$	$2.31\text{e-}1$	${6.89\text{e-}5}$
MMD-EMD	${5.48\text{e-}13}$	${3.32\text{e-}6}$	${4.10\text{e-}4}$
Eval3D-geo	${8.24\text{e-}34}$	${2.70\text{e-}30}$	${9.09\text{e-}4}$

Table 11: One-sided t-test

p

-values comparing our weighted partition function proxy against three alternatives on BigBIRD + YCB. Each cell reports the

p

-value for the hypothesis that our proxy achieves lower regret, averaged over cost coefficient vectors sampled from

\mathcal{C}

( Sec. 4.1 ). An improvement indicates a metric where our proxy achieves statistically significant (

\alpha=0.05

) lower regret than the alternative; a deterioration (

\alpha=0.05

for the reverse one-sided test) indicates metrics where the alternative achieves statistically significant lower regret than ours.

Figure 11: Evaluation of grasp proposals based on the reconstructed meshes from the input image. The figure shows the following from left to right: (1) original input image inputted to SCOUT, (2) 3D reconstructions from the original image with the model chosen by SCOUT boxed in green, (3) grasp proposals on the reconstructed mesh, (4) grasp proposals evaluated on the ground-truth mesh; colliding grasps are shown in red.

Figure 12: Evaluation of grasp proposals based on the reconstructed meshes from the input image. The figure shows the following from left to right: (1) original input image inputted to SCOUT, (2) 3D reconstructions from the original image with the model chosen by SCOUT boxed in green, (3) grasp proposals on the reconstructed mesh, (4) grasp proposals evaluated on the ground-truth mesh; colliding grasps are shown in red.

Figure 13: Evaluation of grasp proposals based on the reconstructed meshes from the input image. The figure shows the following from left to right: (1) original input image inputted to SCOUT, (2) 3D reconstructions from the original image with the model chosen by SCOUT boxed in green, (3) grasp proposals on the reconstructed mesh, (4) grasp proposals evaluated on the ground-truth mesh; colliding grasps are shown in red.

Panel of Figure 14 — Figure 14: Oracle and learned routing policies across object categories.

Panel of Figure 15 — Figure 15: Oracle and learned routing policies across viewpoint elevations. High: $75^{\circ}$ and $85^{\circ}$ ; low: $35^{\circ}$ , $45^{\circ}$ , and $60^{\circ}$ .

3D Mesh Reconstruction

3D mesh reconstruction turns images, depth, or sensor data into explicit surfaces for measurement, editing, and simulation.

See all 109 papers

Feb 26, 2026

UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perception

We present UniScale, a unified, scale-aware multi-view 3D reconstruction framework for robotic applications that flexibly integrates geometric priors through a modular, semantically informed design. In vision-based robotic navigation, the accurate extraction of environmental structure from raw image sequences is critical for downstream tasks. UniScale addresses this challenge with a single feed-forward network that jointly estimates camera intrinsics and extrinsics, scale-invariant depth and point maps, and the metric scale of a scene from multi-view images, while optionally incorporating auxiliary geometric priors when available. By combining global contextual reasoning with camera-aware feature representations, UniScale is able to recover the metric-scale of the scene. In robotic settings where camera intrinsics are known, they can be easily incorporated to improve performance, with additional gains obtained when camera poses are also available. This co-design enables robust, metric-aware 3D reconstruction within a single unified model. Importantly, UniScale does not require training from scratch, and leverages world priors exhibited in pre-existing models without geometric encoding strategies, making it particularly suitable for resource-constrained robotic teams. We evaluate UniScale on multiple benchmarks, demonstrating strong generalization and consistent performance across diverse environments. We will release our implementation upon acceptance.

Mohammad Mahdavian, Gordon Tan, Binbin Xu +3

Huawei Noah’s Ark Lab · University of Toronto

3D Reconstruction Multi-View 3D Reconstruction Feed-Forward 3D Reconstruction Robotic Perception Camera Pose Estimation

Ask PDF

Jun 22, 2026

Can Single-View Mesh Reconstruction Generalize to Robot Camera Rotation?

Single-view mesh reconstruction predicts object meshes and spatial layouts from a single observation, making it attractive for fast robot spatial reasoning and real-to-sim digital twins. However, robot-mounted cameras naturally rotate during manipulation and navigation, while learned single-view reconstruction models often rely on view-dependent priors and may generalize poorly to out-of-distribution camera rotations. Such rotations can introduce 3D inconsistencies, incorrect layouts, and violations of physical constraints, but this failure mode remains under-evaluated. We introduce an evaluation protocol with controlled axis-wise roll, pitch, and yaw sweeps to trace errors in monocular depth estimation (MDE), canonical object meshes, camera-space layout, and physical plausibility within a representative SAM3D-style pipeline. On the Aria Digital Twin dataset and a real Franka wrist-camera sequence, camera rotations induce MDE distortion, layout drift, and collision penetration, while canonical mesh predictions remain relatively stable. A two-stage SAM3D+FoundationPose pipeline is more robust than one-stage feed-forward layout prediction, and our Gravity-Aware Refinement reduces one-stage pairwise ICP-based layout-orientation error by 47.1

\%

. Our evaluation reveals that current single-view mesh reconstruction methods generalize poorly to robot camera rotation, and suggests that explicit gravity cues are important for reliable robotic single-view mesh reconstruction.

Yu Zhan, Guangcheng Chen, Hanjing Ye +4

Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology (SUSTech) · Pengcheng Laboratory

Monocular Depth Estimation 3D Mesh Reconstruction Single-View 3D Reconstruction Robotic Perception

Ask PDF

Sep 8, 2026

RoMa- $Ω$ : What Feed-Forward 3D Models Know About Image Matching

Learned image matching has experienced significant progress in recent years, culminating in robust and accurate matchers such as RoMa, whose robustness is often attributed to its use of frozen DINO features. In a parallel development, feed-forward reconstruction models, such as VGGT, have been trained on ever-growing datasets to accurately regress dense 3D point maps and camera poses. The distinction between matchers and feed-forward reconstruction models has become increasingly blurred with the introduction of matching losses in models such as MASt3R and VGGT-

Ω

. This raises a natural question: what do feed-forward 3D models know about image matching? In this work, we answer this question by analyzing three scenarios: (i) zero-shot matching of patch features, (ii) direct matching of 3D point predictions, and (iii) training a full matcher on top of the learned representations. We find that, despite performing poorly in zero-shot matching, especially in later layers, feed-forward reconstruction models provide strong representations for linear probing and full matching pipelines. We further show that, even without any training, their raw predictions alone enable competitive matching, albeit only under moderate viewpoint changes and modality gaps. Based on these insights, we retrain RoMa v2 by replacing its DINO backbone with VGGT-

Ω

. Our resulting model, \ours, outperforms state-of-the-art matchers on a wide range of benchmarks, e.g. +8.1 mAA compared to RoMa v2 on WxBS.

David Nordström, Xinyue Zhang, Thibaut Loiseau +2

Chalmers University of Technology, Sweden · Mobile Perception Lab, ShanghaiTech University, China · LIGM, Ecole des Ponts, Univ. Gustave Eiffel, CNRS, France

Image Matching 3D Representation Learning Cross-Modal Feature Matching Feed-Forward 3D Reconstruction Local Feature Matching

Ask PDF

Abstract

3D Mesh Reconstruction

Explore similar work

UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perception

Can Single-View Mesh Reconstruction Generalize to Robot Camera Rotation?

RoMa-ΩΩΩ: What Feed-Forward 3D Models Know About Image Matching

RoMa- $Ω$ : What Feed-Forward 3D Models Know About Image Matching