Which Reconstruction Model Should a Robot Use? Routing Image-to-3D Models for Cost-Aware Robotic Manipulation

Akash Anand; Aditya Agarwal; Leslie Pack Kaelbling

cs.ROMar 29, 2026

Which Reconstruction Model Should a Robot Use? Routing Image-to-3D Models for Cost-Aware Robotic Manipulation

Authors: Akash Anand, Aditya Agarwal, Leslie Pack Kaelbling

Organizations: Massachusetts Institute of Technology

Abstract

Robotic manipulation tasks require 3D mesh reconstructions of varying quality: dexterous manipulation demands fine-grained surface detail, while collision-free planning tolerates coarser representations. Multiple reconstruction methods offer different cost-quality tradeoffs, from Image-to-3D models - whose output quality depends heavily on the input viewpoint - to view-invariant methods such as structured light scanning. Querying all models is computationally prohibitive, motivating per-input model selection. We propose SCOUT, a novel routing framework that decouples reconstruction scores into two components: (1) the relative performance of viewpoint-dependent models, captured by a learned probability distribution, and (2) the overall image difficulty, captured by a scalar partition function estimate. As the learned network operates only over the viewpoint-dependent models, view-invariant pipelines can be added, removed, or reconfigured without retraining. SCOUT also supports arbitrary cost constraints at inference time, accommodating the multi-dimensional cost constraints common in robotics. We evaluate on the Google Scanned Objects, BigBIRD, and YCB datasets under multiple mesh quality metrics, demonstrating consistent improvements over routing baselines adapted from the LLM literature across various cost constraints. We further validate the framework through robotic grasping and dexterous manipulation experiments. We release the code and additional results on our website.

Figures & tables

Figure 1: Overview of SCOUT. Given an input image, (a) SCOUT routes to the most suitable reconstruction model and generates the selected 3D reconstruction (compared against other candidate models). (b) Grasp proposals generated on the reconstructed mesh, evaluated on the ground-truth mesh; colliding grasps are shown in red. (c) Utility of SCOUT’s reconstruction-aware routing in downstream robust robot grasping and dexterous manipulation.

Figure 2: Effect of the number of view-invariant methods on routing performance when evaluated on viewpoint-dependent methods. With decoupling, regret over viewpoint-dependent models remains constant regardless of how many view-invariant methods are included during training. Without decoupling, regret increases as the number of view-invariant methods grows.

Method	$\{\mathbf{c}\in\mathcal{C}_{0}\}$	$\{\mathbf{c}\in\mathcal{C}\}$	$\{\mathbf{c}\in\mathcal{C}\cap\,\mathbf{c}_{\mathrm{inv}}=\infty\}$
ZOOTER	$0.6489\pm 0.0038$	N/A	N/A
RouterDC	$0.8013\pm 0.0050$	N/A	N/A
MLP	$0.6716\pm 0.0038$	$0.4653\pm 0.0026$	$0.5004\pm 0.0021$
MF	$0.7557\pm 0.0066$	$0.5163\pm 0.0036$	$0.5814\pm 0.0040$
kNN	$0.6489\pm 0.0033$	$0.4630\pm 0.0022$	$0.4992\pm 0.0019$
LR	$0.6497\pm 0.0033$	$0.4519\pm 0.0021$	$0.5047\pm 0.0019$

Table 1: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret across different cost subspaces on the GSO dataset.

Panel of Figure 3 — Figure 3: Deferral curves for the latency $\odot$ memory (a) and latency (b) cost coefficient vectors, showing the normalized utility achieved by each method as a function of the allocated cost.

Method	Latency $\odot$ Memory	Latency	Memory
MLP	$0.7365\pm 0.0008$	$\mathbf{0.5815\pm 0.0003}$	$0.7911\pm 0.0007$
MF	$0.7233\pm 0.0011$	$0.5795\pm 0.0003$	$0.7736\pm 0.0011$
kNN	$0.7382\pm 0.0007$	$0.5798\pm 0.0002$	$0.7942\pm 0.0006$
LR	$0.7364\pm 0.0007$	$0.5787\pm 0.0002$	$0.7913\pm 0.0006$
Ours (no dc)	$0.7408\pm 0.0007$	$0.5796\pm 0.0003$	$0.7958\pm 0.0006$
Ours	$\mathbf{0.7416\pm 0.0007}$	$0.5814\pm 0.0002$	$\mathbf{0.7976\pm 0.0006}$

Table 2: AIQ (

\uparrow

) for one-dimensional cost families on the GSO dataset, evaluated on novel objects.

Method	DCD	Chamfer L2	Chamfer L1	IoU	MMD-EMD	Eval3D-geo	Eval3D-struct
MLP	$1.1628\pm 0.0147$	$1.1677\pm 0.0252$	$1.0268\pm 0.0156$	$0.7135\pm 0.0100$	$0.9354\pm 0.0147$	$1.1520\pm 0.0431$	$1.2257\pm 0.0209$
MF	$1.3116\pm 0.0152$	$1.3869\pm 0.0247$	$1.2151\pm 0.0169$	$0.8972\pm 0.0172$	$1.3003\pm 0.0222$	$1.6351\pm 0.0791$	$1.3365\pm 0.0218$
kNN	$1.0385\pm 0.0118$	$0.9109\pm 0.0134$	$0.8818\pm 0.0093$	$0.7411\pm 0.0089$	$0.9604\pm 0.0158$	$0.9957\pm 0.0531$	$1.1563\pm 0.0158$
LR	$0.9983\pm 0.0120$	$0.9174\pm 0.0147$	$0.8506\pm 0.0105$	$\mathbf{0.6421\pm 0.0090}$	$0.8554\pm 0.0139$	$\mathbf{0.8246\pm 0.0274}$	$\mathbf{1.0815\pm 0.0159}$
Ours (no dc)	$1.0171\pm 0.0129$	$0.8806\pm 0.0152$	$0.8395\pm 0.0121$	$0.6849\pm 0.0088$	$0.8447\pm 0.0143$	$0.8674\pm 0.0412$	$1.1197\pm 0.0167$
Ours	$\mathbf{0.9767\pm 0.0113}$	$\mathbf{0.8730\pm 0.0146}$	$\mathbf{0.8222\pm 0.0106}$	$0.6609\pm 0.0091$	$\mathbf{0.8136\pm 0.0144}$	$0.8588\pm 0.0413$	$1.1527\pm 0.0188$

Table 3: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret for

\mathcal{C}_{0}

across multiple quality metrics on BigBIRD + YCB. Ours (no dc) denotes our method without decoupling, equivalent to ZOOTER extended to support arbitrary cost vectors.

Method	DCD	Chamfer L2	Chamfer L1	IoU	MMD-EMD	Eval3D-geo	Eval3D-struct
MLP	$0.9944\pm 0.0165$	$1.1708\pm 0.0234$	$0.8983\pm 0.0149$	$0.7054\pm 0.0115$	$0.8153\pm 0.0128$	$3.1921\pm 0.1267$	$0.8975\pm 0.0117$
MF	$1.2518\pm 0.0158$	$2.7886\pm 0.0524$	$1.7604\pm 0.0281$	$0.8378\pm 0.0119$	$1.8527\pm 0.0301$	$5.9481\pm 0.2481$	$1.3899\pm 0.0197$
kNN	$0.9845\pm 0.0187$	$0.7939\pm 0.0096$	$0.8121\pm 0.0125$	$0.7951\pm 0.0121$	$0.7699\pm 0.0157$	$\mathbf{0.8413\pm 0.0122}$	$0.7794\pm 0.0094$
LR	$0.8925\pm 0.0170$	$0.8659\pm 0.0090$	$0.7776\pm 0.0116$	$0.7208\pm 0.0099$	$0.7080\pm 0.0128$	$1.0219\pm 0.0161$	$0.7366\pm 0.0095$
Ours (no dc)	$0.9451\pm 0.0164$	$\mathbf{0.7878\pm 0.0095}$	$0.7450\pm 0.0109$	$\mathbf{0.6767\pm 0.0094}$	$0.7147\pm 0.0119$	$0.9922\pm 0.0227$	$\mathbf{0.7336\pm 0.0094}$
Ours	$\mathbf{0.8880\pm 0.0160}$	$0.7991\pm 0.0107$	$\mathbf{0.7433\pm 0.0105}$	$0.6808\pm 0.0101$	$\mathbf{0.6741\pm 0.0107}$	$0.9253\pm 0.0166$	$0.7564\pm 0.0136$

Table 4: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret for

\mathbf{c}\in\mathcal{C}

across multiple quality metrics on BigBIRD + YCB. Ours (no dc) denotes our method without decoupling.

Object	Hunyuan3D	InstantMesh	TRELLIS	TripoSR	Ours (novel obj.)	Ours (novel view)
Tomato Soup Can	$0.7174$ / $0.0685$	$0.0227$ / $0.9164$	$0.3864$ / $0.2071$	$0.6585$ / $0.1807$	InstantMesh	InstantMesh
Cup (vC)	$0.0$ / $0.4019$	$0.2273$ / $0.1381$	$0.0$ / $0.4372$	$0.1500$ / $0.1487$	Hunyuan3D	Hunyuan3D
Marble	$0.0$ / $0.9580$	$0.0$ / $0.7423$	$0.0$ / $0.1464$	$0.0$ / $0.9537$	TripoSR	TripoSR
Chef Can	$0.0$ / $0.8543$	$0.0$ / $0.8763$	$0.0250$ / $0.9406$	$0.1190$ / $0.7730$	InstantMesh	TRELLIS
Potted Meat	$0.0$ / $0.6589$	$0.6098$ / $0.4604$	$0.0682$ / $0.7573$	$0.3571$ / $0.4632$	InstantMesh	TRELLIS
Cup (vB)	$0.0$ / $0.2960$	$0.0238$ / $0.1423$	$0.1087$ / $0.0338$	$0.3659$ / $0.1080$	Hunyuan3D	Hunyuan3D

Table 5: Grasp collision rate (

\downarrow

) / mesh-IoU (

\uparrow

) for each reconstruction model across 10 YCB objects. Grasps are generated on the reconstructed mesh using a two-fingered gripper [ 23 ] and evaluated against the ground-truth mesh. SCOUT is evaluated in two settings: routing over novel views of known objects, and routing over both novel views and novel objects.

Table 9

Method	$\{\mathbf{c}\in\mathcal{C}_{0}\}$	$\{\mathbf{c}\in\mathcal{C}\}$	$\{\mathbf{c}\in\mathcal{C}\cap\,\mathbf{c}_{\mathrm{inv}}=\infty\}$
Ours w/ smoothing	$0.6386\pm 0.0033$	$0.4319\pm 0.0021$	$0.4831\pm 0.0019$
Ours w/o smoothing	$0.6588\pm 0.0035$	$0.4371\pm 0.0021$	$0.4878\pm 0.0020$

Table 8: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret across different cost subspaces on the GSO dataset.

Method	$\{\mathbf{c}\in\mathcal{C}_{0}\}$	$\{\mathbf{c}\in\mathcal{C}\}$	$\{\mathbf{c}\in\mathcal{C}\cap\,\mathbf{c}_{\mathrm{inv}}=\infty\}$
LR w/ smoothing	$0.6515\pm 0.0034$	$0.4723\pm 0.0022$	$0.5064\pm 0.0019$
LR w/o smoothing	$0.6497\pm 0.0033$	$0.4519\pm 0.0021$	$0.5047\pm 0.0019$
kNN w/ smoothing	$0.6515\pm 0.0034$	$0.4723\pm 0.0022$	$0.5064\pm 0.0019$
kNN w/o smoothing	$0.6489\pm 0.0033$	$0.4630\pm 0.0022$	$0.4992\pm 0.0019$

Table 9: Regret (

\downarrow

) as a proportion of the input-agnostic baseline regret across different cost subspaces on the GSO dataset.

Metric	DCD	Chamfer L2	Chamfer L1	IoU	MMD-EMD	Eval3D-geo	Eval3D-struct
$T$	$0.5$	$0.1$	$0.2$	$2.0$	$0.15$	$0.1$	$0.2$

Table 10: Softmax temperature

T

per metric for BigBIRD + YCB.

Panel of Figure 4 — Figure 4: Degenerate and nondegenerate views of a cracker box.

Panel of Figure 5 — Figure 5: Different lighting conditions for a fixed viewpoint.

Panel of Figure 6 — Figure 6: Sensitivity of hyperparameters for SCOUT evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Figure 7: Sensitivity of alpha for SCOUT evaluated on novel objects in the GSO dataset over cost coefficient vectors sampled from $\mathcal{C}$ .

Panel of Figure 8 — Figure 8: Sensitivity of hyperparameters for SCOUT without decoupling evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Figure 9: Sensitivity of the number of nearest neighbors for the kNN baseline evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Figure 10: Sensitivity of alpha for the linear regression baseline evaluated on novel objects in the GSO dataset over the cost coefficient vector $\mathcal{C}_{0}$ .

Metric	True $z$	Equiweighted	One-hot
DCD	${3.24\text{e-}}4$	$5.44\text{e-}1$	${2.38\text{e-}24}$
Chamfer L2	${4.01\text{e-}35}$	${5.74\text{e-}35}$	$8.11\text{e-}1$
Chamfer L1	${2.03\text{e-}4}$	${3.92\text{e-}2}$	${7.71\text{e-}16}$
IoU	$4.35\text{e-}1$	$2.31\text{e-}1$	${6.89\text{e-}5}$
MMD-EMD	${5.48\text{e-}13}$	${3.32\text{e-}6}$	${4.10\text{e-}4}$
Eval3D-geo	${8.24\text{e-}34}$	${2.70\text{e-}30}$	${9.09\text{e-}4}$

Table 11: One-sided t-test

p

-values comparing our weighted partition function proxy against three alternatives on BigBIRD + YCB. Each cell reports the

p

-value for the hypothesis that our proxy achieves lower regret, averaged over cost coefficient vectors sampled from

\mathcal{C}

( Sec. 4.1 ). An improvement indicates a metric where our proxy achieves statistically significant (

\alpha=0.05

) lower regret than the alternative; a deterioration (

\alpha=0.05

for the reverse one-sided test) indicates metrics where the alternative achieves statistically significant lower regret than ours.

Figure 11: Evaluation of grasp proposals based on the reconstructed meshes from the input image. The figure shows the following from left to right: (1) original input image inputted to SCOUT, (2) 3D reconstructions from the original image with the model chosen by SCOUT boxed in green, (3) grasp proposals on the reconstructed mesh, (4) grasp proposals evaluated on the ground-truth mesh; colliding grasps are shown in red.

Figure 12: Evaluation of grasp proposals based on the reconstructed meshes from the input image. The figure shows the following from left to right: (1) original input image inputted to SCOUT, (2) 3D reconstructions from the original image with the model chosen by SCOUT boxed in green, (3) grasp proposals on the reconstructed mesh, (4) grasp proposals evaluated on the ground-truth mesh; colliding grasps are shown in red.

Figure 13: Evaluation of grasp proposals based on the reconstructed meshes from the input image. The figure shows the following from left to right: (1) original input image inputted to SCOUT, (2) 3D reconstructions from the original image with the model chosen by SCOUT boxed in green, (3) grasp proposals on the reconstructed mesh, (4) grasp proposals evaluated on the ground-truth mesh; colliding grasps are shown in red.

Panel of Figure 14 — Figure 14: Oracle and learned routing policies across object categories.

Panel of Figure 15 — Figure 15: Oracle and learned routing policies across viewpoint elevations. High: $75^{\circ}$ and $85^{\circ}$ ; low: $35^{\circ}$ , $45^{\circ}$ , and $60^{\circ}$ .

3D Mesh Reconstruction

3D mesh reconstruction turns images, depth, or sensor data into explicit surfaces for measurement, editing, and simulation.

See all 109 papers

UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perception
Feb 26, 2026Mohammad Mahdavian, Gordon Tan, Binbin Xu +33D Reconstruction Multi-View 3D Reconstruction
Huawei Noah’s Ark Lab · University of Toronto
Can Single-View Mesh Reconstruction Generalize to Robot Camera Rotation?
Jun 22, 2026Yu Zhan, Guangcheng Chen, Hanjing Ye +4Monocular Depth Estimation 3D Mesh Reconstruction
Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology (SUSTech) · Pengcheng Laboratory
RoMa- $Ω$ : What Feed-Forward 3D Models Know About Image Matching
Sep 8, 2026David Nordström, Xinyue Zhang, Thibaut Loiseau +2Image Matching 3D Representation Learning
Chalmers University of Technology, Sweden · Mobile Perception Lab, ShanghaiTech University, China · LIGM, Ecole des Ponts, Univ. Gustave Eiffel, CNRS, France

Feb 26, 2026J/K move · Enter open · S save

UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perception

Mohammad Mahdavian, Gordon Tan, Binbin Xu, Yuan Ren+2

Huawei Noah’s Ark Lab · University of Toronto

3D Reconstruction Multi-View 3D Reconstruction Feed-Forward 3D Reconstruction Robotic Perception Camera Pose Estimation

Open paper AskPDF arXiv

We present UniScale, a unified, scale-aware multi-view 3D reconstruction framework for robotic applications that flexibly integrates geometric priors through a modular, semantically informed design. In vision-based robotic navigation, the accurate extraction of environmental structure from raw image sequences is critical for downstream tasks. UniScale addresses this challenge with a single feed-forward network that jointly estimates camera intrinsics and extrinsics, scale-invariant depth and point maps, and the metric scale of a scene from multi-view images, while optionally incorporating auxiliary geometric priors when available. By combining global contextual reasoning with camera-aware feature representations, UniScale is able to recover the metric-scale of the scene. In robotic settings where camera intrinsics are known, they can be easily incorporated to improve performance, with additional gains obtained when camera poses are also available. This co-design enables robust, metric-aware 3D reconstruction within a single unified model. Importantly, UniScale does not require training from scratch, and leverages world priors exhibited in pre-existing models without geometric encoding strategies, making it particularly suitable for resource-constrained robotic teams. We evaluate UniScale on multiple benchmarks, demonstrating strong generalization and consistent performance across diverse environments. We will release our implementation upon acceptance.

Abstract

3D Mesh Reconstruction

Explore similar work

UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perception

Can Single-View Mesh Reconstruction Generalize to Robot Camera Rotation?

RoMa-ΩΩΩ: What Feed-Forward 3D Models Know About Image Matching

RoMa- $Ω$ : What Feed-Forward 3D Models Know About Image Matching