Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion

Kartik Bali; Mahish Guru; Yiderigun Borjigin; Alexandra Starostina; Christian J. Cyron; Roland Aydin

cs.CVOct 8, 2026

Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion

Authors: Kartik Bali, Mahish Guru, Yiderigun Borjigin, Alexandra Starostina, Christian J. Cyron, Roland Aydin

Organizations: Helmholtz-Zentrum Hereon · Leuphana Universität Lüneburg · Universität des Saarlandes · Technische Universität Hamburg

Abstract

CAD reconstruction methods assume a luxury reality rarely grants: unrestricted visual access to the object, photographed from any desired angle. Real objects, however, are scene-embedded, bolted against walls, wedged into corners, resting on floors, where the scene renders much of the view sphere unreachable and the remaining views unequally informative. We introduce \textbf{SightCAD}, a framework for parametric CAD reconstruction that treats view feasibility as a first-class constraint. In this work we consider objects from standard CAD benchmarks embedded in realistic indoor scenes with physically derived visibility constraints over a discrete view sphere. A learned view selector must choose $K$ feasible views for a vision--language model (VLM) that generates executable CadQuery code, scored by geometric fidelity of the executed solid. Because reward arrives only after discrete view selection, autoregressive generation, and CAD-kernel execution, we propose a joint training paradigm in which the view selector and the CAD-generation VLM are trained together against this reward. The learned selection policy departs sharply from random, uniform, and coverage-greedy alternatives, outperforming surface-area maximization (SA-max) by up to $6.4$ Intersection-over-Union (IoU) points across budgets $K\in\{1,\dots,5\}$ . The full system surpasses strong external baselines on scene-embedded, occluded multi-view renders of DeepCAD and Fusion360 objects ( $+21$ and $+17$ effective-mIoU points over the best baseline, respectively), as well as on test-time domain-canonicalized real images from the industrial T-LESS benchmark and on both synthetic and real images from the MP6D industrial metal-parts benchmark, while producing the highest rate of executable programs of any method compared (invalid-code rate ${\leq}1.5\%$ ).

Figures & tables

Figure 1: SightCAD joint training paradigm: the view selector actively learns the VLM’s view preferences via GRPO while the VLM learns to understand and exploit view selections that maximize reward via SFT. Here $\psi$ and $\theta$ are the trainable parameters of the view selector and the code-generation VLM, respectively; $S$ is a subset of the calibrated views $\mathcal{V}$ ; $\mathrm{IR}$ is the invalid-code penalty ( $-0.5$ , Eq. 1 ); and $M_{g}$ is the mesh computed by the execution operation $\mathcal{E}$ ( $\mathrm{exec}$ in text) on code $Y_{g}$ .

Figure 2: Scene-embedded CAD objects. Objects from CAD-Recode and Text2CAD are placed into furnished SceneNet office rooms under three occlusion regimes: floor (free-standing), wall (one open half-space), and corner (wedged between two walls). Placement determines which of the $N{=}36$ candidate views are feasible.

Figure 3: SightCAD generator warm-starting on variable- $K$ multi-view subsets: candidate views are sampled under three view policies (random, uniform, SA-max) with $K\sim\mathrm{U}\{1,\dots,5\}$

Figure 4: SightCAD generator RL tuning: the view selector trained in the joint stage is kept frozen while the VLM further learns to understand views and express code better via GRPO on high-temperature code rollouts. All mathematical expressions are as per Fig. 1 . KL regularization toward the SFT reference ( $\beta\,\widehat{\mathbb{D}}_{\mathrm{KL}}[\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}]$ , Eq. 6 ) omitted for clarity.

	SA-max ( a )		fw-SA ( a )		Learned ( a )		Policy-neutral generator ( b )
$K$	SFT	+GRPO	SFT	+GRPO	SFT	+GRPO	Random	Uniform	SA-max	fw-SA	Learned
1	45.80	53.05	44.15	50.98	49.97	55.86	24.36	23.79	40.90	39.23	44.05
2	47.36	54.28	46.72	53.75	50.68	56.95	33.11	36.40	44.44	44.32	46.04
3	47.75	54.72	47.66	54.40	51.69	58.51	36.89	40.57	46.02	46.11	47.46
4	48.15	55.16	48.26	54.99	52.90	60.52	38.82	42.68	46.30	46.59	47.81
5	48.55	55.40	48.89	55.29	53.88	61.79	39.85	43.81	46.01	46.68	48.86

Table 1: What view selection is worth, and what co-adaptation adds. Left ( a ): each policy is paired with the generator trained alongside it, at both stages (SFT, +GRPO). Right ( b ): a single policy-neutral warm-started generator decodes the views chosen by five selection policies fw-SA is a curvature-saliency-reweighted variant of SA-max (Appendix A.4 ). Scene-embedded test split, mIoU eff (%); standard errors are

0.46

–

0.53

throughout (

n{=}4000

objects).

	DeepCAD				Fusion360
Method	IR% $\downarrow$	medCD $\downarrow$	mIoU% $\uparrow$	mIoU eff $\uparrow$	IR% $\downarrow$	medCD $\downarrow$	mIoU% $\uparrow$	mIoU eff $\uparrow$
Ours ( $K{=}4$ , +GRPO)	0.00	5.36	68.49	68.49	1.00	10.42	53.56	53.03
Ours ( $K{=}4$ , SFT)	0.00	6.83	62.18	62.18	1.00	11.96	49.86	49.36
cadrille	4.00	18.30	49.05	47.09	5.00	33.62	37.73	35.85
CAD-Fit	6.00	21.07	47.08	44.26	17.00	26.46	40.32	33.47
CAD-Coder	2.00	37.10	38.29	37.52	3.00	66.61	28.56	27.71

Table 2: DeepCAD-test and Fusion360-test (

n{=}100

each).

	Synth ( $n{=}30$ )				Real ( $n{=}138$ )				Canon ( $n{=}138$ )
Method	IR%	medCD	mIoU%	mIoU eff	IR%	medCD	mIoU%	mIoU eff	IR%	medCD	mIoU%	mIoU eff
Ours ( $K{=}4$ , +GRPO)	0.00	18.79	54.02	54.02	1.45	111.59	8.38	8.26	0.00	43.56	32.48	32.48
Ours ( $K{=}4$ , SFT)	3.33	24.47	46.76	45.20	4.35	101.41	9.68	9.25	11.59	71.36	19.83	17.53
cadrille	10.00	36.45	34.92	31.43	24.64	104.27	13.39	10.09	27.54	81.21	15.11	10.95
CAD-Fit	13.33	12.53	57.73	50.03	57.97	45.54	34.33	14.43	39.86	47.54	29.86	17.96
CAD-Coder	6.67	84.22	17.94	16.74	23.91	94.36	15.58	11.86	2.90	58.36	27.11	26.32

Table 3: T-LESS industrial objects (

n{=}138

real,

n{=}30

synthetic) .

	Synth ( $n{=}20$ )				Real ( $n{=}200$ )
Method	IR%	medCD	mIoU%	mIoU eff	IR%	medCD	mIoU%	mIoU eff
Ours ( $K{=}4$ , +GRPO)	0.00	20.55	48.31	48.31	0.50	57.91	25.71	25.58
Ours ( $K{=}4$ , SFT)	10.00	31.22	40.81	36.73	4.00	72.09	19.37	18.59
cadrille	10.00	26.91	43.03	38.73	30.50	77.94	18.44	12.82
CAD-Fit	20.00	29.51	42.39	33.91	21.50	71.03	21.85	17.16
CAD-Coder	15.00	49.47	27.51	23.38	7.50	66.28	21.45	19.84

Table 4: MP6D industrial objects (

n{=}200

real,

n{=}20

synthetic).

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

	Learned				SA-max				fw-SA
$K$	ortho. #	az-gap ( ∘ )	cov. %	rig %	ortho. #	az-gap ( ∘ )	cov. %	rig %	ortho. #	az-gap ( ∘ )	cov. %	rig %
1	0.30	—	43.5	8.7	0.16	—	47.4	6.5	0.26	—	45.5	5.4
2	0.61	52.5	54.0	16.5	0.31	135.6	75.8	9.4	0.45	131.2	73.5	9.6
3	1.25	15.8	60.7	21.6	0.79	55.3	81.1	16.3	0.92	55.0	80.5	17.2
4	2.15	11.2	66.3	39.2	1.29	31.0	82.5	25.6	1.41	29.8	82.5	26.4
5	2.13	12.6	72.1	43.8	1.70	21.3	82.9	34.8	1.82	20.0	83.1	36.3

Appendix

Table 5: Learned selection policy over the full evaluation set (4,000 test (CAD-Recode and Text2CAD) + 100 DeepCAD + 100 Fusion360, 1365 floor / 977 wall / 1858 corner), per budget

K

. ortho. # is the number of views in a selection whose azimuth is cardinal (

0^{\circ},90^{\circ},180^{\circ},270^{\circ}

on the 12-azimuth grid), averaged over objects. az-gap is the minimum pairwise azimuth gap within a selection; cov. is the fraction of the object’s surface area physically visible from the union of the selected views, rig is the mean fraction of a selection lying on the modal

K

-slot rig (the

K

globally most frequent view slots of that policy at that budget). fw-SA is the feature-weighted variant of SA-max whose greedy coverage objective reweights surface points by local curvature saliency (Appendix A.4 ). All columns span the full 4,200-object set for all three policies.

Figure 5: Three randomly sampled CAD objects from the view analysis split, showing the learned selector’s picks (left) beside the SA-max chain’s first $K$ picks (right) for every budget $K\in\{1,\dots,5\}$ ; each tile is tagged with its (elevation, azimuth) pose. The patterns quantified in the table are directly visible: the learned selector squares views to the cardinal azimuths and stacks elevations at a shared azimuth (small min az-gap), and its per-budget selections are non-nested, whereas SA-max spreads toward antipodal, oblique, surface-grazing views to maximize coverage.

Figure 6: Controlled same-scene study of geometry-dependent selection. Six parts spanning the geometry taxonomy: a ring with a lateral through-hole and a flat plate (top row), a block stack pierced by a circular bore and a U-bracket (middle row), a cylinder–block assembly and a hollow cylinder (bottom row); are placed in the same room at the same corner anchor with an identical shared yaw of $45^{\circ}$ . Each panel shows the frozen per- $K$ selectors’ picks for budgets $K\in\{1,\dots,5\}$ ; each tile is tagged with its (elevation, azimuth) pose. With scene, anchor, lighting, and feasibility held fixed, all differences between panels are attributable to part geometry alone.

Figure 7: Executed per-property correctness under five view-selection policies decoded by the same policy-neutral generator, averaged over

K\in\{1,\dots,5\}

. Program validity is view-policy independent; the graded properties degrade in the ordering of Table 1 b; extrude depth is the major property on which learned selection separates from SA-max.

Panel of Figure 8 — Figure 8: The view-class mechanism, averaged over budgets $K$ (learned-selector picks, token-column ablation with other-views-mean replacement). (a) Deleting the in-plane pick damages extrude-depth tokens while deleting the face-on pick damages sketch-coordinate tokens. (b) Late-readout analysis: extrude-token damage from deleting the in-plane pick remains approximately flat through L16 and collapses to zero by L24, while sketch-token damage from deleting the face-on pick follows the same late collapse around the logit-lens digit-commitment window. Depth and sketch parameters are thus read late, from disjoint view classes, at every budget $K$ .

Figure 9: In-plane and face-on view supply for all view selection policies considered in this work. Occluded (contaminated) views, together with the frequency of in-plane and face-on views, correlate with the loss in accuracy on the extrude depth and sketch parameters, respectively, in Fig. 7 .

Figure 10: DeepCAD-test model inputs. For eight held-out objects, the exact visual input consumed by each method: Ours receives the four scene-embedded views chosen by the learned selector (2 $\times$ 2 montage); cadrille its native yellow four-quadrant view grid with scene-occluded quadrants whited out; CAD-Fit and CAD-Recode $\leftarrow$ TripoSR their native white-background single-view studio render; CAD-Coder and GenCAD their native isometric / edge-lattice render, blanked where the scene occludes the native camera. Rows are ordered by mIoU eff .

Figure 11: DeepCAD-test reconstructions. Executed CAD for the objects of Fig. 10 : ground truth (top) and each method’s solid, rendered semi-transparent so internal features are visible and aligned by its per-object best-fit proper rotation (the same rotation scores the printed IoU), so shape quality is shown independent of coordinate-frame convention.

Figure 12: Fusion360-test model inputs. For eight held-out objects, the exact visual input consumed by each method: Ours receives the four scene-embedded views chosen by the learned selector (2 $\times$ 2 montage); cadrille its native yellow four-quadrant view grid with scene-occluded quadrants whited out; CAD-Fit and CAD-Recode $\leftarrow$ TripoSR their native white-background single-view studio render; CAD-Coder and GenCAD their native isometric / edge-lattice render, blanked where the scene occludes the native camera. Rows are ordered by mIoU eff .

Figure 13: Fusion360-test reconstructions. Ground truth and per-method executed solids for the objects of Fig. 12 , shown translucent and aligned by each object’s best-fit rotation with the corresponding IoU.

Figure 14: T-LESS synth model inputs. For eight held-out objects, the exact visual input consumed by each method: Ours receives the four scene-embedded views chosen by the learned selector (2 $\times$ 2 montage); cadrille its native yellow four-quadrant view grid with scene-occluded quadrants whited out; CAD-Fit and CAD-Recode $\leftarrow$ TripoSR their native white-background single-view studio render; CAD-Coder and GenCAD their native isometric / edge-lattice render, blanked where the scene occludes the native camera. Rows are ordered by mIoU eff .

Figure 15: T-LESS synth reconstructions. Ground truth and per-method executed solids for the objects of Fig. 14 , translucent and aligned by each object’s best-fit rotation with the corresponding IoU.

Figure 16: T-LESS real model inputs. The real photographs fed to each method for eight T-LESS captures, without any test-time domain match (conventions as in Fig. 10 ); ours consumes the real scene-embedded views directly.

Figure 17: T-LESS real reconstructions. Ground truth and per-method executed solids for the objects of Fig. 16 , translucent and aligned by each object’s best-fit rotation with the corresponding IoU. This is the one regime where our framework trails the lift baselines, reflecting the raw-photograph domain gap discussed in the main text.

Figure 18: T-LESS real (canon) model inputs. The same real captures as Fig. 16 after the model-agnostic, test-time-only domain match, composited onto each method’s native plate (conventions as in Fig. 10 ).

Figure 19: T-LESS real (canon) reconstructions. Ground truth and per-method executed solids for the objects of Fig. 18 , translucent and aligned by each object’s best-fit rotation with the corresponding IoU.

Figure 20: MP6D synth model inputs. For eight held-out objects, the exact visual input consumed by each method: Ours receives the four scene-embedded views chosen by the learned selector (2 $\times$ 2 montage); cadrille its native yellow four-quadrant view grid with scene-occluded quadrants whited out; CAD-Fit and CAD-Recode $\leftarrow$ TripoSR their native white-background single-view studio render; CAD-Coder and GenCAD their native isometric / edge-lattice render, blanked where the scene occludes the native camera. Rows are ordered by mIoU eff .

Figure 21: MP6D synth reconstructions. Ground truth and per-method executed solids for the objects of Fig. 20 , translucent and aligned by each object’s best-fit rotation with the corresponding IoU.

Figure 22: MP6D real model inputs. The real photographs fed to each method for eight MP6D captures, without any test-time domain match (conventions as in Fig. 10 ); ours consumes the real multi-view input directly.

Figure 23: MP6D real reconstructions. Ground truth and per-method executed solids for the objects of Fig. 22 , translucent and aligned by each object’s best-fit rotation with the corresponding IoU. Unlike real T-LESS, MP6D captures are acquired on a plain background, and our framework leads every baseline on this real condition (Table 4 ).

Figure 24: View-selector pretraining by SA-max: the greedy surface-area set-cover SA-max is distilled into an image-based scorer. The coverage (cov.) values denote the cumulative coverage of the object as the views increase from left to right. After training a central view selector we then train seperate view selectors for Joint-training for every view budget K= $\{1,..5\}$ .

Figure 25: Align-net training for pose estimation on captured renders of CAD objects, used for the T-LESS (real and canon) and MP6D evaluations to re-anchor view azimuths to the canonical frame before CAD generation.

	Joint training	Generator RL tuning
Group size $G$	4	8
Sampling temperature	$1.0$ (view subsets)	$0.9$ (decodes)
Batch size	64 global (4 GPUs)	12 per GPU (4 GPUs)
Selector optimizer	AdamW, lr $10^{-4}$	frozen
Generator optimizer	AdamW, lr $10^{-5}$ , wd $10^{-2}$	AdamW, lr $10^{-5}$
SFT weight	$0.5$	—

Appendix

Table 6: Optimization hyperparameters of the two RL training phases (Sec. 4.3 ).

Policy	mIoU eff $\uparrow$	IR% $\downarrow$
Random	36.89	2.03
Uniform	43.80	1.85
SA-max	47.75	2.00
fw-SA	47.76	2.00
Learned (Joint-Training)	51.69	1.80

Appendix

Table 7: View policies with co-trained generators at

K{=}3

. Each heuristic policy’s generator is fine-tuned for one epoch on that policy’s views from the same policy-neutral initialization. Scene-embedded 4k test split, mIoU eff (%), standard errors

0.47

–

0.53

$\Delta$ mIoU eff (pp)	$K{=}1$	$K{=}2$	$K{=}3$	$K{=}4$	$K{=}5$
Policy-neutral	$+3.2$ [2.4, 3.8]	$+1.6$ [0.8, 2.4]	$+1.4$ [0.7, 2.2]	$+1.5$ [0.7, 2.3]	$+2.9$ [2.0, 3.7]
Co-adapted +GRPO	$+2.8$ [2.3, 3.3]	$+2.7$ [2.1, 3.2]	$+3.8$ [3.2, 4.4]	$+5.4$ [4.8, 6.0]	$+6.4$ [5.8, 7.0]
floor	$+3.8$	$+2.8$	$+4.1$	$+6.0$	$+7.3$
wall	$+2.2$	$+2.2$	$+4.2$	$+4.7$	$+7.6$
corner	$+2.4$	$+2.8$	$+3.4$	$+5.3$	$+5.1$

Appendix

Table 8: Paired learned

-

SA-max contrasts on the 4k scene-embedded test split.

\Delta

mIoU eff (pp), paired per object, with task-bootstrap 95% CIs; all entries are significant under two-sided Wilcoxon signed-rank tests (

p<0.02

for the policy-neutral row,

p<10^{-4}

for the co-adapted row and every regime cell). Regimes: 1305 floor / 933 wall / 1762 corner.

Method	IR% $\downarrow$	medCD $\downarrow$	mIoU% $\uparrow$	mIoU eff $\uparrow$
Ours ( $K{=}4$ , +GRPO)	0.30	8.10	66.70	66.50
Ours ( $K{=}4$ , SFT)	1.25	11.68	60.00	59.25
cadrille	17.12	13.02	54.30	45.01
CAD-Coder	33.40	48.20	37.75	25.14
CAD-Recode $\leftarrow$ TripoSR	43.23	84.94	18.43	10.46
GenCAD	77.00	73.18	18.72	4.31

Appendix

Table 9: External baselines on the full scene-embedded 4k test split (

n{=}4000

CAD-Recode + Text2CAD objects). Every method receives its native input format and the scene contributes only occlusion: Our method leads all other baselines.

Explore similar work

Jul 5, 2026cs.CV

SOV-CAD: Stepwise Orthographic Views Guided CAD Modeling Sequence Reconstruction

Reconstructing Computer-Aided Design (CAD) modeling sequences from images is crucial for preserving design intent and supporting parametric editing. However, existing methods typically generate full CAD sequences holistically, overlooking the iterative, feedback-driven nature of human design workflows. We address this limitation by introducing the rich stepwise visual supervision: at each modeling step, the system observes the target's orthographic projections, the projections of the incrementally constructed model, and the active sketch, enabling informed action selection. To effectively leverage this on-the-fly feedback, we propose SOV-CAD, a framework that formulates CAD reconstruction as a sequential decision-making task and employs offline reinforcement learning with a Decision Transformer architecture. This design incorporates continuous visual feedback guided by geometric alignment rewards, resulting in a more accurate and human-like modeling process. Extensive experiments show that SOV-CAD surpasses state-of-the-art methods in CAD sequence reconstruction while exhibiting strong data efficiency. Code of SOV-CAD is available at: https://github.com/LukePhong/SOV-CAD

Zhaopeng Feng, Chen Zhi, Xuhong Zhang +2

School of Software Technology, Zhejiang University · Zhejiang Key Laboratory of Digital-Intelligence Service Technology, Zhejiang University · ZWSOFT Co., Ltd.

3D Reconstruction Parametric CAD Modeling

Ask PDF

May 2, 2026cs.CV

CADFit: Precise Mesh-to-CAD Program Generation with Hybrid Optimization

Despite recent progress, recovering parametric CAD construction sequences from geometric input, such as meshes or point clouds, is a key challenge for design and manufacturing, as existing CAD reconstruction and generation methods are largely restricted to difficult-to-edit formats like meshes or Breps or editable simple sketch-and-extrude pipelines and low-complexity datasets. We introduce CADFit, a hybrid optimization-based CAD reconstruction framework that recovers complex, editable CAD construction sequences from meshes by incrementally fitting and validating parametric operations using geometric feedback. Our approach is distinguished by formulating reconstruction as an IoU-driven optimization over structured CAD programs and supporting a rich set of operations, including extrusions, revolutions, fillets, and chamfers. Experiments on multiple CAD benchmarks show that CADFit outperforms state-of-the-art mesh-to-CAD methods in volumetric Intersection-over-Union and Chamfer Distance, while substantially reducing the Invalid Ratio of reconstructed CAD programs, particularly for complex designs. We further present a multimodal pipeline that enables end-to-end reconstruction of CAD construction sequences from images by combining image-based geometry reconstruction with CADFit. By enabling accurate reconstruction of higher-complexity CAD models, CADFit provides a practical foundation for generating richer datasets and advancing future learning-based approaches to CAD reverse engineering. The code is available at: https://github.com/ghadinehme/CADFit.

Ghadi Nehme, Eamon Whalen, Faez Ahmed

Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Siemens Digital Industries Software, Plano, TX 75024, USA

3D Reconstruction Parametric CAD Modeling

Ask PDF

Oct 6, 2026cs.AI

CADFather: Autonomous CAD Reconstruction through Coordinated Tool Use

Reconstructing an editable CAD model from a 3D shape remains a challenging engineering task. Existing methods can propose CAD operations, but no single source of proposals works equally well across different part geometries and stages of reconstruction. We introduce CADFather, an autonomous agentic system that coordinates complementary tools to recover parametric CAD programs from 3D meshes. A vision-language assistant inspects renders of the target and intermediate reconstructions, then decides which candidate CAD programs to extend, which tools to invoke, how many proposals to generate, and when to finish. Learned and algorithmic tools propose CAD operations, while numerical optimization refines the parameters of existing programs. Proposed or refined programs are executed and evaluated to provide feedback for subsequent decisions. The agent maintains alternative candidate programs for each target part and preserves the best valid result throughout reconstruction. CADFather uses pretrained generation and assistant models without additional training. We evaluate reconstruction quality and execution validity on the full DeepCAD, Fusion360, and MCB test sets, as well as on CADENA-Bench, CADBench, and BenchCAD. We additionally analyze computational cost and the trade-off between cost and reconstruction quality.

Gennadiy Savrasov, Maksim Elistratov, Nikita Gavrilov +7

Lomonosov Moscow State University · Innopolis University · FusionBrain Lab

Parametric CAD Modeling 3D Reconstruction Program Synthesis 3D Mesh Reconstruction Agentic AI

Ask PDF