Verbal-prompted segmentation is inherently limited by the expressiveness of natural language and struggles with uncommon, instance-specific, or difficult-to-describe objects: scenarios frequently encountered in manufacturing and 3D printing environments. While image exemplars provide an alternative, they primarily encode appearance cues such as color and texture, which are often unrelated to a part's geometric identity. In industrial settings, a single component may be produced in different materials, finishes, or colors, making appearance-based prompting unreliable. In contrast, such objects are typically defined by precise CAD models that capture their canonical geometry. We propose a CAD-prompted segmentation framework built on SAM3 that uses canonical multi-view renderings of a CAD model as prompt input. The rendered views provide geometry-based conditioning independent of surface appearance. The model is trained using synthetic data generated from mesh renderings in simulation under diverse viewpoints and scene contexts. Our approach enables single-stage, CAD-prompted mask prediction, extending promptable segmentation to objects that cannot be robustly described by language or appearance alone.
Figures & tables
Fig. 2 : Model architecture. Canonical multi-view renderings of a CAD mesh are encoded into geometry-conditioned exemplar embeddings and concatenated as prompt tokens for SAM3’s fusion transformer. The fused features are processed by the detector to produce instance masks in a single forward pass.
Fig. 3 : Geometry-conditioned training pipeline. (a) Canonical CAD renderings serve as prompts. (b) Synthetic training data with instance annotations.
Fig. 4 : Representative samples from the custom 3D printing dataset.
Method
PQ ↑
F1 ↑
mIoU ↑
PerSAM (Original)
0.314
0.337
0.250
PerSAM (Adapted)
0.318
0.341
0.240
Matcher (Original)
0.249
0.280
0.443
Matcher (Adapted)
0.514
0.573
0.446
SAM3 Exemplar Prompt
0.141
0.156
0.132
CAD-Prompted SAM3 (Ours)
0.754
0.803
0.749
TABLE I : Performance on the custom 3D printing dataset.
Method
T-LESS
ITODD
PQ ↑
F1 ↑
mIoU ↑
PQ ↑
F1 ↑
mIoU ↑
PerSAM (Original)
0.168
0.190
0.080
0.347
0.419
0.322
PerSAM (Adapted)
0.166
0.188
0.082
0.363
0.438
0.322
Matcher (Original)
0.163
0.184
0.248
0.263
0.357
0.631
Matcher (Adapted)
0.173
0.219
0.177
0.277
0.379
0.478
SAM3 Exemplar Prompt
0.171
0.189
0.138
0.415
0.507
0.415
TABLE II : Performance on T-LESS and ITODD benchmarks.
Fig. 5 : Qualitative comparision of CAD-Prompted SAM3 against baselines in semantic segmentation task
Object
Ours
COPA
Shaft Mount
9/10
1/10
Gear
8/10
2/10
Square Tube
8/10
3/10
Bolt
6/10
2/10
LEGO Brick
9/10
3/10
TABLE III : Pick-and-place success rates for different objects, comparing our method with COPA [ 24 ] .
Fig. 6 : Zero-shot Manipulation Pipeline based on CAD-Prompted SAM3
Stage
Ours
COPA
Module
Time (s)
Module
Time (s)
Detection
CAD-Prompted SAM3
0.170
SAM
5.80
GPT5.4 API
1.29
Pose Estimation
ICP
0.112
–
–
Grasp Prediction
Physics-Based
0.188
GraspNet
0.172
Total
0.460
7.26
TABLE IV : Per-object average runtime breakdown for our method and COPA. Both methods are evaluated on a computer with NVIDIA RTX 4070 GPU and Intel Core Ultra 9 185H CPU.