Fear of Memorizing Objects: End-to-End Zero-Shot Model-Based Object Detection
Authors: Leon Greiner, Wout Boerdijk, Junwen Huang, Benjamin Busam, Rudolph Triebel, Maximilian Durner
Organizations: German Aerospace Center (DLR), Wessling, Germany · Technical University of Munich (TUM), Munich, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
This paper presents FOMO (Fear of Memorizing Objects), an end-to-end trainable transformer-based architecture for zero-shot model-based object detection. Existing approaches typically follow a proposal-and-match pipeline: target-agnostic mask proposals are generated and then matched to template features in a set of loosely coupled stages, making them susceptible to fragmented proposals in clutter and degraded matching under occlusions. FOMO elegantly replaces this paradigm by injecting object template information early in the detection pipeline: Bidirectional attention between learnable latent tokens, multi-view CAD templates, and query image features produces object-aware image features and conditioned queries which are subsequently decoded into bounding-box and confidence predictions. We train FOMO on Clutter6D, a novel large-scale synthetic dataset with over 500k images featuring more than 10k diverse object models rendered in highly cluttered scenes with multiple instances and varying occlusion levels, which addresses the limited object diversity and lack of realistic dense clutter in existing datasets for training zero-shot multi-instance algorithms. In true zero-shot fashion, FOMO achieves 54.2 mAP on BOP-Classic-Core at 0.23 s per image, ranking second overall with 3.8 pp difference to the top method while being more than 7x faster. Throughout comprehensive ablation studies we investigate and verify architectural design choices.
Figures & tables
Figure 1: FOMO for zero-shot model-based object detection. (a) Conventional zero-shot detectors ( e.g . , CNOS [ 38 ] , top right) require mask proposals ( e.g . , SAM [ 27 ] ) and match these to object descriptors, leading to duplicate instances ( obj_3 ) or removed predictions due to high occlusions ( obj_13 ). In contrast, FOMO (bottom middle) directly predicts amodal bounding boxes for every prompted object in a single forward pass. (b) AP vs. inference time on the BOP-Classic-Core unseen-object 2D detection task [ 3 ] . FOMO achieves the second-highest AP at a fraction of the cost.
Figure 2: FOMO architecture. Rendered template views and the query image are encoded by a shared foundational backbone. The ICMT fuses both modalities, and a detection head produces final predictions. For better visualization only a single object prompt is shown; multi-object prompts are simply batched.
Figure 3: ICMT detail for multi-object settings. Each of L layers alternates latent token updates (attending to templates and image) with image feature conditioning via cross-attention to the latent tokens.
Dataset
Objects
Images
Categories
Inst./Img.
Multi-cls.
Multi-inst.
NOCS CAMERA [ 52 ]
1,085
300k
6
4.0 ∗
✓
✗
ShapeNet6D [ 19 ]
12,490
800k
51
5.5 ∗
✓
✗
MegaPose [ 29 ]
21,000
2M
n/a †
13.2 ∗
✓
✗
Omni6D-xl [ 59 ]
15,957
1.1M
419
6.3 ∗
✓
✗
FoundationPose [ 53 , 39 ]
42,000
1.2M
1,173
18.6 ∗
✓
✗
Omni6DPose SOPE [ 58 ]
4,162
475k
149
15.2 ∗
✓
✗
Table 1: Clutter6D vs. other model-based synthetic RGB datasets. Unlike prior large-scale ( i.e. , #objects) datasets, Clutter6D supports native multi-instances in every scene and achieves the highest number of objects per image visible, indicating significantly denser, more cluttered, and highly occluded scenes. ∗ Inst./Img. computed based on released annotations; † not reported in associated publication.
Figure 4: Clutter6D samples. Left: Clutter6D-HLD contains many object classes with moderate per-class instance counts. Right : Clutter6D-IND is comprised of fewer classes but many instances per class, yielding heavier same-class clutter and occlusion.
Method
Mean
LM-O
T-LESS
TUD-L
IC-BIN
ITODD
HB
YCB-V
Time [s]
ZeroPose [ 6 ]
34.1
36.7
30.0
43.1
22.8
25.0
39.8
41.6
3.82
CNOS (SAM) [ 38 ]
36.1
39.5
33.0
36.8
20.7
31.3
42.3
49.0
1.85
CNOS (FastSAM) [ 38 ]
42.8
43.3
39.5
53.4
22.6
32.5
51.7
56.8
0.22
NIDS-Net [ 35 ]
46.9
45.7
49.3
48.6
25.7
37.9
58.7
62.1
0.49
SAM-6D (FastSAM) [ 30 ]
44.4
43.8
41.7
54.6
23.4
37.4
52.3
57.3
0.25
FOSOD ∗
47.7
46.0
47.9
49.5
23.4
42.6
56.9
67.9
1.06
Table 2: Zero-shot detection on BOP core datasets (RGB only). All data from [ 3 ] . Time measured per query image averaged over all seven datasets; AP values in [%]. Best and second-best highlighted. † A second submission achieves 57.9% mean AP at 1.63 s.
Configuration
Mean
HLD
IND
w/o ICMT Stage 3 (late fusion)
55.4 ↓ 6.2
57.4
53.3
w/o latent-to-image cross-attn (w/o Stage 3)
53.7 ↓ 7.9
55.2
52.1
w/o latent-to-image cross-attn (w/ Stage 3)
58.2 ↓ 3.4
61.5
54.8
w/o Deformable DETR encoder
57.2 ↓ 4.4
61.1
53.2
w/o Deformable DETR encoder + decoder
47.0 ↓ 14.6
51.8
42.1
w/o embedding reconstruction loss
59.7 ↓ 1.9
62.1
57.3
Table 3: Architecture & training ablation on Clutter6D. AP ([%]) on the Clutter6D validation set. All variants were trained for 10 epochs on synthetic Clutter6D.
Configuration
Mean
LM-O
T-LESS
TUD-L
IC-BIN
ITODD
HB
YCB-V
w/o real-data fine-tuning
43.7 ↓ 10.5
44.2
26.2
45.5
28.5
30.9
63.3
67.1
w/o Clutter6d data aug.
48.2 ↓ 6.0
50.5
21.5
59.7
42.0
30.9
63.3
69.6
w/o ICMT Stage 2
52.3 ↓ 1.9
51.0
28.9
72.3
40.6
38.5
63.0
72.1
Full model
54.2
50.6
36.8
68.0
43.0
44.3
65.4
71.2
Table 4: Architecture & training ablation on BOP. AP ([%]) on BOP core datasets. All configurations include 1200 steps of real-data fine-tuning unless stated otherwise.
Figure 5: Query image feature evolution across ICMT layers on a real-world scene with multiple instances and classes. Left: Query image (top) and 12 template views (bottom). Top row: PCA projection of patch features to RGB, showing progressive semantic clustering around the target. Bottom row: L2 norm of feature vectors, revealing concentration at object locations.
Figure 10
Object Type
Baseline
D-S
D-T
D-S+T
No T
D-S+No T
Box
73.8
75.0
60.3
60.5
12.3
14.4
Cylindric
74.1
75.1
53.6
55.3
12.6
4.7
Misc
68.4
66.2
66.5
65.1
59.6
37.6
Mean across categories
72.1
72.1
60.1
60.3
28.2
18.9
Table 5: Mean AP values ([%]) across YCB-V objects merged into super-categories. D-S = degraded shape; D-T = degraded texture.
Parameter
Value
Notes
Backbone
Architecture
DINOv3 ViT-B/16
Hidden dimension
768
ICMT
Number of layers
3
Attention heads
12
Table 6: Model architecture and training hyperparameters.
Figure 8: Effect of augmentation scheduling on training samples. (a) Shape-focused phase applies aggressive augmentations including high rates of grayscale conversion, edge-only rendering, and texture suppression, forcing the model to learn geometric matching. (b) Base augmentation phase shows the decayed augmentation rates after the cosine schedule, allowing texture-based refinement while retaining the geometric foundation. In the query images, ground-truth bounding boxes are shown in green . Next to each query, the first template view of all prompted objects is displayed: templates with a green frame indicate positive prompts (objects present in the scene), while templates with a red frame indicate negative prompts (objects absent from the scene).
LM-O
T-LESS
TUD-L
IC-BIN
ITODD
HB
YCB-V
GPU
Mean
( P=8 )
( P=30 )
( P=3 )
( P=2 )
( P=28 )
( P=33 )
( P=21 )
H200
0.233
0.117
0.375
0.060
0.047
0.353
0.410
0.269
RTX 3090
0.610
0.302
0.975
0.160
0.133
0.923
1.069
0.706
Table 7: Per-image inference time [s] on NVIDIA H200 and RTX 3090 across BOP-Classic-Core datasets. Number of object prompts P per dataset shown in parentheses. Query-image batch size is 4 (H200) and 1 (RTX 3090).
Figure 9: Latent token analysis and inference scaling. (a) Gini coefficient of the ICMT Stage 1 cross-attention from latent tokens → query image patches increases across ICMT layers, showing progressive spatial grounding onto the prompted object. (b) Cumulative detection distribution across tokens shows near-uniform utilization rather than mode collapse. (a, b) computed on the Clutter6D val. set (50k images, 175k matched predictions). (c) Per-image inference time (H200) across the seven BOP-Classic-Core datasets (dots) scales approximately linearly with object prompts P .
Parameter
Value / Range
Distribution
Notes
Scene parameters
Images per scene
25
fixed
Resolution
720×720
fixed
Divisible by 16 (ViT compatibility)
Number of background textures
1,219
U{1,1219}
One sampled per scene
Object parameters (Clutter6D-HLD / Clutter6D-IND)
Classes per scene
10–20 / 2–5
U
Distinct object classes
Table 8: Summary of the dataset generation parameters and sampling distributions. Parameters shown as HLD / IND indicate values that differ between the two dataset subsets.
Figure 10: Clutter6D object catalog overview. (a) Samples from the Clutter6D object catalog, top-left from gso (category: Media Cases), top-right from Objaverse (category: Shoe), bottom-left from oo3d (category: Bucket Noodle), bottom-right from oo3d (category: Figurine). (b) Top 20 categories by frequency in the catalog, selected from 681 total categories with a mean of 14.2 objects per category, unlabelled objects excluded.
Figure 11: Object instance sampling distributions. Probability mass functions controlling the number of object instances per class in synthetic scenes for both dataset variants.
Figure 13: ICMT features on T-LESS. The first four rows show positive object prompts, while the last two rows show prompts of objects that are not in the query image. Note that the ICMT features in layer 3 are highly similar across rows.
We present B2TFPose, a training-free zero-shot method for 6DoF pose estimation of unseen objects from RGB images. Using a single frozen DINOv3 vision transformer as its only pretrained component within the pose estimation pipeline, B2TFPose extracts dense patch-level features that generalize across the synthetic-to-real domain gap without any task-specific fine-tuning, revisiting the classical local feature matching paradigm through the lens of large-scale self-supervised foundation models. Three contributions advance the training-free state of the art. A geodesic non-maximum suppression strategy retrieves a viewpoint-diverse template set for coarse-to-fine correspondence matching. Render-guided Re-Correspondence (RRC) synthesizes object-specific views at the estimated pose and re-establishes dense 2D-3D correspondences to sharpen the initial estimate without additional learned parameters. A multi-mask hypothesis selection strategy jointly scores competing segmentation candidates to resolve segmentation ambiguity. On the seven core datasets of the BOP Benchmark, B2TFPose achieves 40.7 mean AR without refinement and 56.4 with refinement, establishing state-of-the-art performance among training-free RGB methods and outperforming trained counterparts including GigaPose and GenFlow, at competitive inference speed.
Ali Rafiaei, Michael Greenspan
Dept. Electrical & Computer Engineering, Ingenuity Labs Queen’s University Kingston, ON, Canada
Object detection research remains constrained by high annotation costs and the closed-set assumptions of supervised models, limiting scalability to novel categories. We present a target-dataset annotation-free object detection framework that eliminates the need for human labeling on the target dataset by leveraging the complementary strengths of pretrained foundation models, including segmentation and vision-language systems, combined with a self-refining training loop. To further reduce reliance on expensive LVLM queries, we introduce a landmark-based clustering mechanism that amortizes labeling across image clusters, improving label consistency while lowering computational cost. Our method autonomously generates segmentation proposals, assigns semantic labels through multimodal reasoning, and iteratively improves both localization and classification using a lightweight open-set detector. This modular pipeline enables adaptation to completely unseen object classes and domains without any human-provided target-dataset annotations, while maintaining compatibility with existing detection architectures. Evaluations on COCO, LVIS, ODinW-35, and Roboflow-100 demonstrate substantial performance gains over state-of-the-art open-vocabulary and long-tail detection methods, achieving 60.8 AP on COCO and 47.4 AP on LVIS under zero-shot settings. Beyond strong benchmark results, our approach provides a scalable foundation for real-world detection systems in unknown environments, where annotation is infeasible.
Current zero-shot Camouflaged Object Segmentation methods typically employ a two-stage pipeline (discover-then-segment): using MLLMs to obtain visual prompts, followed by SAM segmentation. However, relying solely on MLLMs for camouflaged object discovery often leads to inaccurate localization, false positives, and missed detections. To address these issues, we propose the Discover-Segment-Select (DSS) mechanism, a progressive framework designed to refine segmentation step by step. The proposed method contains a Feature-coherent Object Discovery (FOD) module that leverages visual features to generate diverse object proposals, a segmentation module that refines these proposals through SAM segmentation, and a Semantic-driven Mask Selection (SMS) module that employs MLLMs to evaluate and select the optimal segmentation mask from multiple candidates. Without requiring any training or supervision, DSS achieves state-of-the-art performance on multiple COS benchmarks, especially in multiple-instance scenes.
Yilong Yang, Jianxin Tian, Shengchuan Zhang +1
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.