SOS! : A Streamlined Object-Conditional Transformer for Model-free Segmentation
Authors: Jiaqi Hu, Junwen Huang, Hongli Xu, Peter KT Yu, Nassir Navab, Benjamin Busam, Slobodan Ilic
Organizations: Technical University of Munich Munich, Germany · Siemens AG Munich, Germany · Munich Center for Machine Learning Munich, Germany · ROBOX Cambridge, MA, USA
Foundation segmentation models excel at generating high-quality, class-agnostic masks, but they struggle to associate these proposals with specific target objects. This semantic gap severely hinders their deployment in downstream applications like robotic manipulation, which demand precise unseen objects segmentation. Existing approaches attempt to resolve this by relying on exhaustive 3D object model priors, inherently introducing prohibitive computational overhead and complex, multi-stage pipelines. To address these limitations, we propose SOS (Streamlined Object-conditional Transformer for model-free Segmentation). SOS completely eliminates the reliance on 3D models, requiring only a single reference image per target object. Central to our framework is a novel Object-Conditional Transformer that learns identity-anchored queries, unifying mask generation and target identification into a single feed-forward pass. This streamlined design drastically improves both structural and computational efficiency. Extensive evaluations across multiple benchmarks demonstrate that SOS establishes a new state-of-the-art for model-free unseen objects segmentation, delivering accurate and high-efficiency performance. The project page and code are available at https://sos-seg.github.io/.
Figures & tables
Figure 1: Comparison of unseen objects segmentation pipelines. (a) Model-based two-stage pipeline (e.g., CNOS [ 32 ] , SAM6D [ 23 ] ) uses SAM [ 21 ] for mask proposals and DINO [ 34 ] for feature matching, heavily relying on cumbersome 3D CAD onboarding. (b) Model-based one-stage pipeline (e.g., OC-DiT [ 42 ] ) predicts masks directly but still requires exhaustive multi-view rendering from 3D CAD priors, while iterative denoising severely bottlenecks inference. (c) Model-free one-stage pipeline (SOS \xspace ) completely eliminates 3D CAD reliance. Driven by identity-anchored queries, it requires only a single 2D reference view per object to output identity-aware masks in a single feed-forward pass, enabling real-world deployment.
Figure 2: Overview of the SOS \xspace framework. (1) Feature Encoder : A frozen DINO backbone extracts feature maps ( Fs,Fc ) from the scene and object conditions respectively. The Scene and Condition Embedders subsequently process these maps into token sequences ( Ts,Tc ) augmented with spatial and identity positional encodings. (2) Object Conditional Transformer : Initial queries Q iteratively evolve into deterministic identity-anchored queries. Within each block, queries perform masked cross attention with the condition tokens Tc and condition masks Mc to absorb target identities, followed by masked cross attention with the scene tokens Ts restricted by the preceding attention masks Al−1 for spatial grounding, and self attention to resolve overlapping targets and occlusions. (3) Mask Decoder : The identity-anchored queries Q^ are decoded, interacting with the scene tokens to directly output the final masks for each object.
Method
Model-based
Onboarding (s / per-obj)
Ref. Image(s)
YCB-V
TUD-L
LM-O
HB
Average
Inference (s / per-img)
∗ CNOS [ 32 ]
✓
18.82
42 - CAD
59.9
48.0
39.7
51.1
49.7
0.221
∗ SAM6D [ 23 ]
✓
18.82
42 - CAD
59.5
49.8
44.4
55.7
52.3
0.249
∗ NIDS-Net [ 27 ]
✓
18.82
42 - CAD
65.0
55.6
43.9
62.0
56.6
0.485
∗ MUSE [ 5 ]
✓
18.82
42 - CAD
69.0
57.3
47.7
63.5
59.3
0.505
∗ OC-DiT [ 42 ]
✓
18.82
42 - CAD
71.7
59.4
40.1
61.5
58.2
-
CNOS
×
0.0
1
55.3
45.2
41.1
42.5
46.0
-
Table 1: Results on the BOP challenge datasets. We report the AP for the task unseen objects segmentation. The highest value in each column is denoted in bold , and the second highest is underlined . For baseline methods requiring CAD models, the reported onboarding time includes both template rendering via BlenderProc [ 8 ] and templates feature extraction, measured on a single NVIDIA RTX 4090 GPU. AP and inference time of methods marked with ∗ are sourced directly from the original papers.
Figure 3: Qualitative results on LM-O [ 1 ] , YCB-V [ 46 ] and TUD-L [ 13 ] . The first column shows the input RGB image and the following columns depict the detections produced by CNOS [ 32 ] , SAM6D [ 23 ] and SOS \xspace with confidence scores greater than 0.5. The last column shows the Ground Truth.
Configuration
YCB-V
TUD-L
LM-O
HB
Average
w/o self-attn
59.3
50.3
44.3
56.6
52.6 ↓6.2
w/o scene cross-attn
41.0
51.2
31.2
48.4
43.0 ↓15.8
w/o deep supervision
47.0
52.0
31.1
48.2
44.6 ↓14.2
w/ anonymous query (instead of identity-anchored query)
47.0
25.8
37.5
51.4
40.4 ↓18.4
Full model
61.7
57.6
50.3
65.6
58.8
Table 2: Ablation study of SOS architectural design choices on BOP benchmarks. We report AP for unseen object segmentation on YCB-V, TUD-L, LM-O, and HB, together with the average AP. All ablation variants are trained on the same dataset for the same number of iterations.
Method
Backbone
YCB-V
TUD-L
LM-O
HB
Average
CNOS
DINOv3
63.6
48.3
43.7
50.5
51.5
SAM6D
DINOv3
63.7
43.6
43.7
50.3
50.3
Ours
DINOv3
61.7
57.6
50.3
65.6
58.8
Table 3: Backbone ablation under the single-reference setting. CNOS, SAM6D, and SOS are compared with DINOv3 using the same real cropped reference images.
Figure 7
Figure 6: Visual condition templates. This grid illustrates the four distinct rendered perspectives used for our sensitivity analysis on the LM-O [ 1 ] dataset. For each object, the images arranged from left to right correspond sequentially to view 1, view 2, view 3, and view 4 in Tab. 4 .
View
Asymmetric Objects
Symmetric Objects
Overall Avg.
ape
cat
holepuncher
Avg.
can
driller
duck
eggbox
glue
Avg.
view 1
39.8
47.2
34.7
40.6
55.8
63.8
58.9
47.6
41.7
53.6
48.7
view 2
30.8
46.5
36.2
37.8
56.4
64.1
58.0
47.5
42.0
53.6
47.3
view 3
45.6
41.2
46.3
44.4
56.1
64.0
62.2
48.2
43.1
54.7
50.4
view 4
45.1
44.0
30.3
39.8
55.9
64.3
62.0
47.5
43.1
54.6
49.0
Δ (Max - Min)
14.8
6.0
16.0
6.6
0.6
0.5
4.2
0.7
1.4
1.1
3.1
Table 4: Ablation on view condition sensitivity. We report the average precision across different azimuth angles for eight objects from the LM-O [ 1 ] dataset. Symmetric objects maintain stable scores, while asymmetric objects show significant variance depending on the provided perspective.
Figure 7: Ablation on reference views. (a) Internal scaling : Performance improves significantly by aggregating complementary geometric context, particularly for highly sensitive asymmetric objects, reaching a saturation point around three to four views. (b) Comparative scaling : Compared to baselines, our method achieves higher initial accuracy and reaches peak performance much earlier with only a fraction of the reference views.
Figure 8: Qualitative evaluation in unconstrained environments. Our model accurately isolates target items in highly cluttered everyday scenes, handling severe occlusions and complex backgrounds entirely without domain specific training data.