Promptable segmenters of the SAM family turn a click or a phrase into an instance mask, so the costly part of a label is now its class. However, each annotation project draws its own class boundary, often through objects these models already know: people from cars, sedans from trucks, one sedan model from another. A generic model's boundary is fixed by its training vocabulary. We present SNAP, an annotation pipeline that learns the project's boundary from the labels the annotator draws anyway. On one frozen SAM2.1 pass per image, we fit a per-project head, SnapHead, that classifies every proposal and refits in seconds after each labeled image. Where SAM 3 is available, the same labels decide class by class whether to keep its boundary or ours. Given the true masks of eight look-alike cell lines, SAM 3 assigns classes at Cohen's kappa = 0.038 and SnapHead at 0.754 (60 labeled images), and when only the class names change, SAM 3 falls to chance and SnapHead's score is unchanged. End to end, with 20 labeled images, SNAP beats a SAM 3 oracle tuned on the evaluation labels on five microscopy projects (0.484 against 0.441 F1), and run unchanged on five held-out datasets, it beats SAM 3 as shipped on four. Our results suggest that frozen foundation features already encode the class boundaries a generic vocabulary misses, and that a project's own labels are enough to read them out.
Figures & tables
Figure 1: A generic segmenter has a fixed class boundary, but every project needs its own. Two annotation projects over the same scene, colored by class. SAM 3’s boundary (orange) is fixed by its training vocabulary and stays at the coarse person/car cut. A boundary learned from a project’s own labels (blue) moves to the finer sedan/truck/bus cut that project needs.
Figure 2: Overview of SNAP. Gray blocks are frozen, blue blocks are fit to the labeled images L , and the dashed block is an optional proposal source. Every proposal passes through one frozen SAM2.1 decoder, and each newly labeled image refits SnapHead and the learned proposer.
Project
K
SAM 3
ours
LIVECell, 8 cell lines
8
0.038
0.754
COCO animals
10
0.940
0.392
LVIS rare
20
0.927
0.605
A172, small/medium/large
3
0.125
0.722
A172, type A/B/C
3
−0.007
0.722
Table 1: Class assignment on shared ground-truth masks. Cohen’s κ , zero at chance. SAM 3: zero-shot prompt oracle; ours is SnapHead at N=60 . The last two rows split LIVECell-A172 into area terciles under two class-name sets. Bold: better of the two.
Figure 3: Class assignment on one LIVECell image of SH-SY5Y cells (densest third, 112 of 190 cells). Every panel draws the same ground-truth masks; color shows the assigned line. Titles give the count right in this crop and the accuracy over all 35,458 evaluation cells.
Figure 4: Comparison with SAM 3 on the label ladders and class-assignment projects. Each point is one cut; above the diagonal, the learned boundary wins. Colored markers follow the three ladders, coarse to fine; SnapHead sees 20 labeled images per finest class. Gray diamonds are the single-cut projects of Tab. 1 at N=60 . The dashed arrow renames the A172 size classes: SAM 3 moves, ours stays.
Method
F1↑
F1cl↑
F1nu↑
D↓
E↓
[E]↓
t↓
SAM 3, name
0.219
0.000
0.000
0.858
0.893
0.878–0.936
0
SAM 3, blind
0.106
0.016
0.170
0.911
—
—
0
SAM 3 †
0.441
0.078
0.742
0.609
0.719
0.672–0.904
0
G-DINO †
0.090
0.000
0.359
0.940
0.956
0.950–0.991
0
pixel forest
0.249
0.010
0.806
0.806
0.847
0.830–0.915
0.30c
PerSAM
0.253
0.097
0.704
0.741
0.815
0.785–0.967
0.07c
Table 2: Five microscopy projects at N=20 , mean over projects ( t : median). F1cl , F1nu : F1 on LIVECell’s eight cell lines and on BBBC038 nuclei. D : fraction drawn by hand. E : effort at wcheck=0.1 , wrefine=0.5 ; [E] : min–max of E over the cited box, wcheck∈[0.063,0.473] [ 51 , 35 ] , wrefine∈[0.268,0.5] [ 49 ] . t : seconds to adapt, c CPU, g GPU; bold: fastest in group. Drawing every object by hand gives D=E=1 . † : oracle, phrasing and threshold tuned on evaluation labels. ‡ : proposals from Cellpose, whose training data include LIVECell and BBBC038 ( Sec. D.7 ). Bold: best without † or ‡ .
dataset (classes)
ours
router
SAM 3
oracle
SW
everdaynew (pack sizes)
0.294± 0.030
0.250± 0.042
0.107
0.145
y
aerial-pool (hats, people)
0.188± 0.009
0.317± 0.010
0.052
0.354
y
stomata (open, closed)
0.260± 0.012
0.239± 0.038
0.000
0.154
n
floating-waste (litter)
0.196± 0.027
0.233± 0.003
0.237
0.253
y
floods (human)
0.602± 0.011
0.601± 0.011
0.578
0.622
y
mean
0.308± 0.014
0.328± 0.017
0.195
0.306
—
Table 3: Transfer to five randomly drawn RF100-VL datasets at N=20 . Instance F1 at box IoU 0.5 , mean ± sd over three seeds; SAM 3 is deterministic. ours : SnapHead. SAM 3: class names, default threshold. oracle : best phrasing and threshold on evaluation labels. router : Sec. 3.3 . SW: the automatic switch ( Sec. 3.2 ) is within seed noise of the better fixed proposer. Last row: mean; sd over seeds.
Component
Setting
Encoder
SAM2.1-Hiera-L, frozen, one forward pass per image
Proposer pool
SAM2’s automatic mask generator (AMG), fixed knobs; plus a 97,603 -parameter proposer, architecture fixed, weights fit per project ( auto )
Pool switch
auto : the union of AMG and the learned proposer once AMG’s recall on L falls below 0.5 , AMG alone otherwise ( Sec. 3.2 )
Descriptor
x : the mask-pooled SAM2 image embedding (256-d) joined with the box-prompt object pointer (256-d)
SnapHead
multinomial logistic regression, balanced class weights, standardized descriptors, explicit background class
Decision rule
one flat threshold, 0.5 , on SnapHead’s class probability times SAM’s mask-quality score, the same for every class and every project
Table A1 : Default SNAP configuration, used for every project. auto entries are read from the labeled images by the rule shown; the rest are fixed.
ours, fixed
Dataset
Cut
K
SAM 3
N=5
N=20
Stanford Cars
body type
9
0.481
0.448
0.608
make
48
0.115
0.193
0.380
model
182
0.063
0.088
0.269
FGVC-Aircraft
manufacturer
30
0.045
0.385
0.582
family
70
0.036
0.329
0.577
Table A2 : Cohen’s kappa per cut, Stanford Cars and FGVC-Aircraft. SAM 3: zero-shot prompt oracle; SnapHead, three seeds.
Figure A1 : Class assignment on eight LIVECell cell lines, SAM 3 against SnapHead. Masks fixed; n=35,458 instances, row-normalized, cells below 0.005 blank.
Project
SAM 3
Grounding DINO
YOLOE
coco-animals
0.914
0.806
0.780
voc-20
0.829
0.777
0.735
coco-vehicles
0.766
0.379
0.652
coco-kitchen
0.636
0.279
0.476
lvis-rare20
0.439
0.116
0.207
ade20k-indoor
0.379
0.218
0.305
Table A3 : Oracle F1 for SAM 3, Grounding DINO and YOLOE under one protocol: 120 evaluation images, the same five phrasings per class, each model’s own oracle over seven thresholds. Bold: best per row. SAM 3 uses this shared protocol without the matched- K prompt set of Sec. C.5 , so livecell-8type reads 0.000 here against 0.078 in Tab. 2 . That row is unbolded: all three round to 0.000 (Grounding DINO 0.0002 ).
Project
N=1
N=5
N=20
text
coco-animals
0.884
0.896
0.893
0.916
coco-vehicles
0.726
0.715
0.711
0.765
lvis-rare20
0.520
0.533
0.533
0.439
bbbc038-nuclei
0.756
0.676
0.634
0.742
livecell-A172
0.256
0.219
0.219
0.586
livecell-8type
0.092
0.097
0.083
0.078
Table A4 : SAM 3 few-shot by the spatial-canvas recipe: six projects, N∈{1,5,20} , three seeds, a 60 -image evaluation prefix; full coverage in Sec. D.3 . text : the matched- K zero-shot oracle of Tab. A5 . Bold: best per project.
Project
cls
oracle
exp
Δ
worst
ret.
coco-animals
10
0.916
0.628
0.288
0.353
0.69
voc-20
20
0.822
0.616
0.205
0.274
0.75
coco-vehicles
5
0.765
0.477
0.288
0.082
0.62
coco-kitchen
11
0.636
0.421
0.215
0.194
0.66
lvis-rare20
20
0.439
0.326
0.113
0.199
0.74
ade20k-indoor
22
0.375
0.241
0.133
0.079
0.64
Table A5 : Fragility axis, all 13 projects. SAM 3 under five phrasings per class ( K=5 ), each at its best threshold; 120 evaluation images, 119 for the two aerial projects. oracle : best phrasing on evaluation labels, in bold. exp : expected F1 of a blind pick. Δ : their difference. worst : weakest phrasing. ret. : exp/oracle , kept by a blind pick.
Project
AMG
LAP ∪ AMG
ours ∪ AMG
oracle
livecell-A172
0.167
0.295
0.587
0.880
livecell-8type
0.259
0.309
0.489
0.816
livecell-microsam
0.248
0.291
0.486
0.809
livecell-SHSY5Y
0.108
0.155
0.320
0.581
bbbc038-nuclei
0.876
0.891
0.893
0.972
ade20k-indoor
0.506
0.566
0.529
0.715
Table A6 : Proposal recall at IoU 0.5 , N=5 , mean over 3 seeds and 120 images. LAP : AMG knobs plus a stride-16 box scorer. oracle : decoder ceiling. Bold: best below the oracle.
Figure A2 : Proposals on one LIVECell-A172 image: production AMG, the learned proposer ( N=5 , top- K=512 boxes), and ground truth, chosen by Fig. 3 ’s rule ( Sec. C ). Panels give frame recall.
Project
nGT
AMG
box
point
gap
SnapHead
coco-animals
452
0.912
0.990
0.902
0.078
0.433
bbbc038-nuclei
6951
0.851
0.972
0.889
0.121
0.954
voc-20
396
0.760
0.937
0.740
0.177
0.357
coco-kitchen
667
0.748
0.883
0.709
0.135
0.181
coco-vehicles
687
0.737
0.958
0.710
0.220
0.523
lvis-rare20
1019
0.657
0.948
0.689
0.291
0.360
Table A7 : Recall and SnapHead accuracy per project at IoU 0.5 , image-macro; aerial rows cover 179 images (iSAID-vehicles box oracle: subsample, below). box / point : oracle recall from a ground-truth box or centroid. gap : box oracle minus AMG. SnapHead : accuracy on instances AMG hits.
microscopy (5)
robust (6)
Method
F1
drawn
corr
F1
drawn
manual
—
1.000
1.000
—
1.000
SAM 3, class name
0.219
0.858
0.924
0.629
0.311
SAM 3, blind phrasing
0.106
0.911
0.998
0.452
0.517
SAM 3, oracle
0.441
0.609
1.002
0.661
0.338
G-DINO, oracle
0.090
0.940
1.001
0.429
0.448
Table A8 : Every method, both bins, N=20 . 11 core projects; the two aerial projects are analyzed separately, by recall, in Tab. A7 . drawn : fraction of objects still drawn by hand. corr : corrective actions per object. manual has no F1 by construction. Every ours row uses the same descriptor x . Bold: best F1 and drawn fraction per bin.
method
8 lines
A172
SH-SY5Y
micro-SAM
nuclei
mean
D
Matcher
0.003
0.001
0.001
0.002
0.063
0.014
0.990
PerSAM
0.008
0.008
0.002
0.006
0.203
0.045
0.969
PerSAM-F
0.008
0.009
0.003
0.005
0.171
0.039
0.972
Matcher ∗
0.007
0.007
0.002
0.006
0.290
0.062
0.948
PerSAM ∗
0.037
0.063
0.014
0.087
0.766
0.193
0.804
PerSAM-F ∗
0.029
0.065
0.015
0.062
0.585
0.151
0.837
Table A9 : Exemplar baselines at their published settings, five microscopy projects, N=20 , instance F1 (mean over three seeds). ∗ : object cap lifted.
F1 ↑
D↓
E↓
Project
SnapHead
name
labeled
r. name
router
SAM 3 †
SnapHead
name
labeled
router
SnapHead
name
labeled
router
ADE20K indoor
0.077
0.313
0.362
0.314
0.344
0.379
0.763
0.456
0.467
0.493
0.899
0.947
0.837
0.835
COCO animals
0.177
0.881
0.898
0.881
0.898
0.914
0.411
0.073
0.106
0.106
0.506
0.254
0.267
0.267
COCO kitchen
0.081
0.620
0.629
0.587
0.596
0.636
0.836
0.325
0.404
0.470
0.913
0.514
0.557
0.604
COCO vehicles
0.216
0.766
0.745
0.766
0.745
0.766
0.482
0.198
0.265
0.265
0.621
0.371
0.415
0.415
LVIS rare
0.045
0.408
0.406
0.327
0.335
0.439
0.904
0.682
0.598
0.669
0.976
0.739
0.684
0.754
Table A10 : Router per project, N=20 , mean over three seeds. SnapHead : our head with the gated proposer. name : SAM 3 with the class name at threshold 0.5 . labeled : SAM 3 with phrasing and threshold chosen on the labeled images. r. name : per-class choice between SnapHead and SAM 3 with its class name. router : per-class choice that also picks SAM 3’s phrasing on the labeled images ( Sec. 3.3 ). † : oracle, phrasing and threshold chosen on evaluation labels. D , E as in Tab. 2 . Bold: best F1 without † . ∗ Two seeds.
vs. SAM 3, name
vs. SAM 3 †
Project
ours
Δ
CI
p
Δ
CI
p
Microscopy, instance F1 at mask IoU 0.5
BBBC038 nuclei
0.842
+0.842
[0.797,0.882]
<10−4
+0.100
[0.061,0.147]
<10−4
LIVECell 8 lines
0.216
+0.216
[0.192,0.237]
<10−4
+0.138
[0.109,0.162]
<10−4
LIVECell A172
0.481
−0.022
[−0.065,0.019]
0.85
−0.104
[−0.128,−0.082]
1.00
LIVECell SH-SY5Y
0.236
−0.021
[−0.059,0.013]
0.88
−0.139
[−0.162,−0.117]
1.00
Table A11 : Paired bootstrap over evaluation images, N=20 , 10,000 resamples. Δ : ours minus SAM 3 for F1; SAM 3 minus ours for D , so a positive Δ always favors ours. CI: 95% percentile interval. p : one-sided. name : SAM 3 with its class name at threshold 0.5 (RF100-VL: as shipped). † : oracle, phrasing and threshold chosen on evaluation labels; on RF100-VL it is chosen on box IoU, the reported metric.
Cellpose ‡
SD ‡
CP + head
head
Project
F1
D
E
F1
F1
D
E
F1
BBBC038 nuclei
0.817
0.220
0.366
0.756
0.845
0.180
0.334
0.842
8 lines
0.053
0.257
0.365
—
0.338
0.542
0.668
0.216
A172
0.751
0.335
0.514
—
0.626
0.453
0.680
0.481
SH-SY5Y
0.668
0.444
0.631
—
0.367
0.719
0.868
0.236
micro-SAM split
0.799
0.293
0.446
—
0.545
0.581
0.727
0.390
Table A12 : Microscopy specialists, instance F1 at mask IoU 0.5 , with D and E as in Tab. 2 . Cellpose : cyto3 , zero-shot. SD : StarDist, BBBC038 only. CP + head : Cellpose boxes as proposals for SnapHead, N=20 , mean over three seeds. head : SnapHead with its own proposals ( Tab. 2 ). ‡ : likely trained on these images.
Project
images
x
mask-pooled
pointer
8 lines
8
0.429
0.417
0.344
16
0.558
0.553
0.421
40
0.713
0.708
0.479
60
0.754
0.747
0.501
A172, size split
1
0.068
0.110
0.084
2
0.206
0.163
0.193
Table A13 : Descriptor halves, given ground-truth masks as in Tab. 1 . Cohen’s κ , mean over three seeds. images : labeled images; on the eight lines the budget is set per line, and 20 per line takes the whole 60 -image pool. Bold: best per row.
alternative
result
trade-off
AMG knobs tuned from seed images
On LIVECell-A172 the costliest grid reaches recall 0.358 at 84.9 s per image; the learned proposer reaches 0.575 at 0.41 s.
More compute buys less recall than our proposer.
AMG filters switched off
The raw pool covers 0.694 of A172 cells; SAM2’s predicted-IoU >0.7 filter cuts that to 0.146 , and NMS removes almost nothing.
About 104 masks per image is too many to score.
AMG filter learned from seed images
On single-scale grids it keeps 78 – 99% of the raw pool on cells (A172 0.355 of 0.369 ), but fails on crops and COCO vehicles.
Still below the learned proposer ( 0.575 on A172) at higher cost.
Training-free top-hat blob proposer
Joined with AMG it lifts A172 recall 0.167→0.282 and 8type 0.259→0.330 ; alone it finds almost nothing large ( 0.014 on VOC).
The learned proposer is better wherever top-hat helps.
Native-resolution tiling, aerial tiles
83% of iSAID vehicles are under 8 px after resize, where both methods find about 1% ; above it we match SAM 3. Tiled AMG costs about 110 s.
It lifts SAM 3 and us alike, so we treat it as preprocessing.
Proposer choice by cross-validation
An oracle picking the best proposer set per project beats the one-line switch by only 0.005 ( N=5 ) and 0.002 ( N=20 ) mean F1.
The one-line switch already captures the gain.
Table A14 : Alternatives to the fixed configuration and their trade-offs.
Figure A3 : Effort per project at N=20 , split into drawn ( D ), refined ( wrefineR ) and checked ( wcheckC ) shares. Whiskers: cited ranges, wcheck0.063 – 0.473 [ 51 , 35 ] and wrefine0.268 – 0.423 [ 49 , 71 ] , extended to our 0.5 default. Dashed: manual.
Project
nGT
D↓
E↓
range ↑
bbbc038-nuclei
6951
0.213±0.006
0.357
1.000
coco-animals
452
0.324±0.014
0.453
1.000
coco-vehicles
687
0.455±0.002
0.602
0.988
livecell-A172
38500
0.504±0.003
0.767
0.994
Table A15 : Endpoint of the 36 -round loop, SNAP configuration, mean over three seeds. D : hand-drawn share, ± sd. E : effort at default weights. range : share of the cited range below manual.
Stage
measured
encoder pass, once per image
18.6 – 19.7 ms
encoder peak VRAM
1.64 GB
production AMG proposals
9.0 – 9.9 s/img
AMG at half grid density
0.53 – 0.75 s/img
learned proposer, stage only
0.02 – 0.49 s/img
learned proposer, parameters
97,603
Table A16 : Measured per-image cost, median over 8 images; AMG at production density. stage only : proposer vs. AMG alone. deployed path : adds decode, NMS, re-prompt and descriptor. Bold: deployed learned proposer. Accuracy: Sec. 4.4 .
Figure A4 : Time on the annotator’s clock and in the background. (a) Seconds per object to draw or check [ 71 , 49 ] , plus the browser’s decode wait (one CPU thread). (b) GPU stages per image or update, one H100. Bars: median; whiskers: range; log axis.
Figure A5 : Mean instance F1 over the five microscopy projects against time to adapt, per budget N (log x -axis). no training : prompt-only methods. ours, gated : SnapHead fit (CPU) plus proposer training (GPU, only where the gate fires), as in Tab. 2 .