Where is My Mug? Training-Free Framework for Personalized Image Segmentation
Organizations: Qualcomm AI Research‡ · National University of Singapore · VinFast, Vietnam
Abstract
Personalized Image Segmentation (PIS) aims to segment the specific object in a target image that matches a user-provided reference. Unlike category-based segmentation or few-shot segmentation, PIS must discriminate between multiple visually similar instances, a requirement central to deployed applications such as augmented reality, personalized content editing, robotics manipulation, inventory tracking, and fine-grained visual retrieval. We show that the strong numbers reported by existing PIS methods are largely an artifact of single-instance benchmarks: when evaluated on realistic scenes with several same-category look-alikes, prior point-prompt methods collapse, because the correct instance is frequently never surfaced as a prompt. We trace this to a recall bottleneck rather than a mask-quality one, and address it with a training-free, modular system. The pipeline first generates high-recall object proposals using an off-the-shelf class-agnostic detector, then matches each proposal to the reference using foundation-model features (e.g., BLIP-2), and finally refines the selected box with SAM. For robust matching we propose \textbf{top- Sinkhorn}, a patch-level distance that scores each candidate only on its most confidently matched patches; trimming the Sinkhorn dual potentials recovers the robustness of partial optimal transport at more than lower cost, keeping it inside the per-proposal candidate loop. To evaluate this regime, we introduce \datasetname, a realistic multi-instance benchmark. Across \datasetname, PerSeg, PerMIS, and DAVIS, our system surpasses PerSAM, , and PDM by up to mIoU while matching their runtime and using fewer parameters, establishing a strong, deployable training-free baseline for real-world personalized segmentation.
Figures & tables
| Dataset | # Ref | # Identities | # Categories | # Obj/Img |
| PerSeg [ 57 ] | 40 | 40 | 40 | 1.00 |
| PerMIS [ 42 ] | 140 | 140 | 16 | 3.10 |
| DAVIS [ 37 ] | 91 | 91 | 30 | 3.18 |
| PerMultiSeg | 2,147 | 1,377 | 45 | 3.67 |
| PerSAM [ 57 ] | PDM [ 42 ] | [ 58 ] | Ours | |
| Recall (%) | 34.33 | 30.60 | 34.33 | 89.55 |
| Inf. time (s) | 1.07 | 26.29 | 1.44 | 1.02 |
| Method | Prompt | Backbone | mIoU | bIoU | Recall | Params |
| PerSAM [ 57 ] | Point | SAM | 21.65 | 14.82 | 25.29 | 1.9B |
| PDM [ 42 ] | Point | SDXL-turbo | 23.65 | 12.82 | 43.91 | 2.6B |
| P 2 SAM [ 58 ] | Point | SAM | 31.58 | 27.90 | 24.57 | 1.9B |
| SAM2 ‡ [ 41 ] | 1-shot | SAM2 | 38.52 | 33.10 | 41.20 | 0.6B |
| SAM3 ‡ [ 6 ] | Concept | SAM3 | 44.73 | 40.95 | 48.15 | 2.1B |
| SAM3 + Qwen2.5-VL ‡ [ 6 , 2 ] | Caption | SAM3+Qwen | 70.11 | 64.57 | 71.09 | 9.1B |
| Method | PerSeg | DAVIS2017 | PerMIS | ||
| mIoU | J&F | J | F | mIoU | |
| PerSAM [ 57 ] | 89.30 | 66.90 | 71.30 | 75.10 | - |
| PerSAM-F † [ 57 ] | 95.30 | 76.10 | 74.90 | 79.70 | 16.50 |
| Matcher [ 29 ] | - | 73.50 | 70.00 | 77.50 | - |
| PDM [ 42 ] | 95.40 | 75.80 | 72.90 | 80.10 | 42.30 |
| [ 58 ] | 95.70 | - | - | - | 43.57 |
| Matcher | mIoU | Recall | bIoU | ms/pair |
| Avg. feature map | 65.29 | 66.44 | 60.97 | 0.5 |
| [CLS] token only | 83.86 | 85.58 | 78.77 | 0.07 |
| Full Sinkhorn ( ) | 85.04 | 86.83 | 79.93 | 8 |
| Sinkhorn + mutual-NN filter | 81.70 | 82.92 | 76.92 | 10 |
| Partial OT ( ) [ 8 ] | 87.97 | 89.83 | 82.69 | 848 |
| Top- Sinkhorn ( ) | 87.97 | 89.83 | 82.69 | 8 |
| Encoder | CLS only | Mean pool | Top- | vs CLS |
| BLIP-2 [ 25 ] | 83.86 | 83.16 | 87.97 | +4.11 |
| SigLIP-2 [ 50 ] | 77.02 | 84.54 | 86.50 | +9.48 |
| EVA-02 CLIP [ 13 ] | 71.67 | 72.32 | 75.69 | +4.02 |
| DINOv2 [ 36 ] | 66.96 | 70.92 | 80.71 | +13.75 |
| DINOv3 [ 46 ] | 65.21 | 70.94 | 78.26 | +13.05 |
| CLIP [ 40 ] | 71.35 | 74.76 | 74.82 | +3.47 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Mean (ms) | % of total |
| Object detection | 34.8 | 3.4% |
| Feature extraction | 749.2 | 73.7% |
| Matching (top- Sinkhorn) | 14.8 | 1.5% |
| Mask refinement | 218.1 | 21.4% |
| Total | 1016.9 | 100% |
| mIoU | Recall | bIoU | |
| 0.10 | 85.60 | 87.38 | 80.40 |
| 0.20 | 86.23 | 88.05 | 81.03 |
| 0.25 | 86.67 | 88.50 | 81.47 |
| 1/3 | 87.97 | 89.83 | 82.69 |
| 1/2 | 87.97 | 89.83 | 82.69 |
| 3/4 | 86.67 | 88.50 | 81.47 |
| Method | Params | PerMultiSeg | DAVIS2017 | ||
| mIoU | Recall | mIoU | Recall | ||
| SAM [ 24 ] | 641M | 95.55 | 99.38 | 80.89 | 82.60 |
| CODET [ 31 ] | 45M | 94.09 | 98.51 | 80.37 | 83.79 |
| YOLOW [ 10 ] | 47M | 85.95 | 90.24 | 90.00 | 97.51 |
| YOLOv11 [ 22 ] | 25M | 90.97 | 96.21 | 94.09 | 98.51 |
| Avg props/img | Prop. Rec. | Match (ms) | mIoU | Recall | |
| 4.51 | 74.63 | 17.6 | 65.60 | 67.25 | |
| 9.99 | 88.60 | 17.3 | 82.74 | 84.72 | |
| 27.69 | 91.42 | 18.9 | 87.97 | 89.83 |
| Config | mIoU | ms/pair | |
| Cross-only | 1.00 | 85.04 | 8.3 |
| Cross-only | 1/3 | 87.97 | 7.9 |
| De-biased | 1.00 | 85.04 | 22.9 |
| De-biased | 1/3 | 87.97 | 22.7 |
| Full Sinkhorn | Top- | ||
| 0.00 | 83.86 | 83.86 | 0.00 |
| 0.25 | 85.60 | 85.60 | 0.00 |
| 0.50 | 86.23 | 86.23 | 0.00 |
| 0.75 | 85.04 | 87.97 | +2.93 |
| 0.85 | 85.83 | 87.00 | +1.17 |
| 1.00 | 85.45 | 86.80 | +1.35 |
| mIoU | Recall | bIoU | |
| 0.1 | 86.76 | 89.05 | 81.35 |
| 0.5 | 87.39 | 89.72 | 81.93 |
| 0.9 | 87.05 | 89.36 | 81.68 |
| 1.0 | 86.76 | 89.05 | 81.35 |
| [0–7] | [8+] | Occ | NoOcc | Clut | NoClut | overexp | normal | |
| mIoU | 78.89 | 75.82 | 70.70 | 81.45 | 79.72 | 78.64 | 80.43 | 78.23 |
| Recall | 80.89 | 79.34 | 72.58 | 75.71 | 81.85 | 80.67 | 74.59 | 80.34 |
| Method | Task | mIoU | Recall | bIoU |
| SegGPT [ 53 ] | FSS | 29.34 | 16.86 | 21.23 |
| Matcher [ 29 ] | FSS | 23.22 | 11.29 | 18.53 |
| DCAMA [ 45 ] | FSS | 32.15 | 17.16 | 24.41 |
| GF-SAM [ 55 ] | FSS | 37.99 | 22.11 | 32.98 |
| PerSAM [ 57 ] | PIS | 21.65 | 25.29 | 14.82 |
| PDM [ 42 ] | PIS | 23.65 | 43.91 | 12.82 |
| Encoder | CLS | Mean | Full | Top- |
| SAM ViT-H [ 24 ] | 52.54 | 52.53 | 55.44 | 57.46 |
| SAM2 Hiera-L [ 41 ] | 61.15 | 59.19 | 57.94 | 59.08 |
| SAM3 [ 6 ] | 65.49 | 66.58 | 78.30 | 80.25 |
| Quantity | Value |
| Proposal-recall ceiling | 91.58% |
| Matching accuracy given recall | 85.98% |
| Overall success | 79.55% |
| Unrecoverable (no proposal covers the object) | 8.42% |
| Recalled but mismatched | 12.84% |
| mIoU when recalled | 82.68 |
| Split | Variant | mIoU | Recall | bIoU |
| val | (official default) | 35.52 | 28.77 | 29.96 |
| val | our proposals (mean pool) | 33.56 | 33.69 | 30.22 |
| val | our proposals (max pool) | 32.04 | 32.69 | 29.28 |
| test | (official default) | 31.58 | 24.57 | 27.90 |
| test | our proposals (mean pool) | 22.11 | 22.15 | 20.04 |
Promptable Image Segmentation
Promptable image segmentation uses points, boxes, text, or masks to specify objects and generate pixel-level regions.
See all 160 papers