What You Show Is Not What You Get: Rethinking Demonstration Following in Visual In-Context Segmentation
Abstract
Visual in-context segmentation uses visual demonstrations to guide predictions for a new image. In SegGPT, we identify background color mapping deviation, where non-target regions sharing a background label receive different output colors, and part-to-parent expansion, where predictions extend beyond the requested part into its parent object. These deviations can occur even when the relevant regions remain distinguishable in internal features. Controlled changes to demonstration colors and requested targets support an account in which demonstrations guide role and color assignments over familiar semantic regions, influenced by learned semantic and color preferences. Building on these findings, we introduce Semantic Contrast Prompt Selection (SCPS). Its semantic contrast encoding assigns separate colors to the requested part and the rest of its object while keeping the remaining scene black. SCPS combines existing retrieval scores with quality estimates from labeled development images to select demonstrations and their binary or semantic contrast encodings. Averaged across seven retrieval methods on three benchmarks, SCPS improves target IoU by 0.054 and reduces leakage into the rest of the object by 22.7% relative to the original binary selectors. Our findings offer a new perspective on understanding and improving demonstration following in visual in-context segmentation.
Figures & tables
| Method / benchmark | Original | SCPS | ||||
|---|---|---|---|---|---|---|
| IoU | PL | STS | IoU | PL | STS | |
| Appearance | .2133 | .1522 | .1451 | .3037 | .1153 | .2466 |
| CLIP ( Radford et al., 2021 ) | .2331 | .1596 | .1663 | .3059 | .1110 | .2514 |
| DINOv2 ( Oquab et al., 2023 ) | .2540 | .1512 | .1868 | .3140 | .1112 | .2579 |
| UnsupPR ( Zhang et al., 2023 ) | .2457 | .1486 | .1819 | .3104 | .1093 | .2559 |
| SupPR ( Zhang et al., 2023 ) | .2309 | .1512 | .1660 | .2590 | .1271 | .2055 |
| Candidate encodings | Calibration | Target IoU | PL | STS |
|---|---|---|---|---|
| Binary | No | .2412 | .1512 | .1762 |
| Semantic contrast | No | .2214 | .0716 | .1754 |
| Binary | Yes | .2875 | .1451 | .2243 |
| Semantic contrast | Yes | .2837 | .0815 | .2350 |
| Both (SCPS) | Yes | .2952 | .1168 | .2399 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Background | Mean AUROC | 95% CI |
|---|---|---|
| Red | 0.9760 | |
| Green | 0.9798 | |
| Blue | 0.9755 | |
| Cyan | 0.9792 | |
| Magenta | 0.9787 | |
| Yellow | 0.9810 |
| Black background | Purple background | |||
|---|---|---|---|---|
| Foreground | Color share (%) | Target IoU | Color share (%) | Target IoU |
| Red | 73.5 | .663 | 66.4 | .615 |
| Green | 76.6 | .681 | 74.6 | .672 |
| Blue | 78.0 | .687 | 74.7 | .670 |
| Cyan | 78.6 | .696 | 76.9 | .685 |
| Magenta | 79.4 | .703 | 74.2 | .665 |
| Region | Requested | Not requested | Difference [95% confidence interval] |
|---|---|---|---|
| Person | 73.0 | 75.5 | |
| Dog | 54.1 | 33.0 |
| Measure | Without expansion ( ) | With expansion ( ) |
|---|---|---|
| Body-only purple share | 91.6 | 73.1 |
| Body-only white share | 4.0 | 19.5 |
| Body-only black share | 4.4 | 7.5 |
| Scene purple share | 8.7 | 9.9 |
| Scene black share | 89.9 | 86.8 |
| Face recall, black background | .785 | .938 |
| Condition | Target IoU | Target recall | PL |
|---|---|---|---|
| Unmodified prediction | .2054 | .8553 | .6916 |
| Target enhancement | .2073 | .8569 | .6923 |
| Parent-only suppression | .6063 | .7993 | .0360 |
| Both interventions | .6062 | .8040 | .0372 |
| Random target directions | .2052 | .8537 | .6916 |
| Random parent-only directions | .2058 | .8551 | .6902 |
| Benchmark | Tasks |
|---|---|
| LIP | arms person; face person; legs person; shoes person |
| PartImageNet-OOD | Bicycle Seat; Biped Foot; Biped Hand; Biped Head; Biped Tail; Bird Foot; Bird Head; Bird Tail; Bird Wing; Bottle Mouth; Car Tier; Fish Fin; Fish Head; Fish Tail; Quadruped Foot; Quadruped Head; Quadruped Tail; Reptile Foot; Reptile Head; Reptile Tail; Snake Head |
| Pascal-Part-116 | aeroplane:stern; aeroplane:wing; bird:head; bus:window; car:window; cat:ear; cat:head; cat:leg; cat:neck; cat:paw; cat:tail; cow:head; dog:ear; dog:head; dog:leg; dog:muzzle; horse:head; person:hair; person:head; person:leg; person:lower arm; person:upper arm |
| Decoder | Selection | Target IoU | PL | STS |
|---|---|---|---|---|
| Native | Original | .2412 | .1512 | .1762 |
| Native | SCPS | .2952 | .1168 | .2399 |
| White / black | Original | .2412 | .1512 | .1762 |
| White / black | SCPS | .2956 | .1170 | .2404 |
| White / green / black | Original | .2407 | .1505 | .1761 |
| White / green / black | SCPS | .2949 | .1164 | .2400 |
| Encoding | Target IoU | Precision | Recall | PL |
|---|---|---|---|---|
| Binary | .2140 | .2261 | .8562 | .6681 |
| Correct parent-only region | .1896 | .2853 | .4728 | .3061 |
| Translated control region | .2085 | .2266 | .7827 | .6157 |
| Retrieval method | Target IoU | PL | STS (pp) |
|---|---|---|---|
| Appearance | |||
| CLIP | |||
| DINOv2 | |||
| Original binary | SCPS | |||||
| Method | IoU | PL | STS | IoU | PL | STS |
| LIP | ||||||
| Appearance | .3842 | .0152 | .4125 | .5476 | .0076 | .6292 |
| CLIP | .3594 | .0160 | .3850 | .5321 | .0117 | .5892 |
| DINOv2 | .4072 | .0077 | .4500 | .5501 | .0091 | .6242 |
| UnsupPR | .4165 | .0123 | .4575 | .5458 | .0071 | .6208 |
| Candidate encodings | Target IoU | PL | STS |
|---|---|---|---|
| Binary | .2875 | .1451 | .2243 |
| Binary + uniform background | .2973 | .1458 | .2303 |
| Binary + semantic contrast | .2952 | .1168 | .2399 |
| Binary + uniform background + semantic contrast | .2976 | .1223 | .2398 |
| Features or output | Mean query AUROC |
|---|---|
| Features after interaction with the demonstration | .988 |
| Continuous output score | .874 |
| Thresholded binary mask | .521 |