Matching-Based Few-Shot Semantic Segmentation Models Are Interpretable by Design
Authors: Pasquale De Marinis, Uzay Kaymak, Rogier Brussee, Gennaro Vessio, Giovanna Castellano
Organizations: Department of Computer Science, University of Bari Aldo Moro · Jheronimus Academy of Data Science (JADS) · Eindhoven University of Technology (TU/e)
Few-Shot Semantic Segmentation (FSS) models achieve strong performance in segmenting novel classes with minimal labeled examples, yet their decision-making processes remain largely opaque. Explainable AI has matured on standard computer vision tasks, but not on FSS, where an explanation would tell a practitioner which support pixels the prediction rested on and which examples to provide next. We argue that matching-based FSS models are interpretable by design: their query-support similarity scores, once normalized and processed, attribute the prediction to individual support pixels. Our approach, Affinity Explainer (AffEx), makes this intrinsic interpretability explicit by extracting attribution maps directly from matching scores at multiple feature levels, without requiring gradients or external perturbations. We extend the standard causal metrics for evaluating explanations to the FSS domain and propose additional metrics to better capture the practical utility of explanations in few-shot scenarios. Experiments on two FSS benchmarks, covering seven architectures that range from classical matching-based models to recent pipelines built on vision foundation models (DINOv2/v3, SAM, SAM2) and a prototype-based head, show that AffEx outperforms adapted standard attribution methods at nearly every evaluation point, and that the gain comes from the matching mechanism itself. On pipelines whose segmentation stage transforms the matching output, the same evaluation doubles as a diagnostic of where the decision is made, and on the prototype-based head AffEx subsumes the prototype-style read-out as a special case. Qualitatively, the attribution maps expose how each architecture resolves the support set, and we show how they guide model diagnosis. Code is available at https://github.com/pasqualedem/AffinityExplainer.
Figures & tables
Figure 1: The proposed framework (two examples, one class). AffEx applies to any FSS model adhering to the illustrated structure: it reads the matching scores produced by the matching mechanism MMθ , before Mapθout , normalizes the scores of each query position with a softmax over the support, averages the result over a region R (the predicted mask, the ground truth mask, or a user-specified region), and aggregates the per-level maps; the support masks assign the attribution sign. The resulting maps highlight the support regions most influential in segmenting the query.
COCO 20i
Pascal 5i
Explanation Method
1-shot
5-shot
1-shot
5-shot
@0.01
@0.05
@0.01
@0.05
@0.01
@0.05
@0.01
@0.05
DCAMA
Random
33.79
29.78
41.42
31.62
48.72
34.64
55.24
33.42
Gaussian Noise Mask
27.07
12.96
31.08
17.71
33.60
16.01
35.65
15.86
Saliency
33.93
24.13
37.01
23.73
50.06
32.80
46.42
24.46
Table 1: mIoULoss@ p results for the evaluation of the attributions on COCO 20i and Pascal 5i under 1-way 1-shot and 1-way 5-shot settings. The table reports [email protected]↓ and [email protected]↓ . The best results are highlighted in bold, and the second-best results are underlined. Each block lists the methods applicable to that model: gradient-based attribution does not exist on INSID3, whose decision path contains agglomerative clustering and a discrete nearest-neighbor vote, while SANSA, whose adapter-tuned SAM2 pipeline is differentiable end to end, supports it, with the exception of Deep Lift, whose implementation requires the input–baseline pair batched through every module, which SAM2’s single-sequence pipeline does not support.
COCO 20i
Pascal 5i
Explanation Method
1-shot
5-shot
1-shot
5-shot
@0.01
@0.05
@0.01
@0.05
@0.01
@0.05
@0.01
@0.05
Random
31.13
26.54
37.17
26.55
44.52
37.29
49.76
36.12
Gaussian Noise Mask
24.30
14.36
29.13
18.99
39.72
22.56
42.42
19.83
LIME
20.38
10.76
25.20
15.91
28.09
11.28
28.89
15.82
Prototype read-out
16.51
10.39
19.26
19.16
24.97
11.64
20.34
19.18
Table 2: mIoULoss@ p results on COCO 20i and Pascal 5i for the prototype-based head sharing DCAMA’s frozen encoder; the results of DCAMA’s attention head are in Table 1 . The table reports [email protected]↓ and [email protected]↓ . The prototype read-out upsamples the pixel-to-prototype similarity, as prototype-part methods do with their activation maps. The best results are highlighted in bold, and the second-best results are underlined.
COCO 20i
Pascal 5i
Explanation Method
1-shot
5-shot
1-shot
5-shot
@0.01
@0.05
@0.01
@0.05
@0.01
@0.05
@0.01
@0.05
GF-SAM
Random
46.78
14.28
39.33
5.20
40.81
14.15
36.19
5.69
Gaussian Noise Mask
15.47
2.98
15.42
4.61
11.41
1.74
10.97
3.38
LIME
42.54
27.91
59.25
48.10
44.45
26.89
62.70
53.30
Table 3: mIoULoss@ p results on GF-SAM and Matcher, two decoder-dominated pipelines in which the promptable SAM decoder consumes the whole mask region, in the settings of Table 1 (1000 episodes each). The table reports [email protected]↓ and [email protected]↓ . Gradient-based baselines are inapplicable, as their decision paths contain discrete, non-differentiable steps (point-graph clustering in GF-SAM, Hungarian matching in Matcher). The best results are highlighted in bold, and the second-best results are underlined.
Figure 2: A 1-way 1-shot episode for each of the four evaluated models. The columns show the query and the support image, followed by Saliency, Blur IG, LIME, Unmasked AffEx and AffEx. N/A marks the gradient methods, which do not apply to INSID3. The prediction is highlighted in red over the query image, and the ground truth is shown in blue only when it differs significantly from the prediction. All attribution maps are min-max normalized to [0,1] and drawn with the same colormap.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
1 shot
5 shots
Model
Explanation Method
Expl. Mem. (GB)
Expl. Time (ms)
Expl. Mem. (GB)
Expl. Time (ms)
DCAMA
Forward Pass
1.85 ± 0.00
44 ± 1
6.37 ± 0.00
109 ± 1
Saliency
4.00 ± 0.00
102 ± 1
13.70 ± 0.00
284 ± 2
Integrated Gradients
4.01 ± 0.00
2571 ± 40
13.76 ± 0.00
7154 ± 103
Guided IG
4.00 ± 0.00
3693 ± 34
13.70 ± 0.00
11079 ± 78
Blur IG
4.00 ± 0.00
11904 ± 56
13.70 ± 0.00
35268 ± 163
Appendix
Table 4: Computational cost comparison between AffEx and other explanation methods, including Integrated Gradients, Saliency Maps, and Deep Lift. The table reports the average memory consumption (in GB) and time (in milliseconds) for generating explanations, along with the forward pass time of the models for reference. The forward pass cost is shown in a gray background. The results are presented for both 1-shot and 5-shot scenarios, with standard deviations included. The experiments were conducted on an NVIDIA A100. For INSID3 and SANSA the AffEx rows are measured at similarity map resolution 64×64 , while the gradient-based and perturbation-based baselines operate at the native input resolution of each model.
Superpixels
Model
Kernel
Image-Mask
mIoUL.
width ( σ )
balance ( λ )
@0.01
80
Lasso
0.5
0.2
37.54
0.5
0.5
37.65
0.5
0.8
37.39
Lasso
1.0
0.2
37.24
1.0
0.5
37.19
Appendix
Table 5: Grid search results for adapting LIME to few-shot segmentation. The table reports the [email protected] for different configurations of superpixels, regression models (Lasso or Ridge), kernel width ( σ ), and the balance parameter ( λ ). The best performance is highlighted in bold.
COCO 20i
Pascal 5i
Explanation Method
IAUC mIoU
DAUC mIoU
Diff
IAUC Conf @0.01
IAUC mIoU
DAUC mIoU
Diff
IAUC Conf @0.01
DCAMA
Random
47.50
46.85
0.65
43.36
71.35
70.77
0.59
41.71
Gaussian Noise Mask
52.32
15.46
36.86
53.80
74.67
23.36
51.31
55.32
Saliency
52.02
41.10
10.92
44.97
72.99
66.73
6.27
44.32
Integrated Gradients
49.23
45.97
3.27
45.63
72.38
70.22
2.16
42.94
Appendix
Table 6: Insertion and deletion evaluation under 1-way 5-shot. The table reports IAUC ↑ , deletion DAUC ↓ , their difference (Diff ↑ ), and the insertion AUC computed on the prediction confidence over the first 1% of revealed pixels (IAUC Conf @0.01 ↑ ). No value is highlighted here: the full-curve IAUC/DAUC are dominated by the flat tail of the curves, whose oscillations are of the same order as the differences between methods, so their ranking is not robust; Diff (IAUC − DAUC) summarizes the separation between informative and non-informative regions and is reported for completeness. On INSID3 gradient-based attribution does not exist, as its decision path contains discrete steps. On SANSA it does, but the cost of the gradient family is prohibitive at this scale, so we consider here only XRAI, the strongest gradient baseline on SANSA under mIoULoss@ p in Table 1 , where the whole family is reported.
N. Steps
5
10
25
50
75
100
Metric
Method
DAUC
Saliency
6.44
3.85
1.91
1.16
0.86
0.66
AffEx
9.04
5.37
3.12
1.85
1.37
1.10
IAUC
Saliency
6.33
3.63
2.14
1.32
0.98
0.75
AffEx
6.64
3.22
1.34
0.77
0.59
0.46
Appendix
Table 7: Relative error on the 95th percentile of the Insertion Area Under the Curve (IAUC) and Deletion Area Under the Curve (DAUC) metrics with respect to the ground truth value obtained with 1000 steps. The table shows the relative error for the Affinity Explainer (AffEx) and Saliency methods. The lower the relative error, the more stable the metric is with respect to the number of steps.
Explainer
DCAMA 1-shot
DCAMA 5-shot
DMTNet 1-shot
DMTNet 5-shot
AffEx
3.2 / 5.6
1.0 / 2.2
6.2 / 9.0
4.8 / 8.8
Unmasked AffEx
3.8 / 7.0
1.4 / 3.2
6.2 / 9.0
3.6 / 7.2
LIME
5.2 / 7.8
5.8 / 8.4
3.6 / 6.4
5.0 / 9.8
Gaussian Noise Mask
7.0 / 9.6
5.8 / 10.4
11.2 / 13.0
12.4 / 13.6
Random
9.8 / 10.8
10.0 / 12.4
n/a
n/a
Appendix
Table 8: Saturation points of the insertion curves: smallest p (in percent of support pixels) at which each explainer’s episode-mean insertion curve reaches 90 / 95 percent of its own plateau. Pascal 5i , fine grid of 0.2 percent steps. Random on DMTNet is non-monotone (inserting randomly chosen support pixels lowers the mIoU below its own starting value), so its saturation point is undefined. LIME’s early points on DMTNet come with a visibly lower plateau (0.42 against 0.48 for AffEx at 1-shot): saturating early to a worse value is not a sign of informativeness.
Figure 3: Insertion curves at fine resolution over the early regime (steps of 0.2 percent of support pixels) for DCAMA on Pascal 5i , 5-shot, averaged over episodes. The curves of the strong explainers saturate within the first few percent; p=0.01 falls in the steep segment where methods are most separated, p=0.05 at the knee of the best curves.
Figure 4: Qualitative comparison between smoothed and raw attribution maps. The raw attribution exhibits abrupt transitions at mask boundaries, causing pixels near the object boundary to receive erroneously negative attribution (blue). The smoothed attribution produces more gradual transitions, better reflecting the spatial uncertainty of encoder representations.
Table 9: Ablation study on support mask processing. [email protected] on Pascal 5i with DCAMA in a 1-way 5-shot setting.
Figure 5: Model diagnosis with AffEx on a failure case of DCAMA on COCO 20i . From left to right: attribution map computed over the predicted region, attribution map computed over the ground truth region, attribution-based support mask highlighting the most influential area (circled in red), and the resulting prediction after support mask refinement. Restricting the support mask to the most influential region significantly reduces false positives on the zebras while preserving the elephant segmentation.
Model
Clean mIoU
κ=0.01
0.05
0.1
0.25
0.5
δ=2
DCAMA
64.5
−1.6
−4.7
−9.9
−34.2
−65.3
INSID3
60.7
−10.5
−25.5
−35.5
−54.6
−74.5
GF-SAM
61.5
−0.2
−1.5
−2.2
−8.5
−20.3
Matcher
53.6
−2.4
−8.0
−12.7
−14.6
−11.7
δ=4
DCAMA
64.5
−1.5
−4.5
−9.3
−31.0
−57.7
INSID3
60.7
−10.5
−25.5
−35.8
−56.5
−82.5
Appendix
Table 10: Similarity intervention on 1000 Pascal 5i 1-shot episodes: relative mIoU loss (%) when the k highest in-mask similarities at every query position are lowered by δσS ( Eq. 13 ). Standard errors of the underlying mIoU changes are between 0.3 and 1.1 points.
Figure 6: Qualitative results for AffEx with DCAMA across multiple episodes from both COCO 20i and Pascal 5i . For each episode, the left column shows the query image with the predicted segmentation in red and the ground truth overlaid in blue. The right column presents the corresponding support set alongside the attribution maps produced by the proposed method.
Figure 7: Qualitative results for AffEx with DMTNet across multiple episodes from both COCO 20i and Pascal 5i . For each episode, the left column shows the query image with the predicted segmentation in red and the ground truth overlaid in blue. The right column presents the corresponding support set alongside the attribution maps produced by the proposed method.
Figure 8: Qualitative results for AffEx with INSID3 across multiple episodes from both COCO 20i and Pascal 5i . For each episode, the left column shows the query image with the predicted segmentation in red and the ground truth overlaid in blue. The right column presents the corresponding support set alongside the attribution maps produced by the proposed method. The first episode is shared with Fig. 6 , Fig. 7 and Fig. 9 .
Figure 9: Qualitative results for AffEx with SANSA across multiple episodes from both COCO 20i and Pascal 5i . For each episode, the left column shows the query image with the predicted segmentation in red and the ground truth overlaid in blue. The right column presents the corresponding support set alongside the attribution maps produced by the proposed method. The first episode is shared with Fig. 6 , Fig. 7 and Fig. 8 .
Figure 10: Qualitative results for AffEx with the prototype head of Appendix J across multiple episodes from both COCO 20i and Pascal 5i . For each episode, the left column shows the query image with the predicted segmentation in red and the ground truth overlaid in blue. The right column presents the corresponding support set alongside the attribution maps produced by the proposed method. The first episode is shared with Fig. 6 , Fig. 7 , Fig. 8 and Fig. 9 .
Figure 11: A visually clean episode of the insertion and deletion evaluation for DCAMA on COCO 20i (5-shot) using AffEx. The top row reports the IAUC and DAUC curves for a case with a clearly delineated foreground object. Subsequent rows show the confidence map, the predicted query mask (red) overlaid with the ground truth (blue), and the support images with their masks at different insertion/deletion steps, illustrating stable attribution behavior.
Figure 12: A challenging episode highlighting unstable insertion and deletion behavior for DMTNet on COCO 20i (5-shot) with AffEx. The top row presents IAUC and DAUC curves reflecting the increased difficulty of this scenario. The lower rows display the confidence map, the predicted query mask (red) against the ground truth (blue), and the support images with their masks across insertion/deletion steps, revealing fluctuations in evidence attribution.
Figure 13: An episode of the insertion and deletion evaluation for INSID3 on COCO 20i (5-shot) using AffEx. The top row reports the IAUC and DAUC curves. Subsequent rows show the model output, the predicted query mask (red), and the support images with their masks at different insertion/deletion steps. INSID3 returns a hard mask, so the first column is its binary output rather than a confidence map.
Figure 14: An episode of the insertion and deletion evaluation for SANSA on COCO 20i (5-shot) using AffEx. The top row reports the IAUC and DAUC curves. Subsequent rows show the confidence map, the predicted query mask (red), and the support images with their masks at different insertion/deletion steps.
Figure 15: An episode of the insertion and deletion evaluation for GF-SAM on COCO 20i (5-shot) using AffEx. The top row reports the IAUC and DAUC curves. Subsequent rows show the model output, the predicted query mask (red), and the support images with their masks at different insertion/deletion steps. GF-SAM returns a hard mask, so the first column is its binary output rather than a confidence map.
Figure 16: An episode of the insertion and deletion evaluation for Matcher on COCO 20i (5-shot) using AffEx. The top row reports the IAUC and DAUC curves. Subsequent rows show the model output, the predicted query mask (red), and the support images with their masks at different insertion/deletion steps. Matcher returns a hard mask, so the first column is its binary output rather than a confidence map.
Figure 17: An episode of the insertion and deletion evaluation for the prototype head of Appendix J on COCO 20i (5-shot) using AffEx. The head is the one trained on Pascal 5i with fold 0 held out, so bicycle is a class it never saw in training. The top row reports the IAUC and DAUC curves. Subsequent rows show the confidence map, the predicted query mask (red), and the support images with their masks at different insertion/deletion steps.
Few-Shot Semantic Segmentation (FSS) focuses on segmenting novel object categories from only a handful of annotated examples. Most existing approaches rely on extensive episodic training to learn transferable representations, which is both computationally demanding and sensitive to distribution shifts. In this work, we revisit FSS from the perspective of modern vision foundation models and explore the potential of Segment Anything Model 3 (SAM3) as a training-free solution. By repurposing its Promptable Concept Segmentation (PCS) capability, we adopt a simple spatial concatenation strategy that places support and query images into a shared canvas, allowing a fully frozen SAM3 to perform segmentation without any fine-tuning or architectural changes. Experiments on PASCAL-5i and COCO-20i show that this minimal design already achieves state-of-the-art performance, outperforming many heavily engineered methods. Beyond empirical gains, we uncover that negative prompts can be counterproductive in few-shot settings, where they often weaken target representations and lead to prediction collapse despite their intended role in suppressing distractors. These findings suggest that strong cross-image reasoning can emerge from simple spatial formulations, while also highlighting limitations in how current foundation models handle conflicting prompt signals. Code at: https://github.com/WongKinYiu/FSS-SAM3
Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao Wang
National Yang Ming Chiao Tung University · Institute of Information Science, Academia Sinica, Taiwan
Few-shot segmentation (FSS) aims to segment unseen object categories with a few (e.g., one or five) labeled examples, enabling efficient adaptation to novel classes. Conventional models typically rely on appearance-based visual matching between support and query images for segmentation. While straightforward, these methods often struggle to handle significant appearance discrepancies and occlusions in the query image due to insufficient target knowledge. To mitigate this, we introduce a novel framework that mines target knowledge using the strong reasoning capacity of Multimodal Large Language Models (MLLMs) and employs it to enhance FSS. Specifically, building on SAM 2, our method, named MK-FSS, exploits two forms of complementary knowledge derived from a query image by an MLLM for FSS, including spatial knowledge, which provides a spatial prior indicating the potential target location, and semantic knowledge, which describes the target using text. The spatial knowledge is first encoded into a memory representation, and then resulting memory is integrated with the support-guided memory feature from query image through a carefully designed dual-memory debate-fusion (DMDF) module, yielding a more robust target memory feature. In parallel, the semantic knowledge is encoded into the textual feature, which is fused with multi-scale query features via a progressive cross-modal prompt generator (PCPG), producing a target-aware multimodal prompt for segmentation. Working together, the dual-memory feature and the multimodal prompt provide a comprehensive representation of the target, enabling more robust segmentation. In our extensive experiments, MK-FSS shows promising results and largely surpasses existing methods. Code will be released.
Few-shot medical image segmentation (FS-MIS) aims to segment novel regions of interest (ROIs) from a few annotated support examples. Despite rapid progress, existing FS-MIS solutions span diverse paradigms but are evaluated under inconsistent settings, leaving their relative effectiveness unclear. We introduce FAME, a unified benchmark for evaluating FS-MIS solutions, covering specialists, SAM-based methods, CLIP-based methods, and MLLM-based methods. FAME contains 14,958 test samples across 7 anatomical sites, 9 imaging modalities, and 14 ROI categories, and evaluates models under zero-shot and ten-shot settings with additional assessment of target-absence recognition and generalization under covariate and semantic shifts. Our evaluation reveals several findings. First, effective few-shot segmentation depends on how models exploit support examples: direct visual adaptation generally outperforms prompt-based strategies. Second, increasing support examples improves performance only when models can effectively utilize them. Third, semantic transfer remains substantially more challenging than imaging-domain adaptation, and strong localization ability does not necessarily imply reliable target-absence recognition. We hope FAME provides a comprehensive understanding of current FS-MIS solutions and facilitates the development of more effective and reliable few-shot medical segmentation methods.