State-Message Interaction Adapter for Cross-Domain Few-Shot Segmentation
Authors: Hang Xing, Yan Xia, Yongjiang Liu, Siqi Zhang, Xueming Ding
Organizations: Wuxi Key Laboratory of Photovoltaic Equipment Advanced Manufacturing Technology, Wuxi, China · Wuxi Autowell Technology Co., Ltd., Wuxi, China · Department of Control Science and Engineering, School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China
Cross-domain few-shot segmentation adapts natural-image pretrained encoders to medical or industrial domains from a few pixel-annotated support masks. Within frozen DINOv3, intermediate per-head attention messages already carry target-background distinctions that baseline final-layer similarities and predictions do not consistently reflect. We propose the State-Message Interaction Adapter (SMIA) to learn how per-head messages should update each receiving token representation while keeping pretrained attention and FFN parameters frozen. Inserted right after per-head aggregation, SMIA conditions the transformation of each head's aggregated output (message) on each patch's pre-attention feature (state), allowing similar contextual messages to produce distinct or opposing updates depending on the receiving position. Because scarce support masks cover only limited state-message combinations, a shared low-rank bilinear map confines the learned interactions to low-dimensional state, message, and output subspaces. Added before each adapted block's frozen FFN, these updates yield adapted features that are refined by frozen FFNs and deeper blocks for linear readout. With only 0.51M trainable parameters and single-pass, support-free inference, SMIA achieves six-dataset mean mIoUs of 77.40% at 5 shots and 80.99% at 10 shots across 20 medical and industrial tasks, surpassing the strongest prior methods by +3.61 and +3.93 percentage points. In controlled 10-shot comparisons across three benchmarks on the same DINOv3 backbone, SMIA also outperforms parameter-matched AdaptFormer, LoRA, and ParaX with a better precision-recall balance.
Figures & tables
Figure 1: Intermediate attention messages vs. downstream segmentation responses. Left column: query image with ground-truth mask and marked foreground patch (top), plus single-head message contrast from frozen DINOv3 Block 9 (bottom; Head 2 on BUSI, Head 5 on MVTec Capsule), where teal/plum indicates closer cosine proximity to foreground/adjacent-background references. Right three columns: final-layer cosine similarity to the marked patch with dashed white ground-truth contours (top) and foreground probability from linear readouts fit on the same 10 supports (bottom; diagnostic 4-block SMIA and rank-8 LoRA setup in Supplementary Sec. C.1). Only SMIA recovers clean, localized boundaries.
Figure 2: In-block architecture and state–message interaction of SMIA. Left: Frozen DINOv3 block with SMIA patch correction injected before the FFN sublayer. Right: Conditional update path for patch t , illustrated on a Kvasir-SEG query at Block 12 of 4-block SMIA: heatmaps show cosine similarity to reference patch t (shared [−1,1] scale; brighter is higher) for local state ut , per-head messages mth , and pre-FFN updated features combining the original attention residual with δt , while correction magnitude plots ∥δt∥2 ( [0,545] ).
Medical Imaging Benchmarks (5 Tasks)
Industrial (15)
6-Dataset Summary
Method
Backbone
BUSI
Kvasir-SEG
CVC-ClinicDB
ISIC 2018
Chest X-ray
MVTec AD
Overall mIoU
5-Shot Support Budget ( N=5 )
INSID3 [ 8 ] [CVPR’26]
DINOv3 ViT-L/16
40.01 ± 6.02
81.07 ± 1.03
63.08 ± 8.80
76.13 ± 1.08
85.65 ± 0.27
56.75 ± 1.76
67.11 ± 1.90
RSRM [ 21 ] [ECCV’26]
DINOv3 ViT-B/16
61.49 ± 1.63
81.85 ± 0.70
73.39 ± 4.95
75.86 ± 1.16
90.86 ± 1.64
52.62 ± 0.95
72.68 ± 0.79
DINO-AugSeg [ 24 ] [MLST’26]
DINOv3 ConvNeXt-L
56.83 ± 6.41
76.54 ± 3.75
74.35 ± 5.21
71.15 ± 6.20
92.78 ± 0.90
66.40 ± 1.72
73.01 ± 1.43
DefectSAM [ 25 ] [TNNLS’25]
SAM ViT-B
55.87 ± 6.74
60.88 ± 4.14
58.53 ± 3.71
71.80 ± 2.01
88.30 ± 5.73
62.16 ± 0.73
66.25 ± 1.57
Table 1: Quantitative comparison across 20 medical and industrial tasks under 5-shot and 10-shot support budgets. Values report native-resolution mIoU (%, mean ± SD across five nested random support draws). MVTec AD averages all 15 industrial categories equally per draw, and Overall weights the five medical datasets and MVTec AD equally (green parentheses: gains over the strongest prior method computed before rounding). Best and second-best scores per budget are bolded and underlined .
Figure 3: Qualitative comparison across medical and industrial tasks ( N=10 , shared support draw per row). Top to bottom: BUSI breast ultrasound, Kvasir-SEG polyp, MVTec Bottle contamination, and MVTec Metal Nut scratch (cyan contours: ground-truth boundaries; coral-red overlays: predicted foreground). SMIA delineates fine contours without over-segmenting surroundings.
5-Shot ( N=5 )
10-Shot ( N=10 )
Method
Sup.-Free
Prec.
Rec.
mIoU
Prec.
Rec.
mIoU
INSID3 [ 8 ]
✗
26.39
47.82
56.75
26.78
52.76
57.98
RSRM [ 21 ]
✗
17.22
69.85
52.62
16.67
72.74
52.40
HERA [ 16 ]
✗
22.97
70.80
55.45
28.62
70.65
58.30
MPA [ 17 ]
✗
53.80
40.92
65.14
51.10
54.22
67.62
DefectSAM [ 25 ]
✓
79.01
26.91
62.16
81.60
39.83
68.30
Table 2: Industrial foreground precision–recall and mIoU breakdown across all 15 MVTec AD categories (%, five-draw mean; gains computed before rounding). Sup.-Free denotes query-only inference without support inputs.
Dataset mIoU (%)
Summary
Configuration
Params
MVTec
CVC
ISIC
Mean
Frozen + Linear
0.4K
74.98
74.90
77.95
75.94
AdaptFormer [ 6 ] ( b=55 )
512.5K
76.94
78.69
77.06
77.56
LoRA [ 10 ] ( r=28,α=56 )
516.5K
77.58
78.80
77.66
78.01
ParaX [ 15 ] ( b=28 )
507.7K
77.31
79.91
77.56
78.26
SMIA ( r=4 )
243.5K
78.12
80.09
79.52
79.24 (+0.99)
Table 3: Controlled same-backbone PEFT and subspace ablations on DINOv3 ViT-S+/16 ( N=10 , a shared fixed support–query split per task; gains vs. ParaX computed before rounding). w/o Bh omits off-diagonal mixing.
Adapted Blocks
Params
Medical (5)
MVTec AD (15)
Overall mIoU
Last 4 Blocks
170.9K
81.05 ± 0.99
76.29 ± 1.21
80.26 ± 0.76
Last 8 Blocks
341.4K
81.74 ± 1.22
76.70 ± 0.98
80.90 ± 1.05
All 12 Blocks
511.9K
81.80 ± 1.24
76.96 ± 1.11
80.99 ± 1.10
Table 4: Adaptation depth ablation across all 20 tasks ( N=10 , five draws, r=8 ). Medical and MVTec AD average the 5 medical tasks and 15 industrial categories equally.
Figure 4: Opposing corrections from strongly aligned boundary messages. Rows show MVTec Grid, MVTec Bottle, and ISIC 2018 queries ( N=10 , Block 12). Green overlays mark ground truth; red triangles ( A ) and circles ( B ) mark nearby foreground and background boundary patches (amber zoom insets). Right three columns plot cosine similarity to A ( [−1,+1] ) for concatenated six-head message, state projection, and total correction; aligned messages at A and B map to opposing updates.
Figure 5: Counterfactual state-swap intervention under fixed attention messages. Rows show BUSI lesion ( A , red triangle) and MVTec Hazelnut intact shell ( B , red circle), with amber zoom insets and restored pixel probabilities p ; heatmaps aggregate independent per-patch swaps across matched positions. Holding Block-12 features, six-head messages, and correction ℓ2 -norm fixed, swapping only SMIA’s state projection from opposite-label donors ( d≤0.20 ) flips local predictions.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
MVTec Category
DefectSAM [ 25 ]
DINO-AugSeg [ 24 ]
HERA [ 16 ]
INSID3 [ 8 ]
RSRM [ 21 ]
MPA [ 17 ]
SMIA (Ours)
5-Shot Support Budget ( N=5 )
Bottle
72.12 ± 7.15
75.49 ± 4.61
63.74 ± 14.54
55.51 ± 11.63
49.61 ± 5.28
78.39 ± 3.08
81.96 ± 6.47
Cable
64.44 ± 5.18
66.22 ± 3.49
51.61 ± 8.68
54.58 ± 9.31
49.66 ± 8.71
63.29 ± 5.74
68.72 ± 8.12
Capsule
52.94 ± 2.96
57.09 ± 5.11
52.62 ± 0.88
56.24 ± 1.16
43.16 ± 0.76
53.46 ± 0.85
59.30 ± 3.91
Carpet
56.57 ± 6.90
73.88 ± 6.10
71.13 ± 6.84
56.30 ± 5.71
75.92 ± 6.13
71.10 ± 3.74
78.88 ± 5.14
Grid
53.98 ± 1.70
64.72 ± 1.49
63.18 ± 2.56
70.19 ± 0.84
62.38 ± 1.46
66.42 ± 2.74
73.25 ± 2.99
Appendix
Table S1 : Per-category MVTec AD comparison under N=5 and N=10 support budgets. Values report native-resolution mIoU (%, mean ± SD across five nested random support draws; gains computed before rounding). Best and second-best results per category and budget are bolded and underlined .
Figure S1 : Additional qualitative comparisons across medical and industrial tasks ( N=10 ). Top two rows: ISIC 2018 dermoscopy and CVC-ClinicDB endoscopy; bottom three rows: MVTec Capsule faulty imprint, Cable swap, and Screw neck scratch (cyan contours: ground-truth boundaries; coral-red overlays: predicted foreground; shared 10-shot supports per row).
5-Shot ( N=5 )
10-Shot ( N=10 )
Method
Sup.-Free
Prec.
Rec.
mIoU
Prec.
Rec.
mIoU
INSID3 [ 8 ]
✗
60.74
82.46
69.19
61.28
85.20
69.41
RSRM [ 21 ]
✗
72.48
77.83
76.69
73.36
80.30
77.76
HERA [ 16 ]
✗
76.40
76.73
77.46
77.81
80.82
79.60
MPA [ 17 ]
✗
78.74
61.69
72.46
78.57
68.42
75.53
DefectSAM [ 25 ]
✓
72.46
53.45
67.07
75.64
63.94
72.73
Appendix
Table S2 : Medical foreground precision–recall and mIoU breakdown across all five clinical datasets (%, five-draw mean; gains computed before rounding). Sup.-Free denotes query-only inference without support inputs.
Figure S2 : Extended representation and segmentation diagnostics across six tasks ( N=10 ). Left, top to bottom: BUSI, CVC-ClinicDB, and Kvasir-SEG. Right, top to bottom: ISIC 2018, Chest X-ray, and MVTec AD Carpet. Within each two-row case, the top row shows the query image with ground-truth mask and reference patch in red alongside final-layer anchor cosine similarity for frozen DINOv3, LoRA, and SMIA against white dashed ground-truth contours; the bottom row shows frozen intermediate single-head message contrast from Eq. S2 alongside predicted foreground probabilities under shared 10-shot supports and linear readouts (4-block SMIA vs. 12-block LoRA at r=8 ; Sec. C.1 ).
Figure S3 : Extended counterfactual state-swap interventions across four medical datasets (left) and four MVTec AD categories (right). Top to bottom: CVC-ClinicDB, Kvasir-SEG, ISIC 2018, and Shenzhen Chest X-ray (left); MVTec Bottle, Carpet, Leather, and Wood (right). Donor message cosine distance is ≤0.30 on CVC-ClinicDB, Kvasir-SEG, and Chest X-ray, and ≤0.20 elsewhere; same-label swaps preserve predictions while opposite-label swaps flip four foreground and four background targets.
Figure S4 : Extended forward-pass boundary direction analysis across eight tasks at Block 12 ( N=10 ). Left, top to bottom: BUSI, Kvasir-SEG, CVC-ClinicDB, and Chest X-ray. Right, top to bottom: MVTec Tile, Leather, Capsule, and Cable. Heatmaps plot cosine similarity relative to foreground patch A (red triangle, [−1,+1] ) for concatenated six-head message, state projection, and total correction; amber insets magnify nearby foreground ( A ) and background ( B , red circle) boundary patches.
CVC-ClinicDB
MVTec Bottle
Method
Lat. (ms)
Mem. (MiB)
Lat. (ms)
Mem. (MiB)
INSID3 [ 8 ]
277.57
3,283.30
73.28
2,673.37
RSRM [ 21 ]
509.43
926.43
299.96
762.33
HERA [ 16 ]
249.44
1,603.78
158.85
1,537.80
MPA [ 17 ]
53.56
407.73
60.09
406.44
DefectSAM [ 25 ]
74.52
2,819.17
83.39
2,819.17
Appendix
Table S3 : Batch-1 inference latency and peak allocated GPU memory on one RTX 5090. Median single-image latency (ms) and peak PyTorch allocated memory (MiB) over 80 shared queries per task across three post-warmup runs (lowest bolded , second-lowest underlined ).