cs.CVNov 6, 2026

MESAM: Enhancing SAM2 for Few-Shot Segmentation through Semantic-Visual Memory Generation and Refinement

Authors: Tianyu Zou, Shengwu Xiong, Rui Wu, Ruilin Yao, Yi Rong

Organizations: Sanya Science and Education Innovation Park, Wuhan University of Technology Sanya, Hainan, China · School of Artificial Intelligence, Wuhan University of Technology Wuhan, Hubei, China · Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology Wuhan, Hubei, China · Institute of Automation, Chinese Academy of Sciences Beijing, China

Abstract

This paper investigates how to leverage the powerful capabilities of the Segment Anything Model 2 (SAM2) to tackle few-shot segmentation (FSS) tasks. One major challenge lies in the substantial appearance differences between the support and query images, which may diminish the effectiveness of the memory-matching process in SAM2. Through systematic analysis, we find that the memory generated based on the categorical semantic information is robust to this appearance divergence and is more compatible with SAM2. Based on these observations, we propose a new Memory-Enhanced SAM2 (MESAM) method, which first introduces a Semantic-Visual Memory Generation (SVMG) module to separately construct a query semantic memory, a query foreground visual memory, and a support adaptive visual memory. Then, in our designed Interactive Semantic Memory Refinement (ISMR) module, we further enrich the obtained semantic memory with fine-grained visual details, and suppress its misatcivated background responses. Extensive experiments on the PASCAL-5i5^i, COCO-20i20^i, and LVIS-92i92^i datasets demonstrate that MESAM consistently outperforms existing FSS methods, achieving new state-of-the-art segmentation performance. Our source code is available at: https://github.com/tianyu-zou/MESAM.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Few-Shot Semantic Segmentation Meets SAM3

    Apr 7, 2026Yi-Jen Tsai, Yen-Yu Lin, Chien-Yao WangFew-Shot Semantic SegmentationVision Foundation Models

  2. Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation

    Oct 19, 2026Tianqi Wei, Zhi Chen, Xin Yu +2Few-Shot Semantic SegmentationText-Guided Image Segmentation