Open-Vocabulary Semantic Segmentation

Ask about this topic

Overview

Open-vocabulary semantic segmentation aims to assign meaningful labels to each pixel, including categories not seen during training, making it useful when the set of objects or environments can change. Many methods reuse vision-language models such as CLIP, DINO, and SAM, combining text–image similarity with attention, mask proposals, feature alignment, or prompt and prototype calibration; others train with region-level text, pseudo-labels, or generated data. Systems are tested on general benchmarks such as Pascal, COCO, and ADE20K, as well as remote-sensing, urban-scene, 3D, and other specialized settings. A central challenge is balancing broad category recognition with precise boundaries and small-object localization, especially under domain shifts, while keeping inference efficient and evaluation protocols comparable.

Latest papers

CardsList

87 papers8 papers in the last four weeks, up 14% on the four weeks before. 0.0% of all new papers.

  1. Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes

    Oct 22, 2026Aijing Yu, Zhengbo Wang, Lijun Sheng +2Open-Vocabulary Semantic SegmentationVision-Language Alignment

    School of Cyber Security, University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences +2

  2. OpenEarth-OVS44: A Multimodal Remote Sensing Dataset for Open-Vocabulary Segmentation

    Oct 21, 2026Yimin Wei, Hongruixuan Chen, Hao Zheng +5Remote Sensing Image SegmentationOpen-Vocabulary Semantic Segmentation

    The University of Tokyo · RIKEN AIP · East China Normal University +1

  3. CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation

    Oct 17, 2026Dengke Zhang, Fagui Liu, Quan Tang +1Open-Vocabulary Semantic SegmentationSemantic Segmentation

    School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China. · Department of New Network, Pengcheng Laboratory, Shenzhen, 518000, China.

  4. TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

    Oct 10, 2026Hao Zhu, Heyu Sun, Shuo Jin +2Promptable Image SegmentationOpen-Vocabulary Semantic Segmentation

  5. A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Images

    Oct 3, 2026Jianzheng Wang, Huan Ni, Xiaonan Niu +2Remote Sensing Image SegmentationOpen-Vocabulary Semantic Segmentation

    School of Remote Sensing and Geomatics Engineering, Nanjing University of Information Science and Technology, China, Nanjing, 210044, China · Nanjing Center, China Geological Survey, Nanjing, 210016, China · School of Automation, Southeast University, Nanjing, 210096, China

  6. EmbPASS: Towards Cross-Embodiment Open Panoramic Segmentation

    Oct 2, 2026Pujun Guo, Yuanfan Zheng, Fei Teng +5Semantic SegmentationCross-Embodiment Transfer

    School of Artificial Intelligence and Robotics, Hunan University, China · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China

  7. When to Adapt: Multi-Signal Domain Shift Detection for Efficient Training-Free Adaptation in Open-Vocabulary Segmentation

    Sep 29, 2026Michele Antonazzi, Alejandra C. Hernandez, José Araujo +2Distribution ShiftContinual Test-Time Adaptation

    EECS and Digital Futures, KTH Royal Institute of Technology, Stockholm, Sweden · Ericsson Research, Ericsson AB, Stockholm, Sweden

  8. Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement

    Sep 28, 2026Hyun-Kurl Jang, Jihun Kim, Kuk-Jin YoonWeakly Supervised Semantic SegmentationPreference Optimization

    Visual Intelligence Lab. KAIST

  9. HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space

    Sep 21, 2026Zelin Peng, Zhengqin Xu, Changsong Wen +4VLM AdaptationOpen-Vocabulary Semantic Segmentation

    MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · State Key Laboratory of Infrared Physics, Shanghai Institute of Technical Physics, Chinese Academy of Science, Shanghai, 200083, China · Meituan, Shanghai, 200051, China

  10. SatOV: Restoring Spatial Priors for Training-Free Open-Vocabulary Segmentation in Remote Sensing Imagery

    Sep 19, 2026Changhao Zhao, Linglin Zeng, Hai LiuOpen-Vocabulary Semantic SegmentationRemote Sensing Image Segmentation

    College of Resources and Environment, Huazhong Agricultural University, Wuhan 430070, China

  11. ViCo-SAM3: Vision-Conditioned Alignment for Open-Vocabulary Camouflaged Object Segmentation

    Sep 14, 2026Qiangqiang Zhou, Wenjun Tang, Yong Chen +2Text-Guided Image SegmentationOpen-Vocabulary Semantic Segmentation

    School of Artificial Intelligence, Jiangxi Normal University · Institute of AI Education, East China Normal University

  12. LangStreet: Persistent Language Fields for Anchor-Decoded Street Gaussians

    Sep 10, 2026Runyi Yang, Deheng Zhang, Xiaoye Wang +63D Gaussian SplattingLanguage-Embedded 3DGS

    INSAIT, Sofia University “St. Kliment Ohridski” · vivo Mobile Communication Co., Ltd., Shenzhen, China

  13. Evaluation of Vision-Language Models Across Diverse Coastal Environments

    Sep 9, 2026Seth Knoop, Chad R. Samuelson, Gabriel R. Slade +2VLM EvaluationOpen-Vocabulary Semantic Segmentation

  14. ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

    Sep 3, 2026Javier del Pino, Salvador Rodríguez, Alejandro Garabito +2Video Object SegmentationInstance Segmentation

    SperidLabs

  15. Restrict, Don't Retrain: Inference-Time VLM Guidance for Zero-Shot Aerial Segmentation

    Sep 1, 2026Teresa DiMeola, Charles Walter, Hong XiaoRemote Sensing Image SegmentationVision Foundation Models

    Dept. of Computer and Information Science University of Mississippi

  16. Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation

    Aug 30, 2026Chandler Timm C. Doloriel, Yunbei Zhang, Sarthak Kumar Maharana +5Test-Time Adaptation of VLMsOpen-Vocabulary Semantic Segmentation

    Faculty of Science and Technology (REALTEK), Norwegian University of Life Sciences (NMBU) · Tulane University · University of Texas at Dallas

  17. Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

    Aug 29, 2026Tianrui Hui, Shaofei Huang, Qisong Han +6Audio-Visual UnderstandingOpen-Vocabulary Semantic Segmentation

    Hefei University of Technology Hefei, Anhui, China · Cardiff University Cardiff, Wales, United Kingdom · University of Macau Macau, China

  18. Text-to-seed generation: Training-free open-vocabulary seeded semantic segmentation via re-purposing diffusion as text-guided seed generator

    Aug 27, 2026Kumju Jo, Heesun Jung, Sungyong BaikOpen-Vocabulary Semantic SegmentationSemantic Segmentation

    Dept. of Artificial Intelligence, Hanyang University · Dept. of Data Science, Hanyang University

  19. GeoSeg-OV: Bridging Geospatial Gaps with Structural Guidance for Open-Vocabulary Remote Sensing Segmentation

    Aug 11, 2026Ruizhong Liu, Tingzhang Luo, Zaiyan Zhang +4Remote Sensing Image SegmentationDomain Generalization

    School of Computer Science, China University of Geosciences, Wuhan 430074, Hubei, China · Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, Guangdong, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China +3

  20. Test-Time Prototype Adaptation for Open-Vocabulary Semantic Segmentation

    Aug 8, 2026Haozhe Wang, Jintao Cheng, Weibin Li +1Test-Time AdaptationTest-Time Adaptation of VLMs

    South China Normal University · The Hong Kong University of Science and Technology

  21. Universal Concept Disruption for SAM3 Image Segmentation

    Aug 6, 2026Hao Wang, Yuxuan Zhang, Wei YangPromptable Image SegmentationAdversarial Robustness

    School of Computer Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China +1

  22. MAVISEG: Manifold Propagation and Visual Prototypes for Zero-Shot Open-Vocabulary Segmentation in Diffusion Transformers

    Aug 6, 2026Rajatsubhra Chakraborty, Xujun Che, Ritabrata Chakraborty +2Image SegmentationDiffusion Transformer

    University of North Carolina at Charlotte, Charlotte, NC, USA · Manipal University Jaipur, Jaipur, Rajasthan, India

  23. SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

    Aug 6, 2026Mohamad Zamini, Diksha ShuklaMemory-Augmented VLMsOpen-Vocabulary Semantic Segmentation

    University of Wyoming Laramie, WY

  24. Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation

    Aug 4, 2026Wanli Ma, Jiangwen Lu, Qinmu Peng +1Open-Vocabulary Semantic SegmentationVision-Language Alignment

    School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · National Anti-counterfeit Engineering Research Center, Huazhong University of Science and Technology, Wuhan 430074, China · School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China

  25. Standalone DINOv3 for Training-Free Open-Vocabulary Semantic Segmentation in Remote Sensing

    Aug 4, 2026Changhao Zhao, Haoxiang Li, Yuke Li +2Remote Sensing Image SegmentationOpen-Vocabulary Semantic Segmentation

    College of Resources and Environment, Huazhong Agricultural University, Wuhan 430070, China

  26. Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

    Aug 3, 2026Jiazhen Liu, Mingkuan Feng, Long ChenInstance SegmentationImage Segmentation

    The Hong Kong University of Science and Technology (HKUST), Hong Kong.

  27. EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass

    Aug 3, 2026Haomin Peng, Yongkang Li, Zhaoxiang Liu +4Efficient VLM InferenceOpen-Vocabulary Semantic Segmentation

    Huazhong University of Science and Technology · Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom +1

  28. OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation

    Jul 29, 2026Kaiyu Li, Zepeng Xin, Zixuan Jiang +4VLM EvaluationOpen-Vocabulary Object Detection

    Xi’an Jiaotong University

  29. TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions

    Jul 28, 2026Xinran Liu, Shouqian Shi, Yutong Chen +3Open-Vocabulary Semantic SegmentationVision-Language Alignment

  30. Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion

    Jul 28, 2026Junhyuk Heo, Junghwan ParkRemote Sensing Image SegmentationFew-Shot Semantic Segmentation

    TelePIX, Seoul, Republic of Korea