cs.CVOct 19, 2025

Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting

Authors: Jiazhen Liu, Long Chen

Organizations: Department of Computer Science and Engineering, The Hong Kong University of Science and Technology

Abstract

Ensuring trustworthiness and versatility in Open-World Visual Recognition is a core goal of Multimodal Large Language Models (MLLMs). While integrating segmentation is desirable, prevailing methods require invasive fine-tuning, which often leads to catastrophic forgetting of the model's general-purpose capabilities. This trade-off between task-specific performance and general reliability fundamentally undermines the goal of building unified, trustworthy systems. We introduce LENS (Leveraging kEypoiNts for MLLMs' Segmentation), a novel plug-and-play paradigm designed to equip MLLMs with pixel-level ability without compromising their foundational integrity. LENS attaches a lightweight head to a completely frozen MLLM, interpreting the intrinsic spatial cues embedded in attention maps. It extracts keypoints and translates them into prompt-compatible features for a mask decoder. Extensive experiments validate that LENS achieves segmentation performance competitive with retraining-based methods. Crucially, it largely preserves the MLLM's open-world generalization, avoiding the degradation observed in fine-tuned counterparts. By decoupling segmentation from generation, LENS offers a robust and efficient path toward truly multi-talented and reliable vision models.

Figures & tables

Explore similar work

CardsList
  1. From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

    Mar 18, 2026Boyong Wu, Sanghwan Kim, Zeynep AkataMultimodal Large Language ModelsSemantic Segmentation

  2. Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

    Mar 19, 2026Anqi Zhang, Xiaokang Ji, Guangyu Gao +3Image SegmentationSemantic Segmentation