LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation

Yuqian Yuan; Wenqiao Zhang; Juekai Lin; Yu Zhong; Mingjian Gao; Binhe Yu; Yunqi Cao; Wentong Li

cs.CVApr 13, 2026

LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation

Authors: Yuqian Yuan, Wenqiao Zhang, Juekai Lin, Yu Zhong, Mingjian Gao, Binhe Yu, Yunqi Cao, Wentong Li,

Organizations: Zhejiang University · Nanjing University of Aeronautics and Astronautics

Abstract

Large Multimodal Models (LMMs) have achieved remarkable progress in general-purpose vision--language understanding, yet they remain limited in tasks requiring precise object-level grounding, fine-grained spatial reasoning, and controllable visual manipulation. In particular, existing systems often struggle to identify the correct instance, preserve object identity across interactions, and localize or modify designated regions with high precision. Object-centric vision provides a principled framework for addressing these challenges by promoting explicit representations and operations over visual entities, thereby extending multimodal systems from global scene understanding to object-level understanding, segmentation, editing, and generation. This paper presents a comprehensive review of recent advances at the convergence of LMMs and object-centric vision. We organize the literature into four major themes: object-centric visual understanding, object-centric referring segmentation, object-centric visual editing, and object-centric visual generation. We further summarize the key modeling paradigms, learning strategies, and evaluation protocols that support these capabilities. Finally, we discuss open challenges and future directions, including robust instance permanence, fine-grained spatial control, consistent multi-step interaction, unified cross-task modeling, and reliable benchmarking under distribution shift. We hope this paper provides a structured perspective on the development of scalable, precise, and trustworthy object-centric multimodal systems.

Figures & tables

Figure 1: Overview of object-centric vision in the era of large multimodal models. The modalities considered in this paper include images, videos, and 3D data. Object-centric representations enable four fundamental capabilities: object understanding, object segmentation, object editing, and object generation.

Figure 2: Summary of representative models spanning object-centric visual understanding, segmentation, editing, and generation. For a comprehensive list of related approaches, along with detailed discussions of their specifications, configurations, and technical aspects, please refer to Section 3 , Section 4 , Section 5 , and Section 6 , respectively.

Figure 3: Illustration of three representative paradigms for incorporating visual prompts into multimodal large language models: Text as Prompt, Visual Prompt as Token, and Visual Prompt Fusion with Vision. These paradigms differ in whether visual prompts are expressed as language, injected as prompt tokens, or fused directly with visual representations before language modeling.

Dataset & Benchmark	Year	Modality	Scale	Role	Task Focus
GRIT [ 212 ]	2024		1.1M	D	Region grounding and referring
PAM-Data [ 96 ]	2025		2.1M	D	Region recognition and explanation
LVIS [ 54 ]	2019		164K	D	Large-vocabulary object instances
Visual Genome [ 73 ]	2017		108K	D	Objects, regions, and relations
HC-STVG [ 156 ]	2020		5.6K	D	Human-centric spatio-temporal grounding
ElysiumTrack-1M [ 163 ]	2024		1.27M	D	Object-level video tracking

Table 1: Summary of representative Datasets & Benchmarks for object-centric visual understanding .

\bullet

Modality : : Image, : Video.

\bullet

Role : D : Training Dataset, and B : Evaluation Benchmark.

Figure 4: Four representative language–pixel interfaces for LMM-based object-centric referring segmentation: (a) single embedding prediction, (b) multi embedding prediction, (c) next-token prediction, and (d) attention-based prediction. They differ in whether the segmentation signal is expressed as one prompt embedding, multiple mask-aware queries, a generated mask-related sequence, or an attention-derived spatial field.

Dataset & Benchmark	Year	Modality	Scale	Role	Task Focus
RefCOCO / RefCOCO+ / RefCOCOg [ 110 , 117 ]	2016		142K / 142K / 95K	D B	Classical referring segmentation and object grounding
ReasonSeg [ 77 ]	2024		1.2K	D B	Reasoning segmentation and implicit grounding
gRefCOCO [ 97 ]	2023		278.2K	D B	Generalized segmentation with multi- and no-target cases
MUSE [ 139 ]	2023		246.0K	D B	Open-set multi-target segmentation
GranD [ 135 ]	2024		7500K	D	Grounded concept-mask-text alignment
M3G2 [ 230 ]	2024		2500K	D	Multi-grained grounding tuning

Table 2: Summary of representative Datasets & Benchmarks for object-centric referring segmentation .

\bullet

Modality : Image: , Video: , Audio-Visual: , 3D scene: .

\bullet

Role : D : Training Dataset, B : Evaluation Benchmark.

Figure 5: Four representative paradigms for generative model-based object editing: (a) Diffusion-based Object Editing, (b) Autoregressive-based Object Editing, (c) Video Object Editing, and (d) 3D Object Editing. They differ in their core generative mechanisms and target modalities, specifically utilizing diffusion processes for standard 2D images, autoregressive models to generate tokenized editing sequences, video models for temporal frame sequences, and 3D-aware models for geometry and texture modifications.

Dataset & Benchmark	Year	Modality	Scale	Role	Editing Types	Evaluation Metrics
MagicBrush [ 226 ]	2023		10K triplets	D B	Add, Remove, Replace, Color, Action, Pattern	L1, L2, CLIP-I/T, DINO
TEdBench++ [ 12 ]	2024		120 entries	B	Multi-conditioning, Removal, Style Transfer, Complex Replace	Success Rate, LPIPS
DragBench [ 146 ]	2024		205 images	B	Drag / Point-based	Mean Distance, Image Fidelity
InsViE-1M [ 194 ]	2025		1.02M triplets	D B	Substitution, Color, Style, Add, Remove, Weather	CLIP-T, PickScore, DOVER, Flow EPE
EdiVal-Agent [ 27 ]	2025		572 images, 1,716 instructions	B	Subject Add/Remove/Replace, Color, Material, Position, Count	EdiVal-IF, EdiVal-CC, EdiVal-VQ, EdiVal-O
OBJECT [ 112 ]	2023		400K triplets	D B	Translation, Rotation, Insertion, Removal	PSNR, SSIM, LPIPS, FID

Table 3: Summary of existing Datasets & Benchmarks for training and evaluating object-centric editing methods .

\bullet

Modality : : Image, : Video and : 3D scene.

\bullet

Role : D : Training Dataset, and B : Evaluation Benchmark.

Figure 6: Overview of generative methodologies for multimodal content creation. The figure categorizes mainstream approaches for image, video, and 3D generation, highlighting the evolution from static 2D synthesis to complex spatial and temporal modeling.

Benchmark	Year	Scale	Role	Data Components	Usage
LAION-400M [ 142 ]	2021	400M pairs	D	Image-Text pairs	Text-guided image generation training
LAION-5B [ 143 ]	2022	5B pairs	D	Image-Text pairs	Text-guided image generation training
MS-COCO [ 94 ]	2014	328k images	B	Images, Bbox, Captions, Mask	Generation evaluation with FID, CLIP
InternVid [ 179 ]	2023	7M videos, 243M clips	D	Video-Text pairs	Text-driven video generation
Panda-70M [ 28 ]	2024	70M videos	D	Video-Text pairs	Text-driven video generation
Text2Shape [ 23 ]	2018	75K paris	D	Captions, Colored Voxels	Conditional 3D generation

Table 4: Summary of existing Datasets & Benchmarks for training and evaluating image/video/3D generation methods .

\bullet

Role : D : Training Dataset, and B : Evaluation Benchmark.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

ActiveScope: Actively Seeking and Correcting Perception for MLLMs
Jun 23, 2026Yajing Wang, Chao Bi, Junshu Sun +4Multimodal Large Language Models Vision-Language Grounding
University of Chinese Academy of Sciences · State Key Lab of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China. · Harbin Institute of Technology (Weihai).
MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
Aug 11, 2026Changhao Xiang, Shangyu Xing, Zhen Wu +2Multimodal Grounding Multimodal Pretraining
National Key Laboratory for Novel Software Technology, Nanjing University, China
Is It Time for the Renaissance of Salient Object Detection in the Era of MLLMs?
Jul 31, 2026Wenzhuo Zhao, Xiuzhi Li, Zhongkuan Mao +6VLM Evaluation Salient Object Detection
College of Computer Science, Sichuan University, China · National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, China · School of Information and Communication Engineering, University of ElectronicScience and Technology of China, China

Jun 23, 2026J/K move · Enter open · S save

ActiveScope: Actively Seeking and Correcting Perception for MLLMs

Yajing Wang, Chao Bi, Junshu Sun, Shufan Shen+3

University of Chinese Academy of Sciences · State Key Lab of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China. · Harbin Institute of Technology (Weihai).

Multimodal Large Language Models Vision-Language Grounding Visual Search Fine-Grained Visual Perception

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated impressive vision-language understanding, yet still struggle with fine-grained perception in high-resolution images. While existing training-free methods typically rely on attention-based localization or coarse-to-fine search, they are often misled by distractors and fail to locate multiple targets. Our investigation attributes these failures to Contextual Dominance, where salient distractors overwhelm target attention and cause inaccurate localization, and Semantic Bias, where global semantics cause the model to fixate on the most salient concept, resulting in incomplete localization in multi-object scenarios. Built on these insights, we propose ActiveScope, a training-free framework that enhances MLLMs by actively seeking and correcting perception. ActiveScope features two modules. The Semantic Anchor Localization (SAL) utilizes fine-grained semantic anchors to independently localize key targets, thereby mitigating semantic bias. The Interference-Suppressed Refinement (ISR) refines localization by suppressing attention on salient distractions to overcome contextual dominance. Extensive experiments on high-resolution image understanding benchmarks demonstrate that ActiveScope outperforms existing training-free methods (e.g., 96.34 percent accuracy on

V^{*}

Bench), validating the superiority of the active search and self-correction paradigm. Our code is available at https://github.com/jasmine-ww/ActiveScope.

Abstract

Multimodal Large Language Models

Explore similar work

ActiveScope: Actively Seeking and Correcting Perception for MLLMs

MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment

Is It Time for the Renaissance of Salient Object Detection in the Era of MLLMs?