Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning

Xu Zhang; Jin Yuan; BinHong Yang; Xuan Liu; Qianjun Zhang; Yuyi Wang; Zhiyong Li; Hanwang Zhang

cs.CVMar 21, 2026

Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning

Authors: Xu Zhang, Jin Yuan, BinHong Yang, Xuan Liu, Qianjun Zhang, Yuyi Wang, Zhiyong Li, Hanwang Zhang

Organizations: the College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China. · the School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, P.R. China. · CRRC Zhuzhou Institute Company Ltd., Zhuzhou, Hunan 412001, China. · the School of Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China. ·

Abstract

Recent advancements in multimodal large models have significantly bridged the representation gap between diverse modalities, catalyzing the evolution of video multimodal interpretation, which enhances users' understanding of video content by generating correlated modalities. However, most existing video multimodal interpretation methods primarily concentrate on global comprehension with limited user interaction. To address this, we propose a novel task, Controllable Video Segmentation and Captioning (SegCaptioning), which empowers users to provide specific prompts, such as a bounding box around an object of interest, to simultaneously generate correlated masks and captions that precisely embody user intent. An innovative framework Scene Graph-guided Fine-grained SegCaptioning Transformer (SG-FSCFormer) is designed that integrates a Prompt-guided Temporal Graph Former to effectively captures and represents user intent through an adaptive prompt adaptor, ensuring that the generated content well aligns with the user's requirements. Furthermore, our model introduces a Fine-grained Mask-linguistic Decoder to collaboratively predict high-quality caption-mask pairs using a Multi-entity Contrastive loss, as well as provide fine-grained alignment between each mask and its corresponding caption tokens, thereby enhancing users' comprehension of videos. Comprehensive experiments conducted on two benchmark datasets demonstrate that SG-FSCFormer achieves remarkable performance, effectively capturing user intent and generating precise multimodal outputs tailored to user specifications. Our code is available at https://github.com/XuZhang1211/SG-FSCFormer.

Figures & tables

Fig. 1 : An example to illustrate the difference between the segmentation and captioning method and our approach, which allows users to provide a bounding box to generate multimodal outputs that are tailored to the user’s intent.

Fig. 2 : Framework of our Scene Graph-guided Fine-grained SegCaptioning Transformer (SG-FSCFormer), consisting of three components: a Visual Encoder, a Prompt-guided Temporal Graph Former (PTGFormer), and a Fine-grained Mask-Linguistic Decoder (MLDecoder), where the mathematical symbols are explained in the corresponding modules.

Fig. 3 : Prediction by the revised decoder which is able to output the position of caption words for each mask instance.

Fig. 4 : An example illustrating our data annotation.

Fig. 5 : Quantitative results by different approaches tested on the LV-VIS and OVIS datasets, where the color indicates the matching relationship between masks and words.

Model	LLM	LV-VIS			OVIS			YouCook2
Model	LLM	METEOR	SPICE	CIDEr	METEOR	SPICE	CIDEr	METEOR	SPICE	CIDEr
Vid2Seq CVPR23 [ 60 ]	T5-Base	16.3	24.7	107.4	18.0	33.0	106.8	9.3	7.9	47.1
SMOTer ECCV24 [ 51 ]	GRiT	16.9	25.0	109.6	17.8	32.8	108.1	-	-	-
MA-LMM CVPR24 [ 1 ]	Vicuna-7B	17.1	25.2	110.6	18.6	33.1	109.3	17.6	31.5	131.2
VideoGLaMM CVPR25 [ 52 ]	Phi3-Mini-3.8B	16.7	25.7	112.0	19.5	33.9	112.8	15.4	28.7	124.3
SG-FSCFormer (ours)	Vicuna-7B	19.3	26.8	112.5	21.4	35.2	113.7	20.5	33.4	139.6

TABLE I : Performance comparison with the advanced video captioning methods. where the results on the LV-VIS and OVIS are reproduced and that on the YouCook2 is cited from the references.

Model	LV-VIS			OVIS
Model	${J}\&{F}$	${J}$	${F}$	${J}\&{F}$	${J}$	${F}$
SAMURAI arXiv24 [ 61 ]	87.3	85.6	89.1	72.0	69.2	74.9
SAM2 ICLR25 [ 45 ]	85.6	83.3	87.9	68.7	66.0	71.3
SG-FSCFormer (ours)	87.8	85.9	89.7	74.6	72.5	76.7

TABLE II : Performance comparison between our approach and the two controllable video segmentation methods.

Model	Mask Decoder	LV-VIS			OVIS
Model	Mask Decoder	AP	AP 50	AP 75	AP	AP 50	AP 75
GLEE CVPR24 [ 63 ]	MaskDINO	23.9	24.6	23.3	27.1	45.4	26.3
OVFormer ECCV24 [ 36 ]	Mask2Former	24.7	31.1	26.5	21.3	38.5	20.8
OW-VISCap NIPS24 [ 21 ]	Mask2Former	-	-	-	25.4	48.8	22.8
SG-FSCFormer (ours)	Mask2Former	25.2	31.8	27.5	27.8	49.0	27.2
VideoGLaMM CVPR25 [ 52 ]	SAM2	25.4	32.2	27.6	28.0	49.3	26.9

TABLE III : Performance comparison with several advanced video multimodal interpretation methods, where † denotes the evaluation using the instance-level AP.

Spa.	Tem.	SPICE	CIDEr	${J}\&{F}$	AP
-	-	24.6	108.3	85.0	23.9
$\checkmark$	-	25.7	111.2	86.5	25.4
-	$\checkmark$	25.3	109.8	86.4	25.1
$\checkmark$	$\checkmark$	26.8	112.5	87.8	26.0

TABLE IV : Performance comparison of PTGFormer using the adaptor with different components tested on the LV-VIS validation set, where “spa.” and “tem.” represent the use of spatial and temporal correlations, respectively.

$L_{FA}$	$L_{MC}$	SPICE	CIDEr	${J}\&{F}$	AP
-	-	23.8	106.9	84.5	22.1
$\checkmark$	-	25.4	109.8	86.3	25.6
-	$\checkmark$	25.1	110.4	86.9	25.8
$\checkmark$	$\checkmark$	26.8	112.5	87.8	26.0

TABLE V : Performance comparison by using different components of losses tested on the LV-VIS Val dataset.

$\lambda$	0	1	2	5	10
SPICE	23.8	25.4	26.8	26.3	25.7
CIDEr	106.9	110.6	112.5	112.0	111.3
${J}\&{F}$	84.5	85.7	87.8	86.5	86.1
AP	22.1	25.3	26.0	25.9	25.4

TABLE VI : Performance comparison by using

\mathcal{L_{\text{MC}}}

with different weights tested on the LV-VIS Val dataset.

Fig. 6 : Visualization of several challenging and failure cases, including heavy occlusion, motion blur, visual ambiguity, and long-term target disappearance.

Method	SPICE	CIDEr	${J}\&{F}$	AP
Our (Vicuna-7B&SAM2)	26.8	112.5	87.8	26.0
Without Prompt Box
Our (w/o Prompt)	25.4	110.8	86.5	25.3
Using Different LLMs
Our (OPT-2.7B)	25.9	111.2	87.4	25.7
Using Different Mask Decoders

TABLE VII : Performance comparison of three SG-FSCFormer variants, analyzing the impact of removing the prompt box, replacing the LLM, and using different mask decoders.

Module	PTGFormer	MLDecoder
Params	10.2M	85.7M
FLOPs	1.8G	3.1G

TABLE VIII : Parameters and FLOPs of the trainable modules in SG-FSCFormer.

Method	OVFormer	SAM2	MA-LMM	SMOTer	VideoGLaMM	Ours
FPS	2.98	10.61	7.35	4.82	5.13	4.96

TABLE IX : Inference speed (FPS) comparison.

Video Captioning

Video captioning turns visual and audiovisual sequences into language that describes their events, entities, timing, and context.

See all 68 papers

X2SAM: Any Segmentation in Images and Videos
Apr 27, 2026Hao Wang, Limeng Qiao, Chi Zhang +4Image Segmentation Video Object Segmentation
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc
Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation
Mar 23, 2026Jingnan Luo, Mingqi Gao, Jun Liu +2Video Object Segmentation Referring Video Object Segmentation
Southern University of Science and Technology, Shenzhen, China. · Tencent YouTu Lab, Shenzhen, China.
Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding
Apr 13, 2026Tencent Hunyuan TeamVideo Captioning Temporal Video Understanding

Apr 27, 2026J/K move · Enter open · S save

X2SAM: Any Segmentation in Images and Videos

Hao Wang, Limeng Qiao, Chi Zhang, Lin Ma+3

Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc

Image Segmentation Video Object Segmentation Promptable Image Segmentation Multimodal Large Language Models Referring Image Segmentation Open-Vocabulary Semantic Segmentation+2

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated strong image-level visual understanding and reasoning, yet their pixel-level perception across both images and videos remains limited. Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannot natively interpret complex conversational instructions. Existing segmentation MLLMs narrow this gap, but are usually specialized for either images or videos and rarely support both textual and visual prompts in one interface. We introduce X2SAM, a unified segmentation MLLM that extends any-segmentation capabilities from images to videos. Given conversational instructions and visual prompts, X2SAM couples an LLM with a Mask Memory module that stores guided vision features for temporally consistent video mask generation. The same formulation supports generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded segmentation across image and video inputs. We further introduce the Video Visual Grounded (V-VGD) segmentation benchmark, which evaluates whether a model can segment object tracks in videos from interactive visual prompts. With a unified joint training strategy over heterogeneous image and video datasets, X2SAM delivers strong video segmentation performance, remains competitive on image segmentation benchmarks, and preserves general image and video chat ability.

Abstract

Video Captioning

Explore similar work

X2SAM: Any Segmentation in Images and Videos

Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation

Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding