PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning

Qingyun Liu; Yuankun Liu; Jiaye Feng; Jie Huang; Zonghong Dai; Haochen Li; Jingkun Wang; Weiping Li

cs.CVNov 5, 2026

PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning

Authors: Qingyun Liu, Yuankun Liu, Jiaye Feng, Jie Huang, Zonghong Dai, Haochen Li, Jingkun Wang, Weiping Li,

Organizations: Peking University Beijing, China · AlignBase Beijing, China

Abstract

Panoptic Scene Graph generation (PSG) requires holistic yet fine-grained reasoning over complex relations among objects. However, current Multimodal Large Language Models (MLLMs) applied to PSG suffer from a perception bottleneck: they function as static observers limited by fixed-resolution views. As a result, models often overlook critical relations involving small or peripheral objects and struggle to differentiate ambiguous predicates in complex scenes. To address these limitations, we propose PSGAgent, an agentic framework that enhances visual perception for PSG through a think-act-observe loop. PSGAgent allows MLLMs to dynamically refine their visual perception using two tools: a Region Zoom-In Tool for local exploration of small or boundary relations and a Visual Highlight Tool for relation-focused segmentation that filters out irrelevant contextual noise. To optimize decision-making, we construct a two-stage training pipeline. First, we create a high-quality, error-aware Multimodal Chain-of-Thought (M-CoT) dataset that provides robust supervision for cold-start learning. Then, we design a novel hierarchical reward mechanism integrated with Group Sequence Policy Optimization (GSPO) to enable strategic and adaptive tool usage. Extensive experiments on the PSG benchmark show that PSGAgent achieves SOTA performance, demonstrating strong capability in both holistic and fine-grained scene understanding.

Figures & tables

Figure 1. The perception bottleneck in PSG. (a) Saliency bias : static models are biased toward dominant regions, while neglecting small-scale or off-center relations. (b) Semantic ambiguity : long-range contextual interference and fine-grained predicate confusion lead to relational uncertainty.

Figure 2. Overview of PSGAgent. We reformulate PSG as a multi-turn, tool-augmented decision-making process. (Left) Trajectory Construction. We perform error-aware data mining to identify challenging cases caused by saliency bias (relations occupying small regions or located near image boundaries) and semantic ambiguity (long-range contextual interference, high predicate confusion, or tail relations). These identified cases are used to synthesize error-aware Multimodal Chain-of-Thought (M-CoT) trajectories through a structured generation pipeline. (Right) Cold-start SFT and RL-based Enhancement. The agent is first initialized via SFT to learn basic tool usage, and then optimized with GSPO for strategic reasoning. Through a think–act–observe interaction loop guided by a hybrid reward function ( $R_{\text{outcome}}$ , $R_{\text{causal}}$ , $R_{\text{highlight}}$ , and $R_{\text{format}}$ ), the agent learns to autonomously invoke tools ( Region Zoom-In Tool and Visual Highlight Tool ) to achieve holistic and fine-grained scene understanding.

Specialized PSG Models
Model	Params	SGDet
Model	Params	AP@50 $\uparrow$	Recall $\uparrow$	mRecall $\uparrow$	Mean $\uparrow$
IMP ( Xu et al., 2017 )	–	–	18.60	7.23	12.92
MOTIFS ( Zellers et al., 2018 )	–	–	22.00	9.69	15.85
VCTree ( Tang et al., 2019 )	–	–	22.50	10.20	16.35
GPSNet ( Lin et al., 2020 )	–	–	20.10	7.69	13.90

Table 1. Comparison with state-of-the-art methods on the PSG benchmark under the SGDet setting. * denotes using predefined object classes and relation categories in prompts. Zm. T. and Hl. T. denote Region Zoom-In Tool and Visual Highlight Tool , respectively. Bold and underline results highlight the best and second-best.

Specialized PSG Models
Model	Params	PredCls
Model	Params	Recall $\uparrow$	mRecall $\uparrow$	Mean $\uparrow$
IMP ( Xu et al., 2017 )	–	38.90	11.60	25.25
GPSNet ( Lin et al., 2020 )	–	44.70	18.30	31.50
C-SCC ( Jin et al., 2023 )	–	46.50	36.40	41.45
MOTIFS ( Zellers et al., 2018 )	–	52.40	22.90	37.65

Table 2. Comparison with state-of-the-art methods on the PSG benchmark under the PredCls setting.

Specialized PSG Models
Model	Params	zs-	Open-Set (OvR)
Model	Params	Recall $\uparrow$	Recall $\uparrow$	mRecall $\uparrow$	Mean $\uparrow$
PSGFormer ( Yang et al., 2022 )	–	4.90	3.10	2.70	2.90
PSGTR ( Yang et al., 2022 )	–	6.40	3.30	2.00	2.65
SGTR ( Li et al., 2022 )	–	5.80	–	–	–
PairNet ( Wang et al., 2024b )	–	–	6.50	5.10	5.80

Table 3. Results of zs-Recall and Open-Set (OvR) on PSG under the SGDet setting.

Model	Params	Bottleneck Sets		Frequency Sets		All $\uparrow$
Model	Params	$\mathcal{S}_{\text{sal}}$ $\uparrow$	$\mathcal{S}_{\text{sem}}$ $\uparrow$	Head $\uparrow$	Tail $\uparrow$	All $\uparrow$
Qwen2.5-VL-3B-Instruct	3B	0.27	0.03	0.58	0.41	0.57
+ SFT	3B	7.91	6.58	21.90	5.93	21.67
Qwen2.5-VL-72B-Instruct*	72B	5.89	3.69	12.39	11.01	12.37
+ ZM.T. HL.T.*	72B	8.98	6.26	14.50	13.23	14.48
R1-SGG ( Chen et al., 2025 )	7B	28.53	24.82	43.72	26.85	43.48

Table 4. Recall@100 of each predicate group under the SGDet setting.

\mathcal{S}_{\text{sal}}

and

\mathcal{S}_{\text{sem}}

denote the subsets of samples suffering from saliency bias and semantic ambiguity, respectively. Head and Tail indicate splits based on predicate frequency. All denotes the overall performance.

Model	Pre-	Zm.	Hl.	Failure	PSG Metrics
Model	train	T.	T.	Rate $\downarrow$	AP@50 $\uparrow$	Recall $\uparrow$	mRecall $\uparrow$
SFT	–			0.82	30.10	21.67	10.37
SFT	–			0.22	38.49	25.26	18.70
SFT	–			0.20	38.95	25.47	19.74
SFT	–			0.41	38.26	24.79	18.51
GSPO	SFT			0.01	39.73	39.31	27.71

Table 5. Ablation of tool-use and training strategies on PSG benchmarks under the SGDet setting.

Model	Pre-	Zm.	Hl.	Failure	PSG Metrics
Model	train	Rwd.	Rwd.	Rate $\downarrow$	AP@50 $\uparrow$	Recall $\uparrow$	mRecall $\uparrow$
GSPO	SFT			0.02	34.15	43.63	35.72
GSPO	SFT			0.04	38.92	45.72	41.17
GSPO	SFT			0.03	40.07	48.49	37.83
GSPO	SFT			0.00	49.91	50.27	42.88

Table 6. Ablation of GSPO reward strategies under the SGDet setting. Hl. Rwd and Zm.Rwd denote Highlight and Zoom rewards for tool invocation.

Model	Pre-	M-CoT	Failure	PSG Metrics
Model	train	dataset	Rate $\downarrow$	AP@50 $\uparrow$	Recall $\uparrow$	mRecall $\uparrow$
SFT	–		0.42	36.98	21.51	13.67
SFT	–		0.41	38.26	24.79	18.51
GSPO	SFT		0.00	49.27	48.99	38.64
GSPO	SFT		0.00	49.91	50.27	42.88

Table 7. Ablation of the M-CoT dataset under the SGDet setting. w/o M-CoT denotes randomly sampled data used for trajectory synthesis during the initial cold-start phase.

Highlight mask source
Setting	AP@50	Recall	mRecall	Mean
No Highlight (zoom only)	46.70	48.43	39.54	43.99
OneFormer (weaker segmenter)	49.60	49.67	40.87	45.24
GT panoptic masks (upper bound)	49.12	50.44	46.03	48.24
Mask degradation based on Mask2Former
Clean Mask2Former	–	50.27	42.88	46.58

Table 8. Ablation of mask sources.

Panel of Figure 3 — Figure 3. Qualitative comparison of PSGAgent with existing methods. (a) R1-SGG (7B); (b) PSGAgent (3B, ours).

Panel of Figure 4 — Figure 4. Analysis of the M-CoT dataset. (a) Classification of error types; (b) Rounds of tool calls.

Panel of Figure 5 — Figure 5. Training dynamics of GSPO. (a) Rewards curve; (b) Tool-Call Curve.

Scene Graph Generation

Scene graph generation turns images or videos into structured graphs of objects, their attributes, and their spatial or semantic relations.

See all 48 papers

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
Jul 7, 2026Zhiwei Yang, Yuanchen Wu, Nan Zhang +3Multimodal Large Language Models Multimodal CoT Reasoning
Tencent Youtu Lab, Shanghai, China · Fudan University, Shanghai, China
A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models
Mar 20, 2025Xuanming Cui, Jaiminkumar Ashokbhai Bhoi, Chionh Wei Peng +2Scene Graph Generation Video SGG
University of Central Florida · DSO National Laboratories
Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning
May 27, 2026Xuanzhao Dong, Wenhui Zhu, Peijie Qiu +11Visual Question Answering Tool-Using Vision-Language Agents
Arizona State University, Tempe, AZ, USA · Washington University in St. Louis, St. Louis, MO, USA · Clemson University, Clemson, SC, USA +3

Jul 7, 2026J/K move · Enter open · S save

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

Zhiwei Yang, Yuanchen Wu, Nan Zhang, Yucong Meng+2

Tencent Youtu Lab, Shanghai, China · Fudan University, Shanghai, China

Multimodal Large Language Models Multimodal CoT Reasoning Visually Grounded Reasoning Graph Reasoning Fine-Grained Visual Perception RL for VLMs

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.

Abstract

Scene Graph Generation

Explore similar work

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models

Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning