Through the PRISm: Importance-Aware Scene Graphs for Semantic Image Retrieval

Dimitrios Georgoulopoulos; Nikolaos Chaidos; Angeliki Dimitriou; Giorgos Stamou

cs.CVDec 20, 2025

Through the PRISm: Importance-Aware Scene Graphs for Semantic Image Retrieval

Authors: Dimitrios Georgoulopoulos, Nikolaos Chaidos, Angeliki Dimitriou, Giorgos Stamou

Organizations: National Technical University of Athens Athens, Greece

Abstract

Web-based visual search increasingly handles queries driven by semantic intent, focusing on what is happening in an image rather than merely what it looks like. While global embeddings from vision-language models capture overall appearance, they often blur fine-grained relational structure. Conversely, scene graphs make these relationships explicit, but generating them automatically for unannotated web images produces dense, noisy graphs that retain every detected entity and relation regardless of relevance. To address this, we introduce PRISm (Pruning-based Image Retrieval via Importance Prediction on Semantic Graphs), an image-to-image retrieval framework that learns to focus on what matters in a scene. PRISm employs an Importance Prediction Module, supervised by caption-derived relevance strictly during training, to score and prune non-essential objects and <subject, relation, object> triplets. An Edge-Aware Contextual GNN then aggregates relation conditioned neighbor messages, fusing the refined graph with global visual features into a single, index-ready embedding per image. On PSG, PRISm outperforms the strongest baseline by points in MAP@1. Furthermore, on Flickr with machine-generated scene graphs and captions, PRISm remains best on top-1 metrics, while adding only 33,ms per query beyond scene graph generation.

Figures & tables

Figure 1. Example of top-2 image-to-image retrievals. PRISm (i) leverages scene graphs alongside visual features, (ii) retains important ( green ) / prunes redundant ( red ) elements, and (iii) explicitly encodes object interactions, emphasizing relational structure.

Figure 2. Importance Prediction Module inference. Extracted embeddings of image/SG, depicted objects and object/relation labels are fed to the trained module to predict scores $\hat{s}(u)$ and retain key objects and triplets.

Figure 3. Overall retrieval pipeline of PRISm. Similarity between images $I_{1}$ and $I_{2}$ is computed as the inner product of embeddings that combine projected global visual features with edge-aware multimodal scene graph representations.

	Method	PSG Dataset							Flickr Dataset
	Method	NDCG			MAP			MRR	NDCG			MAP			MRR
		@1	@3	@5	@1	@3	@5	-	@1	@3	@5	@1	@3	@5	-
Vision	DINOv3	78.27	78.35	78.56	73.81	80.41	78.98	83.00	50.84	49.56	49.22	55.10	64.32	63.19	68.33
Vision	MambaVision	75.32	75.77	76.18	68.88	76.28	75.46	79.14	50.32	50.80	50.78	56.20	64.20	60.24	68.76
VL	BLIP-2	80.66	80.86	80.90	77.47	83.78	82.25	85.81	52.34	52.17	51.87	60.23	68.92	67.57	72.57
VL	CLIP	81.28	81.49	81.59	78.96	85.12	83.64	86.90	52.47	52.59	52.40	60.59	68.60	67.79	72.59

Table 1. Comparison across methods on SG retrieval benchmarks. Left: PSG, Right: Automatically generated data with PSGTR and BLIP on Flickr dataset. Best per column in bold .

	PSG	Flickr
Graphs	47,874	12,000
Nodes	535,441	139,577
Edges	268,114	139,811
Obj. classes	133	132
Rel. types	56	48
Nodes / graph	11.18	11.63

Table 2. SG statistics: PSG vs. Flickr with generated SGs. Edges are directed; density and degree are undirected.

Figure 4. Object and relation retention rates by graph size.

Panel of Figure 5 — Figure 5. Retrieval examples from PRISm. (a) Top-3 retrieval comparisons against best-performing SotA methods (CLIP (VL), DINOv3 (Vision), and Hi-SIGIR (Scene-Graph-based)). (b) Additional PRISm retrievals illustrating detail-focused matching via the Edge-Aware GNN, as well as the effects of importance pruning and the synergy between graph semantics and visual cues.

	Setting	NDCG@1	MAP@1	MRR
With Vision	PRISm	88.54	91.62	95.09
	- Importance Pruning	88.05	90.57	94.40
	- Edge-Aware Layer	88.39	91.46	95.04
	- Global Visual Stream	83.21	82.09	88.70
Without Vision	PRISm*	74.09	64.93	76.84
	- Edge-Aware Layer	73.70	64.15	76.01

Table 3. PRISm ablations on PSG. PRISm* is PRISm without any visual information. Best per column and among variants in bold .

Figure 6. Inference time versus Performance of retrieval frameworks on Flickr.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	NDCG@1	MAP@1	MRR
PRISm	88.54	91.62	95.09
No oversampling	87.79	90.81	94.45
No weighted loss	88.02	91.25	94.74

Appendix

Table 4. Effect of oversampling and weighted loss on top-1 retrieval performance. The best result per column is denoted in bold .

Figure 8. Qualitative outputs of the Importance Prediction Module, illustrating the selective retention of semantically critical graph elements ( green ) and the pruning of less relevant background information ( red ).

Panel of Figure 9 — Figure 9. Top-3 retrieval comparisons of our proposed framework PRISm against best-performing SotA methods (CLIP (VL), DINOv3(Vision), and Hi-SIGIR (Scene-Graph-based))

Image Retrieval

Image retrieval finds and ranks images from visual, textual, or combined queries across large and specialized collections.

See all 85 papers

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
May 13, 2026Vladislav Makarov, Mark Gizetdinov, Dmitry YudinScene Graph Generation Vision-Language Models
MIRAI · MIREA · MIRAI, AXXX
Visual Commonsense Driven Knowledge Refinements for Scene Graph Generation
Jun 4, 2026Maëlic Neau, Salim Baloch, Jakob Suchan +2Scene Graph Generation Commonsense Reasoning
Computing Science Department, Umeå University, Umeå, Sweden · School of Computer Science & Engineering, Constructor University, Bremen, Germany · CoDesign Lab EU., Örebro, Sweden +1
Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matching
Aug 27, 2026Rong Shan, Tianyi Xu, Congmin Zheng +9Composed Image Retrieval Multimodal Search Agents
Shanghai Jiao Tong University · Shanghai Innovation Institute · OPPO

May 13, 2026J/K move · Enter open · S save

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Vladislav Makarov, Mark Gizetdinov, Dmitry Yudin

MIRAI · MIREA · MIRAI, AXXX

Scene Graph Generation Vision-Language Models Video SGG Scene Graphs

Open paper AskPDF arXiv

Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.

Abstract

Image Retrieval

Explore similar work

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

Visual Commonsense Driven Knowledge Refinements for Scene Graph Generation

Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matching