cs.CVNov 2, 2026

Discovering Semantic Granularity in Images through Clustering of Pretrained Visual Representations

Authors: Ying Dai

Organizations: Iwate Prefectural University

Abstract

Semantic granularity, which refers to the levels at which visual representations organize image content into meaningful entities, is fundamental to image understanding. However, whether pretrained visual representations inherently encode semantic structures at multiple granularities remains poorly understood. This paper presents a representation-driven approach for discovering semantically meaningful image regions at different granularities through clustering of pretrained visual representations without task-specific training. Specifically, we concatenate feature maps from appropriately deep layers of a pretrained visual model and apply a singular value decomposition (SVD)-based projection to derive a pixel-wise principal representation map (PRM) tensor. The resulting PRM tensor captures latent semantic structures embedded in pretrained visual representations, making them accessible for subsequent semantic clustering. We then introduce the semantic continuity ratio (SCR) and analyze the signal-to-noise ratio (SNR) of SCR profiles to assess the suitability of PRM tensors for semantic granularity discovery. Our results indicate that PRM tensors exhibiting a higher proportion of relatively high SNR values over an initial range of cluster numbers tend to yield more semantically meaningful regions across multiple granularities. Furthermore, once a suitable PRM tensor is identified through SNR-profile analysis, pixel-wise hierarchical clustering enables the level of semantic granularity to be controlled by varying the number of clusters. We evaluate the proposed approach on the COCO-2017 and VOC-2012 validation datasets using Hungarian-matched intersection over union (HIoU). The results suggest competitive segmentation performance among the compared methods, supporting the feasibility of the proposed approach.

Explore similar work

CardsList
  1. Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM

    Mar 9, 2026Junyuan Mao, Qiankun Li, Linghao Meng +5Large Vision-Language ModelsFine-Grained Visual Perception

  2. From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding

    Mar 10, 2026Wenzhao Xiang, Yue Wu, Hongyang Yu +3Self-Supervised Visual Representation LearningMasked Autoencoders

  3. Region-Aware CLS Token Augmentation for Fine-Grained Image Retrieval

    Oct 7, 2026Ian de Holanda Cavalcanti Bezerra, Vivek Trivedy, Lucas Pascotti Valem +1Fine-Grained Image RetrievalImage Retrieval