Visual Representation Learning

Overview

Visual representation learning studies how to turn images and video into features that preserve information useful for tasks such as classification, segmentation, retrieval, and robotic control. Common approaches include contrastive and masked-image learning, predictive methods such as joint-embedding models, generative objectives, and supervision from language, geometry, or action. Vision foundation models are increasingly evaluated beyond ImageNet classification, on dense prediction, robustness to visual shifts, compositional reasoning, and real-world manipulation. A central tension is how to make features broadly transferable and efficient while retaining fine spatial detail, task-relevant information, and reliability under distribution changes.

Latest papers

245 papers26 papers in the last four weeks, down 24% on the four weeks before. 0.2% of all new papers.

Sep 19, 2025

eVGGT: An Efficient Geometry-Aware Vision Encoder for Visuomotor Policies

Geometry-grounded vision models, such as VGGT, have emerged as robust visual encoders, providing essential geometric priors for robotic manipulation. However, the high computational cost of these models often leads to slow inference, limiting their practical applications in real-world robotics. This paper introduces eVGGT, a lightweight geometry-aware vision encoder distilled from the high-performing VGGT. Our findings demonstrate two primary advantages: i) integrating eVGGT into imitation learning frameworks (including ACT and Diffusion Policy) yields up to a 6.3% improvement in success rate over standard 2D encoders across bimanual and single-arm tasks in both simulation and real-world settings with variable viewpoints; ii) eVGGT achieves a nearly 5 times speedup and a 63% reduction in memory usage compared to state-of-the-art geometry-aware encoders while maintaining comparable task performance. These results suggest that eVGGT substantially alleviates the performance-latency bottleneck that has limited geometry-aware visuomotor policies in real-world deployment.
Aug 27, 2025

Disentangling Latent Embeddings with Sparse Linear Concept Cones (SLiCC)

Pretrained vision models encode semantic information in latent embedding spaces that are useful for downstream tasks. We hypothesize that this information can be disentangled to separate the content of complex scenes by decomposing each embedding into multiple concept-specific components. We propose Sparse Linear Concept Cones (SLiCC), a supervised dictionary learning approach that estimates a linear synthesis model consisting of sparse, non-negative combinations of groups of dictionary atoms, whose group-wise activity is guided by multi-label information. Each concept-specific component is a non-negative combination of atoms associated with a label and lies in a corresponding concept cone. We introduce an alternating optimization algorithm and an autoencoder formulation of SLiCC. We apply SLiCC to three types of CLIP and DINOv2 embeddings across five datasets. For CLIP, we additionally exploit image-text alignment to enable a self-supervised SLiCC. We also introduce a fully unsupervised autoencoder that enforces group sparsity based on the norm of neuron activations. Quantitative results show that supervised SLiCC improves concept-filtered image retrieval over unfiltered embeddings, including retrieval of finer-grained information not provided during training. Beyond retrieval of individual concepts, the disentangled components enable concept deletion and recombination across images. We further explore conditional image generation using Image-to-Prompt, with qualitative results illustrating retrieval and generation from manipulated concept representations.
Jun 30, 2025

GViT: Representing Images as Gaussians for Visual Recognition

We introduce GViT, a supervised classification framework that uses patch-conditioned 2D Gaussian primitives as input tokens for ViT backbones. A ViT-S-sized encoder predicts each primitive's position, scale, orientation, color, and opacity in one forward pass. The same parameters are constrained to reconstruct the image through a differentiable renderer, as well as serve as classifier inputs. During training, classifier-gradient guidance shapes primitive geometry and appearance under an image reconstruction constraint. On ImageNet-1K, GViT-B achieves 76.9% top-1 accuracy, compared with 79.8% for ViT-B/16 trained with the same recipe. Gaussian tokens also provide explicit spatial support for explanation maps. Grad-CAM over Gaussians improves on patch-based Grad-CAM in segmentation and token-deletion probes, while other probe comparisons are comparable to a ViT. A covariance-based foreground diagnostic on the PASCAL VOC segmentation task further measures the geometry learned through supervised guidance. These results establish the feasibility of recognition from renderable Gaussian parameters and characterize how task supervision shapes this inspectable representation.
Jun 7, 2025

Employing Discrete Fourier Transform in Representation Learning

Image representation learning via input reconstruction is a common technique in machine learning for generating representations that can be effectively utilized by arbitrary downstream tasks. A well-established approach uses autoencoders to extract latent representations at the network's compression point. These representations are valuable because they retain essential information necessary for reconstructing the original input from the compressed latent space. In this paper, we propose an alternative learning objective. Instead of using the raw input as the reconstruction target, we employ the Discrete Fourier Transform (DFT) of the input. The DFT provides meaningful global information at each frequency level, making individual frequency components useful as separate learning targets. When dealing with multidimensional input data, the DFT offers remarkable flexibility by enabling selective transformation across specific dimensions while preserving others in the computation. Moreover, certain types of input exhibit distinct patterns in their frequency distributions, where specific frequency components consistently contain most of the magnitude, allowing us to focus on a subset of frequencies rather than the entire spectrum. These characteristics position the DFT as a viable learning objective for representation learning. We validate our approach on the CIFAR-10 and STL-10 datasets under constrained training regimes. We discovered that integrating data augmentation during pretraining enhances the learned representations. Furthermore, we demonstrate that training exclusively on the lower-frequency components, those containing the highest magnitudes, produces results comparable to using the full frequency spectrum with only minimal reductions in accuracy.
Feb 20, 2025

Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching

Most self-supervised learning objectives defend against collapse but leave the target representation law unspecified. We formulate representation learning as Distribution Matching (DM), learning an augmentation-invariant encoder whose push-forward law matches an explicit reference law. The reference law specifies what the learned representation distribution should look like, while the matching mechanism determines how this target is matched. Within DM, generative learning and representation learning can be viewed as directionally inverse distributional processes: generative learning maps a tractable reference to data, whereas representation learning maps data to a designed reference law. For the Mallows-distance instantiation studied here, we connect the population objective to class-centre separation and classification error and derive a finite-sample bound on downstream classification error. Simulations and image benchmarks show manifold rectification, fine-grained structure and transferability.