ReCoCo: From Patches to Compositions for Visual Representation Learning
Authors: Juan Gutiérrez, Saman Zadtootaghaj, José Luis Blanco-Murillo, Sara Atito
Organizations: Information Processing and Telecommunications Center (IPTC), Universidad Politécnica de Madrid, ES · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK · Sony Interactive Entertainment, DE · Surrey Institute for People-Centred Artificial Intelligence (PAI), University of Surrey, UK
Self-supervised learning has produced powerful visual representations, yet learning strong dense features remains challenging. While consistency between augmented views provides strong image-level supervision, extending this principle to local visual content is less straightforward. Existing approaches often match individual patches across views or identify coherent regions, introducing uncertainties in correspondence estimation and assumptions about how visual content should be grouped. We introduce ReCoCo (Regional Composition Consistency), a principled approach that extends cross-view consistency to the regional level without requiring such correspondences or semantic grouping. ReCoCo represents the content shared between augmented views through compositions of their patch-level predictions over learned visual prototypes. By aligning these compositions, the model learns from regions containing multiple visual concepts without requiring them to be separated beforehand. ReCoCo integrates naturally with teacher-student self-distillation and introduces no additional learned parameters. Extensive experiments demonstrate substantial improvements in frozen-feature segmentation, clustering, and regional retrieval while maintaining competitive image-level recognition, alleviating the trade-off between dense and global representation quality.
Figures & tables
Figure 2: Overview of ReCoCo’s regional composition consistency. Two augmented views, v1 and v2 , of an image x share an overlapping region o . The student and its EMA teacher process the full views and predict distributions over learned prototypes for individual patches. Only patches fully contained within the overlap are selected, and their predictions are independently averaged to obtain the student and teacher regional compositions, hv1,os and hv2,ot . The regional consistency loss aligns the student composition with the stop-gradient teacher composition, without requiring patch-wise correspondences. Both networks comprise an encoder and a projection head; the projection heads and student masking are omitted for visual simplicity.
Objects (COCO)
Regions (VOC)
Method
mAP
P@50
mAP
R@1
iBOT [ 6 ]
20.9
32.1
49.3
82.3
ReCoCo
38.4
48.4
63.0
86.0
Table 5: Object and region retrieval. mAP, Precision@50 (P@50) and Recall@1 (R@1) (%) of COCO objects and VOC regions retrieved with mean-pooled frozen ViT-S/16 features.
Figure 3: Feature analysis with frozen ViT-S/16 features. (a) Variance of the patch features of VOC validation objects (mean squared cosine distance to the mean) with standard deviation. (b) SPair-71k [email protected] and depth RMSE.
Figure 4
Figure 5: Visualisations of iBOT and ReCoCo. (a) Each patch coloured by its most probable prototype. (b) First three principal components of the patch features as RGB. (c) Cosine similarity to a query patch (red cross).
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.
Self-supervised learning (SSL) has become an effective approach for learning visual representations without manual annotations. Among SSL approaches, contrastive learning has been widely used for visual representation learning. However, existing contrastive SSL methods have focused primarily on image-level or pixel-level representation learning, while region-level representation learning remains less explored. We propose SPARC, a region-level contrastive learning framework that leverages superpixels to establish explicit correspondence between augmented image views. SPARC introduces a region contrastive branch that performs superpixel-based feature pooling and optimizes a region-level contrastive objective jointly with a global image-level objective. Under identical settings, SPARC consistently outperforms previous methods such as MoCo-v2 and DenseCL, achieving improvements of up to +9.79 mIoU for semantic segmentation and +4.88 AP for object detection. Ablation studies further demonstrate that region-level objectives produce the strongest performance. Thus, region-level contrastive learning is an effective approach for improving self-supervised visual pretraining for dense prediction tasks. Code repository can be accessed at https://github.com/xRIPEIx/SPARC.
David Szczecina, Yuanpei Xiang, Jitao Hu +4
Systems Design Engineering University of Waterloo Waterloo, Ontario, Canada
Reliable correspondence estimation is a fundamental problem in image processing, underpinning applications such as Structure from Motion, visual localization, and image registration. Existing learning-based methods have significantly improved local feature representations, yet most still operate at the pixel or patch level and lack explicit modeling of regions that are jointly visible across views. We propose SAMatcher, a feature matching framework that formulates correspondence estimation through co-visibility modeling. Instead of directly matching local features, SAMatcher first predicts co-visible region masks and bounding boxes as structured priors for correspondence estimation. Built upon the Segment Anything Model (SAM), it introduces a symmetric cross-view interaction mechanism that enables bidirectional feature exchange and cross-view semantic alignment. We further develop a unified supervision scheme that jointly optimizes mask prediction and box localization through mask learning, box regression, and mask-box consistency constraints. Extensive experiments on challenging benchmarks demonstrate substantial improvements over existing matching pipelines, particularly under large viewpoint and scale variations. Our results show that foundation models originally designed for monocular segmentation can be effectively extended to multi-view correspondence reasoning through explicit co-visibility modeling, offering a new perspective on structured representation learning for image matching. Code and project page: https://xupan.top/Projects/samatcher
Xu Pan, Qiyuan Ma, Mingyue Dong +3
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, P. R. China · National Key Laboratory of Space Target Awareness, Space Engineering University, Beijing, P. R. China
Self-supervised learning (SSL) at scale produces powerful visual representations. However, most scalable SSL methods rely on ad hoc combinations of multiple objectives and stabilization mechanisms. Taking a step back, we ask if we can design a high-performing, yet principled SSL algorithm. Starting from the multi-view assumption, stipulating that task-relevant content is captured by the information common to different views, we construct an information-theoretic objective decomposing into interpretable terms. This derivation yields JEM, a student-teacher method that learns by aligning corresponding patch representations across views, explicitly regularized by information and structure preservation losses. JEM trains stably from 300M to 7B parameters, and, to our knowledge, is the first latent-space patch-level method demonstrated at 7B scale. Across all scales, JEM reaches strong performance on both global and dense probing tasks, on segmentation benchmarks consistently surpassing the DINOv2 algorithm, an influential foundation for today's strongest visual SSL methods. Notably, at 7B parameters, it exceeds the performance of DINOv3 on panoptic segmentation, despite being trained on 12× less data without refinement stages. These results demonstrate that we can indeed design an SSL algorithm that learns strong representations, is principled and stable.
Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký +5