Déjà Vu: Cycle-Consistent Re-ranking for Semantic Correspondence
Authors: Xiaoran Zhang, Huidong Liu, Yang Liu, Chien-Chih Wang, Jinfa Huang, Lingkai Kong, Bryan Wang, Yan Gao, Hongdong Li, Alex Wong, James S. Duncan
Organizations: Yale University · Amazon.com, Inc. · University of Rochester · Georgia Institute of Technology · Australian National University
We propose Déjà Vu, a test-time decoding module for semantic correspondence. Matching semantically equivalent keypoints across images of the same object category is challenging due to variations in appearance, viewpoint, and scale. While features from large pretrained vision models carry strong zero-shot correspondence ability, existing methods reduce final matching to greedy top-1 nearest-neighbor selection. We show that the correct correspondence appears among the top-ranked candidates far more often than top-1 matching achieves: with 30 candidates, zero-shot features exceed the top-1 accuracy of a supervised model. This reveals a decoding bottleneck rather than a limit of the representation. Motivated by this observation, we formulate semantic correspondence as a recall-then-rerank problem. Our method first nominates an adaptive candidate set, larger when the similarity distribution is ambiguous and smaller when it is confident, then re-ranks the candidates with a graded cycle-consistency score: each candidate is matched back to the source image and penalized by how far its reverse match lands from the original keypoint. The module operates entirely on the similarity matrix already computed by the base method. It consistently improves three representative baselines on SPair-71k, its geometric subset, and PF-Pascal under both unsupervised and weakly supervised settings, with gains of up to +1.8[email protected] on SPair-71k and +3.4[email protected] on PF-Pascal, and +4.9 under Keypoint Average Precision.
Figures & tables
Figure 1: Left: Top- k zero-shot recall on SPair-71k using SD+DINO features. Right: our cycle-consistent re-ranking. The gap between top- 1 accuracy and top- k recall shows that correct matches often appear among top candidates but are missed by similarity-only top- 1 matching. As k increases, performance rises sharply and even surpasses the fully supervised baseline, motivating a recall-then-rerank strategy ( Déjà Vu ). We therefore exploit the bijective structure of semantic correspondence through cycle-consistent re-ranking of candidates (e.g., Patch i,j in the right figure).
Figure 2: Method overview. Given a query source patch, we apply top- p sampling on the row of the similarity matrix to form a candidate pool (forward pass). For each candidate, we perform a reverse lookup on its column to identify the nearest source patch (backward pass). Each candidate’s final score is its forward similarity minus a penalty that grows with the distance between its reverse match and the original query.
Figure 3: Per-category radar plots on PF-Pascal. Each axis represents one object category, and values denote [email protected] ( per-keypoint ). The filled area shows our method applied to TLR (left) and CleanDIFT (right); the outline shows the respective baseline. Our method consistently expands the coverage across the majority of categories.
Figure 4: Qualitative comparison on SPair-71k. For each image pair, correspondences are shown as colored lines: green for correct matches and red for incorrect ones ([email protected]). We apply our cycle-consistent re-ranking to three base methods (TLR, CleanDIFT, DIY-SC). Across all three baselines and diverse object categories, our method consistently reduces incorrect matches.
Figure 5: Qualitative examples showing the benefit of cycle-consistent re-ranking on SPair-71k. For each pair, the left image marks the source keypoint and the right image shows three locations: ground truth (GT), the top-1 nearest-neighbor prediction, and our method’s prediction. While top-1 matching selects an incorrect target point, our cycle-consistent re-ranking successfully recovers the correct correspondence from the top- p candidate pool.
Figure 6: Sensitivity of the cycle-consistency weight λ on SPair-71k ( per-keypoint , [email protected]) for TLR (left), CleanDIFT (middle), and DIY-SC (right). Performance is stable across a broad range of λ values for all three base methods.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Additional qualitative examples showing the benefit of cycle-consistent re-ranking on SPair-71k. Each panel shows the source keypoint (left) and three predicted locations on the target (right): ground truth (GT), the top-1 nearest-neighbor prediction, and our method’s prediction. Our cycle-consistent re-ranking consistently recovers the correct correspondence that top-1 matching fails to identify.
Figure 8: Per-category radar plots on SPair-71k ( per-keypoint , [email protected]). Each axis represents one object category. The filled area shows our method applied to TLR (left), CleanDIFT (middle), and DIY-SC (right); the outline shows the respective baseline. Our method broadly expands coverage across categories for all three base methods.
Figure 9: Sensitivity of the cycle-consistency weight λ on PF-Pascal ( per-keypoint , [email protected]) for TLR (left) and CleanDIFT (right). Performance remains stable across a broad range of λ values for both base methods.
Figure 10: Qualitative comparison on SPair-71k with TLR as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (TLR+Ours) consistently reduces incorrect matches across all 18 categories.
Figure 11: Qualitative comparison on SPair-71k with CleanDIFT as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (CleanDIFT+Ours) consistently produces fewer incorrect matches across all 18 categories.
Figure 12: Qualitative comparison on SPair-71k with DIY-SC as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (DIY-SC+Ours) consistently improves correspondence quality across diverse object categories.
Figure 13: Qualitative comparison on PF-Pascal with TLR as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (TLR+Ours) yields more accurate correspondences across all 20 PF-Pascal categories.
Figure 14: Qualitative comparison on PF-Pascal with CleanDIFT as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (CleanDIFT+Ours) consistently reduces incorrect matches across all 20 PF-Pascal categories.
The strong performance that modern semantic correspondence methods achieve at standard thresholds plateaus sharply at fine-grained thresholds. We argue that this plateau stems not from the representational capacity of backbone features, but from a grid-tied readout. Patch-based vision transformers tokenize images onto discrete grids, introducing two forms of quantization error: querying nearest patch features instead of exact keypoints on the source side, and the absence of grid features representing precise ground-truth locations on the target side. We quantify this quantization ceiling across all 499,188 keypoints in SPair-71k: under the standard 448x448, patch-14 setting, 84.9% of ground-truth keypoints have no grid feature representing their precise location at [email protected]. This is a structural limitation at the representation level, independent of the matching strategy. We address this with ImCorr: Sub-pixel Semantic Correspondence via Implicit Feature Decoding, which formulates correspondence estimation over a continuous feature field queryable at arbitrary continuous coordinates. A FiLM-conditioned decoder is trained to embed sub-pixel positional information into the feature field. Querying the field directly at exact keypoint coordinates theoretically eliminates representation-level quantization error on the source side, while decoding onto a grid denser than the backbone grid substantially reduces quantization error on the target side. On SPair-71k and AP-10K (intra-species, cross-species, and cross-family), ImCorr improves performance at fine-grained thresholds ([email protected]), achieving a 6.2 percentage point gain over the prior state of the art at [email protected] on SPair-71k. These results demonstrate that representational continuity is an effective solution for precise semantic correspondence. Code is available at https://github.com/YusungChoi/ImCorr.
Yusung Choi
Pukyong National University, Busan, Republic of Korea
Recent advances in semantic correspondence rely on dual-encoder architectures, combining DINOv2 with diffusion backbones. While accurate, these billion-parameter models generalize poorly beyond training keypoints, revealing a gap between benchmark performance and real-world usability, where queried points rarely match those seen during training. Building upon DINOv2, we introduce MARCO, a unified model for generalizable correspondence driven by a novel training framework that enhances both fine-grained localization and semantic generalization. By coupling a coarse-to-fine objective that refines spatial precision with a self-distillation framework, which expands sparse supervision beyond annotated regions, our approach transforms a handful of keypoints into dense, semantically coherent correspondences. MARCO sets a new state of the art on SPair-71k, AP-10K, and PF-PASCAL, with gains that amplify at fine-grained localization thresholds (+8.9 [email protected]), strongest generalization to unseen keypoints (+5.1, SPair-U) and categories (+4.7, MP-100), while remaining 3x smaller and 10x faster than diffusion-based approaches. Code is available at https://github.com/visinf/MARCO .
Claudia Cuttano, Gabriele Trivigno, Carlo Masone +1
Establishing semantic correspondence is a challenging task in computer vision, aiming to match keypoints with the same semantic information across different images. Benefiting from the rapid development of deep learning, remarkable progress has been made over the past decade. However, a comprehensive review and analysis of this task remains absent. In this paper, we present the first extensive survey of semantic correspondence methods. We first propose a taxonomy to classify existing methods based on the type of their method designs. These methods are then categorized accordingly, and we provide a detailed analysis of each approach. Furthermore, we aggregate and summarize the results of methods in literature across various benchmarks into a unified comparative table, with detailed configurations to highlight performance variations. Additionally, to provide a detailed understanding on existing methods for semantic matching, we thoroughly conduct controlled experiments to analyse the effectiveness of the components of different methods. Finally, we propose a simple yet effective baseline that achieves state-of-the-art performance on multiple benchmarks, providing a solid foundation for future research in this field. We hope this survey serves as a comprehensive reference and consolidated baseline for future development. Code is publicly available at: https://github.com/Visual-AI/Semantic-Correspondence.
Kaiyan Zhang, Xinghui Li, Jingyi Lu +1
University of Hong Kong, Hong Kong · Department of Engineering, University of Oxford, OX1 3PJ Oxford, U.K.