Déjà Vu: Cycle-Consistent Re-ranking for Semantic Correspondence
Authors: Xiaoran Zhang, Huidong Liu, Yang Liu, Chien-Chih Wang, Jinfa Huang, Lingkai Kong, Bryan Wang, Yan Gao, Hongdong Li, Alex Wong, James S. Duncan
Organizations: Yale University · Amazon.com, Inc. · University of Rochester · Georgia Institute of Technology · Australian National University
We propose Déjà Vu, a test-time decoding module for semantic correspondence. Matching semantically equivalent keypoints across images of the same object category is challenging due to variations in appearance, viewpoint, and scale. While features from large pretrained vision models carry strong zero-shot correspondence ability, existing methods reduce final matching to greedy top-1 nearest-neighbor selection. We show that the correct correspondence appears among the top-ranked candidates far more often than top-1 matching achieves: with 30 candidates, zero-shot features exceed the top-1 accuracy of a supervised model. This reveals a decoding bottleneck rather than a limit of the representation. Motivated by this observation, we formulate semantic correspondence as a recall-then-rerank problem. Our method first nominates an adaptive candidate set, larger when the similarity distribution is ambiguous and smaller when it is confident, then re-ranks the candidates with a graded cycle-consistency score: each candidate is matched back to the source image and penalized by how far its reverse match lands from the original keypoint. The module operates entirely on the similarity matrix already computed by the base method. It consistently improves three representative baselines on SPair-71k, its geometric subset, and PF-Pascal under both unsupervised and weakly supervised settings, with gains of up to +1.8[email protected] on SPair-71k and +3.4[email protected] on PF-Pascal, and +4.9 under Keypoint Average Precision.
Figures & tables
Figure 1: Left: Top- k zero-shot recall on SPair-71k using SD+DINO features. Right: our cycle-consistent re-ranking. The gap between top- 1 accuracy and top- k recall shows that correct matches often appear among top candidates but are missed by similarity-only top- 1 matching. As k increases, performance rises sharply and even surpasses the fully supervised baseline, motivating a recall-then-rerank strategy ( Déjà Vu ). We therefore exploit the bijective structure of semantic correspondence through cycle-consistent re-ranking of candidates (e.g., Patch i,j in the right figure).
Figure 2: Method overview. Given a query source patch, we apply top- p sampling on the row of the similarity matrix to form a candidate pool (forward pass). For each candidate, we perform a reverse lookup on its column to identify the nearest source patch (backward pass). Each candidate’s final score is its forward similarity minus a penalty that grows with the distance between its reverse match and the original query.
Figure 3: Per-category radar plots on PF-Pascal. Each axis represents one object category, and values denote [email protected] ( per-keypoint ). The filled area shows our method applied to TLR (left) and CleanDIFT (right); the outline shows the respective baseline. Our method consistently expands the coverage across the majority of categories.
Figure 4: Qualitative comparison on SPair-71k. For each image pair, correspondences are shown as colored lines: green for correct matches and red for incorrect ones ([email protected]). We apply our cycle-consistent re-ranking to three base methods (TLR, CleanDIFT, DIY-SC). Across all three baselines and diverse object categories, our method consistently reduces incorrect matches.
Figure 5: Qualitative examples showing the benefit of cycle-consistent re-ranking on SPair-71k. For each pair, the left image marks the source keypoint and the right image shows three locations: ground truth (GT), the top-1 nearest-neighbor prediction, and our method’s prediction. While top-1 matching selects an incorrect target point, our cycle-consistent re-ranking successfully recovers the correct correspondence from the top- p candidate pool.
Figure 6: Sensitivity of the cycle-consistency weight λ on SPair-71k ( per-keypoint , [email protected]) for TLR (left), CleanDIFT (middle), and DIY-SC (right). Performance is stable across a broad range of λ values for all three base methods.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Additional qualitative examples showing the benefit of cycle-consistent re-ranking on SPair-71k. Each panel shows the source keypoint (left) and three predicted locations on the target (right): ground truth (GT), the top-1 nearest-neighbor prediction, and our method’s prediction. Our cycle-consistent re-ranking consistently recovers the correct correspondence that top-1 matching fails to identify.
Figure 8: Per-category radar plots on SPair-71k ( per-keypoint , [email protected]). Each axis represents one object category. The filled area shows our method applied to TLR (left), CleanDIFT (middle), and DIY-SC (right); the outline shows the respective baseline. Our method broadly expands coverage across categories for all three base methods.
Figure 9: Sensitivity of the cycle-consistency weight λ on PF-Pascal ( per-keypoint , [email protected]) for TLR (left) and CleanDIFT (right). Performance remains stable across a broad range of λ values for both base methods.
Figure 10: Qualitative comparison on SPair-71k with TLR as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (TLR+Ours) consistently reduces incorrect matches across all 18 categories.
Figure 11: Qualitative comparison on SPair-71k with CleanDIFT as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (CleanDIFT+Ours) consistently produces fewer incorrect matches across all 18 categories.
Figure 12: Qualitative comparison on SPair-71k with DIY-SC as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (DIY-SC+Ours) consistently improves correspondence quality across diverse object categories.
Figure 13: Qualitative comparison on PF-Pascal with TLR as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (TLR+Ours) yields more accurate correspondences across all 20 PF-Pascal categories.
Figure 14: Qualitative comparison on PF-Pascal with CleanDIFT as the base method. Correspondences are shown as colored lines: green for correct and red for incorrect matches ([email protected]). Our method (CleanDIFT+Ours) consistently reduces incorrect matches across all 20 PF-Pascal categories.