Estimating semantic correspondences between different object instances of similar categories in real-world images is a fundamental challenge in computer vision. While the recent progress in foundation models has significantly advanced solutions for such difficult image correspondence problems, respective methods are often insufficiently regularised. For example, common nearest-neighbour matching with foundation model features often results in noisy correspondences, which is particularly prominent when solving for dense correspondences. In this work, we tackle image matching via functional maps, which have been popularised for 3D shape matching due to their powerful spectral formalism utilising an efficient low-dimensional linear representation. However, functional maps require that domains (in our cases the rectangular image grids) are equipped with an informative and non-trivial geometry. To define a meaningful manifold structure for images, we use Galerkin's method from finite elements to derive a discrete Laplace-Beltrami operator (LBO) based on a manifold embedding of deep image features. Further, we leverage a neural adjoint map that allows to represent non-linear mappings. We demonstrate that our method sets the new state of the art among zero-shot image correspondence methods on multiple dense correspondence benchmarks of real-world images. Our code is publicly available: https://github.com/NafieAmrani/NAMI
Figures & tables
Figure 1: We propose a novel zero-shot method for finding correspondences between different object instances from similar categories. In each panel, the left image is the source and the right is the target. Our method is flexible and solves (a) dense image matching, (b) sparse image matching, and due to the zero-shot nature it is directly applicable to other domains, such as (c) medical images.
Figure 2: Overview of our method. Two sets of features are extracted for each input image. One set is used to derive our LBO via a Manifold Embedding used in the spectral branch of our approach (top and bottom). The other set is used in the point-wise matching branch leveraging nearest-neighbour search (middle). Both branches are coupled via the alignment loss (green box, see Eqn. ( 9 )).
Figure 3: Dense correspondences on two datasets. We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our method achieves smoother and more consistent mappings than [ 16 ] .
Setting
Method
FG3DCar
JODS
Pascal
Avg.
Wtd. Avg.
Time [s]
Supervised
SCOT [ 93 ]
95.3
81.3
57.7
78.1
80.8
-
CATs* [ 50 ]
92.1
78.9
64.2
78.4
80.7
-
PWarpC-CATs* [ 94 ]
95.5
85.0
85.5
88.6
90.2
-
Unsupervised
CNNGeo [ 59 ]
90.1
76.4
56.3
74.2
76.8
-
(task-specific)
PARN [ 95 ]
89.5
75.9
71.2
78.8
81.0
-
GLU-Net [ 53 ]
93.2
73.3
71.1
79.2
82.3
-
Table 1: Quantitative results ([email protected]↑ ) on the TSS dataset [ 85 ] . Methods are grouped into supervised, unsupervised and zero-shot. Our approach substantially outperforms all zero-shot baselines (bold face) and is on par with unsupervised task-specific methods. NN stands for nearest-neighbour matching. We report both the unweighted (Avg.) and weighted (Wtd. Avg.) averages over the three data subsets following [ 8 ] and [ 16 ] , respectively. *: backbone fine-tuned on TSS.
Table 2: Sparse keypoint correspondences on SPair-71k [ 89 ] . While our method is on par with the baseline performance at [email protected], it yields superior results for both [email protected] and MSE.
Figure 4: Visualisation of sparse correspondence on Spair-71k [ 89 ] . Correct matches are indicated by blue lines, while incorrect matches are shown in red. We compare our approach against FUSE [ 8 ] , as it represents the strongest baseline in terms of [email protected]. Thanks to its global geometric consistency, our framework produces more robust mappings across challenging examples.
Figure 5: Ablation of our approach without the proposed LBO or NAM on the TSS dataset. LBO + SmoothFM : we replace the NAM with a FM-based variant SmoothFM [ 97 ] . GL + NAM : we replace the LBO with the image graph Laplacian in [ 16 ] . GL + SmoothFM : we combine both alternatives. Our proposed combination of NAM and our LBO achieves the best results overall (red).
Table A3: Impact of spectral resolution ( k ) on performance. We compare our approach against Cheng et al. [ 16 ] across varying numbers of eigenfunctions k∈{50,100,200} . Our method consistently outperforms [ 16 ] at all resolutions, including their optimal setting of k=200 . Our method is roughly two orders of magnitude faster than [ 16 ] at all resolutions. † : Results as reported in the original paper. *: Results reproduced using the official implementation.
Figure A6: Visualization of the eigenfunctions obtained from the graph Laplacian used in [ 16 ] and from our proposed image-based LBO. Our LBO produces smoother low-frequency eigenfunctions, whereas the graph Laplacian retains noticeable high-frequency image details even in the lower modes.
Table A4: Effect of Neural ZoomOut (NZO) refinement [ 14 ] . Without NZO, our method uses a spectral embedding of size 50 . With NZO, we grow the embedding from 50 to 250 in increments of 40 . NZO offers an optional way to improve accuracy across all metrics at the cost of a roughly 10× increase in runtime. In all experiments in our paper, we report results without NZO to characterise the performance of our method on its own.
Figure A7: Laplacian ablation on TSS [ 85 ] . We replace our LBO with the cotangent Laplacian, the robust Laplacian [ 100 ] , and a graph Laplacian [ 16 ] , holding the rest of the pipeline fixed. Our operator achieves the highest PCK and lowest EPE, peaking at k=50 , while the graph Laplacian degrades sharply for small K and the cotangent and robust behave almost identically.
Table A5: Component ablation for sparse correspondence on the Spair-71k dataset [ 89 ] . We vary the operator (graph Laplacian [ 16 ] vs. our LBO) and the map estimator (SmoothFM [ 97 ] vs. NAM [ 14 ] ). Both our LBO and NAM improve all three metrics in isolation, and their combination (shaded) is best overall.
Table A6: Effect of the refinement network used in Cheng et al. [ 16 ] . Adding the refinement network degrades all four metrics.
Figure A8: Dense correspondences on the Faust-2D dataset. We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our method achieves smoother and more consistent mappings than [ 16 ] .
Table A7: Quantitative comparison on Faust-2D . Best results are shown in bold , second best are underlined . Our method outperforms all previous baselines on the Faust-2D dataset.
Table A8: Per-difficulty results on Faust-2D , stratified by the relative rotation between the two shapes in each pair. Best results per column are shown in bold , second best are underlined . Our method achieves the best accuracy on the easy and medium splits, whereas all methods degrade substantially on the hard split.
Figure A9: Ablation of our approach on Faust-2D , removing either the proposed LBO or the NAM. LBO + SmoothFM : we replace the NAM with the functional-map-based SmoothFM [ 97 ] . GL + NAM : we replace our LBO with the image graph Laplacian of [ 16 ] . GL + SmoothFM : we combine both alternatives. Our combination of the NAM and our LBO performs best on [email protected] and end-point error, while SmoothFM yields smoother matchings, as expected given that it is explicitly designed to promote smoothness.
Figure A10: Additional examples of dense correspondences on the NIH Chest X-ray dataset [ 104 ] . We visualise mappings between X-ray images of the same patient captured at different time points. Our method can be applied to medical images, outperforming nearest-neighbour search using backbone image features.
Table A9: Robustness to initialisation. Per-seed metrics on the TSS dataset [ 85 ] , with mean and standard deviation across ten seeds. The low standard deviation shows that our method is robust to the random initialisation of the NAM weights.
Table A10: Ablation of backbone architectures for LBO extraction and NAM training. We evaluate the impact of various feature extractors on both the spectral basis computation and Neural Adjoint Map (NAM) optimisation. While the SD (v3.5) + DINOv2-ViT-L/14 combination achieves the best performance, we adopt the SD (v1.5) + DINOv2-ViT-B/14 configuration as it provides an optimal balance between competitive performance and computational efficiency while enabling direct comparison with SD (v1.5)-based baselines. Best and second-best results are highlighted in bold and underlined , respectively; the default configuration of our method is coloured in gray .
Table A11: Experiment on sparse keypoint correspondences on SPair-71k [ 89 ] . While our method is on par with the baseline performance at [email protected], it yields superior results for both [email protected] and MSE error.
Figure A11: Dense correspondences on the TSS dataset [ 85 ] . We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our framework achieves smoother and more consistent mappings than Cheng et al. [ 16 ] .
Figure A12: Dense correspondences on the SPair-71k dataset [ 89 ] . We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our framework achieves smoother and more consistent mappings than Cheng et al. [ 16 ] .
Figure A13: Visualisation of sparse correspondences on the Spair-71k dataset [ 89 ] . Correct matches are indicated by blue lines, while incorrect matches are shown in red. We compare our approach with FUSE [ 8 ] , and present failure cases of our method in the last two rows.
Figure A14: Visualisation of texture transfer on the TSS [ 85 ] . We transfer the texture from the source image to the target image (third and fourth column). While texture transfer is not the primary focus of our method, our approach maintains superior detail fidelity and consistency compared to Cheng et al. [ 16 ] .
Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.
Jiin Im, Sisung Liu, Je Hyeong Hong
Dept. Electronic Engineering, Hanyang University · Dept. Artificial Intelligence, Hanyang University
Dense correspondence matching has historically been bounded by simplifying spatio-temporal priors, such as smooth motion and rigid geometry. While effective for classical tasks, these assumptions break down in image editing and reference-guided generation (IEG), where transformations can preserve visual identity while breaking physical continuity. To establish identity-preserving correspondence across such transformations, we introduce FreeMatching, a generalizable framework combining generative and semantic foundation representations with heterogeneous supervision from classical datasets, tracked videos, and synthetic scenes. Teacher-guided iterative refinement further improves correspondence in IEG without dense correspondence annotations. Experimentally, a single FreeMatching model substantially improves correspondence quality on challenging IEG image pairs while retaining competitive performance on classical benchmarks. Furthermore, we demonstrate its utility as a quantitative metric for evaluating identity preservation, with scores that correlate with human judgment. The code is available at https://github.com/luping-liu/FreeMatching.
Luping Liu, Bingyi Kang, Yifan Wang +1
The University of Hong Kong · ByteDance Seed · Zhejiang University
Semi-dense local feature matching commonly aggregates contextual information before coarse-to-fine correspondence estimation. However, intra-image and inter-image aggregation serve different roles: the former should propagate spatial evidence within each image while retaining structured two-dimensional dependencies, whereas the latter should gather information from plausible corresponding regions while limiting noise from unrelated locations. We propose WovenAnchor Matcher (WAM), a semi-dense matching framework based on role-specialized context aggregation. WAM introduces two complementary operators. WovenMamba performs intra-image aggregation through horizontal-then-vertical state-space propagation, allowing vertical updates to operate on horizontally contextualized features and thereby encouraging a structured two-dimensional receptive-field bias. AnchorCrossAttention performs inter-image aggregation by using cross-attention to gather context from plausible corresponding regions across the image pair. It performs this exchange over local representative anchors that aggregate feature evidence within each window. On MegaDepth, WAM achieves 66.1 pose AUC@5∘, improving on the JamMa baseline by 2.0 AUC points under the same evaluation setup. Ablations show that removing or replacing either specialized component reduces accuracy, providing evidence that intra-image propagation and inter-image retrieval benefit from role-specific aggregation designs. Code and pretrained weights are available at https://github.com/mmaiLab/WovenAnchorMatcher.
Zhiyang Li, Ruijiang Jin, Thibaut Klenke +2
Institute of Science Tokyo, Tokyo, Japan · Denso IT Laboratory, Tokyo, Japan