Manifold Embedding of Deep Image Features for Image Matching via Neural Adjoint Maps

Nafie El Amrani; Moritz Heep; Florian Bernard

cs.CVNov 3, 2026

Manifold Embedding of Deep Image Features for Image Matching via Neural Adjoint Maps

Authors: Nafie El Amrani, Moritz Heep, Florian Bernard

Organizations: University of Bonn Lamarr Institute

Abstract

Estimating semantic correspondences between different object instances of similar categories in real-world images is a fundamental challenge in computer vision. While the recent progress in foundation models has significantly advanced solutions for such difficult image correspondence problems, respective methods are often insufficiently regularised. For example, common nearest-neighbour matching with foundation model features often results in noisy correspondences, which is particularly prominent when solving for dense correspondences. In this work, we tackle image matching via functional maps, which have been popularised for 3D shape matching due to their powerful spectral formalism utilising an efficient low-dimensional linear representation. However, functional maps require that domains (in our cases the rectangular image grids) are equipped with an informative and non-trivial geometry. To define a meaningful manifold structure for images, we use Galerkin's method from finite elements to derive a discrete Laplace-Beltrami operator (LBO) based on a manifold embedding of deep image features. Further, we leverage a neural adjoint map that allows to represent non-linear mappings. We demonstrate that our method sets the new state of the art among zero-shot image correspondence methods on multiple dense correspondence benchmarks of real-world images. Our code is publicly available: https://github.com/NafieAmrani/NAMI

Figures & tables

Panel of Figure 1 — Figure 1: We propose a novel zero-shot method for finding correspondences between different object instances from similar categories. In each panel, the left image is the source and the right is the target. Our method is flexible and solves (a) dense image matching, (b) sparse image matching, and due to the zero-shot nature it is directly applicable to other domains, such as (c) medical images.

Figure 2: Overview of our method. Two sets of features are extracted for each input image. One set is used to derive our LBO via a Manifold Embedding used in the spectral branch of our approach (top and bottom). The other set is used in the point-wise matching branch leveraging nearest-neighbour search (middle). Both branches are coupled via the alignment loss (green box, see Eqn. ( 9 )).

Panel of Figure 3 — Figure 3: Dense correspondences on two datasets. We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our method achieves smoother and more consistent mappings than [ 16 ] .

Setting	Method	FG3DCar	JODS	Pascal	Avg.	Wtd. Avg.	Time [s]
Supervised	SCOT [ 93 ]	95.3	81.3	57.7	78.1	80.8	-
	CATs* [ 50 ]	92.1	78.9	64.2	78.4	80.7	-
	PWarpC-CATs* [ 94 ]	95.5	85.0	85.5	88.6	90.2	-
Unsupervised	CNNGeo [ 59 ]	90.1	76.4	56.3	74.2	76.8	-
(task-specific)	PARN [ 95 ]	89.5	75.9	71.2	78.8	81.0	-
	GLU-Net [ 53 ]	93.2	73.3	71.1	79.2	82.3	-

Table 1: Quantitative results ([email protected]

\uparrow

) on the TSS dataset [ 85 ] . Methods are grouped into supervised, unsupervised and zero-shot. Our approach substantially outperforms all zero-shot baselines (bold face) and is on par with unsupervised task-specific methods. NN stands for nearest-neighbour matching. We report both the unweighted (Avg.) and weighted (Wtd. Avg.) averages over the three data subsets following [ 8 ] and [ 16 ] , respectively. *: backbone fine-tuned on TSS.

Method	[email protected] $\uparrow$	[email protected] $\uparrow$	MSE $\downarrow$
DINOv2	52.3	68.0	105.0
Stable Diffusion	51.2	64.1	120.5
FUSE [ 8 ]	57.2	72.2	97.2
CleanDIFT [ 78 ]	54.4	66.8	129.6
Fuse2Match [ 92 ]	61.0	72.3	100.1
Cheng et al. [ 16 ]	55.3	72.6	88.0

Table 2: Sparse keypoint correspondences on SPair-71k [ 89 ] . While our method is on par with the baseline performance at [email protected], it yields superior results for both [email protected] and MSE.

Panel of Figure 4 — Figure 4: Visualisation of sparse correspondence on Spair-71k [ 89 ] . Correct matches are indicated by blue lines, while incorrect matches are shown in red. We compare our approach against FUSE [ 8 ] , as it represents the strongest baseline in terms of [email protected]. Thanks to its global geometric consistency, our framework produces more robust mappings across challenging examples.

Panel of Figure 5 — Figure 5: Ablation of our approach without the proposed LBO or NAM on the TSS dataset. LBO + SmoothFM : we replace the NAM with a FM-based variant SmoothFM [ 97 ] . GL + NAM : we replace the LBO with the image graph Laplacian in [ 16 ] . GL + SmoothFM : we combine both alternatives. Our proposed combination of NAM and our LBO achieves the best results overall (red).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	Method	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$	Time [s]
200	Cheng et al. [ 16 ] $\dagger$	78.9	91.1	26.1	5.7	95.6
200	Ours	80.0	91.8	22.9	5.6	0.77
100	Cheng et al. [ 16 ] *	60.4	82.1	34.7	11.6	94.3
100	Ours	82.9	93.2	20.9	4.8	0.77
50	Cheng et al. [ 16 ] *	60.3	82.0	34.9	11.7	93.5
50	Ours	84.3	94.2	20.0	4.3	0.76

Appendix

Table A3: Impact of spectral resolution (

k

) on performance. We compare our approach against Cheng et al. [ 16 ] across varying numbers of eigenfunctions

k\in\{50,100,200\}

. Our method consistently outperforms [ 16 ] at all resolutions, including their optimal setting of

k=200

. Our method is roughly two orders of magnitude faster than [ 16 ] at all resolutions.

\dagger

: Results as reported in the original paper. *: Results reproduced using the official implementation.

Panel of Figure 9 — Figure A6: Visualization of the eigenfunctions obtained from the graph Laplacian used in [ 16 ] and from our proposed image-based LBO. Our LBO produces smoother low-frequency eigenfunctions, whereas the graph Laplacian retains noticeable high-frequency image details even in the lower modes.

Method	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$	Time. $\downarrow$
w/ NZO	86.3	94.7	19.0	3.9	11.3
w/o NZO	84.3	94.2	20.0	4.3	0.76

Appendix

Table A4: Effect of Neural ZoomOut (NZO) refinement [ 14 ] . Without NZO, our method uses a spectral embedding of size

50

. With NZO, we grow the embedding from

50

250

in increments of

40

. NZO offers an optional way to improve accuracy across all metrics at the cost of a roughly

10\times

increase in runtime. In all experiments in our paper, we report results without NZO to characterise the performance of our method on its own.

Panel of Figure 11 — Figure A7: Laplacian ablation on TSS [ 85 ] . We replace our LBO with the cotangent Laplacian, the robust Laplacian [ 100 ] , and a graph Laplacian [ 16 ] , holding the rest of the pipeline fixed. Our operator achieves the highest PCK and lowest EPE, peaking at $k=50$ , while the graph Laplacian degrades sharply for small $K$ and the cotangent and robust behave almost identically.

Operator		Map estimator		Metrics
Graph	Ours	SmoothFM	NAM	[email protected] $\uparrow$	[email protected] $\uparrow$	MSE $\downarrow$
✓		✓		31.76	50.39	135.33
✓			✓	38.96	65.05	104.56
	✓	✓		50.13	69.92	95.88
	✓		✓	56.50	73.00	84.70

Appendix

Table A5: Component ablation for sparse correspondence on the Spair-71k dataset [ 89 ] . We vary the operator (graph Laplacian [ 16 ] vs. our LBO) and the map estimator (SmoothFM [ 97 ] vs. NAM [ 14 ] ). Both our LBO and NAM improve all three metrics in isolation, and their combination (shaded) is best overall.

	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth $\downarrow$
Ours w/ refinement network	81.84	93.64	22.02	4.06
Ours	86.72	95.31	20.16	3.90

Appendix

Table A6: Effect of the refinement network used in Cheng et al. [ 16 ] . Adding the refinement network degrades all four metrics.

Panel of Figure 14 — Figure A8: Dense correspondences on the Faust-2D dataset. We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our method achieves smoother and more consistent mappings than [ 16 ] .

Method	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$
DINOv2 [ 5 ]	83.2	94.0	27.2	17.0
SD [ 77 ]	81.5	89.2	33.5	14.6
FUSE [ 8 ]	84.2	91.2	30.3	13.2
Cheng et al. [ 16 ]	82.9	94.4	26.3	12.9
Fuse2Match [ 92 ]	83.9	93.7	26.1	18.0
CleanDIFT [ 78 ]	85.8	92.4	30.1	16.7

Appendix

Table A7: Quantitative comparison on Faust-2D . Best results are shown in bold , second best are underlined . Our method outperforms all previous baselines on the Faust-2D dataset.

	Easy ( $0^{\circ}$ – $20^{\circ}$ )				Medium ( $20^{\circ}$ – $80^{\circ}$ )				Hard ( $80^{\circ}$ – $140^{\circ}$ )
Method	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$
DINOv2 [ 5 ]	85.9	96.0	16.6	9.7	78.2	91.9	22.3	12.0	25.8	43.3	69.5	47.4
SD [ 77 ]	85.1	92.2	18.3	7.0	73.2	84.1	28.0	10.6	8.2	22.3	93.6	45.3
FUSE [ 8 ]	87.6	94.0	16.2	6.6	77.2	87.5	24.4	9.7	10.0	24.5	86.8	40.2
CleanDIFT [ 78 ]	88.9	95.0	17.3	9.6	81.9	90.3	22.1	12.2	11.9	26.6	82.9	45.6
Cheng et al. [ 16 ]	85.0	96.4	16.9	7.7	76.2	92.7	21.7	9.2	21.8	39.9	63.9	34.8

Appendix

Table A8: Per-difficulty results on Faust-2D , stratified by the relative rotation between the two shapes in each pair. Best results per column are shown in bold , second best are underlined . Our method achieves the best accuracy on the easy and medium splits, whereas all methods degrade substantially on the hard split.

Panel of Figure 17 — Figure A9: Ablation of our approach on Faust-2D , removing either the proposed LBO or the NAM. LBO + SmoothFM : we replace the NAM with the functional-map-based SmoothFM [ 97 ] . GL + NAM : we replace our LBO with the image graph Laplacian of [ 16 ] . GL + SmoothFM : we combine both alternatives. Our combination of the NAM and our LBO performs best on [email protected] and end-point error, while SmoothFM yields smoother matchings, as expected given that it is explicitly designed to promote smoothness.

Panel of Figure 18 — Figure A10: Additional examples of dense correspondences on the NIH Chest X-ray dataset [ 104 ] . We visualise mappings between X-ray images of the same patient captured at different time points. Our method can be applied to medical images, outperforming nearest-neighbour search using backbone image features.

Seed	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$
0	86.7	95.3	20.2	3.9
1	86.7	95.3	20.1	3.9
2	86.7	95.3	20.2	3.9
3	86.6	95.3	20.2	3.9
4	86.7	95.3	20.1	3.9
5	86.7	95.3	20.3	3.9

Appendix

Table A9: Robustness to initialisation. Per-seed metrics on the TSS dataset [ 85 ] , with mean and standard deviation across ten seeds. The low standard deviation shows that our method is robust to the random initialisation of the NAM weights.

LBO extraction	NAM training	[email protected] $\uparrow$	[email protected] $\uparrow$	EPE $\downarrow$	Smth. $\downarrow$
SD (v1.5)	SD (v1.5)	79.3	89.1	30.5	4.2
SD (v3.5)	SD (v3.5)	30.7	51.4	95.4	13.2
DINOv2-ViT-B/14	DINOv2-ViT-B/14	82.1	91.2	28.6	3.9
DINOv3-ViT-B/16	DINOv3-ViT-B/16	85.4	95.0	21.2	4.4
DINOv2-ViT-B/14	SD (v1.5)	78.6	89.3	30.9	4.6
DINOv3-ViT-B/16	SD (v1.5)	78.8	89.2	30.7	4.6

Appendix

Table A10: Ablation of backbone architectures for LBO extraction and NAM training. We evaluate the impact of various feature extractors on both the spectral basis computation and Neural Adjoint Map (NAM) optimisation. While the SD (v3.5) + DINOv2-ViT-L/14 combination achieves the best performance, we adopt the SD (v1.5) + DINOv2-ViT-B/14 configuration as it provides an optimal balance between competitive performance and computational efficiency while enabling direct comparison with SD (v1.5)-based baselines. Best and second-best results are highlighted in bold and underlined , respectively; the default configuration of our method is coloured in gray .

	Method																			All
[email protected] $\uparrow$
	DINOv2 [ 5 ]	67.7	63.3	83.8	39.1	45.6	44.0	41.0	68.3	33.9	67.5	48.8	61.2	64.1	61.3	22.2	60.9	46.1	23.1	52.3
	SD [ 77 ]	56.5	55.0	75.0	32.2	51.9	45.3	34.3	73.4	34.8	70.6	46.3	63.7	44.7	49.6	50.8	51.3	55.7	32.1	51.2
	Fuse2Match [ 92 ]	66.4	65.8	86.0	40.2	65.1	62.0	44.7	81.4	50.4	81.6	60.3	61.7	65.0	60.5	30.7	61.7	66.5	47.5	61.0
	CleanDIFT [ 78 ]	57.0	55.0	77.2	32.1	66.4	50.6	35.0	77.3	36.5	76.6	46.9	60.5	47.5	36.1	40.3	45.2	64.7	75.2	54.4
	FUSE [ 8 ]	67.7	60.8	83.8	40.2	52.2	50.7	37.3	78.9	42.5	68.9	53.9	61.3	61.2	64.6	42.9	57.4	60.0	46.4	57.2

Appendix

Table A11: Experiment on sparse keypoint correspondences on SPair-71k [ 89 ] . While our method is on par with the baseline performance at [email protected], it yields superior results for both [email protected] and MSE error.

Panel of Figure 22 — Figure A11: Dense correspondences on the TSS dataset [ 85 ] . We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our framework achieves smoother and more consistent mappings than Cheng et al. [ 16 ] .

Panel of Figure 23 — Figure A12: Dense correspondences on the SPair-71k dataset [ 89 ] . We visualise mappings from source to target by transferring a coordinate-based colourmap (second column) to the target domain (fourth and fifth columns). Our framework achieves smoother and more consistent mappings than Cheng et al. [ 16 ] .

Panel of Figure 24 — Figure A13: Visualisation of sparse correspondences on the Spair-71k dataset [ 89 ] . Correct matches are indicated by blue lines, while incorrect matches are shown in red. We compare our approach with FUSE [ 8 ] , and present failure cases of our method in the last two rows.

Panel of Figure 25 — Figure A14: Visualisation of texture transfer on the TSS [ 85 ] . We transfer the texture from the source image to the target image (third and fourth column). While texture transfer is not the primary focus of our method, our approach maintains superior detail fidelity and consistency compared to Cheng et al. [ 16 ] .

Image Matching

Image matching finds corresponding points, regions, or objects across images despite changes in viewpoint, scale, lighting, or sensor.

See all 93 papers

Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild
Mar 12, 2026Jiin Im, Sisung Liu, Je Hyeong HongSemantic Correspondence Optimal Transport
Dept. Electronic Engineering, Hanyang University · Dept. Artificial Intelligence, Hanyang University
Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching
Oct 8, 2026Luping Liu, Bingyi Kang, Yifan Wang +1Image Matching
The University of Hong Kong · ByteDance Seed · Zhejiang University
WovenAnchor Matcher: Specialized Intra- and Inter-Image Context Modeling for Feature Matching
Oct 22, 2026Zhiyang Li, Ruijiang Jin, Thibaut Klenke +2Local Feature Matching
Institute of Science Tokyo, Tokyo, Japan · Denso IT Laboratory, Tokyo, Japan

Mar 12, 2026J/K move · Enter open · S save

Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

Jiin Im, Sisung Liu, Je Hyeong Hong

Dept. Electronic Engineering, Hanyang University · Dept. Artificial Intelligence, Hanyang University

Semantic Correspondence Optimal Transport

Open paper AskPDF arXiv

Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.

Abstract

Image Matching

Explore similar work

Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching

WovenAnchor Matcher: Specialized Intra- and Inter-Image Context Modeling for Feature Matching