Improving Local Feature Matching by Entropy-inspired Scale Adaptability and Flow-endowed Local Consistency

Ke Jin; Jiming Chen; Qi Ye

cs.CVApr 8, 2026

Improving Local Feature Matching by Entropy-inspired Scale Adaptability and Flow-endowed Local Consistency

Authors: Ke Jin, Jiming Chen, Qi Ye

Organizations: State Key Laboratory of Industrial Control Technology, Zhejiang University.

Abstract

Recent semi-dense image matching methods have achieved remarkable success, but two long-standing issues still impair their performance. At the coarse stage, the over-exclusion issue of their mutual nearest neighbor (MNN) matching layer makes them struggle to handle cases with scale difference between images. To this end, we comprehensively revisit the matching mechanism and make a key observation that the hint concealed in the score matrix can be exploited to indicate the scale ratio. Based on this, we propose a scale-aware matching module which is exceptionally effective but introduces negligible overhead. At the fine stage, we point out that existing methods neglect the local consistency of final matches, which undermines their robustness. To this end, rather than independently predicting the correspondence for each source pixel, we reformulate the fine stage as a cascaded flow refinement problem and introduce a novel gradient loss to encourage local consistency of the flow field. Extensive experiments demonstrate that our novel matching pipeline, with these proposed modifications, achieves robust and accurate matching performance on downstream tasks.

Figures & tables

Fig. 1: Overview of the proposed method. (1) Given a pair of images, a standard Siamese CNN extracts multi-level features, and the coarsest features $F_{c}^{S}$ and $F_{c}^{T}$ are transformed to be more discriminative after processed by interleaved self and cross attention layers. (2) We approximate the scale ratio based on the entropy of the score matrix $\mathcal{S}$ which is computed by correlation between the transformed features $\tilde{F}_{c}^{S}$ and $\tilde{F}_{c}^{T}$ . Coarse matches $\{\mathcal{M}_{c}\}$ are then obtained according to the adaptive mutual nearest neighbor (AMNN) criterion. (3) Given established coarse matches, the coarse flow and certainty are initialized in a straightforward way. (4) A cascade of depth-wise convolution modules are employed to refine the flow progressively, yielding the final pixel-level correspondences.

Fig. 2: The visualization of matching score heatmaps and inferred co-visibility. We sample four points on the reference image pair (a), where the red markers indicate unmatchable points and the green markers indicate co-visible points. (b) shows their matching score heatmaps with the other image, which exhibit completely different distribution patterns: one-hot and evenly distributed. Motivated by this, the entropy of score vectors can be used to infer the co-visible regions of each image (c), enabling scale-aware matching.

Fig. 3: Illustration for inspection window. The many-to-one case is used to illustrate the benefits of the inspection window introduced by AMNN. For better visualization, the source image in the first row and the target image in the second row are shown in different scales. The green translucent square denotes the inspection window. The solid arrows to the target image indicate the predicted nearest neighbors for patches in the source image, while the dashed arrows in reverse direction indicate the cycle consistency check after that. In this example, AMNN produces a scale-dependent window with size $3$ , which enables the correspondence between B and C to be accepted.

Fig. 4: Illustration for local consistency. (a) When the predictions of two adjacent source pixels deviate from the ground truth in opposite directions, this can lead to unreasonable distortions in the flow field. (b ) While the per-pixel error remains identical to (a), this enhanced local consistency proves beneficial for subsequent downstream tasks.

Definition	Number of Matches
Definition	Sacre Coeur	St. Peter’s Square
Based on MNN	1357.96	1301.22
Ours	2086.15	2137.15

TABLE I: Number of ground truth matches

Category	Method	MegaDepth Dataset			ScanNet Dataset			Time (ms)
Category	Method	AUC@5 $\degree$	AUC@10 $\degree$	AUC@20 $\degree$	AUC@5 $\degree$	AUC@10 $\degree$	AUC@20 $\degree$	Time (ms)
Sparse	SP [ 35 ] + NN	31.7	46.8	60.1	7.5	18.6	32.1	10.8
	SP + SG [ 18 ]	49.7	67.1	80.6	16.2	32.8	49.7	48.3
	SP + LG [ 19 ]	49.9	67.0	80.1	14.8	30.8	47.5	31.9
	Mast3R [ 48 ]	42.4	61.5	76.9	28.0	50.2	68.8	294.0
Dense	DKM [ 20 ]	60.4	74.9	85.1	26.64	47.07	64.17	210.8

TABLE II: Results of Relative Pose Estimation on MegaDepth Dataset and ScanNet Dataset. Following [ 15 ] , we evaluate all methods on both datasets using models trained on the MegaDepth dataset, except for Mast3R, which is a fundamental model trained on massive 3D data. The AUC of pose error at different thresholds, along with the processing time for matching image pair at a resolution of 640 × 480, is presented.

Fig. 5: Qualitative comparison results indoor. Not only the number of the matches but also the matching accuracy demonstrate the superiority of the proposed scale-aware matching mechanism.

Method	Homography est. AUC
Method	@3px	@5px	@10px
SP + SG [ 18 ]	53.9	68.3	81.7
Sparse-NCNet [ 57 ]	48.9	54.2	67.1
DRC-Net [ 49 ]	50.6	56.2	68.3
LoFTR [ 9 ]	65.9	75.6	84.6
AspanFormer [ 11 ]	67.4	76.9	85.4

TABLE III: Results of Homography Estimation on HPatches Dataset.

Method	DUC1			DUC2
Method	(0.25m,2 $\degree$ )/(0.5m,5 $\degree$ )/(1.0m,10 $\degree$ )
SP+SG [ 18 ]	49.0	68.7	80.8	53.4	77.1	82.4
DRC-Net [ 49 ]	40.6	65.6	70.1	52.8	76.0	74.3
LoFTR [ 9 ]	47.5	72.2	84.8	54.2	74.8	85.5
TopicFM [ 14 ]	52.0	74.7	87.4	53.4	74.8	83.2
PATS [ 22 ]	55.6	71.2	81.0	58.8	80.9	85.5

TABLE IV: Results of Visual Localization on InLoc Dataset.

Method	Day			Night
Method	(0.25m,2 $\degree$ )/(0.5m,5 $\degree$ )/(1.0m,10 $\degree$ )
SP+SG [ 18 ]	89.8	96.1	99.4	77.0	90.6	100.0
LoFTR [ 9 ]	88.7	95.6	99.0	78.5	90.6	99.0
TopicFM [ 14 ]	90.2	95.9	98.9	77.5	91.1	99.5
PATS [ 22 ]	89.6	95.8	99.3	73.8	92.1	99.5
AspanFormer [ 11 ]	89.4	95.6	99.0	77.5	91.6	99.5

TABLE V: Results of Visual Localization on Aachen v1.1 Dataset.

Method	Pose Estimation AUC
Method	@5°	@10°	@20°
Ours Full	59.6	74.3	84.7
(1) w/o scale-aware coarse matching	56.1	71.9	83.3
(2) Replace our refine. with vanilla refine.	57.4	72.9	83.5
(3) w/o flow gradient loss	58.9	73.8	84.0

TABLE VI: Effectiveness of Each Design.

Fig. 6: Qualitative comparison results outdoor. The left and right columns show the matching results w/o and w/ scale-aware matching mechanism respectively. It can be observed that scale-aware matching mechanism not only increases the number of matches but also improves the matching accuracy.

Method	Pose Estimation AUC@5 $\degree$
Method	$(0,1)$	$[1,2)$	$[2,3)$	$[3,4)$	$[4,+\infty)$
AspanFormer [ 11 ]	58.7	59.0	54.7	51.6	46.3
AdaMatcher [ 23 ]	58.1	58.8	56.0	54.8	52.5
PATS [ 22 ]	61.6	62.5	59.9	58.0	54.8
EfficientLoFTR [ 15 ]	61.2	60.2	55.1	52.4	46.7
Ours	62.3	63.7	60.3	58.0	55.1

TABLE VII: AUC 5 ∘ vs. Scale ratio.

Fig. 7: AUC@5 ∘ vs. Resolution. Our Method is robust against the variation in image resolution, compared with EfficientLoFTR [ 15 ] and JamMa [ 50 ] . Notably, when the resolution is 640, our method outperforms the baselines by 9.0 and 6.1 on AUC@5 ∘ , respectively.

Process	Time(ms)
Total	56.1
Feature Backbone	9.1
Coarse Feature Transformation	11.7
Coarse Matching	8.7
Cascaded Flow Refinement	26.6

TABLE VIII: Time cost for an image pair of

640\times 480

Fig. 8: Qualitative analysis in repetitive-texture scene. We compare representative matching-score heatmaps of co-visible repetitive points and unmatchable points. The co-visible repetitive points still show locally concentrated responses, while the unmatchable point exhibits broadly diffused responses.

$\theta_{e}$	Outdoor		Indoor
$\theta_{e}$	$\left\|\textstyle\frac{\hat{\mathcal{R}}-\mathcal{R}}{\mathcal{R}}\right\|$ / AUC@5 $\degree$
/	/	51.0	/	20.5
1	0.38	51.4	0.42	20.0
2	0.23	53.3	0.30	20.9
3	0.09	55.1	0.21	21.4
4	0.13	55.0	0.14	21.8

TABLE IX: Ablation of entropy threshold

\theta_{e}

Image Matching

Image matching finds corresponding points, regions, or objects across images despite changes in viewpoint, scale, lighting, or sensor.

See all 87 papers

UltraMatch: Transport Path Routing for Ultra-Fast and Memory-Efficient Image Matching
Sep 29, 2026Jiajun Le, Yifan Lu, Zizhuo Li +3Image Matching Memory-Efficient Inference
Electronic Information School, Wuhan University, China · Xiaomi Corporation, China · School of Computer Science and Technology, Harbin Institute of Technology, China +1
WovenAnchor Matcher: Specialized Intra- and Inter-Image Context Modeling for Feature Matching
Oct 22, 2026Zhiyang Li, Ruijiang Jin, Thibaut Klenke +2Local Feature Matching
Institute of Science Tokyo, Tokyo, Japan · Denso IT Laboratory, Tokyo, Japan
FAST: Flow Any Scene Transformer
Sep 30, 2026Yongjian Zhang, Longguang Wang, Zhuo Song +3Optical Flow Vision Transformer
School of Electronics and Communication Engineering, the Shenzhen Campus of Sun Yat-sen University, Sun Yat-sen University, Shenzhen, China · Hong Kong Polytechnic University, HKSAR, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China

Sep 29, 2026J/K move · Enter open · S save

UltraMatch: Transport Path Routing for Ultra-Fast and Memory-Efficient Image Matching

Jiajun Le, Yifan Lu, Zizhuo Li, Lei Cao+2

Electronic Information School, Wuhan University, China · Xiaomi Corporation, China · School of Computer Science and Technology, Harbin Institute of Technology, China+1

Image Matching Memory-Efficient Inference Local Feature Matching

Open paper AskPDF arXiv

Despite recent advances in accuracy and efficiency, coarse matching remains an indispensable yet costly stage in existing semi-dense matchers due to dense token-level matching. We present UltraMatch, an ultra-efficient and scalable semi-dense matching framework that bypasses the quadratic computation and memory cost of dense token-level matching by routing only a small fraction of candidate matching paths. At its core, a lightweight Transport Path Router operates on coarse block representations to rank candidate target blocks for each source block and retain only a small set, restricting subsequent token-level matching to the selected paths and avoiding the construction of the full token-to-token matching matrix. We further design a sparse global Dual-Softmax that performs matching only over the routed block candidates while retaining global competition across the sparse matching space. Beyond matching acceleration, UltraMatch employs deployment-oriented structural reparameterization for feature extraction and a tiny fine matching head with shared parameters, further reducing inference cost and memory consumption. UltraMatch achieves competitive accuracy among semi-dense matchers, while running 1.67

\times

faster than SuperPoint+LightGlue with only 0.44 GiB peak inference memory. Its scalability enables inference at up to 6K resolution on a single RTX 3090, whereas existing semi-dense matchers run out of memory before reaching 2K. Our routing strategy is also transferable, delivering about 2

\times

end-to-end speedup in EDM and ELoFTR without accuracy loss. The project repository is available at https://github.com/JiajunLe/UltraMatch.

Abstract

Image Matching

Explore similar work

UltraMatch: Transport Path Routing for Ultra-Fast and Memory-Efficient Image Matching

WovenAnchor Matcher: Specialized Intra- and Inter-Image Context Modeling for Feature Matching

FAST: Flow Any Scene Transformer