Visual localization is critical for embodied agents, but deploying state-of-the-art deep feature matchers on edge devices is fundamentally limited by their massive computational cost. Standard deep matchers tightly interleave intra-image context building and inter-image matching at every layer, creating a severe interaction bottleneck. Localizing a single query against K database images forces the system to execute the full network K independent times with no capacity for caching. To break this bottleneck, we advocate for a paradigm shift toward cacheable image matching. We realize this paradigm with LateMatch, a versatile approach that restructures existing matchers by deferring inter-image interaction to their final layers. By isolating intra-image processing into offline-cacheable Context Blocks, only a small set of Interaction Blocks must be executed online. To further optimize LateMatch, we introduce E-SToRE (Entropy-guided Spatial Token Reduction and Expansion). E-SToRE leverages the decreasing Shannon entropy of the late Interaction Blocks to dynamically reduce token counts just prior to cross-image interaction. Evaluated across HPatches, ScanNet, MegaDepth, ETH-3D, and Aachen, LateMatch reduces effective computational cost by over 2x compared to standard deep matchers, while E-SToRE yields an additional 10% to 14% savings with negligible degradation in accuracy.
Figures & tables
Figure 1: Rethinking visual localization with cacheable image matching via LateMatch . (A) Standard visual localization pipelines require matching a single query image against Top- K (e.g., K=100 ) retrieved database images. (B) Traditional matching architectures interleave intra-image context building and inter-image cross-attention across all layers, preventing any caching and causing a severe computational bottleneck. (C) Our proposed LateMatch architecture breaks this bottleneck by deferring cross-image interaction to the final layers, decoupling the network into offline-cacheable Context Blocks (CBs) and online Interaction Blocks (IBs), reducing the effective compute cost by over 2 × without sacrificing much on performance.
Figure 2: Strategic positioning of LateMatch in the feature matching landscape. While direct nearest-neighbor matching ( A ) is extremely fast but highly inaccurate, and lightweight matchers ( B ) are fast but nowhere close to the accuracy of heavy, computationally prohibitive deep matchers ( D ), our proposed LateMatch paradigm ( C ) achieves the high performance of full-capacity models with the online computational footprint of lightweight matchers.
Stage (online)
GF
Lat. (ms)
Lat. (%)
SuperPoint (query, ×1 )
0.3k
7.2
0.4
Matching ( ×K=50 )
27.9k
1558.3
96.2
PnP + RANSAC ( ×1 )
∼ 0.025
54.3
3.4
Total
28.3k
1619.8
100.0
Table 1: Matching dominates 1 -to- K localization cost. Stage-wise cost of a standard LightGlue pipeline on Aachen ( K=50 , desktop GPU). GigaFLOPs (GF) is the total online cost, not the per-pair cost of Sec. 4.3 .
Figure 3: Context Block cost amortizes by K≈10 . LG-late4 latency relative to LG-lite4 on Aachen.
Figure 4: Cross-attention entropy naturally decays across Interaction Blocks, guiding the progressive token reduction in E-SToRE . (A) The cross-attention entropy across the four IBs of LightGlue-late4 . Attention smoothly shifts from diffuse global consensus (high entropy) in early IBs to highly peaked, localized matching (low entropy) in later IBs. (B) Progressively decaying r has significantly better performance (homography estimation accuracy) compared to other r selection approaches.
Figure 5: LateMatch consistently dominates the compute-accuracy Pareto frontier. Performance versus online computational cost (E-GFLOPs) evaluated on the ScanNet (A, B) and MegaDepth (C, D) datasets. Across all capacity levels, the -late models (green) achieve strictly higher matching accuracy than the equivalent -lite baselines (red) at the same computational budget.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: MambaGlue-late consistently and significantly outperforms computationally equivalent MambaGlue-lite baselines. We show the trends on three datasets: A) MegaDepth, B) ScanNet, C) HPatches.
Figure 7: LateMatch outperforms token and model pruning. Performance-compute trade-off on the HPatches dataset. We apply confidence-based token pruning, with different confidence thresholds ( ct ), to our LightGlue-late 4 model (blue) to progressively reduce its E-GFLOPs. We compare it against structurally smaller variants shown in green ( LightGlue-late 2 and LightGlue-late 3). We also compare against attention head pruning (black crosses). LateMatch outperforms both these approaches by a significant margin.
Precise and real-time visual localization is critical for applications like AR/VR and robotics, especially on resource-constrained edge devices such as smart glasses, where battery life and heat dissipation can be a primary concerns. While many efficient models exist, further reducing compute without sacrificing accuracy is essential for practical deployment. To address this, we propose asymmetric visual localization: a large Teacher model processes pre-mapped database images offline, while a lightweight Student model processes the query image online. This creates a challenge in matching features from two different models without resorting to heavy, learned matchers. We introduce AsymLoc, a novel distillation framework that aligns a Student to its Teacher through a combination of a geometry-driven matching objective and a joint detector-descriptor distillation objective, enabling fast, parameter-less nearest-neighbor matching. Extensive experiments on HPatches, ScanNet, IMC2022, and Aachen show that AsymLoc achieves up to 95% of the teacher's localization accuracy using an order of magnitude smaller models, significantly outperforming existing baselines and establishing a new state-of-the-art efficiency-accuracy trade-off.
Mohammad Omama, Gabriele Berton, Eric Foxlin +1
The University of Texas at Austin · Work done as a part of a summer internship at Amazon. · Amazon
Semi-dense local feature matching commonly aggregates contextual information before coarse-to-fine correspondence estimation. However, intra-image and inter-image aggregation serve different roles: the former should propagate spatial evidence within each image while retaining structured two-dimensional dependencies, whereas the latter should gather information from plausible corresponding regions while limiting noise from unrelated locations. We propose WovenAnchor Matcher (WAM), a semi-dense matching framework based on role-specialized context aggregation. WAM introduces two complementary operators. WovenMamba performs intra-image aggregation through horizontal-then-vertical state-space propagation, allowing vertical updates to operate on horizontally contextualized features and thereby encouraging a structured two-dimensional receptive-field bias. AnchorCrossAttention performs inter-image aggregation by using cross-attention to gather context from plausible corresponding regions across the image pair. It performs this exchange over local representative anchors that aggregate feature evidence within each window. On MegaDepth, WAM achieves 66.1 pose AUC@5∘, improving on the JamMa baseline by 2.0 AUC points under the same evaluation setup. Ablations show that removing or replacing either specialized component reduces accuracy, providing evidence that intra-image propagation and inter-image retrieval benefit from role-specific aggregation designs. Code and pretrained weights are available at https://github.com/mmaiLab/WovenAnchorMatcher.
Zhiyang Li, Ruijiang Jin, Thibaut Klenke +2
Institute of Science Tokyo, Tokyo, Japan · Denso IT Laboratory, Tokyo, Japan
Despite recent advances in accuracy and efficiency, coarse matching remains an indispensable yet costly stage in existing semi-dense matchers due to dense token-level matching. We present UltraMatch, an ultra-efficient and scalable semi-dense matching framework that bypasses the quadratic computation and memory cost of dense token-level matching by routing only a small fraction of candidate matching paths. At its core, a lightweight Transport Path Router operates on coarse block representations to rank candidate target blocks for each source block and retain only a small set, restricting subsequent token-level matching to the selected paths and avoiding the construction of the full token-to-token matching matrix. We further design a sparse global Dual-Softmax that performs matching only over the routed block candidates while retaining global competition across the sparse matching space. Beyond matching acceleration, UltraMatch employs deployment-oriented structural reparameterization for feature extraction and a tiny fine matching head with shared parameters, further reducing inference cost and memory consumption. UltraMatch achieves competitive accuracy among semi-dense matchers, while running 1.67× faster than SuperPoint+LightGlue with only 0.44 GiB peak inference memory. Its scalability enables inference at up to 6K resolution on a single RTX 3090, whereas existing semi-dense matchers run out of memory before reaching 2K. Our routing strategy is also transferable, delivering about 2× end-to-end speedup in EDM and ELoFTR without accuracy loss. The project repository is available at https://github.com/JiajunLe/UltraMatch.
Jiajun Le, Yifan Lu, Zizhuo Li +3
Electronic Information School, Wuhan University, China · Xiaomi Corporation, China · School of Computer Science and Technology, Harbin Institute of Technology, China +1