GESS: Multi-cue Guided Local Feature Learning via Geometric and Semantic Synergy

Yang Yi; Xieyuanli Chen; Jinpu Zhang; Hui Shen; Dewen Hu

cs.CVApr 7, 2026

GESS: Multi-cue Guided Local Feature Learning via Geometric and Semantic Synergy

Authors: Yang Yi, Xieyuanli Chen, Jinpu Zhang, Hui Shen, Dewen Hu

Organizations: Y. Yi, X. Chen, J. Zhang, H. Shen, and D. Hu are with the College of Intelligence Science and Technology, National University of Defense Technology, China.

Abstract

Robust local feature detection and description are foundational tasks in computer vision. Existing methods primarily rely on single appearance cues for modeling, leading to unstable keypoints and insufficient descriptor discriminability. In this paper, we propose a multi-cue guided local feature learning framework that leverages semantic and geometric cues to synergistically enhance detection robustness and descriptor discriminability. Specifically, we construct a joint semantic-normal prediction head and a depth stability prediction head atop a lightweight backbone. The former leverages a shared 3D vector field to deeply couple semantic and normal cues, thereby resolving optimization interference from heterogeneous inconsistencies. The latter quantifies the reliability of local regions from a geometric consistency perspective, providing deterministic guidance for robust keypoint selection. Based on these predictions, we introduce the Semantic-Depth Aware Keypoint (SDAK) mechanism for feature detection. By coupling semantic reliability with depth stability, SDAK reweights keypoint responses to suppress spurious features in unreliable regions. For descriptor construction, we design a Unified Triple-Cue Fusion (UTCF) module, which employs a semantic-scheduled gating mechanism to adaptively inject multi-attribute features, improving descriptor discriminability. Extensive experiments on four benchmarks validate the effectiveness of the proposed framework. The source code and pre-trained model will be available at: https://github.com/yiyscut/GESS.git.

Figures & tables

Fig. 1: Schematic of geometric and semantic synergy for keypoint detection and feature description. Left: Re-weighting by incorporating depth discontinuity regions and semantic confidence to synergistically enhance the robustness of keypoint detection. Right: Constructing composite identity features to boost information gain, empowering descriptors with superior discriminative performance.

Fig. 2: Overview of the proposed framework. Built upon the multi-scale feature extraction backbone (MLSNet), our framework predicts multi-dimensional cues—including semantics, normals, and depth stability—through multi-branch heads. The UTCF module then adaptively integrates visual texture, normal, and semantic features to significantly enhance descriptor discriminability. Meanwhile, the SDAK mechanism dynamically reweights the initial heatmap, guiding the model to select keypoints that are semantically meaningful and robustly localized.

Fig. 3: Comparison of Mean Matching Accuracy (MMA) on the HPatches dataset across varying pixel error thresholds. Our model consistently achieves the best overall performance compared to state-of-the-art methods.

Fig. 4: Qualitative comparison of matching performance across various challenging scenarios. Green lines represent correct inlier matches, while red lines denote outliers. Our model consistently identifies a higher number of inliers across variations in illumination, low-texture scenes, and viewpoint changes.

Methods	MMA@3	AUC@2	AUC@5
SuperPoint [ 13 ] (CVPRW’18)	59.38	37.31	55.46
D2-Net [ 15 ] (CVPR’19)	43.51	19.65	41.85
R2D2 [ 8 ] (NeurIPS’19)	71.83	44.87	66.06
DISK [ 16 ] (NeurIPS’20)	77.45	52.02	71.92
ALIKE-N [ 17 ] (TMM’22)	68.10	50.62	64.57
ALIKE-L [ 17 ] (TMM’22)	68.10	50.87	64.50

TABLE I: Quantitative comparison on the HPatches dataset [ 1 ] .

		Aachen V1.0						Aachen V1.1
Group	Methods	Day			Night			Day			Night
		$(2^{\circ},0.25)$	$(5^{\circ},0.5)$	$(10^{\circ},5)$	$(2^{\circ},0.25)$	$(5^{\circ},0.5)$	$(10^{\circ},5)$	$(2^{\circ},0.25)$	$(5^{\circ},0.5)$	$(10^{\circ},5)$	$(2^{\circ},0.25)$	$(5^{\circ},0.5)$	$(10^{\circ},5)$
V	SeLF [ 48 ] (TIP’22)	–	–	–	–	–	–	–	–	–	75.0	86.8	97.6
	LBR [ 49 ] (CVPR’22)	88.3	95.6	98.8	84.7	93.9	100.0	88.3	95.6	98.8	77.0	92.1	99.5
	SFD2 [ 21 ] (CVPR’23)	88.2	96.0	98.7	87.8	94.9	100.0	88.2	96.0	98.7	78.0	92.1	99.5
G	SIFT [ 9 ] (IJCV’04)	82.8	88.1	93.1	30.6	43.9	58.2	72.2	78.4	81.7	19.4	23.0	27.2

TABLE II: Visual localization results on Aachen Day-Night V1.0 and V1.1 datasets [ 3 ] .

Methods	MegaDepth-1500 Benchmark			ScanNet Benchmark
Methods	@5 ∘ $\uparrow$	@10 ∘ $\uparrow$	@20 ∘ $\uparrow$	@5 ∘ $\uparrow$	@10 ∘ $\uparrow$	@20 ∘ $\uparrow$
SuperPoint [ 34 ] (CVPRW’18)	40.8	54.3	65.3	12.8	25.0	37.7
D2-Net [ 15 ] (CVPR’19)	35.4	49.5	61.9	12.8	25.2	39.0
R2D2 [ 8 ] (NeurIPS’19)	48.5	63.4	74.7	11.9	23.3	35.4
DISK [ 16 ] (NeurIPS’20)	48.4	60.9	70.8	7.9	16.9	27.0
ALIKE-N [ 17 ] (TMM’22)	44.6	58.3	69.3	5.9	11.9	18.7

TABLE III: Pose estimation results on the MegaDepth1500 and ScanNet datasets.

Fig. 5: Qualitative comparison of feature stability in complex scenes. Our method effectively suppresses spurious detections in dynamic regions (e.g., clouds) and significantly reduces unstable feature extraction along depth discontinuities.

	Large-Scale			Small-Scale
Methods	Madrid metropolis	Gendarmenmarkt	Tower of London	Herzjesu	Fountain	South building
	Track Length ( $\uparrow$ ) / Reproj. Error ( $\downarrow$ )			Track Length ( $\uparrow$ ) / Reproj. Error ( $\downarrow$ )
SuperPoint [ 34 ] (CVPRW’18)	5.76 / 1.03	4.10 / 1.05	4.35 / 0.99	4.42 / 0.73	4.50 / 0.81	5.52 / 0.65
D2-Net [ 15 ] (CVPR’19)	6.45 / 0.87	2.09 / 0.98	2.16 / 0.89	3.39 / 1.15	3.58 / 1.21	4.74 / 1.13
R2D2 [ 8 ] (NeurIPS’19)	2.09 / 0.83	4.46 / 1.11	2.27 / 0.98	5.24 / 1.15	6.11 / 0.94	5.26 / 0.95
ALIKE-N [ 17 ] (TMM’22)	7.48 / 0.99	6.46 / 1.01	3.49 / 0.95	3.83 / 0.82	4.62 / 0.79	6.31 / 0.53

TABLE IV: 3D reconstruction results on the ETH Dataset.

Fig. 6: Ablation studies on HPatches. (a) Impact of multi-scale backbone configurations. (b) Effect of the Semantic-Normal Head components.

Fig. 7: Qualitative visualization of semantic reliability and geometric stability. From left to right: original images, semantic reliability maps (anchoring static structures), and geometric stability heatmaps (perceiving geometric boundaries).

Multi-Cues					Metrics
Appearance		Semantic	Normal	Stability	MMA@3	AUC@2	AUC@5
$\text{M}_{1}$ *	$\text{M}_{2}$ †	Semantic	Normal	Stability	MMA@3	AUC@2	AUC@5
✓					0.778	0.537	0.727
✓		✓			0.823	0.569	0.767
✓			✓		0.815	0.572	0.762
✓				✓	0.788	0.544	0.736

TABLE V: Ablation experiment results on the HPatches dataset.

Methods	MegaDepth-1500			ScanNet
Methods	@ $5^{\circ}\uparrow$	@ $10^{\circ}\uparrow$	@ $20^{\circ}\uparrow$	@ $5^{\circ}\uparrow$	@ $10^{\circ}\uparrow$	@ $20^{\circ}\uparrow$
Baseline	48.3	61.7	71.8	12.79	25.33	38.53
+ MLSNet	52.0	65.4	75.5	14.00	26.66	39.84
+ UTCF	53.1	66.4	76.5	14.30	27.26	40.22
Full (SDAK)	53.9	67.0	76.7	15.81	29.25	42.87

TABLE VI: Ablation study of pose estimation on MegaDepth-1500 [ 52 ] and ScanNet [ 53 ] datasets.

Keypoint Detection

Keypoint detection locates distinctive image points or meaningful landmarks for matching, pose estimation, tracking, and geometric localization.

See all 28 papers

GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training
Jul 2, 2026Yejun Zhang, Xinjue Wang, Zihan Wang +2Visual Place Recognition Camera Pose Estimation
Aalto University, Espoo, Finland · Tampere University, Tampere, Finland · University of Oulu, Oulu, Finland
Towards Complementary Keypoint Detection via Mixture of Detectors
Oct 12, 2026Xiaoyong Lu, Guobao Xiao, Dong Liang +1Keypoint Detection Mixture of Experts
Southeast University · Tongji University · Nanjing University of Aeronautics and Astronautics
GKDT: General Keypoint Detection Transformer
Jul 1, 2026Changsheng Lu, Yuxin Chen, Haokun Gui +5Keypoint Detection
Hong Kong University of Science and Technology, Hong Kong, China · Australian National University, Canberra, Australia · Tencent Inc., China +1

Jul 2, 2026J/K move · Enter open · S save

GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training

Yejun Zhang, Xinjue Wang, Zihan Wang, Esa Rahtu+1

Aalto University, Espoo, Finland · Tampere University, Tampere, Finland · University of Oulu, Oulu, Finland

Visual Place Recognition Camera Pose Estimation Geometric Representation Learning Robot Localization

Open paper AskPDF arXiv

Descriptor-free visual localization eliminates high-dimensional descriptor storage, preserves scene privacy, and simplifies map maintenance, yet its accuracy still lags far behind descriptor-based pipelines. We identify this gap to insufficient geometric discriminability in geometry-only matching. Without visual appearance, current methods underutilize local geometry cues, lack the global context among keypoints, and overfit to a single keypoint detector. We further observe that descriptor-free matching naturally enables multi-detector training, as heterogeneous keypoints can be optimized in a shared geometry-only space without aligning descriptor spaces. Building on these insights, we propose GeoMix, a descriptor-free 2D-3D matching framework that strengthens geometric discriminability at three levels. Locally, directional and distance-aware embeddings enrich neighborhood aggregation with fine-grained spatial structure. Globally, learnable context nodes aggregate and redistribute scene-wide information via cross-attention to resolve ambiguities beyond local receptive fields. At the training level, Mix-Training exploits this detector-agnostic geometry space to learn representations across multiple keypoint detectors. Extensive experiments on MegaDepth, Cambridge Landmarks, 7Scenes, and Aachen Day-Night show that GeoMix sets a new state of the art among descriptor-free methods, reducing 75th-percentile rotation error by 89% and translation error by up to 90% over the previous best, while generalizing zero-shot to unseen detectors and narrowing the gap to descriptor-based pipelines. Code is available at this links.

Abstract

Keypoint Detection

Explore similar work

GeoMix: Descriptor-Free Visual Localization via Global Context and Multi-Detector Training

Towards Complementary Keypoint Detection via Mixture of Detectors

GKDT: General Keypoint Detection Transformer