Exploiting Textual Semantics for Robust Cross-View Object Correspondence

Bing Fan; Yunhe Feng; Yan Huang; Heng Fan

cs.CVNov 2, 2026

Exploiting Textual Semantics for Robust Cross-View Object Correspondence

Authors: Bing Fan, Yunhe Feng, Yan Huang, Heng Fan

Organizations: Department of Computer Science and Engineering, University of North Texas

Abstract

Cross-view object correspondence (CVOC) aims to establish associations of the same target between query and search videos captured from different viewpoints. Existing methods often rely on visual cues for CVOC. Despite recent progress, these methods struggle in complicated scenarios with substantial viewpoint changes and appearance variations, due to lacking sufficient target information. Addressing this, we introduce TeSCo, a framework that exploits rich Textual Semantics of an object, in addition to its visual cues, for cross-view object Correspondence. Our key insight is, textual description of a target, by capturing diverse attributes such as category, texture, and color, provides viewpoint- and appearance-invariant semantics, which are complementary to visual cues and can hence enhance correspondence robustness in complicated scenes. Inspired by this, TeSCo first generates a textual expression of the target from the given query view using a vision-language model, and then extracts textual feature to enhance visual features for localization in the search view. To realize this, we introduce a text-conditioned cross-modal fusion (TCF) module that incorporates the textual feature into visual features with text-guided modulation, producing more robust multimodal target representation for object correspondence. Since not all textual cues are equally compatible with the target in current search-view frame, we propose an iterative textual feature refinement (ITFR) module that progressively adjusts textual feature using search-view information before the TCF module, enabling search-view-aware textual semantics to be fused into visual features and leading to better performance. In extensive experiments on Ego-Exo4D and HANDAL-X, TeSCo demonstrates state-of-the-art results and largely outperforms other models, validating its efficacy. Our code is released at https://github.com/VAIL-Fan/TeSCo.

Figures & tables

Figure 1: Overview of the proposed TeSCo that exploits rich textual information for robust cross-view correspondence through iterative textual feature refinement (ITFR) and text-conditioned cross-modality fusion (TCF) modules. Best viewed in pdf for all figures throughout the paper .

Figure 2: Illustration of the ITFR module.

Figure 3: Illustration of the TCF module.

Figure 4: Image (a): query-view frames; image (b): search-view frames; image (c): attention maps in search view with textual semantics; and (d) attention maps in search view without textual semantics. The visualized attention maps is obtained by aggregating the attention maps from the two attention modules in TeSCo. We can observe that, the attention map with textual semantics can more accurately focus on the target region, evidencing the effectiveness of textual information.

	Ego $\rightarrow$ Exo			Exo $\rightarrow$ Ego			mIoU ( $\uparrow$ )
Method	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )	mIoU ( $\uparrow$ )
XSegTx [CVPR’24] [ 10 ]	18.90	0.386	0.070	27.10	0.358	0.104	23.00
XMem [CVPR’24] [ 10 ]	19.30	0.262	0.151	16.60	0.240	0.160	17.95
XMem+XSegTx [CVPR’24] [ 10 ]	34.90	0.559	0.038	25.00	0.237	0.117	29.95
PCC [CVPR’25] [ 1 ]	39.01	0.600	0.033	47.06	0.590	0.054	43.04
ObjectRelator [ICCV’25] [ 8 ]	35.30	0.540	0.036	40.30	0.500	0.068	37.80

Table 1: Results of Ego

\rightarrow

Exo and Exo

\rightarrow

Ego on the Ego-Exo4D test set. The best and the second-best results are highlighted in bold and underlined in all state-of-the-art comparison tables.

Method	Dataset	IoU ( $\uparrow$ )
(a) zero-shot learning
PSALM [ECCV’24] [ 43 ]	Ego-Exo4D	39.90
ObjectRelator [ICCV’25] [ 8 ]	Ego-Exo4D	42.80
CCMP [CVPR’26] [ 40 ]	Ego-Exo4D	78.80
V 2 -SAM [CVPR’26] [ 31 ]	Ego-Exo4D	77.20
Baseline	Ego-Exo4D	75.26

Table 2: Results on the HANDAL-X test set.

Table 7

	Inj. to Query View	Inj. to Mem.	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )
❶	✓	-	55.28	0.753	0.022
❷	-	✓	58.94	0.804	0.015
❸	✓	✓	60.40	0.847	0.011

Table 7: Ablation on text injection location.

	Text Generation	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )
❶	Qwen2.5-VL + b	56.28	0.766	0.020
❷	Qwen2.5-VL + m	59.75	0.828	0.013
❸	DAM + m	60.40	0.847	0.011
❹	Manual Text	61.20	0.830	0.010

Table 8: Ablation on TeSCo with different text generation methods. “b”: box; “m”: mask.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Examples of generated textual description.

Figure 6: Illustration of the detailed architecture of our baseline.

Scenario	Ego $\rightarrow$ Exo						Exo $\rightarrow$ Ego
	Baseline			TeSCo			Baseline			TeSCo
	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )
Cooking	44.93	0.685	0.027	57.29	0.829	0.131	53.76	0.665	0.005	68.54	0.805	0.025
Music	72.72	0.789	0.020	88.93	0.940	0.010	60.96	0.708	0.034	74.55	0.843	0.017
Health	48.07	0.669	0.019	61.93	0.847	0.008	57.52	0.678	0.041	74.11	0.859	0.018
Basketball	70.35	0.875	0.019	79.32	0.956	0.007	80.08	0.889	0.010	90.29	0.971	0.004

Appendix

Table 9: Performance on different scenarios in Ego-Exo4D.

	Generation Interval	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )
❶	$r=1$	60.40	0.847	0.011
❷	$r=5$	59.10	0.835	0.012
❸	$r=10$	59.00	0.835	0.013

Appendix

Table 10: Results with different textual description generation interval

r

	Memory Bank	IoU ( $\uparrow$ )	Cont.A ( $\uparrow$ )	Loc.E ( $\downarrow$ )
❶	query-view bank	55.47	0.751	0.022
❷	search-view bank	58.78	0.800	0.016
❸	using both banks	60.40	0.847	0.017

Appendix

Table 11: Impact of memory banks.

	Method	Text Condition	IoU ( $\uparrow$ )
❶	Baseline	No text	52.70
❷	TeSCo	Original	60.40
❸	TeSCo	Noisy	59.22
❹	TeSCo	Wrong	56.20
❺	TeSCo	Partial	57.50

Appendix

Table 12: Results on textual stress tests.

	Evaluation Set	Baseline	TeSCo
❶	Full test set	52.70	60.40
❷	Occlusion subset	48.25	58.50
❸	Distractor subset	47.60	57.33

Appendix

Table 13: Results on challenging visual subsets using IoU.

	Method	IoU
❶	LM-EEC	54.98
❷	LM-EEC + Our Textual Modules	59.25

Appendix

Table 14: Results of integrating our textual modules into LM-EEC.

Figure 7: Representative failure cases of the proposed TeSCo under severe occlusion (see image (a)), similar distractors (see image (b)), and small target objects (see image (c)). The prediction results of TeSCo and groundtruth masks are highlighted in pink and green, respectively.

	Method	IoU
❶	Baseline	52.70
❷	TeSCo with Category	57.82
❸	TeSCo with Category and Color	59.65
❹	TeSCo with Complete Description	60.40

Appendix

Table 15: Results of integrating our textual modules into LM-EEC.

Figure 8: (a) Query view with the target outlined in green. (b) Text-visual attention before ITFR. (c) Text-visual attention after ITFR.

Figure 9: Qualitative results in Ego $\rightarrow$ Exo. Image (a): query-view (Ego-centric); image (b): results of baseline; image (c): results of TeSCo; image (d): search-view (Exo-centric) and groundtruth mask.

Figure 10: Qualitative results in Exo $\rightarrow$ Ego. Image (a): query-view (Exo-centric); image (b): results of baseline; image (c): results of TeSCo; image (d): search-view (Ego-centric) and groundtruth mask.

Figure 11: Qualitative results in HANDAL-X. Image (a): query-view; image (b): results of baseline; image (c): results of TeSCo; image (d): search-view and groundtruth mask.

Multi-View Feature Fusion

See all 33 papers

Oct 27, 2026

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.

Siyoon Jin, Dahyun Chung, Honggyu An +5

KAIST AI · Korea University

Visual Object Tracking Cross-View Image Matching Video Diffusion Models

Ask PDF

Feb 22, 2026

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

We study the task of establishing object-level visual correspondence across different viewpoints in videos, focusing on the challenging egocentric-to-exocentric and exocentric-to-egocentric scenarios. We propose a simple yet effective framework based on conditional binary segmentation, where an object query mask is encoded into a latent representation to guide the localization of the corresponding object in a target video. To encourage robust, view-invariant representations, we introduce a cycle-consistency training objective: the predicted mask in the target view is projected back to the source view to reconstruct the original query mask. This bidirectional constraint provides a strong self-supervisory signal without requiring ground-truth annotations and enables test-time training (TTT) at inference. Experiments on the Ego-Exo4D and HANDAL-X benchmarks demonstrate the effectiveness of our optimization objective and TTT strategy, achieving state-of-the-art performance. The code is available at https://github.com/shannany0606/CCMP.

Shannan Yan, Leqi Zheng, Keyu Lv +7

Tsinghua University · WeChat Vision, Tencent Inc. · USTC

Egocentric Video Understanding Multi-View Consistency

Ask PDF

Jun 2, 2026

SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching

Reliable correspondence estimation is a fundamental problem in image processing, underpinning applications such as Structure from Motion, visual localization, and image registration. Existing learning-based methods have significantly improved local feature representations, yet most still operate at the pixel or patch level and lack explicit modeling of regions that are jointly visible across views. We propose SAMatcher, a feature matching framework that formulates correspondence estimation through co-visibility modeling. Instead of directly matching local features, SAMatcher first predicts co-visible region masks and bounding boxes as structured priors for correspondence estimation. Built upon the Segment Anything Model (SAM), it introduces a symmetric cross-view interaction mechanism that enables bidirectional feature exchange and cross-view semantic alignment. We further develop a unified supervision scheme that jointly optimizes mask prediction and box localization through mask learning, box regression, and mask-box consistency constraints. Extensive experiments on challenging benchmarks demonstrate substantial improvements over existing matching pipelines, particularly under large viewpoint and scale variations. Our results show that foundation models originally designed for monocular segmentation can be effectively extended to multi-view correspondence reasoning through explicit co-visibility modeling, offering a new perspective on structured representation learning for image matching. Code and project page: https://xupan.top/Projects/samatcher

Xu Pan, Qiyuan Ma, Mingyue Dong +3

State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, P. R. China · National Key Laboratory of Space Target Awareness, Space Engineering University, Beijing, P. R. China

Multi-View Geometry Image Matching Local Feature Matching Cross-View Image Matching

Ask PDF

Abstract

Multi-View Feature Fusion

Explore similar work

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching