Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

Shannan Yan; Leqi Zheng; Keyu Lv; Jingchen Ni; Hongyang Wei; Jiajun Zhang; Guangting Wang; Jing Lyu

cs.CVFeb 22, 2026

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

Authors: Shannan Yan, Leqi Zheng, Keyu Lv, Jingchen Ni, Hongyang Wei, Jiajun Zhang, Guangting Wang, Jing Lyu,

Organizations: Tsinghua University · WeChat Vision, Tencent Inc. · USTC

Abstract

We study the task of establishing object-level visual correspondence across different viewpoints in videos, focusing on the challenging egocentric-to-exocentric and exocentric-to-egocentric scenarios. We propose a simple yet effective framework based on conditional binary segmentation, where an object query mask is encoded into a latent representation to guide the localization of the corresponding object in a target video. To encourage robust, view-invariant representations, we introduce a cycle-consistency training objective: the predicted mask in the target view is projected back to the source view to reconstruct the original query mask. This bidirectional constraint provides a strong self-supervisory signal without requiring ground-truth annotations and enables test-time training (TTT) at inference. Experiments on the Ego-Exo4D and HANDAL-X benchmarks demonstrate the effectiveness of our optimization objective and TTT strategy, achieving state-of-the-art performance. The code is available at https://github.com/shannany0606/CCMP.

Figures & tables

Figure 1: Cycle-Consistent Visual Correspondence with Test-Time Training. Our framework learns object-level correspondences by enforcing cycle consistency: the object mask is transferred from source to target view and projected back to reconstruct the original query. This self-supervised constraint enables robust cross-view alignment and supports test-time training to further improve performance during inference.

Figure 2: Model overview. $\mathit{CLS}$ denotes class tokens, and $\mathit{CDT}$ denotes condition tokens. The CLS head determines whether the object in the target image corresponding to a given object mask in the source image is visible. The bottom-left image shows the source image with the object mask, while the bottom-right image shows the target image.

Method	VA $\uparrow$	IoU $\uparrow$	LE $\downarrow$	CA $\uparrow$	VA $\uparrow$	IoU $\uparrow$	LE $\downarrow$	CA $\uparrow$	mIoU $\uparrow$
	Ego Query				Exo Query
XSegTx (random weights) [ 15 ]	50.00	0.48	0.118	0.014	50.00	1.08	0.203	0.024	0.78
XSegTx [ 15 ]	66.31	18.99	0.070	0.386	82.01	27.14	0.104	0.358	23.07
XMem [ 15 , 8 ]	64.39	19.28	0.151	0.262	60.35	16.56	0.160	0.204	17.92
XView-XMem [ 15 ]	61.24	14.84	0.115	0.242	61.72	21.37	0.139	0.269	18.11
XView-XMem (+ XSegTx) [ 15 ]	66.79	34.90	0.038	0.559	59.71	25.00	0.117	0.327	29.95

Table 1: Evaluation results on the Ego-Exo4D correspondence benchmark v2 (test set). The best performance is highlighted in bold , and the second-best is underlined .

Method	Fine-tuning Datasets	IoU $\uparrow$
XSegTx [ 15 ]	$\emptyset$	1.5
SEEM [ 63 ]	$\emptyset$	2.5
PSALM [ 58 ]	$\emptyset$	14.2
PSALM [ 58 ]	Ego-Exo4D	39.9
ObjectRelator [ 12 ]	Ego-Exo4D	42.8
Ours	Ego-Exo4D	78.8

Table 2: Evaluation results on the HANDAL-X benchmark.

Configuration	$\mathcal{L}_{\text{cycle}}$	$\mathcal{L}_{\text{aux}}$	TTT	Ego-IoU $\uparrow$	Exo-IoU $\uparrow$	mIoU $\uparrow$
Ours (full)	✓	✓	✓	41.95	47.18	44.57
w/o $\mathcal{L}_{\text{cycle}}$	✗	✓	✓	40.28	45.82	43.05
w/o $\mathcal{L}_{\text{aux}}$	✓	✗	✓	40.64	43.81	42.90
w/o TTT	✓	✓	✗	41.79	44.18	42.99

Table 3: Ablation on optimization components.

Figure 3: Visualization illustrating the contribution of test-time training.

Training data	Ego-IoU $\uparrow$	Exo-IoU $\uparrow$	mIoU $\uparrow$
Ours	41.95	47.18	44.57
w/o Ego2Ego & Exo2Exo	40.88	45.50	43.19
w/o RTA	40.60	45.45	43.03

Table 4: Ablation on data augmentation. ”RTA” refers to the relaxed temporal alignment strategy used for both tasks.

Framework	Backbone	CBS	$\mathcal{L}_{\text{cycle}}$	TTT	Ego-IoU $\uparrow$	Exo-IoU $\uparrow$	mIoU $\uparrow$
XSegTx	DINOv3	✗	✗	✗	26.52	34.36	30.44
Ours	DINOv3	✓	✓	✓	41.95	47.18	44.57
Ours	DINOv2	✓	✓	✓	41.48	44.50	42.99

Table 5: Disentangling feature quality from method design. CBS denotes the proposed conditional binary segmentation.

$\lambda_{\text{dice}}^{\prime}$	$\lambda_{\text{cycle}}$	Ego-IoU $\uparrow$	Exo-IoU $\uparrow$	mIoU $\uparrow$
0	10	41.95	47.18	44.57
5	10	38.25	42.14	40.20
5	1	40.45	44.57	42.51

Table 6: Ablation on dice loss in

\mathcal{L}_{\text{cycle}}

and TTT.

Setting	Ego-IoU $\uparrow$	Exo-IoU $\uparrow$	mIoU $\uparrow$
Ours	41.95	47.18	44.57
w/o Linear Probing Stage	37.97	43.69	40.83

Table 7: Ablation on the linear probing stage.

Figure 4: (a) Performance per activity scenario; (b) Performance across different object sizes in the target view.

Figure 5: Qualitative results on the Ego-Exo4D correspondence benchmark. Each row corresponds to one sample. From top to bottom, the first and second rows show samples of Ego2Exo, while the third and fourth rows show samples of Exo2Ego.

Figure 6: Qualitative results on the HANDAL-X benchmark. Each column corresponds to one sample.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Fine-tuning Datasets	IoU $\uparrow$
XSegTx [ 15 ]	$\emptyset$	1.5
SEEM [ 63 ]	$\emptyset$	2.5
PSALM [ 58 ]	$\emptyset$	14.2
PSALM [ 58 ]	Ego-Exo4D	39.9
ObjectRelator [ 12 ]	Ego-Exo4D	42.8
Ours (w/o TTT)	Ego-Exo4D	78.8

Appendix

Table 8: Evaluation results on the HANDAL-X benchmark.

Method	Ego-IoU $\uparrow$	Exo-IoU $\uparrow$	mIoU $\uparrow$
Ours	41.95	47.18	44.57
Cosine Prediction	40.29	46.75	43.52

Appendix

Table 9: Ablation on the mask prediction method.

$\lambda_{\text{dice}}$	Ego-IoU $\uparrow$	Exo-IoU $\uparrow$	mIoU $\uparrow$
0	28.22	32.76	30.49
0.5	41.11	46.24	43.68
1	41.84	47.17	44.51
2	41.33	46.70	44.02
5	41.95	47.18	44.57
10	41.54	46.72	44.13

Appendix

Table 10: Ablation of dice weight.

Table 17

Table 18

Figure 7: Performance–latency trade-off under test-time training.

Figure 8: More qualitative results on the Ego-Exo4D correspondence benchmark (Ego2Exo). Each row shows a representative sample. Rows 1–3 correspond to cooking, row 4 to health, row 5 to bike repair, row 6 to music performance, row 7 to basketball, and row 8 to soccer.

Figure 9: More qualitative results on the Ego-Exo4D correspondence benchmark (Exo2Ego). Each row shows a representative sample. Rows 1–3 correspond to cooking, row 4 to health, row 5 to bike repair, row 6 to music performance, row 7 to basketball, and row 8 to soccer.

Figure 10: More qualitative results on the HANDAL-X benchmark. Each row shows a representative sample.

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence
Oct 27, 2026Siyoon Jin, Dahyun Chung, Honggyu An +5Visual Object Tracking Cross-View Image Matching
KAIST AI · Korea University
Cycle Consistency in Video Object-Centric Learning
May 28, 2026Rongzhen Zhao, Zhiyuan Li, Ruonan Wei +2Self-Supervised Learning Object-Centric Representation Learning
Department of Electrical Engineering and Automation, Aalto University, Espoo, Finland · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · Department of Computer Science, Aalto University, Espoo, Finland +1
SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching
Jun 2, 2026Xu Pan, Qiyuan Ma, Mingyue Dong +3Multi-View Geometry Image Matching
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, P. R. China · National Key Laboratory of Space Target Awareness, Space Engineering University, Beijing, P. R. China

Oct 27, 2026J/K move · Enter open · S save

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Siyoon Jin, Dahyun Chung, Honggyu An, Sangbeom Lim+4

KAIST AI · Korea University

Visual Object Tracking Cross-View Image Matching Video Diffusion Models

Open paper AskPDF arXiv

Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.

Abstract

Egocentric Video Understanding

Explore similar work

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Cycle Consistency in Video Object-Centric Learning

SAMatcher: Co-Visibility Modeling with Segment Anything for Robust Feature Matching