Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Siyoon Jin; Dahyun Chung; Honggyu An; Sangbeom Lim; Wonjin Nam; Jaewoo Jung; WonJun Moon; Seungryong Kim

cs.CVOct 27, 2026

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Authors: Siyoon Jin, Dahyun Chung, Honggyu An, Sangbeom Lim, Wonjin Nam, Jaewoo Jung, WonJun Moon, Seungryong Kim

Organizations: KAIST AI · Korea University

Abstract

Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.

Figures & tables

Figure 1: Comparison of cross-view temporal object correspondence paradigms. (Top) (a) propagates the mask in the query view and matches it per frame. (b) starts with initial match, then tracks in each view independently. (c) XTrack jointly models cross-view correspondence and temporal dynamics. (Bottom) Comparison of different cross-view correspondence paradigms.

Figure 2: Overall architecture of XTrack.

Figure 3: 2D pretraining with cycle consistency.

Figure 4: Pseudo-label curation pipeline. (a) performs bidirectional propagation from anchor frames and identifies unreliable regions, which are refined by (b) cross-view geometric verification and further filtered by (c) within-view semantic verification.

Figure 5: Qualitative results on Ego-Exo4D Grauman et al. (2024) . Results are shown for the Exo2Ego setting. Additional Exo2Ego results are provided in Fig. 12 , and the opposite setting is shown in Fig. 13 .

Method	Pipeline	Ego2Exo (Ego as Query)						Exo2Ego (Exo as Query)
Method	Pipeline	$\mathcal{J}(\uparrow)$	$\mathcal{F}(\uparrow)$	$\mathcal{J\&F}(\uparrow)$	LE ( $\downarrow$ )	VA ( $\uparrow)$	mR@IoU ( $\uparrow$ )	$\mathcal{J}(\uparrow)$	$\mathcal{F}(\uparrow)$	$\mathcal{J\&F}(\uparrow)$	LE ( $\downarrow$ )	VA ( $\uparrow)$	mR@IoU ( $\uparrow$ )
Track-then-Match (T $\rightarrow$ M)
PSALM (Zero-shot) Zhang et al. (2024b)	T $\rightarrow$ M	0.126	0.155	0.140	0.221	0.480	0.633	0.149	0.180	0.165	0.168	0.601	0.502
XSegTx Grauman et al. (2024)	T $\rightarrow$ M	0.056	0.108	0.082	0.177	0.448	0.800	0.357	0.389	0.373	0.089	0.809	0.677
XView - XMem Grauman et al. (2024)	T $\rightarrow$ M	0.152	0.211	0.182	0.340	0.366	0.692	0.309	0.356	0.333	0.251	0.403	0.773
XView - XMem + XSegTx Grauman et al. (2024)	T $\rightarrow$ M	0.327	0.413	0.370	0.182	0.423	0.799	0.412	0.460	0.436	0.128	0.463	0.757

Table 1: Quantitative comparison on Ego-Exo4D Grauman et al. (2024) under different pipelines. Track-then-Match (T

\rightarrow

M) and Match-then-Track (M

\rightarrow

T) are reported. XTrack performs joint prediction independent of pipeline choice. N/A indicates results unavailable due to missing checkpoints.

	Method	Exo2Ego (Exo as Query)
	Method	Ego $\mathcal{J\&F}$	Ego LE	Ego VA	Exo $\mathcal{J\&F}$	Exo LE	Exo VA
(I)	Baseline	0.303	0.134	0.674	0.415	0.143	0.859
(II)	(I) + $\mathcal{L}_{\text{align}}$	0.332	0.129	0.755	0.514	0.161	0.906
(III)	(II) + Hard Neg.	0.431	0.106	0.776	0.579	0.073	0.882
(IV)	(III) + Local Crop ( Ours )	0.509	0.090	0.814	0.601	0.060	0.908
(V)	(IV) w/o pretrained weight	0.054	0.189	0.647	0.025	0.243	0.845

Table 2: Ablation studies.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Motivation for temporal reasoning in cross-view object correspondence. Conventional frame-wise matching methods suffer from several failure modes, including empty predictions when the query object is absent and incorrect correspondences under ambiguous appearances. Since predictions are made independently for each frame, the resulting target correspondences often become temporally inconsistent.

Figure 7: 3D full attention in video DiTs.

Figure 8: Difference between noise-to-mask and RGB-to-mask design. (a) Our RGB-to-mask architecture directly predicts mask flow from RGB latents, allowing the transport trajectory to focus entirely on mask prediction. (b) In contrast, the noise-to-mask design predicts masks by denoising from noise while conditioning on RGB latents, following the standard diffusion formulation. As a result, the model implicitly undergoes an intermediate noise-to-RGB transport before mask prediction. Implicit noise-to-RGB transition is further visualized through PCA analysis in Fig. 9 .

Figure 9: Visualization of feature PCA. Results are shown in the exo-query, ego-target setting. (a) shows the input RGB frames, (b) shows the final prediction from the RGB-to-mask formulation, and (c) shows the final prediction from the noise-to-mask formulation. (d) visualizes the RGB latent trajectory $V_{l}$ from the RGB-to-mask architecture in Fig. 8 (a), while (e) visualizes the noise latent trajectory $X_{l}$ from the noise-to-mask architecture in Fig. 8 (b). (e) exhibits an implicit transition through RGB-like representations in the early timesteps before converging toward mask prediction. Columns denote transformer layers and rows denote denoising timesteps.

Figure 10: Visualization of cross-view attention. (a) shows the exocentric query frame and query mask used as input. (b) visualizes the cross-view attention map $A^{\text{ego}\rightarrow\text{exo}}$ , indicating which exocentric regions the ego-side target attends to when establishing correspondence. (c) visualizes the reverse attention map $A^{\text{exo}\rightarrow\text{ego}}$ . (d) shows the resulting ego-side prediction mask. The first and third rows correspond to the baseline without attention alignment, while the second and fourth rows show the results after applying focal attention alignment. Without alignment, attention is broadly dispersed across irrelevant regions, resulting in inaccurate or missing predictions. In contrast, focal attention alignment produces compact and localized attention patterns that better align with the target object, leading to more accurate cross-view correspondence predictions.

Method	Ego2Exo	Exo2Ego
w/o Crop	0.211	0.223
w/ Crop (Ours)	0.376	0.327

Appendix

Table 3: Small-object analysis. J&F on the bottom 25% smallest objects.

Figure 11: Additional qualitative results of comparison with V 2 -SAM. Results are shown in the exo-query, ego-target setting. As a match-then-track pipeline, V 2 -SAM heavily depends on the initial cross-view match, making it difficult to refine incorrect or incomplete matches during tracking. In contrast, our method more reliably localizes the target object’s presence across frames, producing temporally stable masks under challenging cross-view appearance changes.

Figure 12: Additional qualitative results of Exo2Ego. Results are shown in the exo-query, ego-target setting. Each pair of rows corresponds to a single sample, where the first row shows the query exocentric frames and propagated exo predictions, and the second row shows the corresponding ego-side predictions over time.

Figure 13: Additional qualitative results of Ego2Exo. Results are shown in the ego-query, exo-target setting. Each pair of columns corresponds to a single sample, where the left column shows the ego query frames and ego-side predictions, and the right column shows the corresponding exo-side predictions over time.

Method	Pipeline	Ego2Exo				Exo2Ego
Method	Pipeline	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
Track-then-Match (T $\rightarrow$ M)
PSALM	T $\rightarrow$ M	0.637	0.657	0.671	0.568	0.546	0.505	0.488	0.466
XSegTx	T $\rightarrow$ M	0.707	0.694	0.999	NaN	0.721	0.703	0.677	0.607
XView-XMem	T $\rightarrow$ M	0.727	0.697	0.730	0.614	0.815	0.795	0.755	0.729
XView-XMem+XSegTx	T $\rightarrow$ M	0.851	0.814	0.780	0.752	0.766	0.754	0.759	0.748

Appendix

Table 4: Quantitative comparison on EgoExo4D (Recall@p). Track-then-Match (T

\rightarrow

M) and Match-then-Track (M

\rightarrow

T) are reported for prior methods. Our method is independent of pipeline choice. N/A indicates missing checkpoints. NaN indicates that no samples satisfy the IoU threshold

p

at the first frame.

Delayed Target Appearance
		Ego2Exo					Exo2Ego
Method	Pipeline	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J\&F}$ $\uparrow$	LE $\downarrow$	VA $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J\&F}$ $\uparrow$	LE $\downarrow$	VA $\uparrow$
LM-EEC	M $\rightarrow$ T	0.086	0.134	0.110	0.557	0.490	0.075	0.086	0.080	0.664	0.441
ObjectRelator	M $\rightarrow$ T	0.058	0.104	0.081	0.570	0.469	0.065	0.075	0.070	0.660	0.441
V 2 -SAM	M $\rightarrow$ T	0.083	0.112	0.098	0.567	0.481	0.076	0.083	0.079	0.673	0.437
O-MaMa	M $\rightarrow$ T	N/A	N/A	N/A	N/A	N/A	0.038	0.047	0.043	0.715	0.412

Appendix

Table 5: Challenge-based evaluation. Performance under delayed target appearance and temporally disjoint visibility. T

\rightarrow

M and M

\rightarrow

T denote Track-then-Match and Match-then-Track, respectively. N/A indicates unavailable results.

Figure 14: Dataset statistics.

Figure 15: Example sample of curated dataset. Example samples from our curated cross-view temporal correspondence dataset across several tasks. Each pair of rows corresponds to a single sample, showing synchronized ego and exo views with temporally dense correspondence annotations.

Figure 16: Qualitative results of dataset curation pipeline. (a) Forward-only propagation produces temporally inconsistent results, including missing masks and inaccurate object localization. (b) Backward propagation still exhibits similar failure cases, including incorrect propagations and ambiguous predictions. (c) Our cross-view geometric verification stage selects geometrically consistent correspondences from bidirectional propagation results. (d) Our within-view semantic verification stage further refines the selected masks by removing semantically incorrect predictions that cannot be resolved through geometry alone. These results demonstrate that both cross-view geometric verification and within-view semantic verification are necessary for constructing reliable dense correspondence annotations.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction
Feb 22, 2026Shannan Yan, Leqi Zheng, Keyu Lv +7Egocentric Video Understanding Multi-View Consistency
Tsinghua University · WeChat Vision, Tencent Inc. · USTC
Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence
May 5, 2026Zhiyuan Li, Rongzhen Zhao, Wenyan Yang +3Object-Centric Representation Learning Video Object-Centric Learning
Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland
Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching
Oct 8, 2026Luping Liu, Bingyi Kang, Yifan Wang +1Image Matching
The University of Hong Kong · ByteDance Seed · Zhejiang University

Feb 22, 2026J/K move · Enter open · S save

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

Shannan Yan, Leqi Zheng, Keyu Lv, Jingchen Ni+6

Tsinghua University · WeChat Vision, Tencent Inc. · USTC

Egocentric Video Understanding Multi-View Consistency

Open paper AskPDF arXiv

We study the task of establishing object-level visual correspondence across different viewpoints in videos, focusing on the challenging egocentric-to-exocentric and exocentric-to-egocentric scenarios. We propose a simple yet effective framework based on conditional binary segmentation, where an object query mask is encoded into a latent representation to guide the localization of the corresponding object in a target video. To encourage robust, view-invariant representations, we introduce a cycle-consistency training objective: the predicted mask in the target view is projected back to the source view to reconstruct the original query mask. This bidirectional constraint provides a strong self-supervisory signal without requiring ground-truth annotations and enables test-time training (TTT) at inference. Experiments on the Ego-Exo4D and HANDAL-X benchmarks demonstrate the effectiveness of our optimization objective and TTT strategy, achieving state-of-the-art performance. The code is available at https://github.com/shannany0606/CCMP.

Abstract

Visual Object Tracking

Explore similar work

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence

Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching