Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.
Figures & tables
Figure 1: Comparison of cross-view temporal object correspondence paradigms. (Top) (a) propagates the mask in the query view and matches it per frame. (b) starts with initial match, then tracks in each view independently. (c) XTrack jointly models cross-view correspondence and temporal dynamics. (Bottom) Comparison of different cross-view correspondence paradigms.
Figure 2: Overall architecture of XTrack.
Figure 3: 2D pretraining with cycle consistency.
Figure 4: Pseudo-label curation pipeline. (a) performs bidirectional propagation from anchor frames and identifies unreliable regions, which are refined by (b) cross-view geometric verification and further filtered by (c) within-view semantic verification.
Figure 5: Qualitative results on Ego-Exo4D Grauman et al. (2024) . Results are shown for the Exo2Ego setting. Additional Exo2Ego results are provided in Fig. 12 , and the opposite setting is shown in Fig. 13 .
Method
Pipeline
Ego2Exo (Ego as Query)
Exo2Ego (Exo as Query)
J(↑)
F(↑)
J&F(↑)
LE ( ↓ )
VA ( ↑)
mR@IoU ( ↑ )
J(↑)
F(↑)
J&F(↑)
LE ( ↓ )
VA ( ↑)
mR@IoU ( ↑ )
Track-then-Match (T → M)
PSALM (Zero-shot) Zhang et al. (2024b)
T → M
0.126
0.155
0.140
0.221
0.480
0.633
0.149
0.180
0.165
0.168
0.601
0.502
XSegTx Grauman et al. (2024)
T → M
0.056
0.108
0.082
0.177
0.448
0.800
0.357
0.389
0.373
0.089
0.809
0.677
XView - XMem Grauman et al. (2024)
T → M
0.152
0.211
0.182
0.340
0.366
0.692
0.309
0.356
0.333
0.251
0.403
0.773
XView - XMem + XSegTx Grauman et al. (2024)
T → M
0.327
0.413
0.370
0.182
0.423
0.799
0.412
0.460
0.436
0.128
0.463
0.757
Table 1: Quantitative comparison on Ego-Exo4D Grauman et al. (2024) under different pipelines. Track-then-Match (T → M) and Match-then-Track (M → T) are reported. XTrack performs joint prediction independent of pipeline choice. N/A indicates results unavailable due to missing checkpoints.
Method
Exo2Ego (Exo as Query)
Ego J&F
Ego LE
Ego VA
Exo J&F
Exo LE
Exo VA
(I)
Baseline
0.303
0.134
0.674
0.415
0.143
0.859
(II)
(I) + Lalign
0.332
0.129
0.755
0.514
0.161
0.906
(III)
(II) + Hard Neg.
0.431
0.106
0.776
0.579
0.073
0.882
(IV)
(III) + Local Crop ( Ours )
0.509
0.090
0.814
0.601
0.060
0.908
(V)
(IV) w/o pretrained weight
0.054
0.189
0.647
0.025
0.243
0.845
Table 2: Ablation studies.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Motivation for temporal reasoning in cross-view object correspondence. Conventional frame-wise matching methods suffer from several failure modes, including empty predictions when the query object is absent and incorrect correspondences under ambiguous appearances. Since predictions are made independently for each frame, the resulting target correspondences often become temporally inconsistent.
Figure 7: 3D full attention in video DiTs.
Figure 8: Difference between noise-to-mask and RGB-to-mask design. (a) Our RGB-to-mask architecture directly predicts mask flow from RGB latents, allowing the transport trajectory to focus entirely on mask prediction. (b) In contrast, the noise-to-mask design predicts masks by denoising from noise while conditioning on RGB latents, following the standard diffusion formulation. As a result, the model implicitly undergoes an intermediate noise-to-RGB transport before mask prediction. Implicit noise-to-RGB transition is further visualized through PCA analysis in Fig. 9 .
Figure 9: Visualization of feature PCA. Results are shown in the exo-query, ego-target setting. (a) shows the input RGB frames, (b) shows the final prediction from the RGB-to-mask formulation, and (c) shows the final prediction from the noise-to-mask formulation. (d) visualizes the RGB latent trajectory Vl from the RGB-to-mask architecture in Fig. 8 (a), while (e) visualizes the noise latent trajectory Xl from the noise-to-mask architecture in Fig. 8 (b). (e) exhibits an implicit transition through RGB-like representations in the early timesteps before converging toward mask prediction. Columns denote transformer layers and rows denote denoising timesteps.
Figure 10: Visualization of cross-view attention. (a) shows the exocentric query frame and query mask used as input. (b) visualizes the cross-view attention map Aego→exo , indicating which exocentric regions the ego-side target attends to when establishing correspondence. (c) visualizes the reverse attention map Aexo→ego . (d) shows the resulting ego-side prediction mask. The first and third rows correspond to the baseline without attention alignment, while the second and fourth rows show the results after applying focal attention alignment. Without alignment, attention is broadly dispersed across irrelevant regions, resulting in inaccurate or missing predictions. In contrast, focal attention alignment produces compact and localized attention patterns that better align with the target object, leading to more accurate cross-view correspondence predictions.
Method
Ego2Exo
Exo2Ego
w/o Crop
0.211
0.223
w/ Crop (Ours)
0.376
0.327
Appendix
Table 3: Small-object analysis. J&F on the bottom 25% smallest objects.
Figure 11: Additional qualitative results of comparison with V 2 -SAM. Results are shown in the exo-query, ego-target setting. As a match-then-track pipeline, V 2 -SAM heavily depends on the initial cross-view match, making it difficult to refine incorrect or incomplete matches during tracking. In contrast, our method more reliably localizes the target object’s presence across frames, producing temporally stable masks under challenging cross-view appearance changes.
Figure 12: Additional qualitative results of Exo2Ego. Results are shown in the exo-query, ego-target setting. Each pair of rows corresponds to a single sample, where the first row shows the query exocentric frames and propagated exo predictions, and the second row shows the corresponding ego-side predictions over time.
Figure 13: Additional qualitative results of Ego2Exo. Results are shown in the ego-query, exo-target setting. Each pair of columns corresponds to a single sample, where the left column shows the ego query frames and ego-side predictions, and the right column shows the corresponding exo-side predictions over time.
Table 4: Quantitative comparison on EgoExo4D (Recall@p). Track-then-Match (T → M) and Match-then-Track (M → T) are reported for prior methods. Our method is independent of pipeline choice. N/A indicates missing checkpoints. NaN indicates that no samples satisfy the IoU threshold p at the first frame.
Ego2Exo
Exo2Ego
Method
Pipeline
J↑
F↑
J&F↑
LE ↓
VA ↑
J↑
F↑
J&F↑
LE ↓
VA ↑
Delayed Target Appearance
LM-EEC
M → T
0.086
0.134
0.110
0.557
0.490
0.075
0.086
0.080
0.664
0.441
ObjectRelator
M → T
0.058
0.104
0.081
0.570
0.469
0.065
0.075
0.070
0.660
0.441
V 2 -SAM
M → T
0.083
0.112
0.098
0.567
0.481
0.076
0.083
0.079
0.673
0.437
O-MaMa
M → T
N/A
N/A
N/A
N/A
N/A
0.038
0.047
0.043
0.715
0.412
Appendix
Table 5: Challenge-based evaluation. Performance under delayed target appearance and temporally disjoint visibility. T → M and M → T denote Track-then-Match and Match-then-Track, respectively. N/A indicates unavailable results.
Figure 14: Dataset statistics.
Figure 15: Example sample of curated dataset. Example samples from our curated cross-view temporal correspondence dataset across several tasks. Each pair of rows corresponds to a single sample, showing synchronized ego and exo views with temporally dense correspondence annotations.
Figure 16: Qualitative results of dataset curation pipeline. (a) Forward-only propagation produces temporally inconsistent results, including missing masks and inaccurate object localization. (b) Backward propagation still exhibits similar failure cases, including incorrect propagations and ambiguous predictions. (c) Our cross-view geometric verification stage selects geometrically consistent correspondences from bidirectional propagation results. (d) Our within-view semantic verification stage further refines the selected masks by removing semantically incorrect predictions that cannot be resolved through geometry alone. These results demonstrate that both cross-view geometric verification and within-view semantic verification are necessary for constructing reliable dense correspondence annotations.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
We study the task of establishing object-level visual correspondence across different viewpoints in videos, focusing on the challenging egocentric-to-exocentric and exocentric-to-egocentric scenarios. We propose a simple yet effective framework based on conditional binary segmentation, where an object query mask is encoded into a latent representation to guide the localization of the corresponding object in a target video. To encourage robust, view-invariant representations, we introduce a cycle-consistency training objective: the predicted mask in the target view is projected back to the source view to reconstruct the original query mask. This bidirectional constraint provides a strong self-supervisory signal without requiring ground-truth annotations and enables test-time training (TTT) at inference. Experiments on the Ego-Exo4D and HANDAL-X benchmarks demonstrate the effectiveness of our optimization objective and TTT strategy, achieving state-of-the-art performance. The code is available at https://github.com/shannany0606/CCMP.
Shannan Yan, Leqi Zheng, Keyu Lv +7
Tsinghua University · WeChat Vision, Tencent Inc. · USTC
The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors function as expensive approximations of discrete correspondence problems. Modern self-supervised vision backbones already encode instance-discriminative features that distinguish objects reliably. Exploiting these features eliminates the need for learned temporal prediction. We introduce Grounded Correspondence, a framework that replaces learned transition functions with deterministic bipartite matching. Slots initialize from salient regions in frozen backbone features. Frame-to-frame identity is maintained through Hungarian matching on slot representations. The approach requires zero learnable parameters for temporal modeling yet achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS. Project page: https://magenta-sherbet-85b101.netlify.app/
Zhiyuan Li, Rongzhen Zhao, Wenyan Yang +3
Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland
Dense correspondence matching has historically been bounded by simplifying spatio-temporal priors, such as smooth motion and rigid geometry. While effective for classical tasks, these assumptions break down in image editing and reference-guided generation (IEG), where transformations can preserve visual identity while breaking physical continuity. To establish identity-preserving correspondence across such transformations, we introduce FreeMatching, a generalizable framework combining generative and semantic foundation representations with heterogeneous supervision from classical datasets, tracked videos, and synthetic scenes. Teacher-guided iterative refinement further improves correspondence in IEG without dense correspondence annotations. Experimentally, a single FreeMatching model substantially improves correspondence quality on challenging IEG image pairs while retaining competitive performance on classical benchmarks. Furthermore, we demonstrate its utility as a quantitative metric for evaluating identity preservation, with scores that correlate with human judgment. The code is available at https://github.com/luping-liu/FreeMatching.
Luping Liu, Bingyi Kang, Yifan Wang +1
The University of Hong Kong · ByteDance Seed · Zhejiang University