Organizations: School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China
Dense video object captioning (DVOC) localizes, tracks, and describes object trajectories within a video. Existing methods process each camera view independently, producing separate camera-local tracklets and captions for the same person across cameras. These outputs lack a unified description and leave complementary appearance evidence across views unused. We therefore formulate cross-view DVOC (CVDVOC), which associates these tracklets into scene-level global trajectories and generates one caption for each trajectory from synchronized views. To support this task, we construct CRTrack-Cap from CRTrack by auditing text-identity links and converting usable referring expressions into canonical captions. It retains 425 global trajectories and 838 local tracklets across 13 scenes, with canonical captions for 335 identities. We adapt CHOTA to the cross-view setting as CV-CHOTA to jointly evaluate tracking and caption quality. Our reference framework combines View-Exclusive Cross-Camera Association (VECA) for grouping tracklets with Conflict-Evidence Gating (CEG) for integrating caption evidence using its reliability and disagreement across views. Against three single-view methods adapted with MvMHAT for cross-view association, our framework achieves CV-CHOTA scores of 83.43 and 33.48 on the scene-disjoint in-domain and cross-domain tests, respectively. Together, CRTrack-Cap and our framework provide a benchmark and a strong baseline for describing the same person consistently across camera views.
Figures & tables
Figure 1: Cross-view DVOC and construction of CRTrack-Cap. Top: synchronized observations with different camera-local identifiers are linked into one global trajectory with one caption. Bottom: CRTrack videos, boxes, global identities, and referring expressions are converted into an auditable caption layer. Individual text supports a canonical caption for A and B; shared text retains its group scope and does not by itself supply a caption for C. D has no usable text. C and D retain their tracking annotations with null canonical-caption fields.
Split
Scenes
Identities
Captioned
Missing
Coverage
Train
6
256
196
60
76.56
Validation
1
33
14
19
42.42
In-domain test
3
87
76
11
87.36
Cross-domain test
3
49
49
0
100.00
Total
13
425
335
90
78.82
Table 1: CRTrack-Cap statistics. Coverage is the percentage of global trajectories with a canonical caption.
Figure 2: Reference framework for cross-view DVOC. The local tracker exports tracklets, Re-ID descriptors e , and trajectory queries q . VECA forms global identity groups; CEG combines member evidence with reliability and conflict estimates to produce a prompt residual ΔPn . The Q-Former processes each member query with this adapted prompt, and projected prefixes are averaged before frozen OPT generates one caption. We use CRTracker as the local tracker. The bottom row shows the training losses in Equation 14 .
Method
CVIDF1
CV-HOTA
CV-AssA
CV-CHOTA
Global-CapA
CIDEr
In-domain test
DenseVOC
66.28
55.79
51.64
48.17
24.63
26.70
SMOTer
59.83
51.80
44.82
57.43
48.18
66.39
CaptionFormer
77.35
64.28
58.44
76.34
75.54
133.53
Ours
80.65
68.57
62.84
83.43
88.19
157.04
Cross-domain test
Table 2: Main comparison on CRTrack-Cap with predicted local tracklets. DenseVOC, SMOTer, and CaptionFormer use MvMHAT for cross-view association; ours uses VECA. Metric values are shown on the ×100 scale; higher is better.
Setting
VECA
CEG
CVIDF1
CV-HOTA
CV-AssA
CV-CHOTA
Global-CapA
CIDEr
In-domain test
Singleton + MVMean
50.65
51.86
35.96
67.04
80.19
138.03
Singleton + CEG
✓
50.65
51.86
35.96
68.56
85.77
149.01
VECA + MVMean
✓
80.65
68.57
62.84
79.92
77.52
130.20
VECA + CEG
✓
✓
80.65
68.57
62.84
83.43
88.19
157.04
Cross-domain test
Table 3: Association and caption-fusion configurations; metric values are shown on the ×100 scale. The VECA+CEG rows reproduce Table 2 ; the other rows are the recorded component controls.
Split
Views
Δ CV-CHOTA
Δ CIDEr
In-domain
1
+0.11
+0.89
In-domain
2
+6.55
+55.03
In-domain
3
+4.11
+38.48
Cross-domain
3
+2.22
+35.21
Cross-domain
4
+1.88
+29.13
Table 4: View-count comparison with VECA groups fixed. Metric deltas are CEG minus MVMean on the ×100 scale. Sample counts and additional metrics appear in Appendix A.5 .
Figure 3: In-domain (Circle) and cross-domain (Garden2) examples. Rows are synchronized views; columns are methods. Green/red arrows mark correct/incorrect links; no arrow means unmatched. Text below compares generated captions with ground truth.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Rule
CVIDF1
CV-HOTA
CV-DetA
CV-AssA
CV-AssRe
Singleton
50.65
51.86
74.99
35.96
37.09
Greedy view-exclusive
78.22
67.75
75.02
61.18
63.79
Hungarian star
77.03
66.65
74.99
59.38
61.74
Full VECA
80.65
68.57
74.99
62.84
65.39
Appendix
Table 5: Recorded association-rule comparison on the in-domain test. TrackEval metric values are shown on the ×100 scale; best values are bold.
Split
Views
Δ CV-CHOTA
Δ Global-CapA
Δ CIDEr
Δ METEOR
Δ SPICE
In-domain
1
+0.11
+0.29
+0.89
− 0.02
+0.00
In-domain
2
+6.55
+20.06
+55.03
+4.05
+1.10
In-domain
3
+4.11
+14.17
+38.48
+2.76
+1.26
Cross-domain
3
+2.22
+12.87
+35.21
+2.78
+0.62
Cross-domain
4
+1.88
+10.75
+29.13
+1.96
+1.15
Appendix
Table 6: CEG–MVMean gain by reference-view count with VECA groups fixed. Metric deltas are shown on the ×100 scale.
Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.
We present TCAM (Track and Caption Any Motion), a generative framework that watches video without a text query or region prompt and produces open-vocabulary captions, temporal intervals, and pointers to the trajectories supporting each event. Dense point trackers localize motion without describing it, while clip-level video-language representations do not directly identify the trajectories carrying an event. TCAM connects these capabilities through a Caption-Aware Resampler: learned queries compress dense trajectory tokens into a fixed-length context for a language decoder. One autoregressive pass describes sequential events and concurrent subjects, each with its own trajectory pointer. Training uses existing captions and segmentation annotations, without additional event labeling. With over 50K training clips, TCAM outperforms the evaluated dense-captioning baselines on ActivityNet Captions and approaches query-based grounding methods on Ref-YouTube-VOS, without training on either dataset. Event selection uses model confidence; best-of-N selection improves grounding by at most 1.4 J&F points in the reported comparisons. Pointer selection exceeds chance by 5.3-8.5 times, and pointer mass concentrates on moving sub-parts above a mask-uniform reference. These selection diagnostics are distinct from full-query tracking accuracy.
Existing video captioning methods struggle to balance visual fidelity and redundancy: holistic captions are compact but lose fine-grained evidence, whereas segment-wise captions improve coverage but introduce heavy redundancy. We propose CodecCap, a codec-inspired framework for high-fidelity dense video captioning. Analogous to video codecs, CodecCap represents videos using keyframe and residual captions. Keyframe captions exhaustively encode stable visual context, while residual captions capture temporally only localized actions, motions and changes. This effectively preserves fine-grained visual evidence while reducing redundant descriptions. To quantify the fidelity of captions, we introduce VidCapQA, a caption-then-QA benchmark with 1,000 questions across 14 capability dimensions. Results on VidCapQA show that captions directly generated by strong VLMs still miss many visual details, highlighting caption representation as a critical bottleneck. Experiments show that CodecCap significantly surpasses direct captioning with the same underlying VLMs, suggesting keyframe-residual captioning a way for high-fidelity video-language supervision. We further use CodecCap to construct CodecVDC-100K, a large-scale dense captioning dataset with anchor, residual, scene-level, and video-level supervision.
Zihan Lin, Songhe Deng, Shuwei He +6
ERNIE Team, Baidu · College of Artificial Intelligence, Inner Mongolia University