Organizations: School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China
Dense video object captioning (DVOC) localizes, tracks, and describes object trajectories within a video. Existing methods process each camera view independently, producing separate camera-local tracklets and captions for the same person across cameras. These outputs lack a unified description and leave complementary appearance evidence across views unused. We therefore formulate cross-view DVOC (CVDVOC), which associates these tracklets into scene-level global trajectories and generates one caption for each trajectory from synchronized views. To support this task, we construct CRTrack-Cap from CRTrack by auditing text-identity links and converting usable referring expressions into canonical captions. It retains 425 global trajectories and 838 local tracklets across 13 scenes, with canonical captions for 335 identities. We adapt CHOTA to the cross-view setting as CV-CHOTA to jointly evaluate tracking and caption quality. Our reference framework combines View-Exclusive Cross-Camera Association (VECA) for grouping tracklets with Conflict-Evidence Gating (CEG) for integrating caption evidence using its reliability and disagreement across views. Against three single-view methods adapted with MvMHAT for cross-view association, our framework achieves CV-CHOTA scores of 83.43 and 33.48 on the scene-disjoint in-domain and cross-domain tests, respectively. Together, CRTrack-Cap and our framework provide a benchmark and a strong baseline for describing the same person consistently across camera views.
Figures & tables
Figure 1: Cross-view DVOC and construction of CRTrack-Cap. Top: synchronized observations with different camera-local identifiers are linked into one global trajectory with one caption. Bottom: CRTrack videos, boxes, global identities, and referring expressions are converted into an auditable caption layer. Individual text supports a canonical caption for A and B; shared text retains its group scope and does not by itself supply a caption for C. D has no usable text. C and D retain their tracking annotations with null canonical-caption fields.
Split
Scenes
Identities
Captioned
Missing
Coverage
Train
6
256
196
60
76.56
Validation
1
33
14
19
42.42
In-domain test
3
87
76
11
87.36
Cross-domain test
3
49
49
0
100.00
Total
13
425
335
90
78.82
Table 1: CRTrack-Cap statistics. Coverage is the percentage of global trajectories with a canonical caption.
Figure 2: Reference framework for cross-view DVOC. The local tracker exports tracklets, Re-ID descriptors e , and trajectory queries q . VECA forms global identity groups; CEG combines member evidence with reliability and conflict estimates to produce a prompt residual ΔPn . The Q-Former processes each member query with this adapted prompt, and projected prefixes are averaged before frozen OPT generates one caption. We use CRTracker as the local tracker. The bottom row shows the training losses in Equation 14 .
Method
CVIDF1
CV-HOTA
CV-AssA
CV-CHOTA
Global-CapA
CIDEr
In-domain test
DenseVOC
66.28
55.79
51.64
48.17
24.63
26.70
SMOTer
59.83
51.80
44.82
57.43
48.18
66.39
CaptionFormer
77.35
64.28
58.44
76.34
75.54
133.53
Ours
80.65
68.57
62.84
83.43
88.19
157.04
Cross-domain test
Table 2: Main comparison on CRTrack-Cap with predicted local tracklets. DenseVOC, SMOTer, and CaptionFormer use MvMHAT for cross-view association; ours uses VECA. Metric values are shown on the ×100 scale; higher is better.
Setting
VECA
CEG
CVIDF1
CV-HOTA
CV-AssA
CV-CHOTA
Global-CapA
CIDEr
In-domain test
Singleton + MVMean
50.65
51.86
35.96
67.04
80.19
138.03
Singleton + CEG
✓
50.65
51.86
35.96
68.56
85.77
149.01
VECA + MVMean
✓
80.65
68.57
62.84
79.92
77.52
130.20
VECA + CEG
✓
✓
80.65
68.57
62.84
83.43
88.19
157.04
Cross-domain test
Table 3: Association and caption-fusion configurations; metric values are shown on the ×100 scale. The VECA+CEG rows reproduce Table 2 ; the other rows are the recorded component controls.
Split
Views
Δ CV-CHOTA
Δ CIDEr
In-domain
1
+0.11
+0.89
In-domain
2
+6.55
+55.03
In-domain
3
+4.11
+38.48
Cross-domain
3
+2.22
+35.21
Cross-domain
4
+1.88
+29.13
Table 4: View-count comparison with VECA groups fixed. Metric deltas are CEG minus MVMean on the ×100 scale. Sample counts and additional metrics appear in Appendix A.5 .
Figure 3: In-domain (Circle) and cross-domain (Garden2) examples. Rows are synchronized views; columns are methods. Green/red arrows mark correct/incorrect links; no arrow means unmatched. Text below compares generated captions with ground truth.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Rule
CVIDF1
CV-HOTA
CV-DetA
CV-AssA
CV-AssRe
Singleton
50.65
51.86
74.99
35.96
37.09
Greedy view-exclusive
78.22
67.75
75.02
61.18
63.79
Hungarian star
77.03
66.65
74.99
59.38
61.74
Full VECA
80.65
68.57
74.99
62.84
65.39
Appendix
Table 5: Recorded association-rule comparison on the in-domain test. TrackEval metric values are shown on the ×100 scale; best values are bold.
Split
Views
Δ CV-CHOTA
Δ Global-CapA
Δ CIDEr
Δ METEOR
Δ SPICE
In-domain
1
+0.11
+0.29
+0.89
− 0.02
+0.00
In-domain
2
+6.55
+20.06
+55.03
+4.05
+1.10
In-domain
3
+4.11
+14.17
+38.48
+2.76
+1.26
Cross-domain
3
+2.22
+12.87
+35.21
+2.78
+0.62
Cross-domain
4
+1.88
+10.75
+29.13
+1.96
+1.15
Appendix
Table 6: CEG–MVMean gain by reference-view count with VECA groups fixed. Metric deltas are shown on the ×100 scale.