Organizations: School of Software, Northwestern Polytechnical University, Xi’an 710129, China · School of Computer Science, Northwestern Polytechnical University, Xi’an 710129, China · Beijing University Of Technology, Beijing 100124, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
Current video captioning methods usually use an encoder-decoder structure to generate text autoregressively. However, autoregressive methods have inherent limitations such as slow generation speed and large cumulative error. Furthermore, the few non-autoregressive counterparts suffer from deficiencies in generation quality due to the lack of sufficient multimodal interaction modeling. Therefore, we propose a non-autoregressive framework based on Diffusion model for Video Captioning (DiffVC) to address these issues. Its parallel decoding can effectively solve the problems of generation speed and cumulative error. At the same time, our proposed discriminative conditional Diffusion Model can generate higher-quality textual descriptions. Specifically, we first encode the video into a visual representation. During training, Gaussian noise is added to the textual representation of the ground-truth caption. Then, a new textual representation is generated via the discriminative denoiser with the visual representation as a conditional constraint. Finally, we input the new textual representation into a non-autoregressive language model to generate captions. During inference, we directly sample noise from the Gaussian distribution for generation. Experiments on MSVD, MSR-VTT, and VATEX show that our method can outperform previous non-autoregressive methods and achieve comparable performance to autoregressive methods, e.g., it achieved a maximum improvement of 9.9 on the CIDEr and improvement of 2.6 on the B@4, while having faster generation speed. The source code will be available soon.
Figures & tables
Fig. 1: The sample comparison between (a) Previous diffusion-based video captioning approach and (b) Our proposed DiffVC.
Fig. 2: Comparison on generation speed between non-autoregressive and autoregressive video captioning. Autoregressive (AR) Captioner is KG-VCN [ 6 ] .
Fig. 3: Comparison on generation quality between non-autoregressive and autoregressive video captioning, Autoregressive (AR) Captioner is KG-VCN [ 6 ] .
Fig. 4: The overall architecture and key modules of our proposed DiffVC. (a) is the overall architecture of DiffVC, the string in italics denotes the generated caption of a video. (b) is the architecture of the discriminative denoiser. (c) is the architecture of the non-autoregressive language model.
Method
Venue
MSR-VTT
MSVD
VATEX
B@4
M
R
C
B@4
M
R
C
B@4
M
R
C
Autoregressive methods
Two-stream [ 22 ]
TPAMI’20
39.7
27.0
-
42.1
54.3
33.5
-
72.8
-
-
-
-
STAT [ 23 ]
TMM’20
39.3
27.1
-
43.8
52.0
33.3
-
73.8
-
-
-
-
VideoTRM [ 2 ]
ACM MM’20
38.8
27.0
-
44.7
-
-
-
-
-
-
-
-
STGCN [ 24 ]
CVPR’20
40.5
28.3
60.9
47.1
52.2
36.9
73.9
93.0
-
-
-
-
TABLE I: Video captioning model performance on MSR-VTT and MSVD datasets. - denotes that the data is not given in the corresponding literature. The bold number denotes the best results among all non-autoregressive methods.
Fig. 5: Case study on MSR-VTT for video captioning. ‘GT’ denotes the ground-truth caption, and ‘NACF’ denotes the caption generated by NACF [ 12 ] .
Method
MSR-VTT
MSVD
B@4
M
R
C
B@4
M
R
C
DiffVC
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
w/o Discriminative Denoiser
43.1
29
60.5
52.3
52.0
34.3
71.4
90.2
w/o NAR Language Model
42.5
28.2
58.5
51.7
50.4
33.9
70.6
88.7
TABLE II: Ablation study for video captioning on MSR-VTT and MSVD datasets. The bold number denotes the best results among all methods.
N
MSR-VTT
MSVD
B@4
M
R
C
B-4
M
R
C
=10
44
30.5
61.4
52.1
53.2
34.3
70.8
90.2
=12
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
=14
44.2
31.1
63.2
55.8
52.0
36.1
72.2
93.7
TABLE III: Ablation study on the number of denoiser blocks. The bold number denotes the best results among all methods.
N
MSR-VTT
MSVD
B@4
M
R
C
B-4
M
R
C
=4
40
22.6
57.4
41.3
49.2
30.8
63.3
79.6
=6
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
=8
44.2
29.9
63.0
50.7
52.8
36.1
68.2
90.4
=10
43.7
27.4
60.8
48.5
50.9
32.7
68.4
87.7
TABLE IV: Ablation study on the number of LM blocks. The bold number denotes the best results among all methods.
N
MSR-VTT
MSVD
B@4
M
R
C
B-4
M
R
C
=5
41.5
22.6
57.3
47.7
51.0
32.4
67.1
83.5
=20
44.5
31.1
63.9
56.7
53.5
37.1
72.5
95.6
=50
45.0
32.3
64.1
61.8
53.8
37.2
73.8
97. 9
TABLE V: Ablation study on the inference step. The bold number denotes the best results among all methods.