Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, CAS · School of Artificial Intelligence, University of Chinese Academy of Sciences · FKLPRIU, School of Computer and Information Engineering, Xiamen University of Technology
Video Large Language Models (Video-LLMs) require continual learning to adapt to non-stationary real-world data. However, existing benchmarks fall short of evaluating modern foundation models: many still rely on models without large-scale pre-training, and prevailing benchmarks typically partition a single dataset into sub-tasks, resulting in high task redundancy and negligible forgetting on pre-trained Video-LLMs. To address these limitations, we propose CL-VISTA, a benchmark tailored for continual video understanding of Video-LLMs. By curating 8 diverse tasks spanning perception, understanding, and reasoning, CL-VISTA induces substantial distribution shifts that effectively expose catastrophic forgetting. To systematically assess CL methods, we establish a comprehensive evaluation framework comprising 6 distinct protocols across 3 critical dimensions: performance, computational efficiency, and memory footprint. Notably, the performance dimension incorporates a general video understanding assessment to assess whether CL methods genuinely enhance foundational intelligence or merely induce task-specific overfitting. Extensive benchmarking of 10 mainstream CL methods reveals a fundamental trade-off: no single approach achieves universal superiority across all dimensions. Methods that successfully mitigate catastrophic forgetting tend to compromise generalization or incur prohibitive computational and memory overheads. We hope CL-VISTA provides critical insights for advancing continual learning in multimodal foundation models.
Figures & tables
Figure 1: Comparison of CL benchmarks. (a) The significant performance gap between zero-shot and joint training indicates that these tasks require Video-LLMs to learn genuinely new knowledge. On NextQA [ 54 ] and STAR [ 51 ] , sequential fine-tuning (Seq-FT) shows negligible forgetting, nearly matching the joint training upper bound. Conversely, our CL-VISTA benchmark maintains a realistic performance degradation under Seq-FT. (b) Backward Transfer (BWT) measures the impact of new learning on past tasks. Negative BWT values reveal that CL-VISTA consistently induces genuine catastrophic forgetting, whereas existing benchmarks exhibit unrealistic positive BWT.
Figure 2: Overview of the CL-VISTA Benchmark: Architecture and Key Functionalities.
Figure 3: Embedding discriminability analysis. Compared to previous benchmarks where task embeddings are highly entangled, our proposed setting yields clearer task separation. Quantitative results (right) further confirm that our benchmark achieves significantly larger inter-task distances, facilitating better task-incremental evaluation.
Figure 4: Learning trajectory analysis. Compared to individual training ( pink ), CL-VISTA’s continual training ( blue ) exhibits distinct loss spikes at task boundaries. Existing benchmarks show smooth, homogeneous transitions, failing to challenge Video-LLMs with the clear distribution shifts necessary for evaluating continual learning.
Table 1: Overview of downstream tasks in the CL-VISTA benchmark. Tasks are categorized into Perception (Per.), Understanding (Und.), and Reasoning (Rea.). “Recon.” indicates whether the task is curated from original video and annotation sources ( ✗ ) or involves data reconstruction ( ✓ ).
Figure 5: Overview of the data reconstruction pipeline. The pipeline begins with the generation of sentence-level question-answer pairs derived from segmented video clips and their corresponding annotations. These candidates are subsequently refined through a multi-discriminator filtering mechanism to yield a final collection of high-quality and diverse QA pairs.
Method
Count.
Space
Traffic
Movie
GUI
Science
Sports
Reason.
MFT ( ↑ )
MFN ( ↑ )
MAA ( ↑ )
BWT ( ↑ )
Zero-shot
32.24
37.24
41.42
22.78
60.17
56.91
34.40
37.52
40.34
–
–
–
Individual
59.69
72.81
67.04
89.63
79.07
83.43
89.75
89.43
78.86
–
–
–
Joint
60.20
67.07
70.13
86.99
78.11
83.97
90.98
85.98
77.93
–
–
–
LoRA-FT
41.13
54.81
55.84
74.61
68.80
78.20
88.90
86.86
78.35
68.64
63.55
-11.09
Replay-based Methods
Replay
53.39
55.13
59.27
74.78
70.91
79.67
86.7
87.76
78.12
70.95
65.77
-8.19
Table 2: Performance comparison of different methods on Video-LLaVA using standard continual learning metrics. The best results are highlighted in bold , and the second-best results are underlined .
Method
MMVU
MVBench
LongVideo Bench
MMBench- Video
NExTQA
Average
Zero-shot
28.60
41.54
37.70
31.78
23.94
32.71
Joint
36.60
45.68
42.71
34.66
53.50
42.63
LoRA-FT
34.80
46.86
45.03
34.03
58.77
43.90
Replay-based Methods
Replay
37.10 (+2.30)
48.03 (+1.17)
44.65 (-0.38)
33.94 (-0.09)
59.61 (+0.84)
44.67 (+0.77)
MR-LoRA
30.90 (-3.90)
46.78 (-0.08)
40.80 (-4.23)
33.51 (-0.52)
42.79 (-15.98)
38.96 (-4.94)
Table 3: General video understanding performance of various CL methods on Video-LLaVA. We highlight the relative change compared to the LoRA-FT baseline ( green for improvement, red for decline).
Figure 6: Comparative analysis of computational efficiency among various LoRA-based continual learning methods on Video-LLaVA. (a) Average training time cost per task. (b) Average inference latency per task. The dashed line represents the standard LoRA-FT lower bound. The red shaded regions explicitly highlight the additional computational delay introduced by each specific CL method relative to the baseline. Methods are sorted in ascending order of their time costs.
Figure 7: Comparison of parameter size overhead (GB) among ten continual learning methods during (a) training and (b) inference phases. Methods are sorted in ascending order of overhead. Darker colors indicate larger parameter size consumption. Note that (a) uses a logarithmic scale due to the significant overhead introduced by RegLoRA.
Figure 8: CL-VISTA Leaderboard.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Data statistics of CL-VISTA.
Figure A.2 : Visualizations of Counting and Space tasks.
Figure A.3 : Visualizations of Traffic and Movie tasks.
Figure A.4 : Visualizations of GUI and Science tasks.
Figure A.5 : Visualizations of Sports and Reasoning tasks.
Method
LR
Projector LR
Epoch
PEFT
Parameter setting
LoRA-FT
1e-4
1e-5
1
LoRA
rank = 32, expert num = 1
Replay
1e-4
1e-5
1
LoRA
rank = 32, expert num = 1
MR-LoRA
1e-4
1e-5
1
LoRA
rank = 256, expert num = 8
O-LoRA
1e-4
1e-5
1
LoRA
rank = 256, expert num = 8
RegLoRA
1e-4
1e-5
1
LoRA
rank = 32, expert num = 1
MoELoRA
1e-4
1e-5
1
LoRA
rank = 32, expert num = 8
Appendix
Table B.1 : Training configurations and PEFT settings for all compared methods on CL-VISTA. Unless otherwise noted, all settings are kept identical across both base models, Video-LLaVA and VideoLLaMA2.
Figure C.1 : Prompt template used for LLM-as-Judge evaluation.
Method
Count.
Space
Traffic
Movie
GUI
Science
Sports
Reason.
MFT ( ↑ )
MFN ( ↑ )
MAA ( ↑ )
BWT ( ↑ )
Zero-shot
45.82
48.02
46.40
49.69
68.05
65.58
47.72
63.62
54.36
–
–
–
Individual
57.86
64.22
77.11
85.13
80.11
86.79
91.04
84.74
78.38
–
–
–
Joint
58.32
67.44
74.65
85.54
80.24
85.27
91.22
85.40
78.51
–
–
–
LoRA-FT
46.72
50.10
60.89
69.09
67.25
82.05
89.10
84.92
78.44
68.77
64.62
-11.05
Replay-based Methods
Replay
53.65
56.29
66.24
75.14
74.13
84.03
87.12
86.53
77.22
72.89
65.94
-4.94
Appendix
Table D.1 : Performance comparison of different methods on VideoLLaMA2 using standard continual learning metrics. The best results are highlighted in bold , and the second-best results are underlined .
Method
MMVU
MVBench
LongVideo Bench
MMBench- Video
NExTQA
Average
Zero-shot
34.70
52.63
44.43
35.92
38.37
41.21
Joint
40.50
51.61
43.98
38.95
60.31
47.07
LoRA-FT
36.30
50.58
44.28
35.97
55.52
44.53
Replay-based Methods
Replay
39.10 (+2.80)
52.84 (+2.26)
47.57 (+3.29)
38.91 (+2.94)
65.33 (+9.81)
48.75 (+4.22)
MR-LoRA
35.50 (-0.80)
47.00 (-3.58)
44.35 (+0.07)
37.93 (+1.96)
35.33 (-20.19)
40.02 (-4.51)
Appendix
Table D.2 : General video understanding performance of various CL methods on VideoLLaMA2. We highlight the relative change compared to the LoRA-FT baseline ( green for improvement, red for decline).
Table D.3 : Continual learning results of LoRA-FT on different Video-LLM backbones.
Table D.4 : Continual learning results of MoELoRA on different Video-LLM backbones.
Table D.5 : Continual learning results of Replay on different Video-LLM backbones.
Table D.6 : Continual learning results of DISCO on different Video-LLM backbones.
Table D.7 : Continual learning results of HiDe on different Video-LLM backbones.
Table D.8 : Continual learning results of CL-MoE on different Video-LLM backbones.
Table D.9 : Continual learning results of RegLoRA on different Video-LLM backbones.
Table D.10 : Continual learning results of SMoLoRA on different Video-LLM backbones.
Table D.11 : Continual learning results of MR-LoRA on different Video-LLM backbones.
Table D.12 : Continual learning results of OLoRA on different Video-LLM backbones.
Table D.13 : Continual learning results of ModalPrompt on different Video-LLM backbones.
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China +2