Audiovisual (AV) archives in museums and galleries are growing rapidly, but much of this material remains effectively locked away because it lacks consistent, searchable metadata. Existing method for archiving requires extensive manual effort. We address this by automating the most labour intensive part of the workflow: catalogue style metadata curation for in gallery video, grounded in an existing collection database. Concretely, we propose catalogue-grounded multimodal attribution for museum AV content using an open, locally deployable video language model. We design a multi pass pipeline that (i) summarises artworks in a video, (ii) generates catalogue style descriptions and genre labels, and (iii) attempts to attribute title and artist via conservative similarity matching to the structured catalogue. Early deployments on a painting catalogue suggest that this framework can improve AV archive discoverability while respecting resource constraints, data sovereignty, and emerging regulation, offering a transferable template for application-driven machine learning in other high-stakes domains.
Figures & tables
Figure 1: High-level overview of the catalogue-grounded pipeline.
Parameter
VideoLLaMA2-FT
Qwen2-VL-FT (v2)
Matched (identical across models)
Training data
custom_small_clean_2turns
Same
Samples
248
Same
Optimizer
paged_adamw_8bit
Same
Learning rate
5×10−5
Same
LR schedule
Cosine
Same
Table 1: Training configuration comparison. Top section: parameters matched for fair ablation. Bottom section: parameters adapted to hardware constraints and training-inference alignment.
Figure 2: Model architecture and fine-tuning setup: high-level overview of the two independent backbones used in this study VideoLLaMA2.1 with a frozen SigLIP vision tower, trainable MLP projector, and Qwen2-VL with its native frozen vision encoder both adapted with LoRA on the language projection layers using the same image/Q&A supervision.
Config
Eval Mode
Videos
Accept
Correct
FP
Prec.
VL2-Base
Batch
18
1
1
0
1.00
VL2-FT
Batch
18
2
1+1 a
0
1.00
Q2VL-ZS
Batch
18
0
0
0
–
VL2-FT
Interactive
3
1
1
0
1.00
Q2VL-FT
Interactive
3
3
3
0
1.00
Q2VL-FT
Batch b
18
2
0
2
0.00
Table 2: Main results across evaluation modes. VL2-Base and VL2-FT achieve zero false positives in batch evaluation. Q2VL-FT achieves 3/3 correct in interactive evaluation (format-matched), but degrades to 0 correct with 2 false positives in batch mode due to training-inference format mismatch. Both fine-tuned models outperform their zero-shot counterparts.
Evaluation
Correct
FP
Input format
Interactive (Streamlit)
3/3
0
pixel_values_videos (native)
Batch (v1 training)
0/13
2
pixel_values (frame extraction)
Batch (v2 training)
TBD
TBD
pixel_values (format-aligned)
Table 3: Q2VL-FT performance by evaluation mode. The 3/3 interactive result reflects genuine model capability with matched training-inference format. Batch degradation is caused by format mismatch, not model weakness.
Catalogue
Coverage
Precision
Correct
None
0/18
–
0
GT (12 entries)
2/18
1.00
2
SFT (enriched)
2/18
1.00
2
Table 4: Effect of catalogue source on VL2-FT. The catalogue is essential for any identification. GT and SFT catalogues yield equivalent results on this evaluation set.
Ground Truth
VL2-Base
VL2-FT
Q2VL-FT ∗
Best
Entombment / Michelangelo
artist: Michelangelo → ✓
artist: Michelangelo → ✓
title+artist via label → ✓
All
Hay Wain / Constable
no signal → abstain
no signal → abstain
artist: Constable via label → ✓
Q2VL-FT
Red Boy / Lawrence
“Red Dress” → abstain
“Red Boy” → ✓
artist: Lawrence via label → ✓
Both FT
Table 5: Qualitative comparison on the 3 videos tested across both evaluation modes. ∗ Q2VL-FT results from interactive evaluation (format-matched). Both fine-tuned models achieve 100% precision; Q2VL-FT achieves higher coverage by reading wall labels.
Figure 3: Qualitative examples of catalogue-grounded attribution on in-gallery videos.
Figure 4: Videollama base and finetuned models .. the last image represents a threshold updated version
Figure 5: Qwen base and finetuned models. The first is base, the second finetuned without ground-truth json and the last finetuned with json
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Description
Default
Artist-driven regime
τartist
Artist score to activate regime
0.45
τartist,accept
Combined score for acceptance
0.38
Title-driven regime
τt
Direct title score accept
0.52
μt
Title margin required
0.05
Appendix
Table 6: Complete abstention configuration. All parameters are exposed in the Streamlit interface for interactive tuning.
Video
VL2-B
VL2-FT
Q2VL-ZS
Q2VL-FT †
04_POTM (Entombment)
✓
✓
–
–
Charlotte (no GT)
–
–
–
–
Char (Cupid/Cranach)
–
–
–
×
Christine-1 (Hay Wain)
–
–
–
–
Christine-2 (Graham)
–
–
–
–
Christine-3 (Daughters)
–
–
–
–
Appendix
Table 7: Per-video batch results. ✓= correct. × = false positive. – = abstained. Q2VL-FT batch results reflect the v1 format mismatch; the interactive evaluation (3/3 correct) better represents model capability.