cs.AIMar 17, 2026

V-DyKnow: A Dynamic Benchmark for Time-Sensitive Knowledge in Vision Language Models

Authors: Seyed Mahed Mousavi, Christian Moiola, Massimo Rizzoli, Simone Alghisi, Giuseppe Riccardi

Organizations: Signals and Interactive Systems Lab, University of Trento, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-Language Models (VLMs) are trained on data snapshots of documents, including images and texts. Their training data and evaluation benchmarks are typically static, implicitly treating factual knowledge as time-invariant. However, real-world facts are intrinsically time-sensitive and subject to erratic and periodic changes, causing model predictions to become outdated. We present V-DyKnow, a Visual Dynamic Knowledge benchmark for evaluating time-sensitive factual knowledge in VLMs. Using V-DyKnow, we benchmark closed- and open-source VLMs and analyze a) the reliability (correctness and consistency) of model responses across modalities and input perturbations; b) the efficacy of knowledge editing and multi-modal RAG methods for knowledge updates across modalities; and c) the sources of outdated predictions, through data and mechanistic analysis. Our results show that VLMs frequently output outdated facts, reflecting outdated snapshots used in the (pre-)training phase. Factual reliability degrades from textual to visual stimuli, even when entities are correctly recognized. Besides, existing alignment approaches fail to consistently update the models' knowledge across modalities. Together, these findings highlight fundamental limitations in how current VLMs acquire and update time-sensitive knowledge across modalities. We release the benchmark, code, and evaluation data.

Figures & tables

Figure 1: An example of multimodal querying VLMs for factual knowledge that is time-sensitive. Upon a visual stimulus, the VLM first misidentifies the entity and retrieves an incorrect fact. Following the clarification turn about the entity, the model generates an outdated answer. In the final turn, the entity is explicitly stated in the text, and the correct fact is finally returned. This example highlights key issues investigated in this work: the prevalence of outdated knowledge in VLMs and the performance gap in VLMs for visual and textual stimuli.
(Year) ModelAccuracy
(2023) LLaVA-1.569%
(2024) LLaVA-OneVision80%
(2024) PaliGemma 263%
(2024) Molmo60%
(2024) Qwen2-VL91%
(2025) Qwen2.5-VL87%
Table 2: Accuracy on the Visual Entity Recognition task. Models are prompted to identify the entity depicted in each image before answering the factual query. Scores are computed using an Upper-Bound across three prompt variants (presented in § Table 6 ).
(Year) ModelPrompt Agreement
VisualTextual
(2023) LLaVA-1.552%70%
(2024) LLaVA-OneVision66%80%
(2024) PaliGemma 225%100%
(2024) Molmo57%81%
(2024) Qwen2-VL46%79%
Table 3: Output consistency (prompt agreement) across three different lexicalizations of the same query for Visual and Textual inputs. An agreement measures the percentage of cases for which all prompts produce the same answer (i.e., correct, outdated, or irrelevant).
(Year) Model# Outdated FactsKnowledge EditingMulti-modal RAG
WISEGRACEIKERetrieved Doc.Gold Doc.
(2023) LLaVA-1.5692.9%5.5%95.6%73.5%84.4%
(2024) Qwen2-VL833.9%2.4%100.0%80.1%92.8%
Table 4: Effectiveness of existing alignment methods (editing and multimodal RAG) on outdated facts for LLaVA-1.5 and Qwen2-VL. Scores report the harmonic mean of efficacy and paraphrase success. We compare knowledge editing methods with multimodal RAG using either retrieved documents as well as gold documents.
ModelC orrectIncorrect
O utdatedG enericH alluc.
LLaVA-1.5    
WISE3%73%0%24%
GRACE21%63%1%15%
IKE95%0%5%0%
RAG    
Table 5: Qualitative error analysis of knowledge editing (WISE, GRACE, IKE) and retrieval-based alignment methods across two representative LLMs, based on human annotation of generated outputs. Responses are categorized as Correct when the outdated attribute is successfully updated. Incorrect responses are further classified as Outdated when the model continues to return the outdated fact, Generic when the model produces vague answers, and Hallucinated when the model fabricates or confuses attributes.
Figure 2: Temporal distribution of the model responses based on Wikidata. For each VLM, we map their correct and outdated responses (e.g., “The CEO of Apple is Steve Jobs ”) to the time at which the corresponding attribute was valid (e.g., “1997-2011”). By aggregating these intervals using a boxplot, we can approximate the state of the world encoded in the model’s parameters. For example, results show that, while Qwen2-VL responses range between 2004 and 2023 (in line with its reported cutoff), most of them are concentrated between 2013 and 2019.
Figure 3: Mechanistic analysis of Qwen2-VL illustrating how editing methods modify the probability for certain attributes (y-axis) across different layers (x-axis) when asked about an Image-Question pair. In case of a successful update (top), WISE and GRACE affect different layers: WISE primarily edits the final layer, whereas GRACE modifies a broader range. In contrast, neither method successfully updates the basketball team of Paul George (bottom), suggesting that this knowledge may be more deeply embedded or that other stored facts may interfere.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Table 6: List of all subject entities and their corresponding visual prompts used for benchmarking the VLMs.
Table 7: List of all subject entities and their corresponding visual entity recognition prompts used for benchmarking the VLMs.
Model NameVLM CheckpointLLM Checkpoint
LLaVA-1.5 7Bllava-hf/llava-1.5-7b-hflmsys/vicuna-7b-v1.5
LLaVA-OneVision 7Bllava-hf/llava-onevision-qwen2-7b-si-hfQwen/Qwen2-7B-Instruct
PaliGemma 2 10Bgoogle/paligemma2-10b-mix-448google/gemma-2-9b-it
Molmo 7Ballenai/Molmo-7B-O-0924allenai/OLMo-7B-Instruct-hf
Qwen2-VL 7BQwen/Qwen2-VL-7B-InstructQwen/Qwen2-7B-Instruct
Qwen2.5-VL 7BQwen/Qwen2.5-VL-7B-InstructQwen/Qwen2.5-7B-Instruct
Appendix
Table 8: Official HuggingFace model checkpoints for the open-weight VLMs and their corresponding base LLMs used in our evaluation.
Model# Demonstrations (K)
123
LLaVA-1.588.3%95.6%0%
Qwen2-VL100%100%99.1%
Appendix
Table 9: Harmonic mean for IKE with up to K=3K=3 demonstrations. Performance peaks at K=2K=2 , while adding a third demonstration degrades results, suggesting difficulty in retrieving the relevant information from the context. LLaVA-1.5 is an extreme case, as it ends up repeatedly generating \n tokens for K=3K=3 .
Model# Documents (K)
135
LLaVA-1.573.5%44.7%8.7%
Qwen2-VL80.1%69.8%67.7%
Appendix
Table 10: Harmonic mean for RAG when considering K∈{1,3,5}K\in\{1,3,5\} documents as context.
Table 11: Mechanistic analysis of LLaVA-1.5 illustrating how editing methods successfully modify the contribution of different layers during factual recall.
Table 12: Mechanistic analysis of LLaVA-1.5 illustrating how editing methods fail to modify the contribution of different layers during factual recall.
More in this topic

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Explore similar work

CardsList

  1. Chronological Blindness: Benchmarking Temporal Reasoning in Vision-Language Models with CHRONOSIGHT

    Jun 15, 2026Parthaw Goswami, Jaynto Goswami DeepVLM EvaluationTemporal Reasoning

    Department of Computer Science, University of Missouri, Columbia, 65201, MO, USA · SAP, Prague, Czech Republic

  2. Benchmarking Deflection and Hallucination in Large Vision-Language Models

    Apr 13, 2026Nicholas Moratelli, Christopher Davis, Leonardo F. R. Ribeiro +2VLM EvaluationVision-Language Models

    University of Modena and Reggio Emilia · Amazon AGI · University of Cambridge

  3. A Dataset for Dynamic Human Preferences for Vision Language Models

    Jun 2, 2026Hannah Gao, Dylan Hadfield-Menell, Rachel MaVLM EvaluationHuman Preference Evaluation

    Massachusetts Institute of Technology · MIT Cambridge, MA