AnachroBench: Evaluating Historical Consistency in Large Vision-Language Models
Organizations: Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University
Abstract
Existing Large Vision-Language Model benchmarks primarily evaluate visual understanding in general everyday scenarios, leaving historical consistency with a specified era and country largely underexplored. To address this gap, we propose AnachroBench, a controlled discriminative benchmark for evaluating historical consistency in visual scenes. It contains 1,600 image pairs from eight countries spanning the late 19th to the early 20th century, each pairing an original historical image with an anachronistic version containing a seamlessly inserted modern object. AnachroBench evaluates three aspects: Inserted Object Perception, which tests whether models detect the inserted modern object; Era-Country Consistency, which assesses whether the scene is consistent with the specified era and country; and Conditional Era-Country Consistency, which measures consistency judgments only when the inserted object is correctly perceived. A false consistency judgment occurs when a model perceives the modern object but still judges the scene as historically consistent. Overall, AnachroBench reveals a gap between inserted object perception and era-country consistency judgments, providing a targeted evaluation of this behavior.
Figures & tables
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Country | Candidates | Agreement | Cohen’s |
| DE | 898 | 89.0% | 0.779 |
| US | 522 | 84.9% | 0.689 |
| NL | 481 | 89.2% | 0.783 |
| FR | 434 | 95.6% | 0.913 |
| RU | 465 | 82.8% | 0.659 |
| KR | 444 | 97.7% | 0.953 |
| Country | Candidates | Agreement | Cohen’s |
| DE | 441 | 85.9% | 0.694 |
| US | 266 | 78.9% | 0.576 |
| NL | 232 | 85.8% | 0.708 |
| FR | 210 | 81.0% | 0.622 |
| RU | 185 | 87.0% | 0.731 |
| KR | 177 | 89.8% | 0.773 |
| Prompt | Template |
| Original | Given the stated setting of {country} during {period} , is this scene consistent with that time and place? |
| Paraphrase 1 | Could this scene realistically have been photographed in {country} during {period} ? |
| Paraphrase 2 | Consider the historical context of {country} during {period} . Based only on what is visible, is the depicted scene historically consistent with that context? |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,060 papers