MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

Baochen Fu; Yuntao Du; Cheng Chang; Baihao Jin; Wenzhi Deng; Muhao Xu; Hongmei Yan; Weiye Song

cs.CLMar 16, 2026

MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

Authors: Baochen Fu, Yuntao Du, Cheng Chang, Baihao Jin, Wenzhi Deng, Muhao Xu, Hongmei Yan, Weiye Song,

Organizations: Shandong University, Jinan, China · Jinzhong Group, Jinan, China.

Abstract

As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation benchmark for multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation benchmark for multimodal knowledge updating, advancing progress in this field.

Figures & tables

Figure 1: After the knowledge cut-off, models suffer from both outdated information and knowledge gaps.

Figure 2: Overall pipeline of the construction for MMKU-Bench.

Statistic	Updated	Unknown
Knowledge	13,483	12,309
Subfields	156	175
Number of unique images	25,077	24,618
- Injection images	12,506	12,309
- Evaluation images	12,571	12,309
Updated knowledge length

Table 1: Key Statistics of MMKU-Bench.

Figure 3: Category distribution of MMKU-Bench. “Others” represents categories not explicitly displayed.

		InternVL2.5-8B					Qwen2.5-VL-7B
Method		Updated			Unknown		Updated			Unknown
		Correct $\uparrow$	F1 $\uparrow$	Outdated $\downarrow$	Correct $\uparrow$	F1 $\uparrow$	Correct $\uparrow$	F1 $\uparrow$	Outdated $\downarrow$	Correct $\uparrow$	F1 $\uparrow$
Baseline	Vanilla	–	–	100	5.03	11.97	–	–	100	5.17	11.09
Baseline	RAG with Golden Context	67.02	66.28	6.50	76.03	86.55	69.44	67.36	5.01	79.36	88.78
SFT	Full-FT	35.19	11.16	5.53	17.44	14.38	38.40	11.82	6.10	18.16	14.40
SFT	LoRA	29.54	9.64	7.74	12.74	11.92	31.53	9.81	8.45	11.16	10.39

Table 2: Performance comparison between InternVL2.5-8B and Qwen2.5-VL-7B.

Panel of Table 3 — Table 3: General capability results of InternVL 2.5. Red values denote degradation, with darker shades indicating greater severity.

Panel of Table 4 — Table 4: General capability results of Qwen2.5 VL. Red values denote degradation, with darker shades indicating greater severity.

Figure 4: Consistency analysis of models trained with different methods. Left: Cross-modal transfer. Right: Decision consistency.

Figure 5: Case Study. Left: updated knowledge case. Right: unknown knowledge case.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Categories

Fine Grained Types

Updated

Common Plant, Bird, Tree, Flower, Church, Butterfly, Architectural Structure, Other, Castle, Moth, Shrub, Museum, Park, Beetle, Dragonfly, Wasp, Stadium, Bridge, Herb, Temple, Bee, Orchid, Fern, Monument, Residential Building, Lake, Mountain, Monastery, Palace, Cactus, Mosque, Square, Lighthouse, Coast, Skyscraper, Theater, Transportation Site, Hotel, Tower, Island, Insect Larvae, Moss and Lichen, Special Building, Historic Site, Fruit, Entertainment Site, Military Building, Reptile, Fly, Mosquito, Ocean, Ant, Tomb Architecture, Memorial Facility, Gate, Waterfall, Special Landform, Wilderness, Market Building, Protected Area, Industrial Site, Historic Area, Water Structure, Train Station, Synagogue, Library, Station Building, Ladybug, River, Fountain, Bird Extension, Amphitheater, Factory, School, Market, Building Component, Road Facility, Religious Site, Zoo, Beach Resort, Aqueduct, Fungus, Volcano, Tunnel, Cave, Amusement Park, Geographical Feature, Dam, Port Building, Road, Astronomical Facility, Port, Peninsula, Mollusk, Agricultural Site, Conference Facility, Fish, Human Related, Algae, Special Transportation, Fjord, Pavilion, Shopping Mall, Arthropod, City Wall, Special Plant Structure, Waterbody, Wetland, Specialized Plant, Mammal, Amphibian, Aquatic Mammal, Sports Facility, Desert, Cable Car, Ship, Cinema, Bell Tower, Colonnaded Building, Geothermal Feature, Natural Arch, Waterway Facility, Golf Course, Train, Valley, Transportation Hub, Aquatic Plant, Stone Formation, Public Building, Bat, Restaurant Building, Mangrove, Grasshopper, Crustacean, Wind Turbine, Airport, Race Track, Racecourse, Ski Resort, Campsite, Car, Pedestrian Street, Bus Station, Airport Building, Bicycle, Subway, Glacier, Coral Reef, Bus, Hot Air Balloon, Forest, Airplane, Parking Lot, Plain, Hospital, Worm

Unknown

Politician, Song, Musician, Company, Actor, Singer, Election, Book, Manga, Boxer, Anime, Soundtrack, Clergy, Award, Train, Tournament, Author, Car, Journalist, Lawyer, Smartphone, Government, Cricketer, Dinosaur, Festival, Software, Model, Athletics, Gymnast, Stadium, Aircraft, Protest, Diplomat, Ship, Warship, Music Album, Podcast, Wrestling, Bank, Documentary, Restaurant, Airline, Airport, Beverage, Brand, Bridge, Church, Concert, Hotel, Weightlifting, School, Video Game, Camera, Sculpture, Spacecraft, Weapon, Artificial Intelligence, Museum, Helicopter, Hospital, League, Motorcycle, Movie, Newspaper, Tablet, Tennis, Toy, Truck, Website, Computer, Cosmetics, Magazine, Mammal, Painting, Sports Team, Supermarket, TV Series, Airshow, Court, Doctor, Fictional Character, Food, Government Official, Jewelry, Plant, Router, Speaker, Sports Event, Weapon System, Aerospace, American Football Player, Animated Character, Architecture, Bakery, Baseball, Baseball Player, Basketball, Basketball Player, Bicycle, Bird, Board Game, Book Type, Cable Car, Car Manufacturer, Chess Player, Clothing, Consumer Electronics, Cycling Race, Election Results, Electronic Component, Electronics Manufacturer, Fashion Item, Film Director, Film Franchise, Film Studio, Film Type, Football, Football Kit, Football Player, Furniture, Game Character, Game DLC, Game Developer, Headphones, Historical Site, Hockey Player, Ice Cream, Ice Hockey, Language Vocabulary, Legal Case, Literature, Medical Equipment, Medical System, Military Base, Military Branch, Military Personnel, Military Vehicle, Mobile App, Motorsport Event, Mountain, Music Festival, Music Genre, Music Style, Non Fiction Book, Online Platform, Other, Payment Service, Payment System, Photography, Police, Political Party, Port, Printer, Race Track, Racing Driver, Radio Program, Retail Store, Road, Rugby Player, Shopping Mall, Snooker Player, Special Sport, Special Vehicle, Sports Season, Sports Team Season, TV Channel, TV Episode, TV Program, TV Season, Taxi, Tennis Player, Trade Center, Train Station, Volleyball, Volleyball Player

Appendix

Table 5: The fine-grained types in MMKU-Bench.

Panel of Figure 6 — Figure 6: Word Cloud Distributions of MMKU-Bench

Method	InternVL2.5-8B					Qwen2.5-VL-7B
	Updated			Unknown		Updated			Unknown
	Correct $\uparrow$	F1 $\uparrow$	Outdated $\downarrow$	Correct $\uparrow$	F1 $\uparrow$	Correct $\uparrow$	F1 $\uparrow$	Outdated $\downarrow$	Correct $\uparrow$	F1 $\uparrow$
Vanilla	-	-	100	5.03	11.97	-	-	100	5.17	11.09
RAG with Golden Context	67.02	66.28	6.50	76.03	86.55	69.44	67.36	5.01	79.36	88.78
Full - FT M→M	35.19	11.16	5.53	17.44	14.38	38.40	11.82	6.10	18.16	14.40
Full - FT M→T	19.78	6.90	5.99	13.71	11.68	19.89	6.73	7.29	13.57	11.26

Appendix

Table 6: Cross-modal performance across different methods.

InternVL2.5-8B
Method	Updated										Unknown
	Avg		Biology		Architecture		Landmark		Natural landscape		Avg		Celebrity		Film		Album		Sports Event
	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$	CEM $\uparrow$	F1 $\uparrow$
Vanilla	-	-	-	-	-	-	-	-	-	-	5.03	11.97	5.04	11.17	9.85	12.43	6.11	12.65	5.30	11.93
Full-FT	35.19	11.16	32.66	10.15	42.86	13.66	40.15	13.64	35.07	11.15	17.44	14.38	16.87	13.11	16.13	13.28	17.04	15.83	15.23	15.77
LoRA	29.54	9.64	27.79	8.98	33.40	10.89	32.69	11.15	32.12	10.30	12.74	11.92	11.93	10.55	14.40	12.79	9.42	11.17	12.76	14.76

Appendix

Table 7: Performance Comparison of Different Methods Across Categories. The yellow cells indicate the method achieving the best average performance, red cells denote the best-performing results within each sub-category, and blue cells represent the worst performance.

Figure 7: Case Study: Multimodal Knowledge Injection with Multimodal Evaluation

Figure 8: Case Study: Multimodal Knowledge Injection with Text-based Evaluation

Figure 9: Case Study: Text-based Knowledge Injection with Multimodal Evaluation

Figure 10: Case Study: Evaluation on Multiple-choice Question Answering

Figure 11: Prompt for updated knowledge editing.

Figure 12: Prompt for self-checking and verification.

Figure 13: Prompt for paraphrasing.

Figure 14: Prompt for entity replacement.

Figure 15: Prompt for multiple-choice question construction.

Figure 16: Prompt for question–answer generation.

Multimodal Knowledge Editing

See all 10 papers

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
Jul 2, 2026Yuanzhi Liu, Shousheng Zhao, Bo Zhou +2VLM Evaluation Benchmark Construction
1Beijing University of Posts and Telecommunications, Beijing, China
When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs
Apr 3, 2026Linyu Li, Zhi Jin, Yichi Zhang +5Multimodal KGs Continual Learning
Key Laboratory of High Confidence Software Technologies (PKU), Ministry of Education School of Computer Science, Peking University Beijing, China · School of Computer Science Zhejiang University Hangzhou, China · School of Cyber Science and Engineering Wuhan University Wuhan, China +1
Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation
Aug 4, 2026Chunlin Liu, Junnian Chen, Haitong Jiang +7VLM Evaluation VLM Unlearning
1Shenzhen University · 2Hebei University of Technology · 3Ludwig Maximilian University of Munich +1

Jul 2, 2026J/K move · Enter open · S save

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

Yuanzhi Liu, Shousheng Zhao, Bo Zhou, Kongming Liang+1

1Beijing University of Posts and Telecommunications, Beijing, China

VLM Evaluation Benchmark Construction Benchmark Design Multimodal Model Evaluation

Open paper AskPDF arXiv

Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at https://github.com/PRIS-CV/MMBench-Live.

Abstract

Multimodal Knowledge Editing

Explore similar work

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs

Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation