MMCOMET: A Large-Scale Multimodal Commonsense Knowledge Graph for Contextual Reasoning

Eileen Wang; Hiba Arnaout; Dhita Pratama; Shuo Yang; Dangyang Liu; Jie Yang; Josiah Poon; Jeff Pan

cs.AIMar 1, 2026

MMCOMET: A Large-Scale Multimodal Commonsense Knowledge Graph for Contextual Reasoning

Authors: Eileen Wang, Hiba Arnaout, Dhita Pratama, Shuo Yang, Dangyang Liu, Jie Yang, Josiah Poon, Jeff Pan,

Organizations: University of Sydney Sydney, Australia · Technische Universität Darmstadt Darmstadt, Germany · University of Melbourne Melbourne, Australia · University of Edinburgh Edinburgh, United Kingdom ·

Abstract

We present MMCOMET, the first multimodal commonsense knowledge graph (MMKG) that integrates physical, social, and eventive knowledge. MMCOMET extends the ATOMIC2020 knowledge graph to include a visual dimension, through an efficient image retrieval process, resulting in over 900K multimodal triples. This new resource addresses a major limitation of existing MMKGs in supporting complex reasoning tasks like image captioning and storytelling. Through a standard visual storytelling experiment, we show that our holistic approach enables the generation of richer, coherent, and contextually grounded stories than those produced using text-only knowledge. This resource establishes a new foundation for multimodal commonsense reasoning and narrative generation.

Figures & tables

Figure 1. An example of automated visual storytelling: Baseline : Family members enjoyed leisurely moments together. Grandpa shared memories during the trip ; Ours : Family spent the day relaxing in the boat, enjoying beer. Grandpa took the grandson on his lap as he drove, with the grandson’s parents watching nearby.

MMKG	Domain or Focus	Size	Multimodal	Image Source
IMGpedia ( Ferrada et al., 2017 )	Ency	442M	E, R	WMC
ImageGraph ( Oñoro-Rubio et al., 2017 )	Ency	560K	E	WSE
MMKG ( Liu et al., 2019 )	Ency	814K	E	WSE
Richpedia ( Wang et al., 2020 )	Ency, Geo	172M	E, R	WSE, WP
VisualSem ( Alberts et al., 2021 )	Ency, Multi-L	1.5M	E	WP, ImageNet
MMEKG ( Ma et al., 2022 )	CS (Ev)	934M	E, T	imSitu

Table 1. Comparison of existing MMKGs with MMCOMET. Ours uniquely covers domains (social, physical, eventive) with nearly 1M tuples (Ency: Encyclopedic; Geo: Geographic; Multi-L: Multilingual; Soc: Social; Phys: Physical; Ev: Eventive; E: Entity; R: Relation; T: Tuples; CS: Commonsense; WSE: Web Search Engine; WP: Wikipedia; WMC: Wikimedia Commons; OID: open-source image datasets.)

Figure 2. A small subset of MMCOMET (top) and the Construction Pipeline (bottom).

	Web (WSE)		Open-source Data (OID)		Both
Physical	FM+PM	FM	FM+PM	FM	FM+PM	FM
ObjectUse	100	73	96	76	98	74.5
AtLocation	100	92.9	100	100	100	96.4
MadeUpOf	100	100	89.3	89.3	94.1	94.1
HasProperty	92.6	92.6	88.5	84.6	90.6	88.7
CapableOf	95.8	87.5	95.8	70.8	95.8	79.2

Table 2. Human evaluation results showing the percentage of commonsense phrases whose retrieved images were judged as full (FM) or partial matches (PM). Web search (WSE) yields higher matching accuracy than open-source image datasets (OID), particularly for abstract social and eventive relations.

Relation	Avg. Concreteness	Relation	% Heads Searched	Relation	% Tails Searched
React (94228)	2.95	NotDesires	2.82	AtLocation	25.52
Attr (115815)	2.98	Desires	2.85	MadeUpOf	36.47
Causes (376)	3.29	AtLocation	18.78	isAfter	48.98
Intent (49312)	3.40	MadeUpOf	20.03	isBefore	53.50
Effect (113494)	3.53	HasProperty	22.77	CapableOf	59.55
Reason (334)	3.55	CapableOf	23.98	HinderedBy	64.72

Table 3. The left sub-table shows the average concreteness scores for each commonsense relation (sample size in brackets) in ascending order. The second and third sub-tables show the proportion of heads and tail phrases across each relation that required web search. Green, orange and blue highlighting indicate social, event and physical relations, respectively.

Model	R-VG	R-C	R-NR	RoViST (R)	SPICE (S)	BLEURT (B)	MoverScore (M)	UNION (U)	Perplexity	R+S+B+M+U	Story L.
SRL-pmi	70.4	72.8	91.6	78.3	11.5	34.7	56.0	75.9	14.7	256.3	51.2
TGCN-SRL-cos	70.3	72.3	90.5	77.7	10.9	34.9	56.0	84.0	14.9	263.4	52.3
VILT-pmi	71.3	76.4	92.1	79.9	10.2	35.1	54.4	82.2	13.2	261.8	51.2
VILT-cos	70.2	76.3	94.2	80.2	9.3	34.3	52.1	83.3	12.1	259.2	50.1
LLaVa-cos	72.3	78.1	96.1	82.2	9.5	34.1	53.9	84.2	10.5	263.6	51.9
LLaVA-cos-MMCOMET S	73.3	78.4	95.8	82.5	9.9	35.2	54.2	82.6	10.8	264.4	50.8

Table 4. Automatic evaluation results of visual storytelling. MMCOMET-enhanced models (bottom rows) outperform baselines (SRL-pmi and TGCN-SRL-cos) in visual grounding, coherence, and overall story quality metrics, demonstrating the benefit of multimodal commonsense knowledge (for Perplexity , lower score is better.)

Model	VQA	Image Captioning
Model	Acc.	BLEU	CIDEr	BLEURT
Qwen2.5-VL	0.630	0.481	0.258	0.357
Qwen2.5-VL+MMCOMET	0.642	0.498	0.296	0.363

Table 5. Automatic evaluation results of VQA, using

\sim

189k validation data from VQAv2, and Image Captioning tasks using Flickr30k. MMCOMET-enhanced model (bottom row) outperforms the baseline in both tasks and all metrics, demonstrating the benefit of multimodal commonsense knowledge.

Image	Baseline (SRL-pmi)	Ours (LLaVA-cos-MMCOMET S+I )
	Merchants set up their booths.	Merchants set up vibrant booths filled with handmade goods .
	It was a family gathering.	It was a family gathering in a restaurant where everyone enjoyed various delicacies .
	Celebrated with a birthday cake.	Blowing out the candles of her birthday cake, she made a wish for happiness and adventure .

Table 6. Selected story frames and their associated text to showcase the effect of MMCOMET in visual story telling.

Panel of Table 7 — Table 7. Selected visual question-answer exemplars to showcase the effect of MMCOMET as additional contexts for VQA tasks. (GT: Ground Truth, Baseline: Qwen2.5-VL Only, Ours: Qwen2.5-VL-MMCOMET-augmented)

Image	Baseline	Ours
	A woman cradles a sleeping child, both appearing calm and serene.	A woman tenderly holds a sleeping child, symbolizing the comfort and care provided by a mother.
	A person walks down a wet brick sidewalk under a yellow checkered umbrella, wearing a black coat and tall boots.	A person walks down a wet sidewalk under a yellow umbrella, protecting themselves from the rain.
	A young boy wearing a blue cap looks through a telescope, focusing intently on the view.	A young boy observes through a telescope, learning about the wonders of the cosmos.

Table 8. Comparison of image captions from baseline and MMCOMET-enhanced models to provide the impact of MMCOMET in image captioning tasks. (GT: Ground Truth, Baseline: Qwen2.5-VL Only, Ours: Qwen2.5-VL-MMCOMET-augmented)

Visual Storytelling

See all 23 papers

Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment
Mar 17, 2026Pengyu Zhang, Klim Zaporojets, Jie Liu +2Multimodal KGs KG Completion
University of Amsterdam, Amsterdam, The Netherlands · Aarhus University, Aarhus, Denmark · Amazon AGI, Seattle, USA
MMKG-R1: Constructing Multimodal Knowledge Graphs via Reinforcement Learning
Oct 14, 2026Zongyu Wu, Yilong Wang, Xiaochen Wang +5KG Construction Multimodal KGs
The Pennsylvania State University
MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation
Apr 4, 2026Sijun Dai, Qiang Huang, Xiaoxing You +1Cross-Modal Learning Multimodal RAG
School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · School of Computer Science, Hangzhou Dianzi University

Mar 17, 2026J/K move · Enter open · S save

Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment

Pengyu Zhang, Klim Zaporojets, Jie Liu, Jia-Hong Huang+1

University of Amsterdam, Amsterdam, The Netherlands · Aarhus University, Aarhus, Denmark · Amazon AGI, Seattle, USA

Multimodal KGs KG Completion

Open paper AskPDF arXiv

Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images, an automatic data-centric enrichment pipeline with optional human auditing. This pipeline operates in three stages: (1) large-scale retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG datasets and multiple baseline models, we observe consistent gains (up to 7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements (201.35% MRR and 333.33% Hits@1). Additionally, we release a lightweight Text-Image Consistency Check Interface for optional targeted audits, improving description quality and dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code, datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images.

Abstract

Visual Storytelling

Explore similar work

Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment

MMKG-R1: Constructing Multimodal Knowledge Graphs via Reinforcement Learning

MG2^22-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

MG $^2$ -RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation