Enhancing Zero-shot Commonsense Reasoning by Integrating Visual Knowledge via Machine Imagination

Hyuntae Park; Yeachan Kim; SangKeun Lee

cs.AIMar 5, 2026

Enhancing Zero-shot Commonsense Reasoning by Integrating Visual Knowledge via Machine Imagination

Authors: Hyuntae Park, Yeachan Kim, SangKeun Lee

Organizations: Department of Artificial Intelligence, Korea University, 145 Anam-ro, Seungbuk-gu, Seoul, 02841, Republic of Korea · Department of Computer Science and Engineering, Korea University, 145 Anam-ro, Seungbuk-gu, Seoul, 02841, Republic of Korea

Abstract

Recent advancements in zero-shot commonsense reasoning have empowered Pre-trained Language Models (PLMs) to acquire extensive commonsense knowledge without requiring task-specific fine-tuning. Despite this progress, these models frequently suffer from limitations caused by human reporting biases inherent in textual knowledge, leading to understanding discrepancies between machines and humans. To bridge this gap, we introduce an additional modality to enrich the reasoning capabilities of PLMs. We propose Imagine (Machine Imagination-based Reasoning), a novel zero-shot commonsense reasoning framework that supplements textual inputs with visual signals from machine-generated images. Specifically, we enhance PLMs with the ability to imagine by embedding an image generator directly into the reasoning pipeline. To facilitate effective utilization of this imagined visual context, we construct synthetic datasets designed to emulate visual question-answering scenarios. Through comprehensive evaluations on multiple commonsense reasoning benchmarks, we demonstrate that Imagine substantially outperforms existing zero-shot approaches and even surpasses advanced large language models. These results underscore the capability of machine imagination to mitigate reporting bias and significantly enhance the generalization ability of commonsense reasoning models

Figures & tables

Figure 1: An example from the PIQA dataset ( Bisk et al., 2020 ) with model predictions. Imagine performs reasoning by leveraging machine-generated images to enhance understanding of the question.

Panel of Figure 2 — Figure 2: Overall procedures for (a) constructing a Synthetic VQA dataset and (b) the inference/optimization phase of Imagine (ours) using the given QA pair. The process starts with the textual pair consisting of a question and its answers, followed by the generation of visual signals (i.e., imagination) conditioned on the question. The two distinct features from visual and textual models are then utilized to derive a comprehensive prediction.

Figure 3: Examples of the Synthetic VQA $\boldsymbol{+}$ dataset. Our dataset is sourced from AbstractATOMIC ( Wang et al., 2023 ) , VCR ( Zellers et al., 2019 ) , and Sherlock ( Hessel et al., 2022 ) . Bold indicates the correct answer, and Underline denotes the generated image caption.

	Train	Dev	Total
# Images generated from AbsAT	18,838	1,695	20,533
# QA pairs from AbsAT	486,778	46,238	533,016
# Images from VCR	80,418	9,929	90,347
# QA pairs from VCR	212,923	26,534	239,457
# Total Images	99,256	11,624	110,880
# Total QA pairs	699,701	72,772	772,473

Table 1: Statistic of Synthetic VQA dataset.

Figure 4: Examples of less plausible data filtered during the construction of Synthetic VQA

\boldsymbol{+}

. We measured the commonsense plausibility of (question, correct answer) pairs using the VERA model ( Liu et al., 2023b ) . Bold indicates the correct answer.

	Train	Dev	Total
# Images generated from AbsAT	17,674	1,612	19,286
# QA pairs from AbsAT	413,013	38,199	451,212
# Images from VCR	74,025	9,146	83,171
# QA pairs from VCR	162,480	20,137	182,617
# Images from Sherlock	89,762	6,619	96,381
# QA pairs from Sherlock	273,627	19,695	293,322

Table 2: Statistic of Synthetic VQA+ dataset.

Imagine	GPT-2-L	RoBERTa-L	DeBERTa-v3-L
# Params.	792M	362M	443M
# Trainable Params.	7.9M	8.4M	8.4M
Training Time	70h	30h	80h
Image Encoder	CLIP-ViT-L/14
Batch Size	8, 16, 32 , 64
Learning Rate	7e-6, 1e-5 , 3e-5

Table 3: Detailed training settings for Imagine . Bold indicates the chosen hyperparameter.

Pre-trained Language Models
Method	KB	$\alpha$ NLI	CSQA	PIQA	SIQA	WG	Avg.
GPT-2-L ( Radford et al., 2019 )	-	56.5	41.4	68.9	44.6	53.2	52.9
RoBERTa-L ( Liu et al., 2019 )	-	65.6	45.0	67.6	47.3	57.5	56.6
DeBERTa-v3-L ( He et al., 2023 )	-	59.9	25.4	44.8	47.8	50.3	45.6
Self-talk ( Shwartz et al., 2020 )	-	-	32.4	70.2	46.2	54.7	-
COMET-DynGen ( Bosselut et al., 2021 )	AT	-	-	-	50.1	-	-

Table 4: Zero-shot evaluation results on five commonsense reasoning tasks (Accuracy %). Bold and Underline indicate the best and second-best results, respectively. AT, CN, WD, WN, and AbsAT refer to ATOMIC, ConcetNet, WikiData, WordNet, and AbstractATOMIC. The results of the large language models including GPT series are taken from Wang et al. (2024) . (Retrieval) refers to the model inference using retrieved images instead of generating them, as described in Section 3.4 .

Method	KB	QASC	SciQ	ARC-E	ARC-C
SMLM ( Banerjee and Baral, 2020 )		26.6	-	33.4	28.4
GPT-Neo-2.7B ( Black et al., 2021 )	-	29.6	64.0	49.6	31.8
GPT-J-6B ( Wang and Komatsuzaki, 2021 )	-	36.3	73.2	44.1	34.8
CAR-RoBERTa-L ( Wang et al., 2023 )	AbsAT	56.7	60.7	57.0	36.5
CAR-DeBERTa-v3-L ( Wang et al., 2023 )	AbsAT	70.0	76.9	75.3	53.2
OPT-30B ( Zhang et al., 2022 )	-	39.7	72.7	58.2	34.8

Table 5: Zero-shot evaluation results on four science question-answering tasks (Accuracy %). Bold and Underline indicate the best and second-best results, respectively. SMLM (*) used different KBs for the different benchmarks. (Retrieval) refers to the model inference using retrieved images instead of generating them, as described in Section 3.4 .

Figure 5: Examples of generated and retrieved images based on the input question. The first row shows cases where retrieved images are helpful for the inference, while the second row shows cases where retrieved images are not helpful.

	$\alpha$ NLI	CSQA	PIQA	SIQA	WQ	Avg.
DeBERTa-v3-L	59.9	25.4	44.8	47.8	50.3	45.6
+ AbstractATOMIC	79.7	72.6	79.0	68.0	74.3	74.7
+ VCR	82.2	74.0	80.7	66.3	76.7	76.0
+ Sherlock	83.0	72.9	81.2	67.6	78.1	76.6
+ Filtering	83.4	76.3	81.4	69.0	79.2	78.8

Table 6: Ablation results on Synthetic VQA

\boldsymbol{+}

. Bold indicates the best results. Filtering refers to the removal of implausible question-answer pairs using the VERA model ( Liu et al., 2023b ) (Section 3.2 ).

	AOKVQA	VCR	Synthetic VQA	Synthetic VQA $\boldsymbol{+}$	$\alpha$ NLI	CSQA	PIQA	SIQA	WG
Relevance	23.81	21.26	23.59	24.85	30.26	29.38	30.80	29.92	29.26

Table 7: Image-text relevance evaluation using CLIP model.

Imagine-DeBERTa-v3-L	KB	$\alpha$ NLI	CSQA	PIQA	SIQA	WG	Avg.
CLIP Retrieval	Synthetic VQA $\boldsymbol{+}$	83.3	76.2	81.3	69.0	79.2	77.8
Concept-aware Retrieval	Synthetic VQA $\boldsymbol{+}$	83.8	75.5	81.2	70.1	77.6	77.6

Table 8: Comparison of image-text retrieval approaches. CLIP retrieval is performed based solely on CLIP embedding similarities, whereas concept-aware retrieval enhances this process by integrating query-relevant knowledge from ConceptNet prior to similarity computation.

Figure 6: Comparison of model predictions and the correctness from Imagine and the existing model ( Wang et al., 2023 ) on five commonsense reasoning tasks.

Table 15

Method	SST-2	RTE	MNLI
DeBERTa-v3-L	49.1	50.5	32.0
CAR-DeBERTa-v3-L	56.2	54.9	32.0
Imagine -DeBERTa-v3-L	81.1	53.8	33.3

Table 11: Zero-shot evaluation results on natural language understanding tasks.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Comparison of generated images. The sentences are the queries used to generate the images.

Imagine	$\alpha$ NLI	CSQA	PIQA	SIQA	WG	Avg.
Text only	73.2	66.3	71.3	64.5	60.3	67.1
LDM ( $512\times 512$ )	73.2	66.3	71.9	64.3	60.6	67.3
DALL-E 3 ( $384\times 384$ )	74.5	66.8	71.9	64.3	60.6	67.6
DALL-E 3 ( $512\times 512$ )	74.7	67.5	72.3	64.3	61.2	68.0

Appendix

Table 12: Results of using various image synthesis models for evaluation. The numbers in parentheses indicate the image resolution.

Model	$\alpha$ NLI	CSQA	PIQA	SIQA	WG	Avg.
Parallel Adapter	74.7	67.5	72.3	64.3	61.2	68.0
Full	73.0	65.4	71.1	61.5	61.2	66.4

Appendix

Table 13: Evaluation results of Imagine with full fine-tuning (Full) and adapter tuning (Parallel Adapter). We use Synthetic VQA for pre-training.

RoBERTa-Large	$\alpha$ NLI	CSQA	PIQA	SIQA	WG	Avg.
LM	74.3	65.2	71.9	62.3	60.5	66.8
LM+LM	74.3	66.0	72.1	64.2	60.4	67.4
LM+ITM ( Imagine )	74.7	67.5	72.3	64.3	61.2	68.0

Appendix

Table 14: Results of two different ensemble methods. We use Synthetic VQA for pre-training.

Figure 8: Model accuracy variation with different ensemble weights. The optimal $w$ for each task is shown below the plots. The line in the middle indicates the average accuracy. We use Synthetic VQA for pre-training.

Inference	$\alpha$ NLI	CSQA	PIQA	SIQA	WG	Avg.
Ensemble	74.7	67.5	72.3	64.3	61.2	68.0
LM	74.1	66.9	71.8	63.8	61.1	67.1
ITM	71.7	63.1	68.3	59.8	59.4	64.0

Appendix

Table 15: Results of the different inference strategy (LM, ITM). These strategies are evaluated on RoBERTa-L. We use Synthetic VQA for pre-training.

LM	ITM	$\alpha$ NLI	CSQA	PIQA	SIQA	WG	Avg.
✓	✓	74.7	67.5	72.3	64.3	61.2	68.0
✓	-	74.3	65.2	71.9	62.3	60.5	66.8
-	✓	71.7	62.0	68.8	60.0	59.6	64.4
-	-	65.6	45.0	67.6	47.3	57.5	56.6

Appendix

Table 16: Ablation results on pre-training objective of Imagine . We use a RoBERTa-L as a backbone and Synthetic VQA for pre-training.

Figure 9: Randomly sampled examples from Imagine alongside the visualization of image attention from the Abductive NLI, CommonsenseQA, and PIQA validation sets. We use Synthetic VQA for pre-training.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 913 papers

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
Jun 25, 2026Sicheng Zhang, Muzammal Naseer, Binzhu Xie +5Compositional Reasoning Multimodal Reasoning
Khalifa University · University of Western Australia · The Chinese University of Hong Kong +2
Reasoning with Image Generation
Sep 14, 2026Nishad Singhi, Hector Garcia Rodriguez, Aditya Arora +2Tool Use in VLMs VLM Reasoning
Technical University of Darmstadt & hessian.AI
Belief-Aware VLM Model for Human-like Reasoning
Apr 5, 2026Anshul Nayak, Shahil Shaik, Yue WangVLM Reasoning Memory-Augmented VLMs
Mechanical Engineering Department, Clemson University.

Jun 25, 2026J/K move · Enter open · S save

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

Sicheng Zhang, Muzammal Naseer, Binzhu Xie, Naufal Suryanto+4

Khalifa University · University of Western Australia · The Chinese University of Hong Kong+2

Compositional Reasoning Multimodal Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Foundation Models Continual Learning for VLMs+1

Open paper AskPDF arXiv

CLIP and its variants are widely adopted visual backbones in multimodal systems, but their pretraining remains dominated by descriptive image-text alignment. As downstream applications increasingly demand visually grounded commonsense inference and compositional reasoning, it remains unclear whether CLIP-style encoders can support such reasoning without architectural changes. To address this, we present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates reasoning signals while preserving descriptive alignment, followed by category-structured reasoning supervision. To support this framework, we construct two complementary datasets and a benchmark: ReasonLite-42M, with open-form, visually verifiable reasoning captions; ReasonPro-16M, with category-specific reasoning supervision; and RCLIP-Bench for diagnostic evaluation of visually grounded reasoning. We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance. As a drop-in visual encoder for multimodal large language models such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional inference cost, demonstrating that structured reasoning supervision enhances the expressive capacity of CLIP-style visual representations. All datasets, models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.

Abstract

LLM Reasoning

Explore similar work

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

Reasoning with Image Generation

Belief-Aware VLM Model for Human-like Reasoning