SmoGVLM: A Small, Graph-enhanced Vision-Language Model

Debjyoti Mondal; Rituraj Singh; Subhadarshi Panda

cs.CVApr 15, 2026

SmoGVLM: A Small, Graph-enhanced Vision-Language Model

Authors: Debjyoti Mondal, Rituraj Singh, Subhadarshi Panda

Organizations: Samsung R&D Institute India-Bangalore

Abstract

Large vision-language models (VLMs) achieve strong performance on multimodal tasks but often suffer from hallucination and poor grounding in knowledge-intensive reasoning. We propose SmoGVLM, a small, graph-enhanced VLM that integrates structured knowledge with visual and textual modalities, using Graph Neural Networks. We investigate the effects of our method across a range of model sizes, from tiny (1.3B) to large (13B) models. Our results demonstrate that, when trained using our approach, a small model can achieve performance gains upto 16.24%, and surpass its larger counterparts, outperforming larger VLMs and strong fine-tuned baselines. These results highlight the potential of structured knowledge augmentation for efficient, smaller-scale multimodal reasoning systems.

Figures & tables

Figure 1: Overview of SmoGVLM. Language, image, and KG inputs are encoded, fused into a joint representation, and decoded into rationale and answer.

Sub-graph	Time	Similarity of top- $200$
extraction method	Taken	triples with Answer
QA-GNN	7 hrs	0.232 $\pm$ 0.10
Proposed	40 min	0.312 $\pm$ 0.14

Table 1: Comparison of sub-graph extraction methods. Our strategy is much faster and yields higher answer similarity.

Model	Size	Method	Acc. (%)
LLaVA-Tiny	$1.3$ B	LoRA	53.22
LLaVA-Tiny	$1.3$ B	FFT	62.72
SmoGVLM-Tiny	$1.3$ B	LoRA	69.46
SmoGVLM-Tiny	$1.3$ B	FFT	75.27
LLaVA	$7$ B	LoRA	69.41
LLaVA	$7$ B	FFT	69.42

Table 2: Performance on ScienceQA test split.

Model	Size	Method	Acc. (%)
LLaVA-Tiny	$1.3$ B	FFT	70.4
SmoGVLM-Tiny	$1.3$ B	FFT	70.7
LLaVA	$7$ B	FFT	79.3
SmoGVLM	$7$ B	FFT	79.5

Table 3: Performance on A-OKVQA validation split.

Figure 2: Similarity of top- $k$ triples to the correct answer.

Model	Size	Method	# Nodes	Acc. (%)
LLaVA	$7$ B	FFT	-	69.42
			50	83.94
SmoGVLM			100	86.65
			200	86.87

Table 4: Impact of sub-graph size on accuracy. Small graphs are sufficient, with performance saturating beyond 200 nodes.

Figure 3: A few qualitative samples showing the strengths and limits of structured knowledge with VLMs.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Distilling Visual Reasoning into Text Space
Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT Distillation VLM Distillation
University of California, Los Angeles · Cisco Research
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Feb 25, 2026Yongchang Zhang, Oliver Ma, Tianyi Liu +2Multimodal Reasoning VLM Reasoning
Southeast University · AIIA, Ministry of Education, China · University of Oxford
Mixture of Cognitive Experts in Large Vision-Language Models
Jul 12, 2026Robert Wijaya, Ngai-Man CheungVision-Language Models Large Vision-Language Models
Singapore University of Technology and Design (SUTD), Singapore

Sep 28, 2026J/K move · Enter open · S save

Distilling Visual Reasoning into Text Space

Wenhan Yang, Nilay Naharas, Ali Payani, Baharan Mirzasoleiman

University of California, Los Angeles · Cisco Research

CoT Distillation VLM Distillation Large Vision-Language Models VLM Reasoning

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher's attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42x faster to train than state-of-the-art baselines.

Abstract

VLM Reasoning

Explore similar work

Distilling Visual Reasoning into Text Space

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

Mixture of Cognitive Experts in Large Vision-Language Models