ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering

Anoop Senthil

cs.CLAug 27, 2026

ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering

Authors: Anoop Senthil

Organizations: School of Electronic Engineering and Computer Science Queen Mary University of London London, United Kingdom

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and region-level representations with the LLM's embedding space. The image bridge maps final transformer block features into image tokens. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, into K region tokens for every bounding box. ReVA uses a detector stack that supplies automatic zero-shot bounding boxes that are both question-agnostic and question-dependent, using RAM++ (Recognize Anything Model), spaCy, and Grounding DINO. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene-level context and fine-grained regional evidence when answering questions. Evaluated on VQAv2, MMBench, POPE, and SEED-Bench, ReVA achieves 82.85% mean F1 on POPE, compared with 81.14% for an image-token baseline without region tokens. These results demonstrate that explicit region-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs.

Figures & tables

Fig. 1: Overview of ReVA. The Vision Transformer is CLIP ViT-L/14. Image bridge: final transformer block features pass through the Image Feature Projector to image tokens. Region bridge: intermediate transformer block features are enriched (CoordConv, per-level projection, multi-head self-attention), cropped with RoI Align using RAM++ (Recognize Anything Model) tags localised by Grounding DINO, then pooled and mapped by the Region Feature Projector to region tokens.

Stage	Objective	Trainable module(s)	Training data
1	Image–text alignment	Image bridge (Image Feature Projector)	LLaVA-Pretrain (LCS-558K) †
2	Region–description alignment	Region bridge (per-level projection, multi-head self-attention, multi-token attention pooling, Region Feature Projector)	COCO, RefCOCO/+/g, Visual Genome, GRIT-20M shard 0 ( $\leq$ 250K rows) †
3	Grounded VQA instruction tuning	LoRA adapters (Qwen)	Visual7W (35%), GQA (30%), VQAv2 (15%), VCR (10%), A-OKVQA (10%) †‡

Table I: Overview of the three-stage ReVA training procedure. † Stage 1: pHash (Hamming

\leq 4

) against COCO and Visual Genome; flagged pairs treated as false positives; full 558,128-image pool retained. Stage 2: pHash (Hamming

\leq 4

) against POPE, VQAv2 test2015, MMBench, and SEED-Bench. Stage 3: hard-ID exclusion then pHash against the same eval image union. ‡ Stage 3 region crops use up to 20 boxes per image across all sources: VQAv2 and A-OKVQA use COCO train2014 ground-truth boxes; GQA, Visual7W, and VCR use their own box annotations.

Hyperparameter	Stage 1	Stage 2	Stage 3
Trainable module(s)	Image bridge	Region bridge	Qwen LoRA adapters ( $q/k/v/o$ , gate/up/down)
Epochs / optimiser steps	1 epoch	3 epochs	10,000 steps
Per-device / micro-batch size	16	16	1
Gradient accumulation	16	8	16
Effective batch size	256	128	16
Peak learning rate	$10^{-3}$	$7{\times}10^{-5}$	$10^{-4}$

Table II: Principal hyperparameters for the three-stage ReVA training pipeline.

Model	Backbone	GQA val	VQAv2 val	GQA test-dev	VQAv2 test-dev
ReVA (image tokens) ∗	CLIP-L/14 224	39.06	37.24	–	–
ReVA (image tokens) ∗	CLIP-L/14 336	42.96	40.99	39.74	40.22
ReVA (image tokens) ∗	DINOv2-L/14	37.84	35.81	–	–

Table III: Image backbone ablation (Stage 1, image alignment only). All models use Qwen2.5-7B-Instruct. ∗ Stage 1 zero-shot: frozen CLIP/Qwen; only the image bridge (Image Feature Projector) is trained.

Model	LLM	VQAv2 test-dev	VQAv2 test-std	MMBench	SEED-Bench
BLIP-2	OPT-6.7B	82.19	82.30	—	—
InstructBLIP ∗	Vicuna-7B	—	—	36.0 †	58.8 ♯
LLaVA-1.5	Vicuna-7B	78.5	—	64.3 ∗	66.1 ♯
LLaVA-1.5	Vicuna-13B	80.0	—	67.7 ∗	68.2 ♯
Shikra	Vicuna-7B	77.36	77.51	—	—
Ferret-v2	Vicuna-7B	81.5	—	—	—

Table IV: Comparison with state-of-the-art multimodal LLMs on VQAv2, MMBench, and SEED-Bench. Values are accuracy (%); SEED-Bench is image-only (dims 1–9), not SEED All.

\Delta

is the difference between ReVA with image and region tokens and ReVA with image tokens only. ∗ Benchmark zero-shot; unmarked VQAv2 rows observe VQAv2 train images. † LLaVA-1.5 re-evaluation ( Liu et al. 2024a ) ; § DeepSeek-VL ( Lu et al. 2024 ) ; ‡ DeepSeek-VL2 ( Wu et al. 2024 ) ; ♯ SEED-Bench ( img ) from LLaVA-1.5 ( Liu et al. 2024a ) Table 4.

Model	LLM	POPE
Model	LLM	Random	Popular	Adversarial	Mean
LLaVA-1.5 ∗	Vicuna-7B	87.3	86.1	84.2	85.9
LLaVA-1.5 ∗	Vicuna-13B	87.1	86.2	84.5	85.9
Shikra ∗	Vicuna-7B	86.19	83.16	82.49	83.9
Ferret ∗	Vicuna-7B	89.76	84.21	82.00	85.3
Ferret-v2 ∗	Vicuna-7B	—	—	—	87.8

Table V: POPE F1 (%) on COCO (random / popular / adversarial; Mean is the unweighted average). ∗ Benchmark zero-shot; § from DeepSeek-VL ( Lu et al. 2024 ) .

\Delta

is the difference between ReVA with image and region tokens and ReVA with image tokens only. Baselines from ( Liu et al. 2024a ; You et al. 2024 ; Zhang et al. 2024 ; Lu et al. 2024 ) .

Benchmark	$n$	Base%	Final%	$\Delta$	$p_{t}$	Sig?
POPE	9,000	83.63	84.81	$+1.18$	$3.165{\times}10^{-6}$	Yes †
MMBench	4,329	77.96	77.32	$-0.65$	$0.1615$	No
SEED-Image	14,232	66.57	66.75	$+0.18$	$0.5492$	No
VQAv2	—	72.89	73.28	$+0.39$	—	—

Table VI: Paired

t

-test (Base: image tokens; Final: image + region). † Significant at

\alpha{=}0.01

Benchmark	$n$	$b_{01}$	$b_{10}$	$p_{\mathrm{McN}}$	Sig?
POPE	9,000	206	312	$3.961{\times}10^{-6}$	Yes †
MMBench	4,329	214	186	$0.177$	No
SEED-Image	14,232	929	955	$0.5646$	No

Table VII: McNemar test (Base: image tokens; Final: image + region) on discordant pairs.

b_{01}

: base correct, final incorrect (region tokens flipped a previously correct answer);

b_{10}

: base incorrect, final correct (region tokens fixed a previously wrong answer). † Significant at

\alpha{=}0.01

Stage	Original	Kept	Removed
1 (LCS-558K)	558,128	558,128	0.00%
2 (region corpora)	2,951,772	2,940,403	0.39%
3 (VQA pool)	1,704,599	1,699,447	0.30%

Table VIII: Training-pool size after decontamination.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

Jun 22, 2026

Faithful Grounded Visual Reasoning via Learned Proxy-Tokens

Multimodal Large Language Models (MLLMs) have achieved remarkable success in Visual Question Answering (VQA), yet their "black-box" nature hinders deployment in critical domains. Grounded Visual Reasoning (GVR) approaches attempt to improve interpretability by explicitly couple textual rationales with visual grounding information, which are typically textual coordinates. This mechanism lacks a learnable semantic link to the visual features, often resulting in a semantic-spatial gap where the model hallucinates coordinates that do not correspond to image evidences. In this work, we introduce Composer, a MLLM that leverages a novel visual grounding mechanism based on learned proxy-tokens to promote faithful interpretability. These discrete symbolic pointers explicitly index the image latent space, allowing the model to manipulate visual regions as addressable, semantically manipulable sets. To rigorously validate our novel grounding mechanism, we constructed ComposerGCoT, a dataset synthesized to enable holistic assessment of reasoning consistency and grounding accuracy. Experimental results indicate that Composer achieves performance parity with its coordinate-based counterpart in final answer accuracy, while improving visual grounding accuracy by +9.0 points. By demonstrating that discrete proxy-tokens capture spatial semantics more effectively than typical textual coordinates, we establish that visual grounding mechanisms with learnable semantic links represent a promising path toward trustworthy and reliable MLLMs.

Tom Hodemon, Mohamed Chaouch, Aboubacar Tuo +1

Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France

Visual Question Answering Visual Reasoning Vision-Language Grounding VLM Reasoning Visually Grounded Reasoning Visual Grounding

Ask PDF

May 5, 2026

Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

With advances in multimodal research and deep learning, Multimodal Large Language Models (MLLMs) have emerged as a powerful paradigm for a wide range of multimodal tasks. As a core problem in vision-language research, Visual Question Answering (VQA) has increasingly employed MLLMs to improve performance, particularly in open-domain settings where external knowledge is essential. In this work, we aim to further enhance retrieval-based VQA by more effectively integrating MLLMs with structured reasoning and knowledge acquisition. We introduce a logical prompting strategy that fuses Chain-of-Thought (CoT) reasoning with Visual Question Decomposition (VQD), termed CoVQD, to guide retrieval toward more accurate and relevant knowledge for MLLM inference. Building on this idea, we propose a new framework, CoVQD-guided RAG (CgRAG), which enables MLLMs to access more comprehensive and coherent external knowledge while benefiting from structured visual-text reasoning guidance, thereby improving generalization and reliability in complex cross-domain VQA scenarios. Extensive experiments on E-VQA, InfoSeek, and OKVQA benchmarks demonstrate the effectiveness of the proposed method.

Quanxing Xu, Ling Zhou, Xian Zhong +3

School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3

Multimodal RAG Visual Question Answering Multimodal Large Language Models Multimodal CoT Reasoning Knowledge-Based VQA

Ask PDF

Oct 19, 2026

Look Before You Reason: Implicit Visual Thinking for Efficient Multimodal Reasoning

Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet they remain limited on tasks requiring fine-grained understanding of small or easily overlooked image regions. Recent thinking with images approaches mitigate this limitation by allowing models to inspect images through operations such as zooming, cropping, or code execution. While effective, their multi-turn interactions with VLMs introduce redundant token computation and slow inference, and separate region inspection from the reasoning step that depends on it. To address these issues, we propose Look Before You Reason (LoRe), an implicit visual thinking framework for efficient visual reasoning within a single-turn VLM interaction. Given an image and a question, LoRe predicts the most relevant image region, aligns it with the corresponding image patch features, and strengthens the associated key-value (KV) cache entries, enabling subsequent decoding to directly exploit the selected visual cues without additional visual inputs or explicit tool calls. We further employ reinforcement learning to guide the VLM toward identifying informative image regions and leveraging them during reasoning. To support fine-grained evaluation of multimodal reasoning, we introduce COCO-ObjQA, a large-scale benchmark for object-centric multimodal reasoning. Experiments on three challenging multimodal benchmarks demonstrate that LoRe consistently improves VLM reasoning performance with low additional inference overhead.

Zerui Chen, Changrui Chen, Fei Ni +1

MVP Lab, Imperial College London

Multimodal Reasoning Efficient Multimodal Inference Latent Visual Reasoning

Ask PDF

Abstract

Visual Question Answering

Explore similar work

Faithful Grounded Visual Reasoning via Learned Proxy-Tokens

Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

Look Before You Reason: Implicit Visual Thinking for Efficient Multimodal Reasoning