Fusing Visual and Textual Representations via Multi-layer Fusing Transformers for Vietnamese Visual Question Answering

Cong Phu Nguyen; Huy Tien Nguyen; Tung Le

cs.CVOct 1, 2026

Fusing Visual and Textual Representations via Multi-layer Fusing Transformers for Vietnamese Visual Question Answering

Authors: Cong Phu Nguyen, Huy Tien Nguyen, Tung Le

Organizations: Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Abstract

In recent decades, artificial intelligence has made significant progress in understanding and interacting with images. One of the important applications of this technology is Visual Question Answering (VQA), a research field that requires computers to understand and answer questions about images in a natural manner. Despite extensive research and development in VQA for English, there have been very few similar efforts made for other languages, especially Vietnamese. This gap presents a significant challenge and opportunity for the advancement of VQA technology in the Vietnamese language context. By bridging this gap, the field of Vietnamese VQA not only enriches the diversity of research in artificial intelligence but also enables practical applications in various domains, such as education, healthcare, and entertainment, catering to Vietnamese-speaking populations worldwide. Thus, the exploration and development of Vietnamese VQA systems hold immense potential for advancing both research and practical applications in the intersection of computer vision and natural language processing. In this paper, we propose a Multi-layer Fusing Transformer model utilizing a cross attention module to combine multiple modality features of images and texts from different layers in an aggregated representation. Our architecture allows us extract information from low level to high level. Through detailed experiments and ablation studies, our model achieves promising results against the competitive baselines in ViVQA dataset for Vietnamese language.

Figures & tables

Figure 1: Architecture of Cross-Attention

Figure 2: Multi-layer Fusing Transformer: use cross-attention on different layers of pre-trained language model and vision model

	Training	Test
Number of samples	11,999	3001
Object	41.55%	41.56%
Number	14.81%	14.81%
Color	20.82%	20.82%
Location	22.82%	22.82%

Table 1: Statistics of the ViVQA dataset

Component	Value
Visual Embedding Module	google/vit-base-patch16-384
Lanuguage Embedding Module	vinai/phobert-base-v2
No. Connections	i $\in$ { $0,0$ , $6,6$ , $12,12$ }
Batch size	32
Optimizer	AdamW(lr = 1e-4, eps = 1e-8)
Learning rate Scheduler	OneCycleLR(lr = 1e-4, pct_start = 0.05)

Table 2: Detail of Component Setting

Model	Precision	Recall	F1-score	Exact Match
${ViHieCoAtt-syllable}_{100dims}$ [ 9 ]	0.2271	0.2954	0.2234	-
${ViHieCoAtt-word}_{100dims}$ [ 9 ]	0.2049	0.3044	0.2256	-
${ViHieCoAtt-syllable}_{300dims}$ [ 9 ]	0.2185	0.3104	0.2277	-
${ViHieCoAtt-word}_{300dims}$ [ 9 ]	0.2192	0.3236	0.2436	-
${PAT}$ [ 10 ]	-	-	-	0.6055
${MCA}$ [ 15 ]	-	-	-	0.6076

Table 3: The comparison of our model and competitive baselines (test-standard)

Connection layers	Precision	Recall	F1-score	Exact Match
(0, 0), (6, 6), (12, 12)	0.7057	0.7060	0.7044	0.6485
(0, 12), (6, 6), (12, 0)	0.7007	0.7003	0.6990	0.6415
(0, 6), (6, 12), (12, 0)	0.6956	0.6954	0.6939	0.6351
(0, 12), (6, 0), (12, 6)	0.7012	0.7004	0.6992	0.6415
(0, 0), (1, 1), (2, 2)	0.6217	0.6207	0.6192	0.5548
(5, 5), (6, 6), (7, 7)	0.6602	0.6587	0.6578	0.5975

Table 4: The effect of using cross-attention on different layer pairs

Num layers	Connection layers	Precision	Recall	F1-score	Exact Match
2	(0, 0), (12, 12)	0.7033	0.7032	0.7017	0.6441
3	(0, 0), (6, 6), (12, 12)	0.7057	0.7060	0.7044	0.6485
4	(0, 0), (4, 4), (8, 8), (12, 12)	0.7009	0.7012	0.6996	0.6451
5	(0, 0), (3, 3), (6, 6), (9, 9), (12, 12)	0.7054	0.7053	0.7039	0.6468

Table 5: The effect of using cross-attention on number of cross-attention layers

Explore similar work

Dec 16, 2025cs.CL

Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models

Focusing on low-resource languages is an essential step toward democratizing generative AI. In this work, we contribute to reducing the multimodal NLP resource gap for Romanian. We translate the widely known Flickr30K dataset into Romanian and further extend it for visual question answering by leveraging open-source LLMs. We demonstrate the usefulness of our datasets by fine-tuning open-source VLMs on Romanian visual question answering. We select VLMs from three widely used model families: LLaMA 3.2, LLaVA 1.6, and Qwen2. For fine-tuning, we employ the parameter-efficient LoRA method. Our models show improved Romanian capabilities in visual QA, as well as on tasks they were not trained on, such as Romanian image description generation. The seven-billion-parameter Qwen2-VL-RoVQA obtains top scores on both tasks, with improvements of +2.29% and +4.45% in BERTScore F1 on VQA and captioning, respectively, over its original version. Finally, the models show substantial reductions in grammatical errors compared to their original forms, indicating improvements not only in language understanding but also in Romanian fluency.

George-Andrei Dima, Răzvan-Alexandru Smădu, Dumitru-Clementin Cercel

National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania

Recent Vision-Language Models Multimodal Continual Instruction Tuning Qwen3 Generative Artificial Intelligence

Ask PDF

May 5, 2026cs.CV

Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

With advances in multimodal research and deep learning, Multimodal Large Language Models (MLLMs) have emerged as a powerful paradigm for a wide range of multimodal tasks. As a core problem in vision-language research, Visual Question Answering (VQA) has increasingly employed MLLMs to improve performance, particularly in open-domain settings where external knowledge is essential. In this work, we aim to further enhance retrieval-based VQA by more effectively integrating MLLMs with structured reasoning and knowledge acquisition. We introduce a logical prompting strategy that fuses Chain-of-Thought (CoT) reasoning with Visual Question Decomposition (VQD), termed CoVQD, to guide retrieval toward more accurate and relevant knowledge for MLLM inference. Building on this idea, we propose a new framework, CoVQD-guided RAG (CgRAG), which enables MLLMs to access more comprehensive and coherent external knowledge while benefiting from structured visual-text reasoning guidance, thereby improving generalization and reliability in complex cross-domain VQA scenarios. Extensive experiments on E-VQA, InfoSeek, and OKVQA benchmarks demonstrate the effectiveness of the proposed method.

Quanxing Xu, Ling Zhou, Xian Zhong +3

School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3

Knowledge-Based Visual Question Answering Multimodal Large Language Models Vision-Language Foundation Models LLM Inference Optimization

Ask PDF

Aug 19, 2025cs.CL

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) interpret and reason over visual and textual information beyond English. VMMU consists of 2.5k multimodal questions across 7 tasks, covering a diverse range of problem contexts, including STEM problem solving, data interpretation, rule-governed visual reasoning, and abstract visual reasoning. All questions require genuine multimodal integration, rather than reliance on text-only cues or OCR-based shortcuts. We evaluate a diverse set of state-of-the-art proprietary and open-source VLMs on VMMU. Despite strong Vietnamese OCR performance, proprietary models achieve only 66% mean accuracy. Further analysis shows that the primary source of failure is not OCR, but instead multimodal grounding and reasoning over text and visual evidence. Code and data are available at https://vmmu-bench.github.io/

Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4

KAIST · MBZUAI

Multimodal Reasoning Benchmarks Multimodal Understanding Visual Reasoning Multi-Modal Data Reasoning Benchmark

Ask PDF