QwenVLConnector: A Fast, Unified Medical VLM Chatbot for Fine-Grained Clinical Perception and Text Generation

Le Thien Phuc Nguyen; Thien Nguyen; Thanh-Huy Nguyen; Gia Minh Hoang; Anh Mai Vu; Ulas Bagci

cs.CVSep 19, 2026

QwenVLConnector: A Fast, Unified Medical VLM Chatbot for Fine-Grained Clinical Perception and Text Generation

Authors: Le Thien Phuc Nguyen, Thien Nguyen, Thanh-Huy Nguyen, Gia Minh Hoang, Anh Mai Vu, Ulas Bagci

Organizations: University of Wisconsin - Madison, USA · University of North Carolina - Chapel Hill · Carnegie Mellon University, USA · Northwestern University, USA ·

Abstract

Most medical vision-language models (VLMs) excel at open-ended report generation and VQA but provide limited support for structured, fine-grained clinical perception within a unified interface. We present QwenVLConnector, a Qwen2.5-VL-based medical chatbot that unifies classification, multi-label classification, textualized detection, counting, regression, and free-form report generation under a single next-token objective. Our key component is a lightweight dense multi-layer Connector that aggregates low- and high-level visual features, aligns them through the pretrained vision Merger, and fuses them with the final visual representation without increasing sequence length. This design enriches visual tokens with complementary spatial and semantic cues while preserving efficiency. On FLARE-2D, QwenVLConnector improves detection F1 from 0.55 to 0.85, raises single-label classification from 0.37 to 0.51, and boosts report-generation GREEN by up to 18.3 points over the Qwen2.5-VL baseline. We further explore multimodal in-context learning for report generation, showing additional improvements without updating model parameters. Overall, QwenVLConnector offers a unified and efficient framework for combining structured medical perception with open-ended clinical text generation. Our code can be found at https://github.com/plnguyen2908/QwenConnector.

Figures & tables

Figure 1: High-level architecture of QwenVLConnector. A Vision Encoder extracts features from medical images; a lightweight dense multi-layer Connector fuses low/high-level cues and projects them to the LLM without increasing sequence length. The fused visual tokens (green) are prepended to the text tokens (pink), and a decoder-only LLM performs next-token generation for every task.

Figure 2: Dense multi-layer connector in QwenVLConnector. Early and late ViT features are depth-averaged, aligned via the backbone Merger , concatenated channel-wise with the encoder output, and projected by a lightweight MLP to LLM visual tokens.

Prompt
Instruction: [Task Instruction]
User: [Demonstration 1]
Response: [Answer 1]
…
User: [Demonstration k]
Response: [Answer k]

Table 1: Prompt template with ICL mechanism.

	Alignment	Instruction	FLARE tuning
Epochs	1	1	3
Learning rate	2e-3	2e-4	2e-4
Per-device batch	1	1	1
Number of devices	8	8	8
GPU type	A40	A40	A40
Training Time	3 days	7 days	1 day

Table 2: Key hyperparameters per training stage.

Model	Classification $\uparrow$	Multi-label classification $\uparrow$
4-bit QwenVL 2.5 7B [ 28 ]	0.37	0.57
8-bit QwenVL 2.5 7B [ 28 ]	0.36	0.56
16-bit QwenVL 2.5 7B [ 28 ]	0.35	0.55
4-bit QwenVLConnector 7B (Ours)	0.51	0.49
8-bit QwenVLConnector 7B (Ours)	0.46	0.53
16-bit QwenVLConnector 7B (Ours)	0.50	0.54

Table 3: Classification evaluation. The result is reported on validiation-hidden subset on Codabench. All metrics are reported as fractions in [0,1].

Model	Detection $\uparrow$	Instance Detection $\uparrow$
4-bit QwenVL 2.5 7B [ 28 ]	0.51	0
8-bit QwenVL 2.5 7B [ 28 ]	0.55	0
16-bit QwenVL 2.5 7B [ 28 ]	0.53	0
4-bit QwenVLConnector 7B (Ours)	0.76	0
8-bit QwenVLConnector 7B (Ours)	0.85	0
16-bit QwenVLConnector 7B (Ours)	0.81	0

Table 4: Detection evaluation. The result is reported on validiation-hidden subset on Codabench. All metrics are reported as fractions in [0,1].

Model	Regression $\downarrow$	Counting $\downarrow$	Report Generation $\uparrow$
4-bit QwenVL 2.5 7B [ 28 ]	15.51	287.3	65.74
8-bit QwenVL 2.5 7B [ 28 ]	15.19	276.77	55.77
16-bit QwenVL 2.5 7B [ 28 ]	15.43	268.24	74.78
4-bit QwenVLConnector 7B (Ours)	20.91	284.44	76.01
8-bit QwenVLConnector 7B (Ours)	21.13	275.81	74.05
16-bit QwenVLConnector 7B (Ours)	19.04	266.7	74.93

Table 5: Regression, Counting, and Report Generation evaluation. The result of Regression is reported on validation-hidden subset on Codabench. The result of Counting and Report Generation is reported on validation-public subset. Report Generation is reported as decimal number in [0, 100].

Panel of Figure 3 — Figure 3: Effect of in-context learning (ICL) on QwenVLConnector for the report-generation task: BLEU, GREEN Clinical Significance, GREEN Entity Matching, and overall GREEN Score on the validation-public split. Incorporating $k$ demonstrations consistently improves performance over no-ICL prompts.

Figure 4: Qualitative examples on Ultrasound images. Left: multiple-choice classification task. Right: lesion detection task where green is our detection, blue is ground truth, and red is QwenVL’s [ 28 ] detection.

Requirements	Answer
A meaningful title	Yes
The number of authors ( $\leq$ 6)	6
Author affiliations and ORCID	Yes
Corresponding author email is presented	Yes
Validation scores are presented in the abstract	Yes
Introduction includes at least three parts: background, related work, and motivation	Yes

Table 6: Checklist Table. Please fill out this checklist table in the answer column.

Medical Image Object Detection

See all 11 papers

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models
Aug 11, 2026Yuan Wang, Hualiang Wang, Yixin Chen +6Unified Multimodal Models Vision-Language Grounding
Zhejiang University · Fudan University
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Mar 2, 2026Yuan Wu, Zongxian Yang, Jiayu Qian +5Medical VLMs VLM Reasoning
Department of Computer Science, City University of Hong Kong (Dongguan) · College of Computing and Data Science (CCDS), Nanyang Technological University · School of Computer Science, Northwestern Polytechnical University

Apr 10, 2026J/K move · Enter open · S save

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan+3

Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Medical VQA Medical VLMs Latent Visual Reasoning RL for VLMs

Open paper AskPDF arXiv

Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.

Abstract

Medical Image Object Detection

Explore similar work

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models

Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine