V-REX: Efficient Specialist VLM Training for Veterinary X-Rays

Tim Elsner; Nicole McNally; Andre Dourson; Michael Fitzke

cs.CVAug 20, 2026

V-REX: Efficient Specialist VLM Training for Veterinary X-Rays

Authors: Tim Elsner, Nicole McNally, Andre Dourson, Michael Fitzke

Organizations: Vyyo AI · Mars Petcare

Abstract

While generalist VLMs are expensive to train, creating domain experts is widely assumed to require fine-tuning increasingly large foundation models. We show that, in veterinary radiology, this assumption is misguided. By rethinking the entire VLM pipeline - from text tokenisation and pre-training to grounding and inference - we demonstrate that careful engineering can yield models that outperform much larger foundation models from scratch, without relying on any other data. Our approach introduces new strategies for generative pre-training and grounding that improve training efficiency, increasing data utilisation and downstream performance. Using only a fraction of the parameters, data, and compute of contemporary generalist models, we develop the first VLM capable of generating diagnostic reports for veterinary radiographs, surpassing open foundation models on this task by significant margin.

Figures & tables

Figure 1: Advantage of using RAPTOR: With the right pre-trained data embedding, our V-REX performs much better, however, still trains with just a linear embedding.

Figure 2: Our proposed pipeline follows a fully autoregressive and unified architecture: For the encoded, frozen embeddings of the image, autoregressively learn to predict the next (quantised) image token as a pre-training. We then autoregressively learn to produce text. All of our model is using the same space, with a simple sequence with image tokens row by row followed by the text. Extensions like multi-image support or IFG for better image adherence are applied during inference.

Figure 3: Visualisation of visual token impact: The model decides for a next token piece containing the word "cardiac", with the computed Integrated Gradients highlighting the heart.

Figure 4: F1 scores for the 30 most common conditions when using our V-REX model.

Model	Days	Trainable	F1 $\uparrow$
	on H100	Parameters	Score
o3	unknown	unknown	$0.2$
PaliGemma, LorA	$1$	$3.3$ M	$0.11$
PaliGemma, LorA	$5$	$3.3$ M	$0.13$
PaliGemma, Finetuning	$1$	$3$ B	$0.34$
PaliGemma, Finetuning	$5$	$3$ B	$0.38$

Table 1: We measure training time of using one H100 on our dataset, with PaliGemma being fine-tuned to our data. Note that these are micro F1 scores (globally across all classes of findings). Both LorA and Finetuning for Paligemma did not improve on their train loss anymore.

Dataset size	$0.1\,\text{M}$	$0.5\,\text{M}$	$3\,\text{M}$	$15\,\text{M}$
Accuracy (top-1) $\uparrow$	$0.210$	$0.327$	$0.351$	0.394

Table 2: Top-1 accuracy for next text token prediction on the test set after 30M samples seen, across different dataset sizes.

Measure	None	512	$4096$	$32768$
Accuracy (top-1) $\uparrow$	$.3769$	.3898	$.3806$	$.3753$
Accuracy (top-5) $\uparrow$	$.6739$	.6857	$.6741$	$.6694$
Perplexity $\downarrow$	$14.81$	13.85	$15.48$	$15.26$

Table 3: Generative pre-training effect, measure as the best value occurring on the

3m

subset. We pre-train for 15 epochs, giving the number of cluster centres used for discretisation to allow autoregressive pre-training. ’None’ shows without pre-training.

Figure 5: We show the effectiveness of our additional grounding through comparing performance with and without IFG.

Efficient VLM Training

See all 10 papers

CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
Aug 4, 2026Mercy Prasanna Ranjit, Anirban Porya, Sathvik Joel +8Radiology Report Generation Chest X-Ray Classification
Microsoft Research India · Medha AI, Narayana Health, India. · RTIICS, Narayana Health, India
Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
Apr 10, 2026Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3Medical VLMs VLM Reasoning
Brown University · University of Washington · University of Tübingen +1
Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models
May 24, 2026Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois +7Radiology Report Generation Healthcare
University of Zurich and University Hospital of Zurich, Switzerland · Zurich University of Applied Sciences, Switzerland · Kobe University, Japan +3

Aug 4, 2026J/K move · Enter open · S save

CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Mercy Prasanna Ranjit, Anirban Porya, Sathvik Joel, Niharika Vadlamudi+7

Microsoft Research India · Medha AI, Narayana Health, India. · RTIICS, Narayana Health, India

Radiology Report Generation Chest X-Ray Classification Radiology VLMs Visual Grounding RL for VLMs

Open paper AskPDF arXiv

A clinically useful chest X-ray system must go beyond fluent report generation: it should classify findings with tunable decision thresholds, localize them spatially, and derive the anatomical measurements upon which many diagnoses depend. Today's Vision-Language Models (VLMs) treat these as separate problems, if they address them at all, leaving a gap between what radiologists need and what generative models provide. We introduce CARE-X, a chest X-ray VLM that narrows this gap by unifying auxiliary discriminative supervision with reward-aligned generation. CARE-X augments its generative backbone with focal-loss classification and composite-loss grounding heads, co-trained alongside the language-modeling objective. This auxiliary supervision produces discriminative diagnostic predictions with tunable decision thresholds and precise spatial localization while also improving report quality, providing evidence that structured prediction and generation reinforce one another. Building on this foundation, Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) leverages task-specific reward signals for report generation, visual question answering (VQA), and spatial grounding, directly optimizing the clinical quality metrics that matter in practice. The result is state-of-the-art performance on the majority of metrics across four report-generation benchmarks, 94.0% VQA accuracy on ReXVQA (+6.0 pp over the next-best baseline), and generative spatial decoding that reaches near parity with dedicated detection heads. Separately, to address measurement-dependent diagnoses, we couple Qwen3-VL-4B-Instruct with native tool-calling capabilities for invoking deterministic measurement tools, while retaining full visual access to the image. This hybrid inference yields +43.6 pp average F1 over perception-only baselines across five measurement-dependent conditions.

Abstract

Efficient VLM Training

Explore similar work

CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models