Vision-Language Models for Analog Gauge Reading: An Empirical Study of Specialization, Transfer and Reliability

Abdul Mueez; Aaditya Baranwal; Junior Chaj-Mejia; Guneet Bhatia; Jason T. Voelker; Shruti Vyas

cs.CVAug 18, 2026

Vision-Language Models for Analog Gauge Reading: An Empirical Study of Specialization, Transfer and Reliability

Authors: Abdul Mueez, Aaditya Baranwal, Junior Chaj-Mejia, Guneet Bhatia, Jason T. Voelker, Shruti Vyas

Organizations: University of Central Florida, Orlando, FL, United States · Siemens Energy, Orlando, FL, United States

Abstract

Analog gauges remain common in industrial environments where manual inspection is costly or hazardous. The engineering application addressed here is direct numerical reading of single-target analog-gauge images, while the artificial-intelligence contribution is a systematic evaluation of specialization, transfer, robustness and reliability for a general-purpose vision-language model (VLM) without an explicit pointer-segmentation and geometric-reading pipeline. The Qwen2.5-VL-7B-Instruct model is evaluated using zero-shot prompting, in-context learning (ICL) and parameter-efficient fine-tuning with Quantized Low-Rank Adaptation (QLoRA) on a public synthetic dataset, a video-derived Pressure Gauge dataset and a proprietary industrial dataset. All fine-tuning experiments use a fixed 20-epoch protocol with the final epoch used for analysis; separate models with and without supplied gauge ranges remove prompt-setting confounds. The primary metric is range-normalized mean percentage error (MPE). The best fine-tuned MPE values are 2.39% on the synthetic dataset, with a 95% bootstrap confidence interval (CI) of 1.43-3.90%; 2.61% on the Pressure Gauge dataset, with a CI of 1.66-3.80%; and 4.43% on the proprietary industrial dataset, with a CI of 2.31-7.14%. Leave-one-dataset-out experiments reveal substantial transfer degradation on held-out synthetic and proprietary data, while robustness tests identify Gaussian blur as the strongest tested corruption. Reliability analysis shows that high-confidence errors remain possible, motivating abstention and independent validation in safety-critical use. These results support QLoRA-specialized VLMs for direct single-gauge reading but not yet a deployment-ready plant-monitoring pipeline.

Figures & tables

Figure 1: Examples from the public Synthetic Data for Precision Gauge Reading dataset [ 6 ] .

Figure 2: Example frames from the Pressure Gauge video collection [ 5 ] .

Figure 3: Examples from the proprietary SE dataset, illustrating variation in gauge appearance, background, scale and viewpoint.

Dataset	Train	Test	# Ranges (train/test)	Scale Endpoints (min–max)	Value Range (test)
Synthetic	375	126	7 / 7	$-2$ to $10$	$-1.9$ to $9.0$
Pressure	57	36	1 / 3	$-1$ to $6$	$-0.42$ to $3.2$
SE (Ours)	152	43	32 / 18	$-30$ to $5000$	$0$ to $2300$

Table 1: Summary of the final dataset splits, with scale-range and value statistics. “# Ranges” is the number of distinct (min, max) scale configurations present in the split; “Value Range” is the span of ground-truth needle readings.

Figure 4: General VLM structure: a vision encoder extracts visual tokens, a projector/merger maps them to the language-model representation and the autoregressive language model produces the numerical answer.

Task	Role / Condition	Template
Value Reading	System	You are an assistant that reads analog gauges. Respond with only the numerical value the needle is pointing to. For example, if the needle points to 5.5, respond ‘5.5’.
	User (With range)	What is the current reading on this analog gauge? The scale ranges from {min_val} to {max_val}.
	User (No range)	What is the current reading on this analog gauge?
Scale Probe	System	You are a vision assistant that reads analog dial gauges. Look only at the printed scale numbers on the dial face. Report the smallest and largest numbers printed on the scale. Respond ONLY as JSON: {"min": <number>, "max": <number>}.
Scale Probe	User	What are the minimum and maximum scale values printed on this gauge’s dial?

Table 2: Prompt templates used in the reported experiments. The same range condition is used during training and inference for the fine-tuned models.

Panel of Figure 5 — Figure 5: Zero-shot (0 examples) and ICL sensitivity to the number of contextual examples. Lower MPE is better.

Dataset	Method	With Range	Without Range
Synthetic	Zero-shot	21.33	18.12
	ICL (best)	18.43 ( $k=20$ )	15.83 ( $k=20$ )
	Fine-tuned	2.39 [1.43, 3.90]	2.73 [1.57, 4.58]
Pressure	Zero-shot	25.29	15.69
	ICL (best)	8.34 ( $k=5$ )	5.89 ( $k=10$ )
	Fine-tuned	3.31 [2.27, 4.57]	2.61 [1.66, 3.80]

Table 3: MPE (%) for zero-shot, best-case ICL and fine-tuned Qwen2.5-VL-7B models. Fine-tuned entries include the 95% bootstrap CI in brackets and come from separately trained with-range and without-range models; all fine-tuned results use the final checkpoint after 20 fixed epochs.

Panel of Figure 6 — Figure 6: Best observed MPE for each learning strategy. For Pressure, the best QLoRA result is the separately trained no-range model; for Synthetic and SE, the best QLoRA result uses range metadata.

Dataset	Mean difference (percentage points)	95% CI	Wilcoxon $p$
SE	-1.85	$[-7.17,\ 1.72]$	0.974
Synthetic	-0.34	$[-0.80,\ 0.03]$	0.356
Pressure	0.70	$[-0.22,\ 1.74]$	0.106

Table 4: Paired range-ablation statistics for the separately trained fine-tuned Qwen2.5-VL-7B models. The mean difference is with-range minus without-range capped per-image percentage error; the corresponding MPE values are reported in Table 3 .

Dataset	Range	Uncapped MPE (%)	MAE	RMSE	P95 FS (%)	Max FS (%)	Pass $\pm 1\%$	Pass $\pm 2\%$	Pass $\pm 5\%$
SE	with	4.43	48.39	176.47	19.90	41.25	44.2%	60.5%	79.1%
Synthetic	with	2.39	0.17	0.61	5.00	80.00	37.3%	64.3%	96.0%
Pressure	with	3.31	0.13	0.17	9.38	16.40	19.4%	36.1%	83.3%
SE	without	13.42	62.25	278.93	26.44	407.14	46.5%	55.8%	81.4%
Synthetic	without	5.94	0.45	3.61	9.06	505.00	37.3%	64.3%	92.9%
Pressure	without	2.61	0.10	0.16	8.53	16.50	38.9%	61.1%	91.7%

Table 5: Extended error, tail-error and operational tolerance metrics for fine-tuned Qwen2.5-VL-7B models. MAE and RMSE use dataset-specific engineering units; P95 FS and Max FS indicate 95th-percentile and maximum errors relative to full scale (FS). Pass rates denote the percentage of predictions within operational tolerances (

\pm 1\%

\pm 2\%

\pm 5\%

FS). No parsing failures occurred across evaluated samples (0/205 per model condition).

Held-out dataset	LODO MPE [95% CI]	Increase (pp)	Increase 95% CI	Wilcoxon $p$
SE	13.25 [8.38, 19.00]	8.82	[4.07, 14.20]	$<0.001$
Synthetic	14.99 [11.14, 19.25]	12.60	[8.37, 17.21]	$<0.001$
Pressure	4.99 [3.09, 7.26]	1.68	[-0.06, 3.93]	0.360

Table 6: Leave-one-dataset-out (LODO) cross-domain transfer evaluation for fine-tuned Qwen2.5-VL-7B. LODO MPE is the range-normalized mean percentage error on the held-out test partition with its 95% nonparametric bootstrap confidence interval [CI]. “Increase (pp)” represents the performance degradation in percentage points (LODO MPE minus the all-domain with-range baseline from Table 3 ), where positive values indicate increased error from domain exclusion. Wilcoxon

p

is the two-tailed significance value from a paired Wilcoxon signed-rank test on per-image errors.

Corruption	Severity 1	Severity 2	Severity 3	Fail $>5\%$ at S3
Gaussian blur	9.24	15.51	19.92	65.12%
Low light	4.05	4.27	4.00	20.93%
Occlusion	4.68	5.69	5.47	23.26%
Perspective	5.07	3.11	4.14	16.28%
Reflection	4.26	4.47	4.86	23.26%

Table 7: SE corruption robustness for the fine-tuned with-range Qwen2.5-VL-7B model. Values are capped MPE (%). The final column is the severity-3 failure rate above 5% full scale.

Figure 7: MPE under synthetic adverse-condition corruptions on the SE test set. Blur is the dominant tested degradation.

Model	Dataset	Accuracy @5% (%)	ECE	MCE	AURC	Mean confidence	HC errors
With range	SE	79.1	0.158	0.311	0.177	0.920	1
With range	Synthetic	96.0	0.036	0.357	0.021	0.925	1
With range	Pressure	83.3	0.138	0.198	0.175	0.803	0
With range	Pooled	90.2	0.053	0.219	0.064	0.902	2
No range	SE	81.4	0.131	0.353	0.066	0.945	0
No range	Synthetic	92.1	0.045	0.068	0.049	0.926	1

Table 8: Confidence calibration and reliability metrics for the fine-tuned Qwen2.5-VL-7B models evaluated on individual datasets and the pooled test set (

N=205

). Accuracy@5% denotes the proportion of predictions meeting the

\pm 5\%

full-scale operational correctness threshold. Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) quantify average and worst-case calibration gaps across 10 equal-width confidence bins. AURC represents the Area Under the Risk-Coverage Curve measuring error rejection efficacy under confidence filtering. Mean Confidence denotes the average token-sequence confidence (

C

) and High-Confidence (HC) errors count critical failures where absolute error exceeds 5% of full scale despite sequence confidence

C\geq 0.99

Panel of Figure 8 — Figure 8: Calibration analysis for the final with-range model. Corresponding no-range plots will be included in the public result package described in Section .

Video	$n$	Reference $\mu_{R}$	Fine-tuned Qwen2.5-VL-7B $\mu_{R}$ [95% CI]
test1/man1	13	0.003	0.0096 [0.0058, 0.0138]
test2/man2	12	0.010	0.0319 [0.0137, 0.0596]
test3/man3	5	0.036	0.0320 [0.0240, 0.0408]
test4/man4	1	0.008	0.0080 [0.0080, 0.0080]
test5/man5	2	No prediction	0.0520 [0.0320, 0.0720]
test6/man6	1	0.045	0.1250 [0.1250, 0.1250]

Table 9: Per-video Pressure Gauge comparison with the reference method [ 7 ] . Values are mean relative error

\mu_{R}

(lower is better). The Qwen column includes a 95% bootstrap CI. Sampling protocols are not identical.

Model	MPE (%)
Model	With range	Without range
Qwen2.5-VL-3B-Instruct	4.98	7.33
SmolVLM2-500M-Instruct	24.14	21.12
SmolVLM2-256M-Instruct	26.22	17.78
MobileNetV2 Regressor	11.12	27.82

Table 10: Additional model baselines on the SE dataset. MPE is reported in percent. The primary Qwen2.5-VL-7B values (4.43% with range; 6.28% without range) are reported in Table 3 and are not duplicated here.

Model	Mean latency (ms)	P95 latency (ms)	Peak GPU memory (MB)	Training time per epoch (min)
Qwen2.5-VL-7B	594.98	597.30	8101.53	8.05
Qwen2.5-VL-3B	599.74	604.12	6668.66	7.65
SmolVLM2-500M	582.55	587.18	2882.85	16.81
SmolVLM2-256M	536.53	539.52	1132.43	16.17
MobileNetV2	3.95	4.07	143.73	0.65

Table 11: Protocol-matched computational benchmark on an Nvidia RTX A5500. Peak memory is peak allocated GPU memory during inference; training time is measured per epoch under the benchmark configuration.

Panel of Figure 9 — Figure 9: Interpretability analysis for three representative SE examples. Top: original inputs. Middle: blurred-patch sensitivity maps. Bottom: attention-rollout maps. The two successful examples emphasize similar gauge-face regions, though attention rollout is more spatially diffuse. The anomalous example shows a nearly flat sensitivity map despite attention on the gauge face, indicating that attention alone does not guarantee a correct numerical reading.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Apr 19, 2026

Lost in Motion: Vision Language Models Fail the Dynamic Gauges Test

The digital transformation of industrial manufacturing increasingly relies on the ability of autonomous robots to interact with legacy infrastructure, particularly analog gauges. Vision-Language Models (VLMs) have the potential to provide a general solution for gauge reading and have already shown good performance in instrument recognition. However, performing accurate, real-time gauge readings is a more complex task. This paper evaluates state-of-the-art models, including versions from the GPT-5 (5.4 Thinking and 5.3 Instant) and Gemini 3 (Pro and Flash) families, against a set of simple but realistic dynamic gauge reading scenarios. To facilitate this evaluation, we introduce a novel dataset comprising video sequences of three instruments of different gauge types: circular, linear, and Vernier, under diverse motion and speed profiles. Our findings indicate that the evaluated frontier VLMs, under our specific testing conditions, exhibit a limited ability to interpret needle trajectories and scale semantics, failing to provide the traceability and reliability needed for safety-critical monitoring. The results demonstrate that these models have not yet achieved the performance necessary to be classified as trustworthy synthetic instruments under existing IEEE and ISO standards.

Tairan Fu, Francisco Javier Santos-Martín, Javier Conde +2

VLM Evaluation Industrial Inspection Temporal Video Understanding

Ask PDF

Sep 3, 2026

InSituMeasure: Probing Situated Measurement Grounding in Industrial Scenes with Multimodal Large Language Models

For trained operators, gauge reading requires little specialized knowledge, low cognitive effort, and high repeatability. Yet Multimodal Large Language Models (MLLMs) remain unreliable in continuous-valued measurement despite strong results on general multimodal benchmarks. Existing benchmarks expose this weakness but isolate measurement from realistic, knowledge-grounded settings, with limited situated context, specialized instruments, real-world noise, and matched diagnostic annotations, reducing realism and constraining root-cause analysis. We introduce InSituMeasure to evaluate situated measurement grounding. It contains 2,922 real industrial monitoring scenes across eight functional categories of professional engineering instruments, with dense gauge-attribute annotations and noise tags for failure diagnosis. We define metrics for numerical accuracy under predefined tolerances and unit consistency, rejection of fake or unanswerable tasks, and alignment between model failures and annotated error factors. Across 24 state-of-the-art MLLMs, the best model reaches only 25.7% joint value-unit accuracy and 51.8% confidence-diagnosis F1, revealing a substantial gap between general multimodal competence and reliable situated measurement. Further analysis identifies failures from text-induced shortcuts, overconfident responses, and authentic industrial noise, including mixed disturbances, viewpoint deviation, occlusion, and environmental interference.

Chao Shen, Xinyuan Li, Yunfan Zhou +4

Shanghai Key Laboratory of Scalable Computing and Systems · School of Computer Science, Shanghai Jiao Tong University · Qwen Team, Alibaba Group

Multimodal Grounding Multimodal Robustness Multimodal Large Language Models Industrial Inspection Multimodal Model Evaluation

Ask PDF

Mar 24, 2026

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions. A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework. A text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability, compositional reasoning is probed with structured negation probes across four templates, and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse (-33.2pp vs. -20.8pp, Wald 95% CI [8.2, 16.8]pp, p < 10^-8), driven almost entirely by COCO while the VQAv2 gap is not statistically significant (4.5pp, p=0.19). The most discriminating template is false_yn: SMOLVLM2-500M responds "Yes" (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Q WEN 2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment.

Mehmet Kaan Erol

Marmara University, Institute of Pure and Applied Sciences

Compositional Reasoning in VLMs VLM Evaluation VLM Robustness VLM Reasoning VLM Quantization VLM Calibration

Ask PDF

Abstract

VLM Evaluation

Explore similar work

Lost in Motion: Vision Language Models Fail the Dynamic Gauges Test

InSituMeasure: Probing Situated Measurement Grounding in Industrial Scenes with Multimodal Large Language Models

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption