Toward Unified Fine-Grained Vehicle Classification and Automatic License Plate Recognition

Gabriel E. Lima; Valfride Nascimento; Eduardo Santos; Eduil Nascimento; Rayson Laroca; David Menotti

cs.CVApr 7, 2026

Toward Unified Fine-Grained Vehicle Classification and Automatic License Plate Recognition

Authors: Gabriel E. Lima, Valfride Nascimento, Eduardo Santos, Eduil Nascimento, Rayson Laroca, David Menotti

Organizations: Department of Informatics, Federal University of Paraná, R. Evaristo F. Ferreira da Costa 391, Jardim das Américas, Curitiba, PR, 81530-090, Brazil.

Abstract

Extracting vehicle information from surveillance images is essential for intelligent transportation systems, enabling applications such as traffic monitoring and criminal investigations. While Automatic License Plate Recognition (ALPR) is widely used, Fine-Grained Vehicle Classification (FGVC) offers a complementary approach by identifying vehicles based on attributes such as color, make, model, and type. Although there have been advances in this field, existing studies often assume well-controlled conditions, explore limited attributes, and overlook FGVC integration with ALPR. To address these gaps, we introduce UFPR-VeSV, a dataset comprising 24,945 images of 16,297 unique vehicles with annotations for 13 colors, 26 makes, 136 models, and 14 types. Collected from the Military Police of Paraná (Brazil) surveillance system, the dataset captures diverse real-world conditions, including partial occlusions, nighttime infrared imaging, and varying lighting. All FGVC annotations were validated using license plate information, with text and corner annotations also being provided. A qualitative and quantitative comparison with established datasets confirmed the challenging nature of our dataset. A benchmark using five deep learning models further validated this, revealing specific challenges such as handling multicolored vehicles, infrared images, and distinguishing between vehicle models that share a common platform. Additionally, we apply two optical character recognition models to license plate recognition and explore the joint use of FGVC and ALPR. The results highlight the potential of integrating these complementary tasks for real-world applications. The UFPR-VeSV dataset is publicly available at: https://github.com/Lima001/UFPR-VeSV-Dataset.

Figures & tables

Panel of Figure 2 — Figure 2: Examples of images featuring multiple vehicles due to camera perspective. The background vehicle is highlighted with a green border, while the main vehicle is shadowed to enhance contrast.

Panel of Figure 3 — Figure 3: Examples of infrared images under varying conditions: (a) optimal visibility with enhanced perception, (b) reduced contrast affecting detail clarity, and (c) overexposure caused by vehicle headlights.

Panel of Figure 1 — Figure 1: Distribution of vehicles across the attributes of color (a), make (b), model (c) and type (d) in the UFPR-VeSV dataset. For better visualization, only the $30$ most common vehicle models are displayed in (c), representing $63.7$ % of the total images.

Panel of Figure 4 — Figure 4: Example of LPs cropped from images captured under diverse conditions, showcasing variations in resolution, perspective, and image quality. The corresponding annotated LP text is shown below each image.

Figure 5: Distribution of LP character classes in the UFPR-VeSV dataset.

Panel of Figure 6 — Figure 6: Example of border standardization. (a) original image with a green border; (b) result after removing a $5$ -pixel margin from all sides.

Panel of Figure 7 — Figure 7: Examples of vehicles assigned to the “unknown” class for color, make, and model annotations: (a) a red Yamaha motorcycle; (b) a white Volkswagen truck; (c) a red Mercedes-Benz truck. In all cases, the main body of the vehicle is obstructed, rendering the identification of color, make, and model impossible.

Dataset	Images	Vehicles	Source	Viewpoint	Annotations
Dataset	Images	Vehicles	Source	Viewpoint	Color	Make/Model	Type	ALPR
Ferryman et al., [1995]	$176^{\phantom{\ast}}$	unk.	Field	Frontal/Rear	-	-	✓	-
Jolly et al., [1996]	$393^{\phantom{\ast}}$	unk.	Field	n/a	-	-	✓	-
Lai et al., [2001]	unk. ${}^{\phantom{\ast}}$	unk.	Field	Rear	-	-	✓	-
Wu et al., [2001]	$800^{\phantom{\ast}}$	unk.	Field	Frontal	-	-	✓	-
Ma and Grimson, [2005]	unk. ${}^{\phantom{\ast}}$	unk.	Field	Frontal	-	-	✓	-

Table 1: Comparison between UFPR-VeSV (proposed in this work) and related datasets.

Panel of Figure 8 — Figure 8: Representative images from three public datasets and UFPR-VeSV. Our dataset features significantly more challenging scenarios, with vehicles captured from diverse viewpoints, environments, lighting conditions, image quality levels, and nighttime infrared imaging.

Results for color [ Chen et al.,, 2014 ] and model [ Yang et al.,, 2015 ] recognition benchmarks.
Classifier	Chen et al., [2014]			CompCars [ Yang et al.,, 2015 ]
Classifier	Mi-Acc	Ma-Acc	F1	Mi-Acc	Ma-Acc	F1
EfficientNet-V2	$85.5$	$85.6$	$86.8$	$73.5$	$70.5$	$72.2$
MobileNet-V3	$88.1$	$90.2$	$90.4$	$82.0$	$79.2$	$81.2$
ResNet-50	$92.7$	$93.4$	$93.8$	$87.4$	$82.5$	$85.0$
Swin Transformer-V2	$91.8$	$93.0$	$93.6$	$92.4$	$89.0$	$90.1$

Table 2: Performance metrics (%) achieved by all classifiers using the transfer-learning approach on the Chen et al., [2014] dataset (vehicle color recognition), the CompCars dataset [ Yang et al.,, 2015 ] (vehicle model recognition), and the proposed UFPR-VeSV dataset (vehicle color, make, model, and type recognition). Results on the proposed dataset are averaged over ten runs, with standard deviations shown in parentheses.

Classifier	UFPR-VeSV Color			UFPR-VeSV Type
Classifier	Mi-Acc	Ma-Acc	F1	Mi-Acc	Ma-Acc	F1
EfficientNet-V2	$\mathbf{93.5}$ $\mathbf{(0.6)}$	$\mathbf{71.5}$ $\mathbf{(2.3)}$	$\mathbf{73.8}$ $\mathbf{(2.3)}$	$\mathbf{96.1}$ $\mathbf{(0.7)}$	$\mathbf{89.0}$ $\mathbf{(2.0)}$	$\mathbf{90.2}$ $\mathbf{(1.6)}$
MobileNet-V3	$\mathbf{93.2}$ $\mathbf{(0.7)}$	$\mathbf{71.2}$ $\mathbf{(2.9)}$	$\mathbf{73.2}$ $\mathbf{(2.5)}$	$95.2$ $(0.7)$	$85.6$ $(1.9)$	$87.4$ $(1.6)$
ResNet-50	$93.1$ $(0.4)$	$69.6$ $(2.6)$	$71.8$ $(2.3)$	$95.5$ $(0.6)$	$86.8$ $(2.3)$	$88.4$ $(1.9)$
Swin Transformer-V2	$87.2$ $(0.4)$	$56.5$ $(1.5)$	$60.2$ $(1.7)$	$84.4$ $(0.67$	$72.8$ $(1.8)$	$76.8$ $(1.9)$
ViT-b16	$87.5$ $(0.4)$	$60.2$ $(1.8)$	$63.3$ $(1.6)$	$86.3$ $(0.7)$	$75.0$ $(2.7)$	$77.1$ $(2.3)$

Table 3: Performance metrics (%) achieved by all classifiers using the end-to-end fine-tuning approach on vehicle color, make, model, and type recognition (averaged over 10 runs). Standard deviations are shown in parentheses.

Panel of Figure 9 — Figure 9: Examples of misclassified multicolored vehicle from different viewpoints. The predicted color is shown below each image. Depending on the camera angle and illumination, a single color can appear dominant, which causes the classifier to classify the vehicle based on that one color.

Panel of Figure 10 — Figure 10: Examples of misclassified vehicle models from different manufacturers. Original make and model are displayed below each image. In (a), a rear-view Boxer was misclassified as a Ducato, while in (b) a rear-view Ducato was misclassified as a Boxer. Aside from minor brand-specific markings, the vehicles have a very similar structure, making accurate differentiation challenging.

Task	Top-1			Top-2			Top-3
Task	Mi-Acc	Ma-Acc	F1	Mi-Acc	Ma-Acc	F1	Mi-Acc	Ma-Acc	F1
Color	$93.5$ $(0.6)$	$71.5$ $(2.3)$	$73.8$ $(2.3)$	$98.0$ $(0.3)$	$87.0$ $(1.7)$	$89.4$ $(1.5$ )	$99.1$ $(0.1)$	$93.5$ $(1.4)$	$95.0$ $(1.0)$
Make	$94.4$ $(0.6)$	$85.0$ $(1.6)$	$86.4$ $(1.4)$	$96.8$ $(0.4)$	$90.8$ $(1.3)$	$91.6$ $(1.3)$	$97.8$ $(0.3)$	$92.9$ $(1.3)$	$93.7$ $(1.2)$
Model	$90.9$ $(0.6)$	$86.2$ $(1.1)$	$87.3$ $(0.8)$	$95.3$ $(0.5)$	$91.5$ $(1.1)$	$92.3$ $(0.9)$	$96.7$ $(0.5)$	$93.8$ $(0.9)$	$94.5$ $(0.7)$
Type	$96.1$ $(0.7)$	$89.0$ $(2.0)$	$90.2$ $(1.6)$	$99.2$ $(0.2)$	$97.2$ $(1.2)$	$97.7$ $(0.8)$	$99.7$ $(0.1)$	$98.9$ $(0.8)$	$99.0$ $(0.6)$

Table 4: Top-1, top-2, and top-3 performance metrics (%) for EfficientNet-V2 on vehicle color, make, model, and type recognition (UFPR-VeSV). Results are reported as the mean over 10 runs. Standard deviations are in parentheses.

Attribute set	Accuracy
Type	$96.1$ $(0.7)$
Type & Make	$91.5$ $(1.0)$
Type & Make & Model	$85.5$ $(1.1)$
Type & Make & Model & Color	$80.2$ $(1.2)$

Table 5: EfficientNet-V2 simultaneous accuracy (%) for conjunctive attribute recognition, detailing the performance as the set of required attributes expands. Results are reported as mean (standard deviation in parentheses) over 10 runs.

Panel of Figure 11 — Figure 11: Grad-CAM [ Selvaraju et al.,, 2017 ] attention maps for a Volkswagen Parati. (a) Original image. (b) The make recognition map, focusing on the manufacturer’s badge. (c) The model recognition map, relying on other features, such as the headlights.

Model	LP -level accuracy	Char-level accuracy
GP ALPR	$93.8~(0.3)$	$98.7~(0.1)$
ParSeq-Tiny	$\mathbf{98.0~(0.4)}$	$\mathbf{99.7~(0.1)}$

Table 6: Comparison of LP -level and character-level accuracy (%) achieved by the GP- ALPR and ParSeq-Tiny models on the UFPR-VeSV dataset for the LPR task. Results represent the average performance over 10 runs using different dataset splits, with standard deviations reported in parentheses.

Panel of Figure 12 — Figure 12: Example of misrecognized LPs . For each image, the ground-truth label is shown above the model’s prediction, with incorrectly recognized characters highlighted in red. The failure cases include severely degraded characters (a, h, i), illumination-induced obstructions (b), low-contrast characters (c, j), blurring (d, g), and physically deformed LPs (e, f).

Tasks	Validation Rate $\uparrow$	Conflict Rate $\downarrow$	Recovery Rate $\uparrow$
LPR & Color	$93.6$ $(0.5)$	$\phantom{0}6.5$ $(0.5)$	$\mathbf{92.7}$ $\mathbf{(0.7)}$
LPR & Make	$94.6$ $(0.5)$	$\phantom{0}5.4$ $(0.5)$	$85.7$ $(0.8)$
LPR & Model	$91.1$ $(0.6)$	$\phantom{0}8.9$ $(0.6)$	$82.1$ $(1.4)$
LPR & Type	$\mathbf{96.2}$ $\mathbf{(0.6)}$	$\mathbf{\phantom{0}3.8}$ $\mathbf{(0.6)}$	$92.1$ $(0.9)$
LPR & Color & Make	$88.7$ $(0.7)$	$11.3$ $(0.7)$	$80.1$ $(0.9)$
LPR & Color & Model	$85.4$ $(0.8)$	$14.6$ $(0.8)$	$76.8$ $(1.4)$

Table 7: Performance metrics of the integrated ALPR-FGVC system. All values are percentages (%). Validation and Conflict Rates are conditional on a correct LPR; Recovery Rate is conditional on an incorrect LPR. Results are reported as mean (standard deviation in parentheses) over 10 runs.

Panel of Figure 13 — Figure 13: Examples of ALPR failures where the LP is illegible due to (a) light glare and (b) occlusion. Despite these failures, our FGVC classifier acts as a fail-safe, correctly identifying the type, make, and model for both vehicles. Color recognition is correct for (b), while the infrared image (a) is classified as “unknown” due to the absence of color information.

Fine-Grained Image Classification

Fine-grained image classification distinguishes visually similar categories, such as species, product varieties, and medical findings.

See all 81 papers

LPLCv2: An Expanded Dataset for Fine-Grained License Plate Legibility Classification
Apr 9, 2026Lucas Wojcik, Eduardo A. F. Machoski, Eduil Nascimento +2Automatic License Plate Recognition
Federal University of Paran´a, Curitiba, Brazil · Department of Technological Development and Quality, Paran´a Military Police, Curitiba, Brazil · Pontifical Catholic University of Paran´a, Curitiba, Brazil
Revisiting Vehicle Color Recognition in Long-Tailed Surveillance Scenarios
Jun 11, 2026Vinícius Orrú, Bruno H. Foggiatto, Gabriel E. Lima +2Synthetic Data Augmentation Long-Tailed Classification
Pontifical Catholic University of Paraná, Curitiba, Brazil · National High Court of Brazil, Brasília, Brazil · Federal University of Paraná, Curitiba, Brazil
Evaluating 2D and 3D-Aware Vision Foundation Models for Vehicle Attribute Recognition
Aug 30, 2026Alexandre V. Delazeri, Gabriel E. Lima, Eduil Nascimento +2Vision Foundation Models Fine-Grained Visual Perception
Department of Informatics, Federal University of Paraná, Curitiba, Brazil · Department of Technological Development and Quality, Paraná Military Police, Curitiba, Brazil · Graduate Program in Informatics, Pontifical Catholic University of Paraná, Curitiba, Brazil

Apr 9, 2026J/K move · Enter open · S save

LPLCv2: An Expanded Dataset for Fine-Grained License Plate Legibility Classification

Lucas Wojcik, Eduardo A. F. Machoski, Eduil Nascimento, Rayson Laroca+1

Federal University of Paran´a, Curitiba, Brazil · Department of Technological Development and Quality, Paran´a Military Police, Curitiba, Brazil · Pontifical Catholic University of Paran´a, Curitiba, Brazil

Automatic License Plate Recognition

Open paper AskPDF arXiv

Modern Automatic License Plate Recognition (ALPR) systems achieve outstanding performance in controlled, well-defined scenarios. However, large-scale real-world usage remains challenging due to low-quality imaging devices, compression artifacts, and suboptimal camera installation. Identifying illegible license plates (LPs) has recently become feasible through a dedicated benchmark; however, its impact has been limited by its small size and annotation errors. In this work, we expand the original benchmark to over three times the size with two extra capture days, revise its annotations and introduce novel labels. LP-level annotations include bounding boxes, text, and legibility level, while vehicle-level annotations comprise make, model, type, and color. Image-level annotations feature camera identity, capture conditions (e.g., rain and faulty cameras), acquisition time, and day ID. We present a novel training procedure featuring an Exponential Moving Average-based loss function and a refined learning rate scheduler, addressing common mistakes in testing. These improvements enable a baseline model to achieve an 89.5% F1-score on the test set, considerably surpassing the previous state of the art. We further introduce a novel protocol to explicitly addresses camera contamination between training and evaluation splits, where results show a small impact. Dataset and code are publicly available at https://github.com/lmlwojcik/LPLCv2-Dataset.

Abstract

Fine-Grained Image Classification

Explore similar work

LPLCv2: An Expanded Dataset for Fine-Grained License Plate Legibility Classification

Revisiting Vehicle Color Recognition in Long-Tailed Surveillance Scenarios

Evaluating 2D and 3D-Aware Vision Foundation Models for Vehicle Attribute Recognition