cs.CVMar 20, 2026
SaveTemplate-based Object Detection Using a Foundation Model
Organizations: Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany · e.solutions GmbH, 91058 Erlangen, Germany
Abstract
Most currently used object detection methods are learning-based, and can detect objects under varying appearances. Those models require training and a training dataset. We focus on use cases with less data variation, but the requirement of being free of generation of training data and training. Such a setup is for example desired in automatic testing of graphical interfaces during software development, especially for continuous integration testing. In our approach, we use segments from segmentation foundation models and combine them with a simple feature-based classification method. This saves time and cost when changing the object to be searched or its design, as nothing has to be retrained and no dataset has to be created. We evaluate our method on the task of detecting and classifying icons in navigation maps, which is used to simplify and automate the testing of user interfaces in automotive industry. Our methods achieve results almost on par with learning-based object detection methods like YOLO, without the need for training.
Figures & tables
Figure 1: Overview of Our Workflow. We use a SAM model to segment all objects in the input image and use each segment as an object proposal. The object proposals can either be generated from the input images or the images with removed text which are generated as shown in Fig. 4 . To reduce the number of icon proposals we first compute the correlation of the color histograms with each histogram of the templates and reject candidates without matching templates. For the remaining templates we compare the metric values (LPIPS or CLIP) between the candidate and the template crops. If there are values below a certain threshold the template with the best metric value is predicted as match.
Figure 2: Example SAM Segment Images. The segmentation masks are colored and surrounded by a black bounding box for visualization. This includes icons, text, and also parts of the background. These proposals will afterwards be filtered during the classification process. The left image shows the result from SAM2.1, the right one the result from SAM3.
Figure 3: Comparisons Between the Color Histogram Correlation of Icons and Non-icons . The best correlations for proposals without complete icons, and the correlation values for each icon with the best matching proposal are counted. The correlations are grouped into bins with a width of 0.1 and the number of icons are scaled logarithmically. As none of the best matches to icons has a correlation below 0.5, this part can be filtered by applying a correlation threshold (dotted line), reducing the number of icon proposals.
Figure 4: Overview of the Font Removal Method. We start computing color clusters from optical character recognition (OCR) bounding boxes of a sample image. Then, we apply an independent component analysis (ICA) and cluster the projected colors. Similar clusters are then merged, and the best fitting cluster is selected by comparing the area filled with colors from each cluster inside and outside of OCR bounding boxes. Given the best cluster, we create image masks for each input image by filtering by the best color cluster and post-process by dilation and erosion. These image masks are afterwards used to inpaint the parts containing text before classification.
Figure 5: Qualitative Result of Template and Feature Matching Algorithms. Some qualitative results to template and feature matching approaches showcasing the problems when applying the methods to the given use case.
| Model | Std. Thresh. | Icons w/o Seg. Mask | #Predictions | |
| FastSAM small | no | 170 / 34,122 (0.498%) | 339,302 | 92.3 |
| FastSAM large | no | 10 / 34,122 (0.029%) | 230,944 | 376.5 |
| SAM2.1 small | yes | 1593 / 34,122 (4.669%) | 71,591 | 2,699.2 |
| SAM2.1 large | yes | 884 / 34,122 (2.591%) | 85,613 | 2,863.2 |
| SAM2.1 small | no | 43 / 34,122 (0.126%) | 235,718 | 3,947.0 |
| SAM2.1 large | no | 5 / 34,122 (0.015%) | 226,061 | 4,007.6 |
Table 1: Percentage of Icons Not Detected by the Segmentation Model . Some icons are not detected as objects by SAM, for example, if the icon is covered by much text, or if something close is predicted to be more certainly an object. The quality of the segments is given by the percentage of not detected icons and the mean intersection of the bounding box of the segment with the ground truth bounding box. The SAM2.1 models are evaluated with an equally distributed point grid having 64 points along the longer image dimension. The models are evaluated on dataset A .
Figure 6: Comparison Between Text Coverage and Detection Rate. The ground truth bounding boxes are grouped by the ratio of text coverage. The total number of by the classification undetected icons per text coverage bin is displayed for the method using LPIPS with and without inpainting.
| Method | in ms |
| SAM2.1 (large) | 4,007.6 |
| SAM3 | 640.8 |
| Mask Generation | 676.5 |
| Inpaint Anything | 5,383.0 |
Table 2: Time Consumption For Each Method Step. We measure the average time consumption for each part of the algorithm. For SAM2, mask generation, and Inpaint Anything, we measure the average time per image , as it scales not linearly with the icon count. For the classification and non-maximum suppression, we measure the average time per icon, as the number of icons varies between images. Additionally, for the classification only templates are compared that have an correlation higher than 90% of the highest correlation between template and the current icon proposal.
| no add. training (Ours) finetuned | Method Precision Recall Misclassifications SAM2.1+CLIP 0.9724 0.9640 0.0276 SAM2.1+LPIPS 0.9900 0.9876 0.0100 SAM2.1+IA+CLIP 0.9768 0.9711 0.0232 SAM2.1+IA+LPIPS 0.9958 0.9940 0.0042 SAM3+LPIPS 0.9875 0.9869 0.0125 SAM3+IA+LPIPS 0.9942 0.9937 0.0058 YOLOv8 0.9991 0.9996 0.0009 YOLOv11 0.9986 0.9994 0.0013 |
Table 3: Quantitative Results Dataset A. We compare the precision, recall, and number of misclassifications across the different methods. Our method is evaluated once with cosine comparison between the CLIP features and once with LPIPS between the candidate, generated with the SAM3 model/large SAM2.1 model, and the templates. Furthermore, we evaluate both variants after inpainting with InpaintAnything(IA) the text that covers the icon proposals before classification. The presented results are evaluated with the testing set from dataset A .
| no add. training (Ours) finetuned | Method Precision Recall Misclassifications SAM2.1+CLIP 0.9676 0.9686 0.0324 SAM2.1+LPIPS 0.9863 0.9844 0.0122 SAM2.1+IA+CLIP 0.9840 0.9794 0.0148 SAM2.1+IA+LPIPS 0.9948 0.9953 0.0038 SAM3+LPIPS 0.9894 0.9869 0.0099 SAM3+IA+LPIPS 0.9972 0.9975 0.0022 YOLO8 0.9993 0.9999 0.0007 YOLO11 0.9995 1.0000 0.0005 |
Table 4: Quantitative Results Dataset B. We compare the precision, recall, and number of misclassifications across the different methods with and without inpainting with InpaintAnything (IA). The presented results are evaluated with the testing set from dataset B and in case of SAM2.1 with the large model.
Figure 7: Icon Bounding Boxes in Comparison. The first two image rows show the SAM segments and ground truth bounding boxes of the icons of dataset A , the third and fourth row of the icons of dataset B .
Figure 8: Failure cases. The first row contains the icon proposals by the SAM model. The second and third row, the predictions and ground truth labels.
Figure 9: Text Inpainting Results. The images are from left to right the original image, the computed text segmentation masks, and the inpainted image.
Figure 10: Annotated Result Image. The image contains annotations for all detected icons. Each detected icon class is shown in the lower part of the image. The image was classified by using the LPIPS variant of our method.
Figure 11: Additional Results Template Matching. The images show results for the template matching with all templates from the dataset with different thresholds. The templates are scaled between 50% and 200%. All results are gathered with the normalized squared difference as a metric. Therefore, lower metric values are better.
Figure 12: Additional Results Template Matching. The images show results for the template matching with one template each from the dataset with different thresholds. The templates are scaled between 50% and 150%. All results are gathered with normalized cross correlation as a metric. Therefore, higher metric values are better.
| Metric | No Optimization | Area Comp. | Area Comp.& Correlation Comp. |
| LPIPS | 45.8 ms | 22.0 ms | 8.4 ms |
| CLIP | 16.2 ms | 10.9 ms | 7.6 ms |
Table 5: Time Comparison For Classification With And Without Optimization . We evaluated the mean time consumption per icon proposal with different optimizations. One optimization is the filtering of icons that are less than 1/4 or more than 2 times the size of the given templates (Area Comp.). The other is the filtering of templates that have a correlation below 90% of the highest correlation between the icon proposal and a template.
Figure 13: Precision And Recall Using LPIPS With And Without Using Histogram Correlation. The images show the precision and recall while changing the threshold using the LPIPS method once with applying the histogram correlation and once without. The left image shows the results without inpainting, the right one with inpainting.
Figure 14: Comparisons Between the Color Histogram Correlation of Icons, Partial Icons, and Non-icons . The best correlation values for proposals without intersections with icons, and the correlation values for each icon with the best matching proposal are counted, the same is done for partial matches between icon proposals and icons. The correlation values are grouped into bins with a width of 0.1 and the number of icons are scaled logarithmically. As none of the best matches to icons has a correlation below 0.5, this part can be filtered by applying a correlation threshold (dotted line), reducing the number of icon proposals.
Figure 15: Error Case Visualization. Some error cases are depicted with red bounding boxes. The corresponding templates are annotated in the bottom line and each detected icon is labeled with the predicted class id. The results are from the not inpainted images, while using LPIPS as metric.
Figure 16: Precision And Recall Using CLIP. The images show the precision and recall while changing the threshold using the CLIP method. The left image shows the results without inpainting, the right one with inpainting.
Figure 17: Precision And Recall Using LPIPS. The images show the precision and recall while changing the threshold using the LPIPS method. The left image shows the results without inpainting, the right one with inpainting.
More in this topic
Object Detection
Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.
See all 624 papers