Most currently used object detection methods are learning-based, and can detect objects under varying appearances. Those models require training and a training dataset. We focus on use cases with less data variation, but the requirement of being free of generation of training data and training. Such a setup is for example desired in automatic testing of graphical interfaces during software development, especially for continuous integration testing. In our approach, we use segments from segmentation foundation models and combine them with a simple feature-based classification method. This saves time and cost when changing the object to be searched or its design, as nothing has to be retrained and no dataset has to be created. We evaluate our method on the task of detecting and classifying icons in navigation maps, which is used to simplify and automate the testing of user interfaces in automotive industry. Our methods achieve results almost on par with learning-based object detection methods like YOLO, without the need for training.
Figures & tables
Figure 1: Overview of Our Workflow. We use a SAM model to segment all objects in the input image and use each segment as an object proposal. The object proposals can either be generated from the input images or the images with removed text which are generated as shown in Fig. 4 . To reduce the number of icon proposals we first compute the correlation of the color histograms with each histogram of the templates and reject candidates without matching templates. For the remaining templates we compare the metric values (LPIPS or CLIP) between the candidate and the template crops. If there are values below a certain threshold the template with the best metric value is predicted as match.
Figure 2: Example SAM Segment Images. The segmentation masks are colored and surrounded by a black bounding box for visualization. This includes icons, text, and also parts of the background. These proposals will afterwards be filtered during the classification process. The left image shows the result from SAM2.1, the right one the result from SAM3.
Figure 3: Comparisons Between the Color Histogram Correlation of Icons and Non-icons . The best correlations for proposals without complete icons, and the correlation values for each icon with the best matching proposal are counted. The correlations are grouped into bins with a width of 0.1 and the number of icons are scaled logarithmically. As none of the best matches to icons has a correlation below 0.5, this part can be filtered by applying a correlation threshold (dotted line), reducing the number of icon proposals.
Figure 4: Overview of the Font Removal Method. We start computing color clusters from optical character recognition (OCR) bounding boxes of a sample image. Then, we apply an independent component analysis (ICA) and cluster the projected colors. Similar clusters are then merged, and the best fitting cluster is selected by comparing the area filled with colors from each cluster inside and outside of OCR bounding boxes. Given the best cluster, we create image masks for each input image by filtering by the best color cluster and post-process by dilation and erosion. These image masks are afterwards used to inpaint the parts containing text before classification.
Figure 5: Qualitative Result of Template and Feature Matching Algorithms. Some qualitative results to template and feature matching approaches showcasing the problems when applying the methods to the given use case.
Model
Std. Thresh.
Icons w/o Seg. Mask ↓
#Predictions
tˉimg↓
FastSAM small
no
170 / 34,122 (0.498%)
339,302
92.3
FastSAM large
no
10 / 34,122 (0.029%)
230,944
376.5
SAM2.1 small
yes
1593 / 34,122 (4.669%)
71,591
2,699.2
SAM2.1 large
yes
884 / 34,122 (2.591%)
85,613
2,863.2
SAM2.1 small
no
43 / 34,122 (0.126%)
235,718
3,947.0
SAM2.1 large
no
5 / 34,122 (0.015%)
226,061
4,007.6
Table 1: Percentage of Icons Not Detected by the Segmentation Model . Some icons are not detected as objects by SAM, for example, if the icon is covered by much text, or if something close is predicted to be more certainly an object. The quality of the segments is given by the percentage of not detected icons and the mean intersection of the bounding box of the segment with the ground truth bounding box. The SAM2.1 models are evaluated with an equally distributed point grid having 64 points along the longer image dimension. The models are evaluated on dataset A .
Figure 6: Comparison Between Text Coverage and Detection Rate. The ground truth bounding boxes are grouped by the ratio of text coverage. The total number of by the classification undetected icons per text coverage bin is displayed for the method using LPIPS with and without inpainting.
Method
tˉimg in ms
SAM2.1 (large)
4,007.6
SAM3
640.8
Mask Generation
676.5
Inpaint Anything
5,383.0
Table 2: Time Consumption For Each Method Step. We measure the average time consumption for each part of the algorithm. For SAM2, mask generation, and Inpaint Anything, we measure the average time per image tˉimg , as it scales not linearly with the icon count. For the classification and non-maximum suppression, we measure the average time per icon, as the number of icons varies between images. Additionally, for the classification only templates are compared that have an correlation higher than 90% of the highest correlation between template and the current icon proposal.
Table 3: Quantitative Results Dataset A. We compare the precision, recall, and number of misclassifications across the different methods. Our method is evaluated once with cosine comparison between the CLIP features and once with LPIPS between the candidate, generated with the SAM3 model/large SAM2.1 model, and the templates. Furthermore, we evaluate both variants after inpainting with InpaintAnything(IA) the text that covers the icon proposals before classification. The presented results are evaluated with the testing set from dataset A .
Table 4: Quantitative Results Dataset B. We compare the precision, recall, and number of misclassifications across the different methods with and without inpainting with InpaintAnything (IA). The presented results are evaluated with the testing set from dataset B and in case of SAM2.1 with the large model.
Figure 7: Icon Bounding Boxes in Comparison. The first two image rows show the SAM segments and ground truth bounding boxes of the icons of dataset A , the third and fourth row of the icons of dataset B .
Figure 8: Failure cases. The first row contains the icon proposals by the SAM model. The second and third row, the predictions and ground truth labels.
Figure 9: Text Inpainting Results. The images are from left to right the original image, the computed text segmentation masks, and the inpainted image.
Figure 10: Annotated Result Image. The image contains annotations for all detected icons. Each detected icon class is shown in the lower part of the image. The image was classified by using the LPIPS variant of our method.
Figure 11: Additional Results Template Matching. The images show results for the template matching with all templates from the dataset with different thresholds. The templates are scaled between 50% and 200%. All results are gathered with the normalized squared difference as a metric. Therefore, lower metric values are better.
Figure 12: Additional Results Template Matching. The images show results for the template matching with one template each from the dataset with different thresholds. The templates are scaled between 50% and 150%. All results are gathered with normalized cross correlation as a metric. Therefore, higher metric values are better.
Metric
No Optimization
Area Comp.
Area Comp.& Correlation Comp.
LPIPS
45.8 ms
22.0 ms
8.4 ms
CLIP
16.2 ms
10.9 ms
7.6 ms
Table 5: Time Comparison For Classification With And Without Optimization . We evaluated the mean time consumption per icon proposal with different optimizations. One optimization is the filtering of icons that are less than 1/4 or more than 2 times the size of the given templates (Area Comp.). The other is the filtering of templates that have a correlation below 90% of the highest correlation between the icon proposal and a template.
Figure 13: Precision And Recall Using LPIPS With And Without Using Histogram Correlation. The images show the precision and recall while changing the threshold using the LPIPS method once with applying the histogram correlation and once without. The left image shows the results without inpainting, the right one with inpainting.
Figure 14: Comparisons Between the Color Histogram Correlation of Icons, Partial Icons, and Non-icons . The best correlation values for proposals without intersections with icons, and the correlation values for each icon with the best matching proposal are counted, the same is done for partial matches between icon proposals and icons. The correlation values are grouped into bins with a width of 0.1 and the number of icons are scaled logarithmically. As none of the best matches to icons has a correlation below 0.5, this part can be filtered by applying a correlation threshold (dotted line), reducing the number of icon proposals.
Figure 15: Error Case Visualization. Some error cases are depicted with red bounding boxes. The corresponding templates are annotated in the bottom line and each detected icon is labeled with the predicted class id. The results are from the not inpainted images, while using LPIPS as metric.
Figure 16: Precision And Recall Using CLIP. The images show the precision and recall while changing the threshold using the CLIP method. The left image shows the results without inpainting, the right one with inpainting.
Figure 17: Precision And Recall Using LPIPS. The images show the precision and recall while changing the threshold using the LPIPS method. The left image shows the results without inpainting, the right one with inpainting.
Industrial object detection systems typically rely on large annotated datasets, which are expensive to collect and challenging to maintain in industrial scenarios where the inventory of objects changes frequently. This work addresses the challenge of few-shot object detection in such industrial scenarios, where only a limited number of labeled samples are available for newly introduced objects. We present a detection framework that leverages vision foundation models to recognize objects with minimal supervision. The method constructs class prototypes from a small set of reference samples by extracting feature representations. For a given query scene during inference, object regions are generated using a segmentation model, and feature embeddings are extracted and matched with class prototypes using similarity matching. We evaluate the detection method on three established industrial datasets from the Benchmark for 6D Object Pose Estimation benchmark following the official 2D object detection evaluation protocol. We demonstrate competitive detection performance, improving AP by 6.9% compared to the state-of-the-art training-free detection methods. Furthermore, the presented method is able to onboard new objects using only a few reference images, without requiring any CAD models or large annotated datasets. These properties make the approach well-suited for real-world industrial applications.
Hari Prasanth S. M., Nilusha Jayawickrama, Risto Ojala
Department of Energy and Mechanical Engineering, Aalto University, Espoo, 02150, Finland.
Autonomous vehicles operating in open-world scenarios are inevitably confronted with previously unknown objects, such as exotic animals or loose cargo. The reliable detection and segmentation of these out-of-distribution (OOD) objects is therefore crucial for a safe understanding of the environment and decision-making. Most existing approaches require access to OOD training samples, retraining of the segmentation backbone, or dedicated auxiliary architectures, limiting their practical applicability. We propose a training-free method that derives dense OOD scores directly from the confidence predictions of a foundation segmentation model, without any task-specific fine-tuning or access to anomalous data. To improve the robustness of our OOD segmentation, geometric information from monocular depth estimation is incorporated into the decision process, providing complementary cues to uncertainty-based predictions. We evaluate the proposed method on the SegmentMeIfYouCan benchmark and additionally assess its performance on OOD tracking in video sequences, reflecting the temporal nature of real-world perception systems. The method performs strongly on road-centered benchmarks.
Open-vocabulary object detection test-time adaptation (OVOD-TTA) aims to address the performance degradation that pre-trained base models suffer when encountering image-domain shifts. Existing source-free OVOD-TTA methods rely either on refined test-time information for re-scoring or on pseudo-labels for self-training, leading to significant accuracy degradation when initial predictions are poor. Meanwhile, most conventional source-domain estimation methods recover abstract, sparse representations suitable for the classification task, but fail to capture the dense, concrete features required for detection. To address these issues, we propose PISA, a novel source-free OVOD-TTA method that can be seamlessly integrated into open-vocabulary visual backbones. The core components of our method are the Corruption-Invariant Feature Extractor (CIFE), the Feature Alignment Module (FAM), and a multi-scale alignment framework (BAA). To capture detection-suitable features, we develop CIFE to exploit the invariance of CLIP's visual features across corrupted images, ensuring robustness against various corruptions. We further develop FAM and BAA for the pre-training and adaptation to transform the corruption-invariant features into pseudo-individual source-domain features that are close to the original source-domain features. In this way, dense and concrete pseudo-individual source-domain features are used for supervision instead of unreliable pseudo-label signals. Experiments on the corrupted VOC-C, COCO-C, and LVIS-C benchmarks across three base models demonstrate that PISA substantially improves both the localization precision and the category recognition accuracy of the original models. Notably, PISA achieves state-of-the-art performance without requiring access to source-domain data, surpassing existing methods by 3.92% in AP@50% on COCO-C.