PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation
Organizations: Chair of Cyber Physical Sytems, Technical University of Leoben, 8700 Leoben, Austria
Abstract
Detecting unseen anomalies in unstructured environments presents a critical challenge for industrial and agricultural applications such as material recycling and weeding. Existing perception systems frequently fail to satisfy the strict operational requirements of these domains, specifically real-time processing, pixel-level segmentation precision, and robust accuracy, due to their reliance on exhaustively annotated datasets. To address these limitations, we propose a weakly supervised pipeline for object segmentation and classification using weak image-level supervision called 'Patch Aggregation for Segmentation of Targets and Anomalies' (PASTA). By comparing an observed scene with a nominal reference, PASTA identifies Target and Anomaly objects through distribution analysis in self-supervised Vision Transformer (ViT) feature spaces. Our pipeline utilizes semantic text-prompts via the Segment Anything Model 3 to guide zero-shot object segmentation. Evaluations on a custom steel scrap recycling dataset and a plant dataset demonstrate a 75.8% training time reduction of our approach to domain-specific baselines. While being domain-agnostic, our method achieves superior Target (up to 88.3% IoU) and Anomaly (up to 63.5% IoU) segmentation performance in the industrial and agricultural domain.
Figures & tables
| Dataset | |||
| PhenoBench | 1407 | 1407 | 772 |
| SteelDS | 3831 | 900 | 956 |
| SteelDS Extended | 16854 | 3677 | 3742 |
| Approach | Backbone | IoU B | IoU T | IoU A | mIoU |
| PB (WaW) | BioCLIP | 98.85 | 68.06 | 11.13 | 59.34 |
| SDS (Ours) | BioCLIP | 99.35 | 64.20 | 35.67 | 66.41 |
| CLIP ViT-B-16 | 99.35 | 74.81 | 8.80 | 60.98 | |
| CLIP ViT-B-32 LAION2b | 99.35 | 86.20 | 54.34 | 79.96 | |
| DINOv3 ConvNeXt-Tiny | 99.35 | 44.86 | 27.84 | 57.35 | |
| DINOv3 ViT-Small | 99.35 | 39.93 | 24.06 | 54.44 |
| Dataset (Pipeline) | Seg. Model | Patches | Time [h] | ||
| PhenoBench (WaW [ 19 ] ) | SAM | 64330 | 100 | 10 | 2.8 |
| SteelDS (Ours) | SAM 3 | 11247 | 260 | 10 | 0.8 |
| PhenoBench (Ours) | SAM 3 | 15565 | 120 | 10 | 0.7 |
| K=10 | K=15 | K=20 | K=25 | |
| SteelDS | ||||
| CLIP ViT-B-16 | 0.0 (0.0) | 0.3 (0.3) | 0.3 (0.4) | 1.6 (1.0) |
| CLIP ViT-B-32 L.2b | 2.7 (1.5) | 2.4 (2.1) | 2.6 (1.4) | 2.7 (1.5) |
| CLIP ViT-B-32 | 0.0 (0.0) | 0.2 (0.3) | 0.3 (0.5) | 0.4 (0.6) |
| DINOv3 ConvN.-T. | 15.3 (11.7) | 31.1 (5.3) | 25.1 (11.7) | 23.8 (13.5) |
| DINOv3 ViT-Small | 20.7 (15.8) | 29.6 (10.1) | 35.4 (2.2) | 34.0 (2.0) |
| Model Setup [s] | Inference Ev. A [ms] | Inference Ev. B [ms] | |
| SteelDS | |||
| CLIP ViT-B-16 | 27.55 | 112 | 346 |
| CLIP ViT-B-32 L.2b | 25.30 | 96 | 332 |
| CLIP ViT-B-32 | 27.08 | 96 | 333 |
| DINOv3 ConvN.-T. | 29.57 | 97 | 332 |
| DINOv3 ViT-Small | 30.21 | 104 | 339 |
| Target IoU | Anomaly IoU | |||||||
| Model | K=10 | K=15 | K=20 | K=25 | K=10 | K=15 | K=20 | K=25 |
| SteelDS | ||||||||
| CLIP ViT-B-16 | 76.5 (0.0) | 65.3 (13.0) | 64.9 (15.2) | 31.7 (17.8) | 0.0 (0.0) | 1.7 (2.4) | 2.2 (3.4) | 12.6 (4.6) |
| CLIP ViT-B-32 L.2b | 62.0 (7.1) | 62.9 (14.5) | 59.9 (16.0) | 58.9 (13.3) | 13.7 (5.6) | 11.1 (9.2) | 13.1 (4.1) | 13.2 (4.7) |
| CLIP ViT-B-32 | 76.4 (0.1) | 75.4 (2.3) | 75.9 (0.9) | 74.8 (1.7) | 0.1 (0.2) | 1.2 (2.2) | 1.5 (2.7) | 2.5 (3.7) |
| DINOv3 ConvN.-T. | 46.6 (28.9) | 80.3 (15.2) | 83.6 (8.4) | 82.7 (9.2) | 23.2 (17.3) | 49.2 (11.3) | 42.7 (16.8) | 39.7 (22.9) |
Weakly Supervised Image Segmentation
Weakly supervised image segmentation learns pixel-level regions from cheaper, incomplete labels such as image tags, boxes, points, scribbles, or reports.
See all 46 papers