PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation

Melanie Neubauer; Elmar Rueckert; Christian Rauch

cs.CVApr 7, 2026

PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation

Authors: Melanie Neubauer, Elmar Rueckert, Christian Rauch

Organizations: Chair of Cyber Physical Sytems, Technical University of Leoben, 8700 Leoben, Austria

Abstract

Detecting unseen anomalies in unstructured environments presents a critical challenge for industrial and agricultural applications such as material recycling and weeding. Existing perception systems frequently fail to satisfy the strict operational requirements of these domains, specifically real-time processing, pixel-level segmentation precision, and robust accuracy, due to their reliance on exhaustively annotated datasets. To address these limitations, we propose a weakly supervised pipeline for object segmentation and classification using weak image-level supervision called 'Patch Aggregation for Segmentation of Targets and Anomalies' (PASTA). By comparing an observed scene with a nominal reference, PASTA identifies Target and Anomaly objects through distribution analysis in self-supervised Vision Transformer (ViT) feature spaces. Our pipeline utilizes semantic text-prompts via the Segment Anything Model 3 to guide zero-shot object segmentation. Evaluations on a custom steel scrap recycling dataset and a plant dataset demonstrate a 75.8% training time reduction of our approach to domain-specific baselines. While being domain-agnostic, our method achieves superior Target (up to 88.3% IoU) and Anomaly (up to 63.5% IoU) segmentation performance in the industrial and agricultural domain.

Figures & tables

Fig. 2 : Dataset comparison for SteelDS and PhenoBench. Input images and ground truth segmentations (blue: target , red: anomaly ) highlight the structural difference between the mixed baseline ( $D_{\mathcal{T}\cup\mathcal{A}}$ ) and Anomaly-free reference ( $D_{\mathcal{T}}$ ).

Fig. 3 : Reconstructed train and inference pipeline. Training (top): SAM 3 extracts object patches, which are embedded and sorted by frequency. High-frequency embeddings define the Target Feature Bag, whereas low-frequency embeddings are isolated as anomalies. Inference (bottom): Extracted features are classified against the target bag.

Fig. 4 : Architecture of PASTA. The training phase (top) identifies anomaly-specific feature clusters via distribution analysis between a mixed baseline and an anomaly-free reference dataset. The inference phase (bottom) applies the trained model for dense patchwise anomaly classification (Evaluation A in Section IV-B ), which is subsequently refined by SAM 3 semantic prompting for precise object segmentation (Evaluation B in Section IV-C ).

Dataset	$\|\mathcal{D}_{\mathcal{T}\cup\mathcal{A}}\|$	$\|\mathcal{D}_{\mathcal{T}}\|$	$\|\mathcal{D}_{test}\|$
PhenoBench	1407	1407	772
SteelDS	3831	900	956
SteelDS Extended	16854	3677	3742

TABLE I : Number of samples used for training (

\mathcal{D}_{\mathcal{T}\cup\mathcal{A}}

\mathcal{D}_{\mathcal{T}}

) and testing (

\mathcal{D}_{test}

) on the two datasets.

Approach	Backbone	IoU B	IoU T	IoU A	mIoU
PB (WaW)	BioCLIP	98.85	68.06	11.13	59.34
SDS (Ours)	BioCLIP	99.35	64.20	35.67	66.41
	CLIP ViT-B-16	99.35	74.81	8.80	60.98
	CLIP ViT-B-32 LAION2b	99.35	86.20	54.34	79.96
	DINOv3 ConvNeXt-Tiny	99.35	44.86	27.84	57.35
	DINOv3 ViT-Small	99.35	39.93	24.06	54.44

TABLE II : Average results (mIoU for B ackground, T arget, A nomaly) of the reconstructed baseline and its variations ( P heno B ench, S teel DS ) over five random seeds.

Dataset (Pipeline)	Seg. Model	Patches	$k_{sphere}$	$k_{v}$	Time [h]
PhenoBench (WaW [ 19 ] )	SAM	64330	100	10	$\approx$ 2.8
SteelDS (Ours)	SAM 3	11247	260	10	$\approx$ 0.8
PhenoBench (Ours)	SAM 3	15565	120	10	$\approx$ 0.7

TABLE III : Feature patch extraction methods for PhenoBench and SteelDS. Our approach produce fewer but more relevant patches, resulting in a reduction of the training time by more than 70%.

Panel of Figure 7 — Fig. 5 : Backbone Robustness: Impact of hypersphere density ( $k_{sphere}$ ) on Target and Anomaly IoU. Shaded regions represent the standard deviation over 5 seeds. In the industrial domain (SteelDS, top), both Target and Anomaly accuracy generally improve with larger $k_{sphere}$ . In the agricultural domain (PhenoBench, bottom), while Target accuracy generally improves, Anomaly accuracy often degrades as the expanding hypersphere starts encompassing anomaly features.

Panel of Figure 8 — Fig. 6 : Parameter Sensitivity Heatmaps illustrating Anomaly IoU sensitivity to Density ( $k_{sphere}$ ) versus Feature Voting K ( $k_{v}$ ). (a) SteelDS benefits from high density, whereas (b) PhenoBench requires constrained density to maintain discriminative power against anomalies.

	K=10	K=15	K=20	K=25
SteelDS
CLIP ViT-B-16	0.0 (0.0)	0.3 (0.3)	0.3 (0.4)	1.6 (1.0)
CLIP ViT-B-32 L.2b	2.7 (1.5)	2.4 (2.1)	2.6 (1.4)	2.7 (1.5)
CLIP ViT-B-32	0.0 (0.0)	0.2 (0.3)	0.3 (0.5)	0.4 (0.6)
DINOv3 ConvN.-T.	15.3 (11.7)	31.1 (5.3)	25.1 (11.7)	23.8 (13.5)
DINOv3 ViT-Small	20.7 (15.8)	29.6 (10.1)	35.4 (2.2)	34.0 (2.0)

TABLE IV : Patch-level Anomaly IoU [%] on SteelDS and PhenoBench across varying cluster counts (

K

). Results are averaged over 5 seeds, with the standard deviation provided in brackets. DINOv3 architectures significantly outperform contrastive models, peaking at 35.4% IoU for SteelDS.

Fig. 7 : Qualitative segmentation results. Top: original image, middle: ground truth segments, bottom: PASTA predictions. blue: target , red: anomaly .

	Model Setup [s]	Inference Ev. A [ms]	Inference Ev. B [ms]
SteelDS
CLIP ViT-B-16	27.55	112	346
CLIP ViT-B-32 L.2b	25.30	96	332
CLIP ViT-B-32	27.08	96	333
DINOv3 ConvN.-T.	29.57	97	332
DINOv3 ViT-Small	30.21	104	339

TABLE V : Training and inference time analysis. Model Setup covers the training time, while Inference is the processing time per image during test on the two approaches ( Ev aluation A and Ev aluation B ).

	Target IoU				Anomaly IoU
Model	K=10	K=15	K=20	K=25	K=10	K=15	K=20	K=25
SteelDS
CLIP ViT-B-16	76.5 (0.0)	65.3 (13.0)	64.9 (15.2)	31.7 (17.8)	0.0 (0.0)	1.7 (2.4)	2.2 (3.4)	12.6 (4.6)
CLIP ViT-B-32 L.2b	62.0 (7.1)	62.9 (14.5)	59.9 (16.0)	58.9 (13.3)	13.7 (5.6)	11.1 (9.2)	13.1 (4.1)	13.2 (4.7)
CLIP ViT-B-32	76.4 (0.1)	75.4 (2.3)	75.9 (0.9)	74.8 (1.7)	0.1 (0.2)	1.2 (2.2)	1.5 (2.7)	2.5 (3.7)
DINOv3 ConvN.-T.	46.6 (28.9)	80.3 (15.2)	83.6 (8.4)	82.7 (9.2)	23.2 (17.3)	49.2 (11.3)	42.7 (16.8)	39.7 (22.9)

TABLE VI : Hierarchical IoU evaluation for Target and Anomaly performance. All values are given in percent (%), with standard deviations in brackets. Best and second best values per dataset and metric are marked in bold and underlined.

Weakly Supervised Image Segmentation

Weakly supervised image segmentation learns pixel-level regions from cheaper, incomplete labels such as image tags, boxes, points, scribbles, or reports.

See all 46 papers

SuperADD: Training-free Class-agnostic Anomaly Segmentation -- CVPR 2026 VAND 4.0 Workshop Challenge Industrial Track
May 14, 2026Lukas Roming, Felix Lehnerer, Jonas V. Funk +4OOD Generalization Industrial Anomaly Detection
Fraunhofer IOSB, Fraunhoferstr. 1, 76131 Karlsruhe, Germany · Independent Research
DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection
Jul 27, 2026Jyun-Ze Tang, Po-Han Huang, Ming-Ching Chang +2Few-Shot Anomaly Detection Anomaly Localization
Inventec Corporation, Taipei, Taiwan · University at Albany, State University of New York, NY, USA
SAFe: Segment-guided Aggregation of Feature Densities for Anomaly-aware Segmentation
Sep 21, 2026Anja Delić, Jurica Runtas, Marin Oršić +2Anomaly Localization Density Estimation
University of Zagreb Faculty of Electrical Engineering and Computing Laboratory for Autonomous Systems and Mobile Robotics Zagreb, Croatia

May 14, 2026J/K move · Enter open · S save

SuperADD: Training-free Class-agnostic Anomaly Segmentation -- CVPR 2026 VAND 4.0 Workshop Challenge Industrial Track

Lukas Roming, Felix Lehnerer, Jonas V. Funk, Andreas Michel+3

Fraunhofer IOSB, Fraunhoferstr. 1, 76131 Karlsruhe, Germany · Independent Research

OOD Generalization Industrial Anomaly Detection Anomaly Localization

Open paper AskPDF arXiv

Visual anomaly detection (AD) for industrial inspection is a highly relevant task in modern production environments. The problem becomes particularly challenging when training and deployment data differ due to changes in acquisition conditions during production. In the VAND 4.0 Industrial Track, models must remain robust under distribution shifts such as varying illumination and their performance is assessed on the MVTec AD 2 dataset. To address this setting, we propose a training-free and class-agnostic anomaly detection pipeline based on the work of SuperAD. Our approach improves generalization through several modifications designed to enhance robustness under distribution shifts. These adaptations include using a DINOv3 backbone, overlapping patch-wise processing, intensity-based augmentations, improved memory-bank subsampling for better coverage of the data distribution, and iterative morphological closing for cleaner and more spatially consistent anomaly maps. Unlike methods that rely on class-specific architectures or per-class hyperparameter tuning, our method uses a single architecture and one shared hyperparameter configuration across all object classes. This makes the approach well suited for industrial deployment, where product variants and appearance changes must be handled with minimal adaptation effort. We achieve segmentation F1 scores of

62.61\%

57.42\%

, and

54.35\%

on test public, private, and private mixed of MVTec AD 2 respectively, thereby outperforming SuperAD and other state-of-the-art methods. Code is available at https://github.com/LukasRoom/SuperADD.

Abstract

Weakly Supervised Image Segmentation

Explore similar work

SuperADD: Training-free Class-agnostic Anomaly Segmentation -- CVPR 2026 VAND 4.0 Workshop Challenge Industrial Track

DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection

SAFe: Segment-guided Aggregation of Feature Densities for Anomaly-aware Segmentation