Prompt-Driven Lightweight Foundation Model for Instance Segmentation-Based Fault Detection in Freight Trains

Guodong Sun; Qihang Liang; Xingyu Pan; Moyun Liu; Yang Zhang

cs.CVMar 13, 2026

Prompt-Driven Lightweight Foundation Model for Instance Segmentation-Based Fault Detection in Freight Trains

Authors: Guodong Sun, Qihang Liang, Xingyu Pan, Moyun Liu, Yang Zhang

Organizations: the School of Mechanical Engineering, Hubei University of Technology, Wuhan 430068, China · The Hubei Key Laboratory of Modern Manufacturing Quality Engineering, Hubei University of Technology, Wuhan 430068, China · the School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China

Abstract

Accurate visual fault detection in freight trains remains a critical challenge for intelligent transportation system maintenance, due to complex operational environments, structurally repetitive components, and frequent occlusions or contaminations in safety-critical regions. Conventional instance segmentation methods based on convolutional neural networks and Transformers often suffer from poor generalization and limited boundary accuracy under such conditions. To address these challenges, we propose a lightweight self-prompted instance segmentation framework tailored for freight train fault detection. Our method leverages the Segment Anything Model by introducing a self-prompt generation module that automatically produces task-specific prompts, enabling effective knowledge transfer from foundation models to domain-specific inspection tasks. In addition, we adopt a Tiny Vision Transformer backbone to reduce computational cost, making the framework suitable for real-time deployment on edge devices in railway monitoring systems. We construct a domain-specific dataset collected from real-world freight inspection stations and conduct extensive evaluations. Experimental results show that our method achieves 74.6 $AP^{\text{box}}$ and 74.2 $AP^{\text{mask}}$ on the dataset, outperforming existing state-of-the-art methods in both accuracy and robustness while maintaining low computational overhead. This work offers a deployable and efficient vision solution for automated freight train inspection, demonstrating the potential of foundation model adaptation in industrial-scale fault diagnosis scenarios. Project page: https://github.com/MVME-HBUT/SAM_FTI-FDet.git

Figures & tables

Panel of Figure 1 — Fig. 1: Visualization results of activation maps from various detectors. (a) Ground truth annotations for each image. (b) Activation maps of Mask R-CNN. (c) Activation maps of SAM. (d) Our proposed prompter-driven SAM detection method. As the activation maps exemplify, our method can more accurately capture the region of interest due to the efficient prompt mechanism.

Fig. 2: An overview of our proposed SAM-based framework for visual fault detection of freight trains is presented. The network is designed to leverage a prompt generator to provide high-quality prompts for TinyViT-SAM, thereby enhancing the segmentation capability of the model. Furthermore, spatial and hierarchical embeddings are incorporated to improve the image perception ability of TinyViT-SAM, enabling efficient and low-cost optimization.

Panel of Figure 3 — Fig. 3: The visualization comparison results of feature maps with and without prompts in the mask decoder stage are presented based on three typical viewpoints. (a) Ground truth annotations. (b) Feature maps generated using a conventional detection head without prompt guidance. (c) Feature maps obtained after embedding prompts in the mask decoder stage. With the proposed architecture, the target regions in (a) can be reliably detected.

Fig. 4: Image acquirement in the wild. (a) Side view. (b) Bottom view.

Fig. 5: Visualization samples of the dataset, where “T” indicates Normal and “F” indicates Damaged or Missing. For brake shoes, the primary criterion is their thickness. For other parts, the main criterion is the presence of damage or absence.

Part	Type	Total Images		Part	Type	Total Images
Part	Type	train	val	Part	Type	train	val
Bogie Block Key	T	725	244	Cutoff Plug	T	665	286
Bogie Block Key	F	201	101	Cutoff Plug	F	382	371
Brake Shoe Pin	T	442	302	Clamping Bolt	T	646	224
Brake Shoe Pin	F	363	53	Clamping Bolt	F	157	124
Bearing Saddle	T	720	244	Dust Collector	T	443	247

TABLE I: Freight Train Fault Detection Dataset.

Method		Backbone	$AP^{box}$	$AP^{box}_{50}$	$AR^{box}_{50-95}$	$AP^{mask}$	$AP^{mask}_{50}$	$AR^{mask}_{50-95}$	Model Size	GFLOPs	FPS	Params
CNN	Mask R-CNN [ 15 ]	ResNet50	70.1	94.8	78.2	70.7	93.7	77.9	336.4	234	44.6	44.0
	PointRend [ 20 ]	ResNet50	71.3	94.2	78.4	71.3	94.2	78.4	431.6	186	38.1	56.4
	YOLACT [ 16 ]	ResNet50	63.6	93.0	71.2	68.4	93.0	76.0	272.0	337	42.8	31.4
	SOLOv2 [ 18 ]	ResNet50	–	–	–	69.8	93.8	76.8	353.9	283	33.7	46.3
	CondInst [ 21 ]	ResNet50	69.4	93.2	74.7	67.3	92.8	73.3	272.0	250	27.1	48.2
	SparseInst [ 22 ]	ResNet50	–	–	–	66.5	87.9	72.2	399.8	221	76.3	47.9

TABLE II: Comparison of accuracy with the state-of-the-art methods on the freight trains dataset.

Panel of Figure 8 — Fig. 6: Comparative visualization of segmentation results for image samples. The “ * ” denotes the Swin-T architecture. Evidently, the results produced by SAM FTI-FDet exhibit the closest alignment with the ground truth annotations. This advantage primarily stems from the proposed efficient prompting and knowledge transfer strategy, which substantially enhances the model’s capability to capture instance-specific semantics.

Method	Prompt Type	$AP^{box}$	$AP^{box}_{50}$	$AP^{mask}$	$AP^{mask}_{50}$
SAM [ 9 ]	gd-bbx	–	–	66.3	91.5
MobileSAM [ 10 ]	gd-bbx	–	–	64.2	90.9
FastSAM [ 30 ]	gd-bbx	–	–	63.0	89.8
SAM-det [ 8 ]	bbox	72.0	93.2	57.7	73.2
RSPrompter [ 8 ]	anchor	67.1	93.2	68.4	92.6
RSPrompter [ 8 ]	query	72.7	92.7	71.9	93.2

TABLE III: Prompt evaluation on the freight trains dataset.

Fig. 7: Comparison of total training loss between SAM FTI-FDet and RSPrompter. The proposed method exhibits markedly faster convergence, indicating that the self-prompt mechanism provides more efficient and semantically aligned guidance during optimization. This superior convergence behavior highlights the higher prompt-generation efficiency of our approach compared with query-based prompting.

Method	Head	$AP^{box}$	$AP^{box}_{50}$	$AP^{mask}$	$AP^{mask}_{50}$
SAM	Mask R-CNN	72.5	94.7	71.8	94.7
SAM	Mask2Former	72.9	92.9	72.1	94.0
SAM FTI-FDet-PF	Mask R-CNN	70.9	94.9	72.1	94.7
SAM FTI-FDet-PF	Mask2Former	73.2	93.4	72.9	93.8

TABLE IV: Decoder-head evaluation on the freight trains dataset.

Backbone	Pretrain	$AP^{box}$	$AP^{box}_{50}$	$AP^{mask}$	$AP^{mask}_{50}$	Params
ResNet18	ImageNet	68.5	93.3	68.8	93.4	11M
ResNet50	ImageNet	70.0	93.7	70.8	94.0	26M
ResNet101	ImageNet	70.7	94.0	70.3	93.8	45M
Swin-T	MS-COCO	73.2	93.1	73.2	94.2	28M
SAM-B	SA-1B	73.6	93.2	73.4	93.7	86M
TinyViT-5m	SA-1B	74.6	93.9	74.2	94.8	5M

TABLE V: Comparison of backbones and pretrained models on the freight train dataset.

Select Layers	$AP^{box}$	$AP^{box}_{50}$	$AR^{box}_{50-90}$	$AP^{mask}$	$AP^{mask}_{50}$	$AR^{mask}_{50-90}$
$[0,1,2,3]$	73.8	92.2	80.1	73.2	93.3	79.2
$[1,2,3]$	74.1	92.7	80.9	73.6	93.7	79.4
$[1,3]$	73.6	92.7	80.5	73.8	93.9	79.5
$[\textbf{2},\textbf{3}]$	74.6	93.9	81.4	74.2	94.8	79.9
$[3]$	72.6	93.0	79.6	72.6	93.8	79.6

TABLE VI: Segmentation performance with different levels of features fed into the aggregator, where [a, b, c] denotes the indices of the retained feature map levels.

En	De	$AP^{box}$	$AP^{box}_{50}$	$AR^{box}_{50-90}$	$AP^{mask}$	$AP^{mask}_{50}$	$AR^{mask}_{50-90}$
f	f	66.9	90.9	71.4	65.7	91.1	71.0
uf	f	74.6	93.9	81.4	74.2	94.8	79.9
f	uf	71.4	92.8	79.2	70.8	93.0	78.6
uf	uf	72.2	91.9	79.7	72.2	91.9	79.0

TABLE VII: Comparisons of frozen different part.

Channels	$AP^{box}$	$AP^{box}_{50}$	$AR^{box}_{50-90}$	$AP^{mask}$	$AP^{mask}_{50}$	$AR^{mask}_{50-90}$
64	67.3	90.4	76.2	67.5	92.1	76.3
128	70.9	91.7	78.7	71.5	93.8	77.5
256	74.6	93.9	81.4	74.2	94.8	79.9

TABLE VIII: Comparisons of different channels in feature aggregator and prompt generator.

$N_{q}$	$N_{p}$	$AP^{box}$	$AP^{box}_{50}$	$AR^{box}_{50-90}$	$AP^{mask}$	$AP^{mask}_{50}$	$AR^{mask}_{50-90}$
30	4	73.2	92.1	82.1	72.8	93.7	79.9
20	4	73.7	93.3	80.9	73.2	94.6	78.6
10	4	74.6	93.9	81.4	74.2	94.8	79.9
5	4	72.2	91.8	79.6	72.7	93.3	78.6
1	4	63.0	79.0	66.9	62.4	79.9	67.1
10	10	72.9	92.8	80.7	72.8	93.4	79.0

TABLE IX: Segmentation performance with varying query numbers and prompt embedding numbers.

Panel of Figure 17 — Fig. 8: Brake shoe wear condition visualization.

Method	Slight wear	Moderate wear	Severe wear
SAM FTI-FDet	96.7	94.8	97.5
Mask R-CNN	91.6	91.2	93.9
Bounding Box	82.3	84.1	88.7
Edge Detection	75.6	78.4	81.2

TABLE X: Comparison of detection accuracy across different wear levels using various methods

Method	$AP^{box}$	$AP^{box}_{50}$	$AR^{box}_{50-90}$	$AP^{mask}$	$AP^{mask}_{50}$	$AR^{mask}_{50-90}$
FastSAM [ 30 ]	56.3	77.9	60.5	57.1	83.8	58.3
RSPrompter [ 8 ]	55.6	78.6	58.8	55.1	77.9	56.4
Mask R-CNN	51.1	70.4	53.6	50.6	71.8	53.8
SAM FTI-FDet	60.8	87.6	65.7	59.6	86.6	62.8

TABLE XI: Quantitative results of different methods with noise and visual artifacts.

Panel of Figure 20 — Fig. 9: Instance detection under various noise and visual artifacts. (a) Motion blur and semi-transparent spots; (b) Gaussian noise and blur; (c) Cutout occlusions and color/contrast variations.

Method	$AP^{box}$	$AP^{box}_{50}$	$AR^{box}_{50-90}$	$AP^{mask}$	$AP^{mask}_{50}$	$AR^{mask}_{50-90}$
FastSAM [ 30 ]	37.9	53.6	50.4	32.6	51.8	49.0
SAM-seg [ 8 ]	37.2	56.1	47.1	33.2	53.3	42.0
RSPrompter [ 8 ]	37.6	54.5	56.3	35.2	54.3	51.0
SAM FTI-FDet	38.5	57.3	54.8	36.0	56.6	49.5

TABLE XII: Comparisons of different methods on the MS-COCO dataset.

Fault Detection

Fault detection identifies abnormal behavior in machines, sensors, infrastructure, software, and autonomous systems before it causes harm or disruption.

See all 104 papers

SAM3-LoRA: Parameter-Efficient Adaptation of a Concept-Promptable Foundation Model for Multi-Class Structural Defect Segmentation
Aug 31, 2026P. Malaisree, S. Youwai, S. Janrungautai +3Promptable Image Segmentation Vision Foundation Model Adaptation
MAA Consultants Co., Ltd. · AI Research Group, Department of Civil Engineering, Faculty of Engineering, King Mongkut´s University of Technology Thonburi (KMUTT). · School of Engineering, University of Phayao. +2
Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain
Apr 24, 2026Annika Bätz, Pavel Klasek, Seo-Young Ham +3Visual Odometry Object Detection
Karlsruhe Institute of Technology, Engler-Bunte-Ring 21, 76131 Karlsruhe, Germany · Deutsches Zentrum für Schienenverkehrsforschung (DZSF), August-Bebel-Straße 10, 01219 Dresden, Germany · DB InfraGO AG, EUREF-Campus 17, 10829 Berlin, Germany
FedTR: Federated Learning Framework with Transfer Learning for Industrial Visual Inspection
Jul 9, 2026Vikash Sathiamoorthy, Shuo Huai, Hao Kong +7Industrial Inspection Optical Character Recognition
HP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University, Singapore · School of Computer Science and Engineering, Nanyang Technological University, Singapore · Department of Computer Science, Norwegian University of Science and Technology, Norway +2

Aug 31, 2026J/K move · Enter open · S save

SAM3-LoRA: Parameter-Efficient Adaptation of a Concept-Promptable Foundation Model for Multi-Class Structural Defect Segmentation

P. Malaisree, S. Youwai, S. Janrungautai, D. Amorndechaphon+2

MAA Consultants Co., Ltd. · AI Research Group, Department of Civil Engineering, Faculty of Engineering, King Mongkut´s University of Technology Thonburi (KMUTT). · School of Engineering, University of Phayao.+2

Promptable Image Segmentation Vision Foundation Model Adaptation Semantic Segmentation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Promptable segmentation foundation models such as SAM3 accept an open-vocabulary text concept and return every instance matching it, but adapting them to a specialized domain by full fine-tuning is computationally prohibitive for the organizations that would benefit most. This study applies Low-Rank Adaptation (LoRA) to SAM3 for multi-class structural defect segmentation and examines both how such a model can be supervised from conventional annotation and whether the resulting efficiency gain transfers across datasets. Two contributions are methodological. First, we describe a supervision procedure that trains a concept-promptable model directly from COCO-style class-labeled instance segmentation by using the category name itself as the prompt, requiring no prompt templates, no synonym expansion, and no learned class embeddings. Second, we identify and mitigate a failure mode specific to this setting: because a conventional annotation file yields positive prompts exclusively, the model's presence prediction decouples from the text condition and degenerates into responding to any prompt, a collapse that is invisible to every metric computed on positive prompts alone. Exhaustive hard-negative prompting, in which every dataset category absent from an image is issued as a zero-detection query, addresses this at no annotation cost. Two adapter placements were compared under an identical protocol, updating 0.121% and 1.341% of model parameters. On a purpose-built tunnel lining dataset, pixel intersection-over-union improved from 0.017 to 0.338 and instance-level recall from 0.375 to 0.672; on the independent public Structural Defects Dataset, from 0.017 to 0.855 and from 0.574 to 1.000. Improvements were directionally consistent across ten metrics on both datasets, and the largest per-category gains occurred precisely where zero-shot competence was absent.

Abstract

Fault Detection

Explore similar work

SAM3-LoRA: Parameter-Efficient Adaptation of a Concept-Promptable Foundation Model for Multi-Class Structural Defect Segmentation

Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain

FedTR: Federated Learning Framework with Transfer Learning for Industrial Visual Inspection