DetM-Tracker: Deterministic and Memory-Efficient VIS for Autonomous Industrial Inspection

You-Rui Lin; Jen-Chieh Wu; Tzu-Ti Wei; Jen-Jee Chen; Yu-Chee Tseng

cs.CVOct 21, 2026

DetM-Tracker: Deterministic and Memory-Efficient VIS for Autonomous Industrial Inspection

Authors: You-Rui Lin, Jen-Chieh Wu, Tzu-Ti Wei, Jen-Jee Chen, Yu-Chee Tseng

Organizations: College of Artificial Intelligence National Yang Ming Chiao Tung University · Department of Computer Science National Yang Ming Chiao Tung University

Abstract

Autonomous industrial inspection requires strictly deterministic real-time inference under tight memory and computational constraints. However, existing Video Instance Segmentation (VIS) models suffer from prohibitive scaling: dense temporal memory grows with spatial resolution and sequence length, while fixed-capacity decoders fail to exploit the inherent sparsity of industrial scenes, creating a practical memory wall on resource-constrained edge platforms. We present DetM-Tracker, a streamlined VIS framework for autonomous industrial inspection, where target objects often exhibit highly similar appearance under constrained "micro" fields of view. DetM-Tracker eliminates architectural redundancy through three synergistic components: (1) Compact Scalar Memory, which condenses high-dimensional instance features into four-scalar descriptors, reducing temporal memory complexity from $O(T \cdot N \cdot R)$ to $O(T \cdot \hat{N})$ . This mechanism effectively compresses frame-level representations into a bank of active instances, where $R$ denotes resolution, $N$ and $\hat{N}$ represent global and window-specific instance counts, respectively and $T$ the temporal window size; and (2) Dynamic Instance Channels, which adaptively allocate query capacity according to the actual number of visible objects; and (3) Entropy-Gated Skip Connection, which concentrates computational budget on high-uncertainty spatial regions to refine object boundaries in a single forward pass without auxiliary refinement heads. Our approach effectively bridges the gap between modern VIS models and the practical demands of industrial edge deployment, reducing inference latency by approximately 20 $\times$ and VRAM consumption by 6 $\times$ compared to state-of-the-art dense-feature baselines.

Figures & tables

Figure 1: Overview of the DetM-Tracker Architecture. The framework decouples VIS into persistent identity and dynamic temporal reasoning. Past states are distilled into Compact Scalar Memory $\mathbf{s}$ , which is processed by a memory encoder to produce resolution-invariant temporal tokens $X_{M}$ . These tokens provide context for Memory Attention to refine current-frame image features. The Mask Decoder then facilitates mutual refinement between image-conditioned features and Global Object Queries $\mathbf{P}_{obj}$ to generate initial masks. Finally, Entropy-Gated Skip Connection H( $H(\cdot)$ ) selectively integrates high-resolution cues to resolve boundary ambiguities in a single forward pass.

Figure 2: Reference Encoding. Initial identity anchoring is performed by processing geometric configuration (Conf.) and a reference image (

I_{ref}

) through parallel encoding streams to produce structural embeddings

\mathbf{F}_{obj}

and visual features

\mathbf{F}_{ref}

. These modalities are integrated by an Object Decoder using a query-key/value mechanism to generate Global Object Queries

\mathbf{P}_{obj}

. This initialization is executed only once per sequence, providing persistent prototypes that maintain identity across the temporal horizon while minimizing per-frame computational overhead.

Figure 3: Compact Scalar Memory. Instance attributes $[x_{c},y_{c},a,\tau]$ are extracted from $T$ frames and stacked into a resolution-invariant bank $\mathbf{S}\in\mathbb{R}^{(\hat{N}\times 4)}$ . This scalar abstraction ensures $O(T\cdot\hat{N})$ memory complexity, decoupling VRAM overhead from spatial resolution.

Figure 4: Bidirectional Mask Decoder. Bidirectional Mask Decoder. Memory tokens

\mathbf{T}_{mem}

and object queries

\mathbf{P}_{obj}

undergo mutual refinement through symmetric decoder layers to yield image-conditioned features

\mathbf{z}_{mask}

and instance embeddings

\mathbf{z}_{obj}

. A final scaled dot-product generates the predicted masks

\mathbf{z}_{mask}\in\mathbb{R}^{N\times N^{\prime}}

. This design decouples tracking capacity from network dimensions, enabling high-cardinality segmentation in a single pass.

Panel of Figure 5 — Figure 5: Dataset Statistics. Our million-frame pipeline supports a three-stage curriculum: (a) Synthesis for geometric anchoring (972.9k frames), (b) Real-Unplugged for environment alignment, and (c) Real-Plugged for modeling complex occlusions and cable routing. The table (right) details the datasets statistics.

Model	Conf.	mIoU $\uparrow$	@.5 $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	Inf. (ms) $\downarrow$	Param	VRAM (MB) $\downarrow$
SAM2.1 Ravi et al. (2024)	Tiny	0.276	0.289	0.375	404.9	38.96M	3359
SAM2.1 Ravi et al. (2024)	Small	0.157	0.137	0.229	407.0	46.06M	3454
SAM2.1 Ravi et al. (2024)	Base	0.252	0.269	0.336	425.0	80.85M	3565
SAM2.1-FT Ravi et al. (2024)	Base	0.360	0.416	0.474	425.0	80.85M	3565
SAM2.1 Ravi et al. (2024)	Large	0.290	0.316	0.383	456.3	224.45M	4128
SAM3 Carion et al. (2025)	–	0.346	0.442	0.418	973.1	860.06M	17506

Table 1: Quantitative Evaluation of Segmentation Capability. We compare our DetM-Tracker against foundation models (SAM 2/3), fine-tuned variants, and VOS baselines. Our model achieves superior segmentation fidelity while maintaining significantly lower inference latency and memory footprint.

Model	Conf.	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	AP@50 $\uparrow$
SAM-2.1 Ravi et al. (2024)	Base-1P	0.188	0.161	0.170	0.053
SAM-2.1 Ravi et al. (2024)	Base-3P	0.277	0.269	0.269	0.138
SAM-2.1 Ravi et al. (2024)	Base-5P	0.312	0.311	0.308	0.167
SAM-2.1 Ravi et al. (2024)	Large-1P	0.255	0.222	0.233	0.092
SAM-2.1 Ravi et al. (2024)	Large-3P	0.332	0.316	0.320	0.192
SAM-2.1 Ravi et al. (2024)	Large-5P	0.351	0.343	0.344	0.223

Table 2: Detection Performance and Instance Robustness. Comparison against various foundation models, fine-tuned variants, and VOS baselines. DetM-Tracker achieves the highest stability and accuracy across all metrics.

Panel of Figure 6 — Figure 6: Qualitative Comparison Across Environmental Complexities. While foundation models (SAM 2/3) exhibit mask bleeding and fragmentation in dense industrial arrays, DetM-Tracker maintains high topological fidelity. Especially in "Hard" scenarios with severe occlusions, our framework precisely resolves individual instance boundaries where baselines collapse into monolithic or incomplete masks.

Figure 7: Evaluation of Long-term Temporal Stability. Per-frame precision measured over a 300-frame industrial sequence. While foundation models like SAM 2.1 and LiVOS exhibit rapid precision decay and drift after initial frames, DetM-Tracker maintains near-optimal, consistent performance. Note the high-frequency fluctuations in SAM 3/3.1, indicating a lack of stable temporal memory for thin structures, whereas our framework effectively mitigates cumulative error.

Dataset	Method	[email protected]	[email protected]:0.95	mIoU
Synthesis	Baseline (w/o)	0.9343	0.6219	0.7692
Synthesis	Entropy-Gated (Ours)	0.9347	0.6236	0.7702
Real-World	Baseline (w/o)	0.9112 $\pm$ 0.0038	0.4200 $\pm$ 0.0019	0.6614 $\pm$ 0.0027
	+ 0.5 $\times$ Skip	0.9118 $\pm$ 0.0039	0.4170 $\pm$ 0.0018	0.6599 $\pm$ 0.0028
	+ Skip	0.9099 $\pm$ 0.0033	0.4101 $\pm$ 0.0023	0.6562 $\pm$ 0.0035
	Entropy-Gated (Ours)	0.9122 $\pm$ 0.0024	0.4206 $\pm$ 0.0013	0.6627 $\pm$ 0.0017

Table 3: Impact of Entropy-Gating on Segmentation Performance.

Scenario	#Obj	[email protected] $\uparrow$	[email protected]:0.95 $\uparrow$	mIoU $\uparrow$
All	2	0.8849 $\pm$ 0.0044	0.3998 $\pm$ 0.0019	0.6452 $\pm$ 0.0030
	4	0.8632 $\pm$ 0.0038	0.3685 $\pm$ 0.0017	0.6298 $\pm$ 0.0027
	6	0.9497 $\pm$ 0.0283	0.3985 $\pm$ 0.0124	0.6681 $\pm$ 0.0196
	8	0.9892 $\pm$ 0.0018	0.4363 $\pm$ 0.0022	0.6929 $\pm$ 0.0005
Env1	2	0.8378 $\pm$ 0.0058	0.3590 $\pm$ 0.0024	0.6076 $\pm$ 0.0042
	4	0.8385 $\pm$ 0.0075	0.3391 $\pm$ 0.0036	0.6046 $\pm$ 0.0054

Table 4: Tracking Robustness Across Different Cabinet Layouts. Performance under increasing occlusion difficulty and instance density. Env1 and Env2 feature different server/switch arrangements under realistic camera noise.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Specification
CPU	2 × AMD EPYC 7V12 (96 Cores)
RAM	1.8 TB
GPU	8 × NVIDIA A100-SXM4 (80GB)
Driver / CUDA	580.105.08 / 13.0
OS	Ubuntu 22.04 LTS

Appendix

Table 5: Hardware Infrastructure.

Hyperparameter	Stage-1	Stage-2	Stage-3
Architecture Settings
Backbone Architecture	Base (ViT-based)
Model Dimension ( $d_{model}$ )	512
Attention Heads ( $n_{heads}$ )	8
Transformer Layers	6
Patch Size	16

Appendix

Table 6: Multi-stage Training Configurations and Hyperparameters.

Model	Random	First	Max-coverage	Single-run
SAM 2	0.275 ( $N{=}10$ )	0.277 ( $N{=}1$ )	0.252 ( $n$ pts, $N{=}10$ )	–
Cutie	0.335 ( $N{=}10$ )	0.276 ( $N{=}1$ )	0.277 ( $N{=}1$ )	–
LiVOS	0.102 ( $N{=}10$ )	0.079 ( $N{=}1$ )	0.077 ( $N{=}1$ )	–
Tracker	–	–	–	0.697 (No prompt, $N{=}1$ )
SAM 3	–	–	–	0.346 (Text prompt, $N{=}1$ )

Appendix

Table 7: Evaluation strategies and prompt frame sampling (reported metric: mIoU;

N

denotes the number of runs).

Video Instance Segmentation

See all 12 papers

Sep 4, 2026

LUMIN: Lifecycle-Utility-aware and Metadata-Informed Network for Visual Anomaly Detection

Industrial anomaly detection faces two engineering bottlenecks: memory bank construction latency and inference efficiency. Traditional sampling algorithms (Farthest Point Sampling, K-Means, etc.) rely on numerous backbone forward passes and iterative distance computations, with construction times ranging from minutes to hours; heavy computation components such as multi-scale feature extraction struggle to meet the millisecond-level real-time requirements of production lines. This paper focuses on sampling efficiency and inference optimization for industrial deployment with two core contributions: (1) PSP (Plugin Sampler Pipeline)---a four-stage adaptive memory bank sampling pipeline based on 18-dimensional pixel metadata and five complementary visual plugins. PSP completes all sampling with zero backbone forward passes; coarse filtering is sub-second numerical sorting, and metadata extraction is a one-time offline cost. PSP supports progressive deployment and incremental updates. (2) Two engineering optimization strategies---parallel memory bank similarity computation (reducing inference memory and latency by over 95%) and stratified pixel sampling for large-scale evaluation (reducing computation time by 20

\times

while keeping metrics stable). As a vehicle for validation, we introduce LUMIN (Lightweight Universal Manufacturing Inspection Network) with extreme segmentation-head compression, systematically exploring the accuracy-efficiency frontier against strong baselines. Experiments on five benchmarks demonstrate that PSP matches state-of-the-art sampling accuracy at near-random construction cost (341

\times

faster than FPS), while inference optimizations reduce evaluation time by 20

\times

with negligible accuracy loss.

Pengfei Yang

Intelligent Precision Instrument, Shenzhen, Guangdong, China

Industrial Anomaly Detection Anomaly Detection Efficient Inference

Ask PDF

Mar 18, 2026

Efficient Visual Anomaly Detection at the Edge: Enabling Real-Time Industrial Inspection on Resource-Constrained Devices

Visual Anomaly Detection (VAD) is essential for industrial quality control, enabling automatic defect detection in manufacturing. In real production lines, VAD systems must satisfy strict real-time and privacy requirements, necessitating a shift from cloud-based processing to local edge deployment. However, processing data locally on edge devices introduces new challenges because edge hardware has limited memory and computational resources. To overcome these limitations, we propose two efficient VAD methods designed for edge deployment: PatchCore-Lite and Padim-Lite, based on the popular PatchCore and PaDiM models. PatchCore-Lite runs first a coarse search on a product-quantized memory bank, then an exact search on a decoded subset. Padim-Lite is sped up using diagonal covariance, turning Mahalanobis distance into efficient element-wise computation. We evaluate our methods on the MVTec AD and VisA benchmarks and show their suitability for edge environments. PatchCore-Lite achieves a remarkable 79% reduction in total memory footprint, while PaDiM-Lite achieves substantial efficiency gains with a 77% reduction in total memory and a 31% decrease in inference time. These results show that VAD can be effectively deployed on edge devices, enabling real-time, private, and cost-efficient industrial inspection.

Arianna Stropeni, Fabrizio Genilotti, Francesco Borsatti +3

University of Padua, Padua, Italy

Industrial Anomaly Detection Edge Inference

Ask PDF

Oct 29, 2025

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a Deep Semantic Injector (DSI) module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a Gradient-guided Adaptive Modulation (GAM) strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of

49.8/53.7/55.4/57.0

at corresponding speeds of

273/169/124/78

FPS. Code is publicly available at https://github.com/RT-DETRs/RT-DETRv4.

Zijun Liao, Yian Zhao, Xin Shan +5

School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China

Real-Time Object Detection Efficient Object Detection Detection Transformer Knowledge Distillation

Ask PDF

Abstract

Video Instance Segmentation

Explore similar work

LUMIN: Lifecycle-Utility-aware and Metadata-Informed Network for Visual Anomaly Detection

Efficient Visual Anomaly Detection at the Edge: Enabling Real-Time Industrial Inspection on Resource-Constrained Devices

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models