DetM-Tracker: Deterministic and Memory-Efficient VIS for Autonomous Industrial Inspection

You-Rui Lin; Jen-Chieh Wu; Tzu-Ti Wei; Jen-Jee Chen; Yu-Chee Tseng

cs.CVOct 21, 2026

DetM-Tracker: Deterministic and Memory-Efficient VIS for Autonomous Industrial Inspection

Authors: You-Rui Lin, Jen-Chieh Wu, Tzu-Ti Wei, Jen-Jee Chen, Yu-Chee Tseng

Organizations: College of Artificial Intelligence National Yang Ming Chiao Tung University · Department of Computer Science National Yang Ming Chiao Tung University

Abstract

Autonomous industrial inspection requires strictly deterministic real-time inference under tight memory and computational constraints. However, existing Video Instance Segmentation (VIS) models suffer from prohibitive scaling: dense temporal memory grows with spatial resolution and sequence length, while fixed-capacity decoders fail to exploit the inherent sparsity of industrial scenes, creating a practical memory wall on resource-constrained edge platforms. We present DetM-Tracker, a streamlined VIS framework for autonomous industrial inspection, where target objects often exhibit highly similar appearance under constrained "micro" fields of view. DetM-Tracker eliminates architectural redundancy through three synergistic components: (1) Compact Scalar Memory, which condenses high-dimensional instance features into four-scalar descriptors, reducing temporal memory complexity from $O(T \cdot N \cdot R)$ to $O(T \cdot \hat{N})$ . This mechanism effectively compresses frame-level representations into a bank of active instances, where $R$ denotes resolution, $N$ and $\hat{N}$ represent global and window-specific instance counts, respectively and $T$ the temporal window size; and (2) Dynamic Instance Channels, which adaptively allocate query capacity according to the actual number of visible objects; and (3) Entropy-Gated Skip Connection, which concentrates computational budget on high-uncertainty spatial regions to refine object boundaries in a single forward pass without auxiliary refinement heads. Our approach effectively bridges the gap between modern VIS models and the practical demands of industrial edge deployment, reducing inference latency by approximately 20 $\times$ and VRAM consumption by 6 $\times$ compared to state-of-the-art dense-feature baselines.

Figures & tables

Figure 1: Overview of the DetM-Tracker Architecture. The framework decouples VIS into persistent identity and dynamic temporal reasoning. Past states are distilled into Compact Scalar Memory $\mathbf{s}$ , which is processed by a memory encoder to produce resolution-invariant temporal tokens $X_{M}$ . These tokens provide context for Memory Attention to refine current-frame image features. The Mask Decoder then facilitates mutual refinement between image-conditioned features and Global Object Queries $\mathbf{P}_{obj}$ to generate initial masks. Finally, Entropy-Gated Skip Connection H( $H(\cdot)$ ) selectively integrates high-resolution cues to resolve boundary ambiguities in a single forward pass.

Figure 2: Reference Encoding. Initial identity anchoring is performed by processing geometric configuration (Conf.) and a reference image (

I_{ref}

) through parallel encoding streams to produce structural embeddings

\mathbf{F}_{obj}

and visual features

\mathbf{F}_{ref}

. These modalities are integrated by an Object Decoder using a query-key/value mechanism to generate Global Object Queries

\mathbf{P}_{obj}

. This initialization is executed only once per sequence, providing persistent prototypes that maintain identity across the temporal horizon while minimizing per-frame computational overhead.

Figure 3: Compact Scalar Memory. Instance attributes $[x_{c},y_{c},a,\tau]$ are extracted from $T$ frames and stacked into a resolution-invariant bank $\mathbf{S}\in\mathbb{R}^{(\hat{N}\times 4)}$ . This scalar abstraction ensures $O(T\cdot\hat{N})$ memory complexity, decoupling VRAM overhead from spatial resolution.

Figure 4: Bidirectional Mask Decoder. Bidirectional Mask Decoder. Memory tokens

\mathbf{T}_{mem}

and object queries

\mathbf{P}_{obj}

undergo mutual refinement through symmetric decoder layers to yield image-conditioned features

\mathbf{z}_{mask}

and instance embeddings

\mathbf{z}_{obj}

. A final scaled dot-product generates the predicted masks

\mathbf{z}_{mask}\in\mathbb{R}^{N\times N^{\prime}}

. This design decouples tracking capacity from network dimensions, enabling high-cardinality segmentation in a single pass.

Panel of Figure 5 — Figure 5: Dataset Statistics. Our million-frame pipeline supports a three-stage curriculum: (a) Synthesis for geometric anchoring (972.9k frames), (b) Real-Unplugged for environment alignment, and (c) Real-Plugged for modeling complex occlusions and cable routing. The table (right) details the datasets statistics.

Model	Conf.	mIoU $\uparrow$	@.5 $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	Inf. (ms) $\downarrow$	Param	VRAM (MB) $\downarrow$
SAM2.1 Ravi et al. (2024)	Tiny	0.276	0.289	0.375	404.9	38.96M	3359
SAM2.1 Ravi et al. (2024)	Small	0.157	0.137	0.229	407.0	46.06M	3454
SAM2.1 Ravi et al. (2024)	Base	0.252	0.269	0.336	425.0	80.85M	3565
SAM2.1-FT Ravi et al. (2024)	Base	0.360	0.416	0.474	425.0	80.85M	3565
SAM2.1 Ravi et al. (2024)	Large	0.290	0.316	0.383	456.3	224.45M	4128
SAM3 Carion et al. (2025)	–	0.346	0.442	0.418	973.1	860.06M	17506

Table 1: Quantitative Evaluation of Segmentation Capability. We compare our DetM-Tracker against foundation models (SAM 2/3), fine-tuned variants, and VOS baselines. Our model achieves superior segmentation fidelity while maintaining significantly lower inference latency and memory footprint.

Model	Conf.	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	AP@50 $\uparrow$
SAM-2.1 Ravi et al. (2024)	Base-1P	0.188	0.161	0.170	0.053
SAM-2.1 Ravi et al. (2024)	Base-3P	0.277	0.269	0.269	0.138
SAM-2.1 Ravi et al. (2024)	Base-5P	0.312	0.311	0.308	0.167
SAM-2.1 Ravi et al. (2024)	Large-1P	0.255	0.222	0.233	0.092
SAM-2.1 Ravi et al. (2024)	Large-3P	0.332	0.316	0.320	0.192
SAM-2.1 Ravi et al. (2024)	Large-5P	0.351	0.343	0.344	0.223

Table 2: Detection Performance and Instance Robustness. Comparison against various foundation models, fine-tuned variants, and VOS baselines. DetM-Tracker achieves the highest stability and accuracy across all metrics.

Panel of Figure 6 — Figure 6: Qualitative Comparison Across Environmental Complexities. While foundation models (SAM 2/3) exhibit mask bleeding and fragmentation in dense industrial arrays, DetM-Tracker maintains high topological fidelity. Especially in "Hard" scenarios with severe occlusions, our framework precisely resolves individual instance boundaries where baselines collapse into monolithic or incomplete masks.

Figure 7: Evaluation of Long-term Temporal Stability. Per-frame precision measured over a 300-frame industrial sequence. While foundation models like SAM 2.1 and LiVOS exhibit rapid precision decay and drift after initial frames, DetM-Tracker maintains near-optimal, consistent performance. Note the high-frequency fluctuations in SAM 3/3.1, indicating a lack of stable temporal memory for thin structures, whereas our framework effectively mitigates cumulative error.

Dataset	Method	[email protected]	[email protected]:0.95	mIoU
Synthesis	Baseline (w/o)	0.9343	0.6219	0.7692
Synthesis	Entropy-Gated (Ours)	0.9347	0.6236	0.7702
Real-World	Baseline (w/o)	0.9112 $\pm$ 0.0038	0.4200 $\pm$ 0.0019	0.6614 $\pm$ 0.0027
	+ 0.5 $\times$ Skip	0.9118 $\pm$ 0.0039	0.4170 $\pm$ 0.0018	0.6599 $\pm$ 0.0028
	+ Skip	0.9099 $\pm$ 0.0033	0.4101 $\pm$ 0.0023	0.6562 $\pm$ 0.0035
	Entropy-Gated (Ours)	0.9122 $\pm$ 0.0024	0.4206 $\pm$ 0.0013	0.6627 $\pm$ 0.0017

Table 3: Impact of Entropy-Gating on Segmentation Performance.

Scenario	#Obj	[email protected] $\uparrow$	[email protected]:0.95 $\uparrow$	mIoU $\uparrow$
All	2	0.8849 $\pm$ 0.0044	0.3998 $\pm$ 0.0019	0.6452 $\pm$ 0.0030
	4	0.8632 $\pm$ 0.0038	0.3685 $\pm$ 0.0017	0.6298 $\pm$ 0.0027
	6	0.9497 $\pm$ 0.0283	0.3985 $\pm$ 0.0124	0.6681 $\pm$ 0.0196
	8	0.9892 $\pm$ 0.0018	0.4363 $\pm$ 0.0022	0.6929 $\pm$ 0.0005
Env1	2	0.8378 $\pm$ 0.0058	0.3590 $\pm$ 0.0024	0.6076 $\pm$ 0.0042
	4	0.8385 $\pm$ 0.0075	0.3391 $\pm$ 0.0036	0.6046 $\pm$ 0.0054

Table 4: Tracking Robustness Across Different Cabinet Layouts. Performance under increasing occlusion difficulty and instance density. Env1 and Env2 feature different server/switch arrangements under realistic camera noise.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Specification
CPU	2 × AMD EPYC 7V12 (96 Cores)
RAM	1.8 TB
GPU	8 × NVIDIA A100-SXM4 (80GB)
Driver / CUDA	580.105.08 / 13.0
OS	Ubuntu 22.04 LTS

Appendix

Table 5: Hardware Infrastructure.

Hyperparameter	Stage-1	Stage-2	Stage-3
Architecture Settings
Backbone Architecture	Base (ViT-based)
Model Dimension ( $d_{model}$ )	512
Attention Heads ( $n_{heads}$ )	8
Transformer Layers	6
Patch Size	16

Appendix

Table 6: Multi-stage Training Configurations and Hyperparameters.

Model	Random	First	Max-coverage	Single-run
SAM 2	0.275 ( $N{=}10$ )	0.277 ( $N{=}1$ )	0.252 ( $n$ pts, $N{=}10$ )	–
Cutie	0.335 ( $N{=}10$ )	0.276 ( $N{=}1$ )	0.277 ( $N{=}1$ )	–
LiVOS	0.102 ( $N{=}10$ )	0.079 ( $N{=}1$ )	0.077 ( $N{=}1$ )	–
Tracker	–	–	–	0.697 (No prompt, $N{=}1$ )
SAM 3	–	–	–	0.346 (Text prompt, $N{=}1$ )

Appendix

Table 7: Evaluation strategies and prompt frame sampling (reported metric: mIoU;

N

denotes the number of runs).

Video Instance Segmentation

See all 12 papers

LUMIN: Lifecycle-Utility-aware and Metadata-Informed Network for Visual Anomaly Detection
Sep 4, 2026Pengfei YangIndustrial Anomaly Detection Anomaly Detection
Intelligent Precision Instrument, Shenzhen, Guangdong, China
Efficient Visual Anomaly Detection at the Edge: Enabling Real-Time Industrial Inspection on Resource-Constrained Devices
Mar 18, 2026Arianna Stropeni, Fabrizio Genilotti, Francesco Borsatti +3Industrial Anomaly Detection Edge Inference
University of Padua, Padua, Italy
RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
Oct 29, 2025Zijun Liao, Yian Zhao, Xin Shan +5Real-Time Object Detection Efficient Object Detection
School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China

Sep 4, 2026J/K move · Enter open · S save

LUMIN: Lifecycle-Utility-aware and Metadata-Informed Network for Visual Anomaly Detection

Pengfei Yang

Intelligent Precision Instrument, Shenzhen, Guangdong, China

Industrial Anomaly Detection Anomaly Detection Efficient Inference

Open paper AskPDF arXiv

Industrial anomaly detection faces two engineering bottlenecks: memory bank construction latency and inference efficiency. Traditional sampling algorithms (Farthest Point Sampling, K-Means, etc.) rely on numerous backbone forward passes and iterative distance computations, with construction times ranging from minutes to hours; heavy computation components such as multi-scale feature extraction struggle to meet the millisecond-level real-time requirements of production lines. This paper focuses on sampling efficiency and inference optimization for industrial deployment with two core contributions: (1) PSP (Plugin Sampler Pipeline)---a four-stage adaptive memory bank sampling pipeline based on 18-dimensional pixel metadata and five complementary visual plugins. PSP completes all sampling with zero backbone forward passes; coarse filtering is sub-second numerical sorting, and metadata extraction is a one-time offline cost. PSP supports progressive deployment and incremental updates. (2) Two engineering optimization strategies---parallel memory bank similarity computation (reducing inference memory and latency by over 95%) and stratified pixel sampling for large-scale evaluation (reducing computation time by 20

\times

while keeping metrics stable). As a vehicle for validation, we introduce LUMIN (Lightweight Universal Manufacturing Inspection Network) with extreme segmentation-head compression, systematically exploring the accuracy-efficiency frontier against strong baselines. Experiments on five benchmarks demonstrate that PSP matches state-of-the-art sampling accuracy at near-random construction cost (341

\times

faster than FPS), while inference optimizations reduce evaluation time by 20

\times

with negligible accuracy loss.

Abstract

Video Instance Segmentation

Explore similar work

LUMIN: Lifecycle-Utility-aware and Metadata-Informed Network for Visual Anomaly Detection

Efficient Visual Anomaly Detection at the Edge: Enabling Real-Time Industrial Inspection on Resource-Constrained Devices

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models