HyperDAM: Hyperspectral Distractor-Aware Memory with Amodal Expansion for SAM 3 Tracking

Ryoga Yuzawa; Tasuku Takagi

cs.CVSep 28, 2026

HyperDAM: Hyperspectral Distractor-Aware Memory with Amodal Expansion for SAM 3 Tracking

Authors: Ryoga Yuzawa, Tasuku Takagi

Abstract

Hyperspectral video provides material cues that can disambiguate targets with similar false-color appearance, yet foundation-model trackers update memory primarily from spatial and appearance evidence. We present HyperDAM, a DAM4SAM3-based hyperspectral tracker with three principal contributions. First, HOTC2026-Modal adds human-verified frame-wise modal masks and mask-tight boxes to all 481 organizer-provided HOTC 2026 videos. Second, a frame-zero-calibrated HSI gate rejects spectrally inconsistent updates to the distractor-resolving memory (DRM) without altering the current prediction. Third, a causal spatiotemporal expander adds outward-only amodal corrections from frozen SAM features. Static-scene recovery and empty-mask RTS smoothing address target switches and full occlusion. Model selection prioritizes cross-domain robustness over leaderboard-specific optimization. The final system ranked second in HOTC 2026, achieving 68.0093% AUC and 87.7703% DP@20 in the organizer's private evaluation.

Figures & tables

Figure 1: HyperDAM through an occlusion on nir-rider6 . From left to right and top to bottom, the target is visible, partially occluded, fully occluded, and visible again. The amodal expander corrects partial visibility, while empty-mask RTS bridges full occlusion. Green, blue, and pink denote ground truth, DAM4SAM3, and HyperDAM, respectively.

Figure 2: Overview of the proposed HyperDAM architecture. The HSI gate filters requested DRM updates using raw hyperspectral observations; DAM4SAM3 supplies modal masks and features for amodal expansion, followed by offline RTS smoothing.

Figure 3: Runtime trace of the HSI gate on vis-clamp2 from the HOTC 2026 dataset. The rows show aligned false-color frames and raw 16-band mosaics. At $t=124$ , DAM4SAM3 accepts the DRM update, whereas DAM4SAM3+HSI rejects it without changing the current output. By $t=236$ – $237$ , the native tracker follows the visually similar distractor while the gated tracker remains on the target. Orange, blue, and green dashed boxes denote DAM4SAM3, DAM4SAM3+HSI, and ground truth, respectively.

ID	Configuration	Public score (AUC)
B0	DAM4SAM3 native	0.69039
B1	B0 + HSI gate	0.69214
B2	B1 + static-scene recovery	0.69513
B3	B2 + amodal expander	0.69732
B4	B3 + RTS smoothing	0.69768
Organizer’s private LB		AUC: 68.0093% DP@20: 87.7703%

Table 1: Cumulative ablation on Public-LB75.

Method	Public-LB75 AUC $\uparrow$	Peak VRAM (MiB) $\downarrow$	Params (M) $\downarrow$	Precision	Comment
OSTrack [ 49 ]	0.49512	443.24	92.83	FP32	–
SeqTrack [ 6 ]	0.54142	1601.95	308.98	FP32	–
HIPTrack [ 2 ]	0.57374	557.88	120.41	FP32	–
ODTrack [ 52 ]	0.54442	599.99	92.83	FP32	–
SUTrack [ 5 ]	0.57169	2205.59	746.61	FP32	–
SAM2 [ 33 ]	0.65041	1634.95	224.45	FP32	–

Table 2: Unified benchmark on Public-LB75 (75 videos; 26,860 frames). Public AUC is evaluated on the Kaggle public leaderboard. Available peak-VRAM measurements use the same NVIDIA L4 GPU. The best AUC value is shown in bold.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	HOTC406			Public-LB75
Category	Title families	V	F	Title families	V	F
Vehicles and traffic	automobile, bus, car, dashcam, drive, excavator, high_car, high_truck, l_car, rainystreet, surround_car, taxi, toy_car, truck	137	33,829	bicycle, car, dashcam, motorcycle, rccar, truck	21	6,234
People and human activity	face, foot, hand, high_person, high_rider, l_basketball_person, l_person, l_runner, pedestrain, pedestrian, people, player, rider, s_jump, s_person, s_runner, s_walker, s_warmup, student, surround_person, trucker, worker	65	26,593	athlete, head, walker	4	1,266
Sports and balls	badminton, ball, ball_holder, basketball, football, high_playground, l_basketball, pingpong, playground, pool, s_soccer, s_volleyball, snow_table_tennis, snow_tennis	43	23,120	pingpong	4	1,114
Animals	ant, duck, kangaroo, turkey	10	4,508	bee, cat, dog	7	3,373
Vegetation and food	apple, coke, cranberries, fake_orange, forest, fruit, grove, leaf, leaves, oranges, real_pear	30	17,712	dryleaf, flower, folium, herbs, moss, setaria, vegetation	15	6,433

Appendix

Table 3: Complete assignment of normalized title families to broad categories. Numeric title variants within a listed family are grouped together. V and F are the total numbers of videos and frames, respectively, in each category and split.

Explore similar work

SpecTrack: Spectral Prompt Guided Adaptive Experts for Multispectral Object Tracking
Jul 7, 2026Xingyu Tan, Yunrong Qin, Mengjie HuMulti-Object Tracking Hyperspectral Image
Beijing University of Posts and Telecommunications, China
End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking
May 20, 2026Xu Han, Mohammad Aminul Islam, Lei Wang +5Hyperspectral Image Multi-Object Tracking
School of Information and Communication Technology, Griffith University, Australia · Department of Computer Science and Mathematics, Bangladesh Agricultural University, Mymensingh 2202, Bangladesh · School of Engineering and Built Environment, Griffith University, Australia +1
OAMVOS:2nd Report for 5th PVUW MOSE Track
Apr 20, 2026Deshui Miao, Xingsen Huang, Yameng Gu +3Multi-Object Tracking Segment Anything Model
1Pengcheng Laboratory · 2Guangzhou Hengyan Technology · University of California at Merced

Jul 7, 2026, cs.CVJ/K move · Enter open · S save

SpecTrack: Spectral Prompt Guided Adaptive Experts for Multispectral Object Tracking

Xingyu Tan, Yunrong Qin, Mengjie Hu

Beijing University of Posts and Telecommunications, China

Multi-Object Tracking Hyperspectral Image Speckle

Open paper AskPDF arXiv

Multispectral image(MSI) and hyperspectral image(HSI) object tracking object tracking exploits recorded band-wise observations to improve target--background discrimination under similar RGB appearance, mixed pixels, illumination variation, occlusion, and clutter. However, existing trackers commonly process all search regions through a fixed capacity spectral--spatial path, ignoring that tracking difficulty varies substantially across frames and target states. Clear regions may require only lightweight local discrimination, whereas ambiguous boundaries and spectrally similar distractors often demand stronger contextual reasoning. To address this limitation, we propose SpecTrack, a spectral--spatial complexity-aware tracker that formulates MSI tracking as search-region-level adaptive capacity allocation. Its core component, the Spectral Adaptive Mixture-of-Experts (SAMoE) module, provides a capacity-ordered expert pool with progressively increasing latent rank, receptive field, and depth. Expert selection is guided by a Spectral Prompt Router, which fuses semantic context, spatial boundary cues, and a latent channel-variation cue computed after multispectral patch embedding to activate a sparse subset of SAMoE experts for each search region. In parallel, a Shared Global Expert supplies common latent spectral--spatial context to reduce fragmented sparse-routing decisions. Experiments on MUST, MSITrack, and HOTC20 demonstrate a favorable accuracy--efficiency trade-off. The accuracy-oriented SpecTrack-L384 achieves state-of-the-art or highly competitive AUCs of 65.2%, 51.9%, and 72.6% on the three benchmarks, while the balanced SpecTrack-B224 reaches 62.4% AUC at 43.7 FPS on MUST. An additional GOT-10k evaluation indicates RGB-domain architectural generalization, with SpecTrack-L384 achieving 79.3% AO.

Abstract

Explore similar work

SpecTrack: Spectral Prompt Guided Adaptive Experts for Multispectral Object Tracking

End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking

OAMVOS:2nd Report for 5th PVUW MOSE Track