P-SRM: Selective Recovery of Rejected Predictions in Visual Tracking

Youbin He; Siwei Wang

cs.CVSep 30, 2026

P-SRM: Selective Recovery of Rejected Predictions in Visual Tracking

Authors: Youbin He, Siwei Wang

Organizations: The Hong Kong Polytechnic University · The University of Hong Kong

Abstract

Many visual tracking methods use rejection mechanisms to suppress unreliable predictions. However, these mechanisms can also reject correctly localized candidates, leaving useful information unused. We investigate how to identify and recover these candidates while preserving native accepted outputs and candidate coordinates. To this end, we propose P-SRM (Post-rejection Selective Recovery Method), which combines spatial responses, past accepted states, and native decision margins to reassess candidates and selectively restore reliable predictions. We evaluate P-SRM on six trackers and four datasets spanning category-specific, point, and generic object tracking. Across all nine configurations, P-SRM improves rejected-candidate ranking and overall tracking performance. These results show that post-rejection verification can identify and recover useful predictions discarded by native rejection, demonstrating the value of reusing rejected information. Project repository: https://github.com/PalestyHR/P-SRM.

Figures & tables

Figure 1: Overview of P-SRM. Rejected candidates are selectively recovered using spatial, historical, and native decision evidence. Dashed arrows indicate training-only paths.

Method / Source	Dataset	Setting	Acc.	Pre.	Rec.	F1	AP r	AJ	OA	$N_{C}$	$\bar{n}_{C}$ (recovery %)	FPS	Extra latency (ms/frame)
Category-specific tracking
TrackNetV1 AVSS 2019 [ 2 ]	Shuttlecock	Native	52.06	88.99	46.48	61.06	62.83	–	–	125		25.3	7.14
	Shuttlecock	+P-SRM	52.89	88.14	47.92	62.09	73.72	–	–	125	124.3 (99.47)	21.5
	RacketVision	Native	55.40	79.57	59.18	67.88	49.21	–	–	23		25.3
	RacketVision	+P-SRM	55.61	78.79	60.06	68.16	54.02	–	–	23	22.0 (95.65)	21.5
TrackNetV2 ICPAI 2020 [ 3 ]	Shuttlecock	Native	83.62	97.60	82.35	89.33	53.92	–	–	771		39.8	4.27

Table 1: Tracking results (%). RTX 4090 inference (KCF: CPU).

N_{C}

: correct rejected candidates;

\bar{n}_{C}

: three-seed mean correct recoveries (recovery %).

\ddagger

: TAP-Net FPS uses clip-amortized latency. Overhead includes clip-to-frame evidence preparation; separate profiling measures 3.94 ms/frame for preparation and 1.08 for core recovery.

Figure 2: Recovery examples across three tracking categories. Gray dashed markers show native-rejected candidates; blue markers show the same candidates after recovery. Examples are from RacketVision, Shuttlecock, Kinetics, and OTB2013.

Comparison	$\Delta t$	$\Delta$ AP r	$\Delta$ F1 pool	$\Delta$ F1 video [95% CI]
TAP-Net
$Q\!\to\!Q{+}H$	1.00	3.92	0.020	$0.017\;[-0.046,0.076]$
$Q{+}H\!\to\!\mathrm{Full}$	0.30	0.11	-0.011	$-0.016\;[-0.024,0.007]$
$Q{+}M\!\to\!\mathrm{Full}$	1.36	4.04	-0.001	$-0.010\;[-0.070,0.044]$
$M\!\to\!\mathrm{Full}$	6.26	32.89	0.588	$0.917\;[0.620,1.258]$
$M{+}H\!\to\!\mathrm{Full}$	5.51	20.16	0.508	$0.715\;[0.460,1.008]$

Table 2: Cue ablations (pp); CIs: 10,000 paired video bootstraps with fixed models/thresholds.

\Delta t

: five-run mean end-to-end change (ms/frame, RTX 4090).

	TAP-Net		TrackNetV3
Setting	AP r (%)	F1 (%)	AP r (%)	F1 (%)
Default	51.27	82.85	55.22	95.34
$\lambda_{a}=0.25$	51.18	83.17	55.20	95.35
$\lambda_{a}=1.0$	47.29	82.91	55.18	95.36
$\kappa=0.4$	51.18	82.57	55.20	95.35
$\kappa=0.8$	51.33	82.89	55.23	95.34

Table 3: Hyperparameter sensitivity analysis. Defaults:

(\lambda_{a},\kappa,\Lambda)=(0.5,0.6,1)

. Seed: 3407.

Explore similar work

RELO: Reinforcement Learning to Localize for Visual Object Tracking
May 8, 2026Xin Chen, Chuanyu Sun, Jiao Xu +4Multi-Object Tracking Indoor Localization
City University of Hong Kong · Dalian University of Technology · Hunyuan Team, Tencent
TRAIL: Trajectory-Aware Visual Place Recognition against Unordered Databases
Sep 7, 2026Dominik A. Kloepfer, Patrick WenzelVisual Place Recognition Schema
Visual Geometry Group (VGG), University of Oxford · Helsing
SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking
Jun 23, 2026Mohamad Alansari, Yonathan Michael, Hasan AlMarzouqi +3Multi-Object Tracking Memory-Augmented Framework
Khalifa University, Abu Dhabi, UAE · University of Western Australia, Australia

May 8, 2026, cs.CVJ/K move · Enter open · S save

RELO: Reinforcement Learning to Localize for Visual Object Tracking

Xin Chen, Chuanyu Sun, Jiao Xu, Houwen Peng+3

City University of Hong Kong · Dalian University of Technology · Hunyuan Team, Tencent

Multi-Object Tracking Indoor Localization Localization Spatial Priors

Open paper AskPDF arXiv

Conventional visual object trackers localize targets using handcrafted spatial priors, often in the form of heatmaps. Such priors provide only surrogate supervision and are poorly aligned with tracking optimization and evaluation metrics, such as intersection over union (IoU) and area under the success curve (AUC). Here, we introduce RELO, a REinforcement-learning-to-LOcalize method for visual object tracking that formulates target localization as a Markov decision process. Specifically, RELO replaces handcrafted spatial priors with a localization policy learned over spatial positions via reinforcement learning, with rewards combining frame-level IoU and sequence-level AUC. We additionally introduce layer-aligned temporal token propagation to improve semantic consistency across frames, with negligible computational overhead. Across multiple benchmarks, RELO achieves superior results, attaining 57.5% AUC on LaSOText without template updates. This confirms that reward-driven localization provides an effective alternative to prior-driven localization for visual object tracking.

Abstract

Explore similar work

RELO: Reinforcement Learning to Localize for Visual Object Tracking

TRAIL: Trajectory-Aware Visual Place Recognition against Unordered Databases

SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking