Organizations: College of Mechanical and Electrical Engineering, Harbin Engineering University, No. 145 Nantong Street, Nangang District, Harbin, 150001, Heilongjiang Province, P. R. China · Department of Thoracic Surgery I, Hainan Cancer Hospital, No. 6 Changbin West 4th Street, Xiuying District, Haikou, 570312, Hainan Province, P. R. China
Surgical action-center localization requires both an accurate reference point and a decision about whether to report it. When frames with and without a reportable center are mixed, the benefit of improved localization depends on which correct candidates survive selection. We present a selective localization pipeline combining heatmap and coordinate supervision with a logistic-regression scorer. The scorer learns to identify predictions with a reportable center and a localization error within a specified tolerance. The internal evaluation uses 5,000 frames from 50 surgical sources, five source-grouped folds and two random seeds. Matched locator and scoring comparisons track correct candidates and their rejection. At an 80% acceptance quota per source and a tolerance of 0.10 times the image diagonal, our method retains 46.08 correct outputs per 100 inputs. The baseline locator with the same scoring approach retains 44.20. The gain is 1.88 (descriptive 95% source-paired bootstrap interval, 0.94 to 2.79). Under entropy-based selection, increased rejection of correct predictions offsets the higher supply of correct predictions on average. For the coordinate-supervised locator, supervised scoring reduces these rejections from 6.15 to 2.99 per 100 inputs. Evaluating retained correct outputs reveals whether a selection rule preserves or erodes an improvement in localization.
Figures & tables
Figure 1: Mixed reportability in surgical action-center localization. (a, c) POINT frames with an annotated action center, indicated by a yellow ring. (b, d) NA frames without a reportable center in the dataset annotation. Each row shows frames from the same surgical source at different times.
Figure 2: Selective surgical action-center localization. (a) The locator C generates a sigmoid heatmap, an expectation-based coordinate, a global image representation and an auxiliary reportability logit. The 523-dimensional feature vector is standardized and passed to the logistic-regression scorer S. GAP denotes global average pooling, and L2 denotes the squared-weight regularization penalty. Candidates are ranked within each source, and the top 80% are reported at the primary operating point. (b) Offline data roles separate locator training (L25), checkpoint selection (V5), scorer fitting (R10) and evaluation (T10). Numbers indicate sources per role. C is fixed when fitting S; labels identify POINT predictions within the 0.10-diagonal tolerance.
Figure 3: Selective localization across acceptance budgets. (a) Correct accepted outputs Y per 100 original inputs and (b) accepted risk at τ=0.10 . The vertical dashed line marks the primary 80% quota. Risk is undefined at zero acceptance. (c) Mean output differences between C+S and B+S across tolerances and quotas. Values average two seeds within each of 50 sources, followed by an equal-source average. C+S is our method; B and C denote the baseline and coordinate-supervised locators, and E and S denote entropy selection and learned success scoring.
Configuration
G ↑
Y ↑
M ↓
Risk (%) ↓
B+E
46.58
43.50
3.08
45.63
B+S
46.58
44.20
2.38
44.75
C+E
49.07
42.92
6.15
46.35
Ours (C+S)
49.07
46.08
2.99
42.40
Comparison
Δ Y ↑
95% interval
C+S − B+S
+1.88
[0.94, 2.79]
Table 1: Selective localization at τ=0.10 and q=0.80 . G is correct localization supply, Y is correct accepted output and M is rejected correct predictions, all per 100 original inputs. Risk is the failure percentage among accepted outputs. Means weight the 50 sources equally after averaging their two seeds. B and C denote the baseline and coordinate-supervised locators; E and S denote entropy selection and learned success scoring. The lower panel reports source-paired differences in Y ( Δ Y) with descriptive 95% bootstrap intervals. Bold identifies the proposed C+S configuration.
Figure 4: Correct coordinate supply and selection outcomes at τ=0.10 and q=0.80 . (a) Open diamonds mark supply G, filled circles mark retained correct outputs Y, and the right-hand values give rejected correct predictions M. (b) Accepted outputs comprise correct predictions, NA frames, POINT errors with 0.10<d≤0.20 , and severe POINT errors with d>0.20 . The four segments sum to 80 accepted outputs per 100 inputs. All values are source-macro means after averaging the two seeds within source.
Figure 5: Source-paired changes in correct accepted outputs at τ=0.10 and q=0.80 . Each dot represents one source after averaging its two seeds. (a) C+S minus B+S. (b) C+S minus C+E. Diamonds and horizontal bars show the mean and descriptive 95% percentile-bootstrap interval from 20,000 source resamples. Positive, zero and negative source counts are stated above each distribution; vertical stacking separates coincident points.
Figure 6: Scoring comparisons at τ=0.10 . (a) Correct accepted outputs per 100 inputs for eight scores on B and C at q=0.80 . Scores are defined in Section 4.3 . (b) Mean difference in Y between success supervision S and reportability supervision pR across quotas. (c) Their source-paired mean differences and descriptive 95% intervals at 80% acceptance. S and pR use the same 523-feature schema with target-specific fitting. All summaries average seeds within source before aggregating 50 sources.
Locator + S
G ↑
Y ↑
SE ↓
B+S
46.58
44.20
7.60
Ours (C+S)
49.07
46.08
5.66
J+S
48.40
45.96
5.60
SegFormer+S
44.18
42.04
7.99
Table 2: Locator configurations with separately fitted S at τ=0.10 and q=0.80 . G and Y are correct supply and correct accepted outputs. SE counts accepted POINT predictions with d>0.20 . B is the baseline, C is the coordinate-supervised locator, and J also lets the auxiliary loss update the shared encoder. All values are per 100 original inputs and use the same source-macro aggregation.
Configuration
Coverage (%)
Y ↑
Risk (%) ↓
B+E
79.48
44.18
45.64
B+S
79.23
44.70
45.09
C+E
78.37
42.83
46.30
Ours (C+S)
78.97
46.85
42.45
Table 3: Frozen-threshold evaluation at τ=0.10 . Thresholds were developed at 80% acceptance on R and applied numerically to T. Coverage is the actual accepted percentage, Y is per 100 original inputs, and risk is the source-macro failure percentage among accepted outputs. All 100 source-seed units had at least one accepted prediction.
Figure 7: Coverage after transferring frozen development thresholds to T, at τ=0.10 . (a) Source coverage for each configuration, with median ticks and interquartile-range bars. (b) Source coverage and accepted risk for C+S, with marginal source counts. Each point averages the two seeds within one source. The dashed line marks 80%, the development acceptance target. Risk is calculated within each source and seed before averaging.
Figure 8: Examples of localization and selection at τ=0.10 , q=0.80 and seed 20260901. Yellow circles denote POINT annotations, cyan crosses B predictions and orange plus signs C predictions. Panels a and d show coordinates entering tolerance; b and e show correct C coordinates retained by S after rejection by E; c shows NA rejection; f shows a residual correct rejection. Full frames retain the original view. White boxes locate the detail crops, whose extents vary to contain the annotation and both predictions; distances beside each crop use the original-image diagonal. The three decisions beneath each frame correspond to B+S, C+E and C+S.
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
Natural-language point-cloud localization typically retrieves candidate map cells and then predicts a position inside each candidate, yet the final output is still determined by retrieval order. A retrieval score measures region-description relevance, not whether the coordinates a region yields are accurate: a relevant region may produce a displaced prediction, and a lower-ranked region may contain a successful one. We study the final selection problem after the candidate pool and Fine predictions are fixed---how to choose the coordinate most likely to localize successfully, and under which conditions the gain holds. We propose SCAR (Spatial Consistency-Aware Reranking). SCAR treats each cell--Fine-coordinate pair as a localization hypothesis, interprets directional relations as functions of position, and reads spatial support at the backbone's actual Fine point (Fine-conditioned Evidence Readout). It organizes this readout with center support, semantic presence, retrieval score, and geometric offsets into a 23-D representation, trains a shared scorer with localization-success labels, and at inference reranks existing hypotheses and returns their original coordinates. Experiments cover four public backbones and a corrected Des4Pos reimplementation across five domains. Excluding one protocol-reference configuration, exploratory TEST point estimates on the remaining 24 configurations show Top-1@5,m above Original on 23 and Cell R@1 above Original on all 24. On development-set VAL, the Fine readout improves Top-1@5,m by 5.77 percentage points (pp) over a center surrogate on corrected Des4Pos*KITTI, and the complete scoring scheme improves it by 13.98,pp over FSR on MambaPlace×nuScenesFull; the latter measures the complete scheme, not learned ranking alone.
Shuhan Dong, Tianyi Shang, Zhenyu Li
Qilu University of Technology (Shandong Academy of Sciences), Jinan, China · Purdue University, West Lafayette, IN, USA
Vision foundation models such as SAM 3 can provide transferable object-level structure across diverse surgical video conditions, but segmentation outputs do not explicitly encode the action-conditioned semantics that define functional surgical landmarks. Estimating instrument extent and geometry differs from localizing the tip or anchor relevant to clipping, grasping, or dissecting. We investigate vision foundation model-enabled sparse action-aware landmark localization, using zero-shot, point-prompted structural masks to provide dense instrument-level context without manual pixel-level mask annotations. We propose a lightweight refinement framework that uses SAM 3 as a structural prior. A coarse multi-frame network predicts tip and anchor prompts, generating non-oracle masks that are fused with visual and heatmap features to refine functional landmark predictions. We compare direct mask-augmented supervision, prediction-derived mask-prior refinement, and auxiliary mask supervision to examine how vision foundation model-derived structure should enter a precision-oriented localization system. Experiments on 7,867 clips from 60 surgical videos spanning YouTube, Cholec80, HeiChole, SurgVU, and CRCD evaluate the approach under heterogeneous conditions. Without manual pixel-level mask annotations for training, the proposed model achieves overall F1 scores of 72.4% for tip and 58.0% for anchor localization. Directly imposing masks on heatmap targets biases learning toward broad tool regions, whereas prediction-derived priors and auxiliary supervision provide effective intermediate structural guidance for action-dependent landmark prediction.
We present RIC-Loc (Reference-Induced Consensus localization), a scene-training-free posed-reference localizer that is SfM-point-map-free in its main estimator: it uses known reference poses, but not precomputed SfM 3D map points, query-to-map 2D-3D matches, or query-to-map PnP. A frozen VGGT pass predicts local camera poses, depth, and query-reference tracks for a query and selected references. Each reference induces one map-frame SE(3) query-pose hypothesis, robust consensus estimates the pose, and the preserved hypothesis structure yields two reliability scores: spatial dispersion and a track-conditioned covariance score. On the covariance-eligible set, the two scores are complementary for held-out, ground-truth-free failure detection across indoor, outdoor, and large-scale low-texture benchmarks: the joint policy is strongest in textured scenes and the covariance score in the low-texture regime, and the hypothesis-derived scores consistently outperform the standard retrieval-score gap and random rankings. Without per-scene training the consensus estimator remains accurate -- competitive with structure-based localization indoors and improving over a comparable feed-forward baseline -- giving an effective selective operating regime for posed-reference localization. Code is available at https://github.com/SNU-DLLAB/ric_loc.
Wonseok Kang, Jaehyun Kim, Jeongmin Lee +1
DeepFine Co., Ltd., Seoul, South Korea · Seoul National University, Department of Naval Architecture and Ocean Engineering, Seoul, South Korea