Detect Anything in Real Time: From Single-Prompt Segmentation to Multi-Class Detection

Mehmet Kerem Turkcan

cs.CVMar 12, 2026

Detect Anything in Real Time: From Single-Prompt Segmentation to Multi-Class Detection

Authors: Mehmet Kerem Turkcan

Organizations: Columbia University

Abstract

Recent advances in vision-language modeling have produced promptable detection and segmentation systems that accept arbitrary natural language queries at inference time. Among these, SAM3 achieves state-of-the-art accuracy by combining a ViT-H/14 backbone with cross-modal transformer decoding and learned object queries. However, SAM3 processes a single text prompt per forward pass. Detecting N categories requires N independent executions, each dominated by the 439M-parameter backbone. We present Detect Anything in Real Time (DART), a training-free framework that converts SAM3 into a real-time multi-class detector by exploiting a structural invariant: the visual backbone is class-agnostic, producing image features independent of the text prompt. This allows the backbone computation to be shared between all classes, reducing its cost from O(N) to O(1). Combined with batched multi-class decoding, detection-only inference, and TensorRT FP16 deployment, these optimizations yield 5.6x cumulative speedup at 3 classes, scaling to 25x at 80 classes, without modifying any model weight. On COCO val2017 (5,000 images, 80 classes), DART achieves 55.8 AP at 15.8 FPS (4 classes, 1008x1008) on a single RTX 4080, surpassing purpose-built open-vocabulary detectors trained on millions of box annotations. For extreme latency targets, adapter distillation with a frozen encoder-decoder achieves 38.7 AP with a 13.9 ms backbone. Code and models are available at https://github.com/mkturkcan/DART.

Figures & tables

Panel of Figure 1 — Figure 1: From SAM3 to DART. (a) SAM3 runs the full pipeline once per class. The backbone accounts for 78% of compute and is repeated $N$ times despite being class-independent. (b) DART shares backbone features across all classes, batches the encoder-decoder, removes the mask head, and deploys both stages as TRT FP16 engines. The backbone uses restructured attention (explicit operations, real-valued RoPE) to enable correct FP16 export (§ 3.3 ). (c) For video, backbone and encoder-decoder run on separate CUDA streams: the encoder-decoder for frame $t$ overlaps with the backbone for frame $t{+}1$ , reducing per-frame latency from 70 ms to 53 ms (4 classes, 1008px).

Level	Optimization	BB	E-D	ms	Speedup
0	Naive ( $N$ full passes)	$N{\times}$	$N{\times}$	336	1 $\times$
1	Shared backbone	1 $\times$	$N{\times}$	162	2.1 $\times$
2	+ Batched + det-only	1 $\times$	1 $\times$	112	3.0 $\times$
3	+ Graph restructure + TRT BB	TRT	1 $\times$	78	4.3 $\times$
4	+ TRT enc-dec + pipeline	TRT	TRT	60	5.6 $\times$

Table 1: Cumulative optimization hierarchy. 1008

\times

1008, 3 classes, RTX 4080. Each row includes all optimizations from rows above.

$N$	BB (ms)	E-D (ms)	Sum	Seq. FPS	Pipe. FPS	Gain
1	53.2	7.9	61.1	16.3	18.7	+15%
2	53.2	11.4	64.6	15.5	17.6	+14%
4	53.2	19.2	72.4	13.8	15.8	+14%
8	53.2	34.7	87.9	11.5	12.5	+9%

Table 2: Latency breakdown by class count (1008

\times

1008, RTX 4080). Backbone latency is constant; encoder-decoder scales linearly. Pipelined FPS follows Eq. 1 .

Panel of Figure 2 — Figure 2: FPS vs. class count. At 644px, all tested class counts exceed 30 FPS pipelined. At 1008px, up to 4 classes reach $\geq$ 15 FPS. Pipelining provides 8-15% improvement with diminishing returns as encoder-decoder cost approaches backbone cost.

Configuration	Res.	Removed	AP	AP 50	AP 75	AP S	AP M	AP L
Full TRT FP16	1008	0	55.8	73.4	61.5	40.3	59.8	70.7
SBP-16 TRT FP16	1008	16	47.6	63.5	52.7	32.5	51.8	62.0
Full TRT FP16	644	0	39.1	63.9	39.9	12.4	44.4	65.4
SBP-16 TRT FP16	644	16	32.8	54.5	33.4	9.9	37.9	56.7

Table 3: COCO val2017 detection quality (5,000 images, 80 classes). Resolution is the dominant quality lever, and small-object recall drives the gap.

Panel of Figure 3 — Figure 3: Qualitative results. DART detections on COCO val2017 images using 6 open-vocabulary classes ( person , car , dog , bicycle , chair , cat ) at 1008px resolution with confidence threshold 0.45. All optimizations are structural: DART produces identical outputs to the per-class SAM3 baseline.

Method	Latency	Cosine	Status
Explicit-attn TRT FP16	53 ms	0.999	✓
Fused-SDPA TRT FP16	26 ms	0.058	×
Fused-SDPA mixed (attn FP32)	128 ms	0.999	✓
torch.compile FP16	75 ms	1.000	✓
PyTorch eager FP16	87 ms	1.000	✓

Table 4: Backbone deployment comparison. 1008px, RTX 4080. Only the explicit-attention TRT path achieves both correctness and competitive latency.

Teacher (training-free, this work)
Method	Backbone	Params	AP	AP 50	AP 75	AP S	AP L	BB (ms)
DART (ViT-H)	ViT-H/14	439M	55.8	73.4	61.5	40.3	70.7	53
Adapter distillation (this work): frozen SAM3 encoder-decoder
DART (student) ‡	RepViT-M2.3	8.2M	38.7	53.1	42.3	22.6	49.9	13.9
DART (student) ‡	TinyViT-21M	21M	30.1	42.4	32.6	17.4	37.8	12.2
DART (student) ‡	EfficientViT-L2	9.2M	21.7	31.5	23.5	13.7	24.2	10.7

Table 5: Backbone distillation comparison on COCO val2017 (80 classes, 1008

\times

1008, 5,000 images). Adapter distillation preserves the original SAM3 encoder-decoder, while full-pipeline distillation replaces it. EfficientSAM3 numbers are with presence-gating disabled. † TRT FP16 backbone + enc-dec (

N_{\max}{=}16

, 5 passes).

Trained on detection data (zero-shot on COCO) †
Method	Backbone	Det. Data	AP
GLIP-L [ 6 ]	Swin-L	O365+GoldG	49.8
Grounding DINO-T [ 7 ]	Swin-T	O365+GoldG	48.1
Grounding DINO-L [ 7 ]	Swin-L	O365+GoldG+Cap4M	52.5
YOLO-World-L [ 4 ]	YOLOv8-L	O365+GoldG	44.9
YOLO-World-X [ 4 ]	YOLOv8-X	O365+GoldG	46.7

Table 6: Context among open-vocabulary detectors (COCO val2017, 80 classes). DART uses frozen SAM3 weights with no detection training; all other methods train on large-scale detection data. † Zero-shot transfer (no COCO images in training). ‡ LVIS zero-shot; COCO zero-shot not reported.

Panel of Figure 4 — Figure 4: Speed-quality Pareto front. Numbers indicate class count. Teacher ViT-H dominates quality (55.8 AP) but is limited to $<$ 19 FPS. Distilled students reach 45–51 FPS at 4 classes, trading AP for 2.5–3 $\times$ throughput. Among student backbones, RepViT-M2.3 achieves the highest AP (38.7) at 45 FPS.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 611 papers

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection
Oct 7, 2026Yupeng Zhang, Ziyi Zhao, Juntao Cheng +4Open-Vocabulary Object Detection Real-Time Object Detection
Tianjin University. · Shenzhen University of Advanced Technology. · Shanghai Jiaotong University. +1
Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design
Feb 24, 2026Jiannan Huang, Aditya Kane, Fengzhe Zhou +2Real-Time Object Detection Efficient Object Detection
Beijing Jiaotong University
RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
Oct 29, 2025Zijun Liao, Yian Zhao, Xin Shan +5Real-Time Object Detection Efficient Object Detection
School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Department of Automation and BNRist, Tsinghua University, Beijing, China

Oct 7, 2026J/K move · Enter open · S save

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

Yupeng Zhang, Ziyi Zhao, Juntao Cheng, Sheng Wang+3

Tianjin University. · Shenzhen University of Advanced Technology. · Shanghai Jiaotong University.+1

Open-Vocabulary Object Detection Real-Time Object Detection Detection Transformer Efficient Object Detection Cross-Modal Knowledge Distillation

Open paper AskPDF arXiv

Open-vocabulary detection (OVD) recognizes categories unseen during training through textual category queries, yet achieving strong generalization with real-time efficiency remains challenging. Beyond vocabulary scaling, zero-shot generalization may benefit from reusable visual--semantic cues learned from seen data, including attributes, actions, states, and contextual relations. Existing real-time OVD methods primarily emphasize vocabulary coverage and efficient region/query--text matching; under strict efficiency constraints, compact detectors may struggle to absorb rich instance semantics and scene context. We propose RT-DETR-World, a compact DETR-style detector that transfers the rich semantics conveyed by descriptions during training while retaining lightweight query--text matching at inference. We construct GroundingCapv2 with three levels of supervision: category names for standard OVD, object descriptions conveying instance-level semantics, and image descriptions conveying object relations and scene context. These descriptions serve only as training-time semantic supervision. To help the compact detector absorb these semantics, we propose Dual-Path Description Alignment (DDA), combining a deployment-consistent MiniLM pathway with a training-only LLM teacher. MiniLM provides query--category supervision and object-description alignment, while offline teacher features supervise matched queries and global visual representations at the object and image levels, respectively. All teacher features are precomputed, and the teacher-side modules are removed after training. We further propose Relation-Aware Negative Relaxation (RNR), which uses teacher-derived semantic similarities to relax related negatives while preserving exact positives. Experiments demonstrate competitive zero-shot accuracy and a favorable accuracy--efficiency trade-off. The code will be released.

Abstract

Object Detection

Explore similar work

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design

RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models