Bridging Object Detection and Segmentation with Polygon Detection Transformers

Jiacheng Sun; Jiaqi Lin; Wenlong Hu; Haoyang Li; Xinghong Zhou; Chenghai Mao; Xinliang Zhang; Jianya Guo

cs.CVMar 10, 2026

Bridging Object Detection and Segmentation with Polygon Detection Transformers

Authors: Jiacheng Sun, Jiaqi Lin, Wenlong Hu, Haoyang Li, Xinghong Zhou, Chenghai Mao, Xinliang Zhang, Jianya Guo,

Organizations: Shanghai University · SenseTime Research

Abstract

Box detection and mask segmentation are two dominant paradigms for foreground representation: boxes are efficient but too coarse for object shapes, while masks are accurate but over-modeled for compact geometry. To bridge this gap, we present a Polygon Detection Transformer (Poly-DETR) built upon Polar Representation, where object queries regress a starting point and its fixed number of radial distances to directly construct the contour-approximating polygon. This formulation can be integrated into most DETR-like detectors by linear extension, since box is a degenerate case of Polar Representation with four rays. Furthermore, we propose two simple but necessary designs, Polar Deformable Attention and Position-Aware Training Scheme, to align feature sampling and polygon supervision. As a DETR-oriented advancement of Polar Representation, Poly-DETR outperforms existing polar-based methods by 4.7 mAP on MS COCO. Moreover, we explore the application regimes of polygon detection in geometry-driven domains, including remote sensing, medical imaging, and autonomous driving. In particular, Poly-DETR shows stronger scalability than its mask-based counterpart in high-resolution scenarios. Additional experiments show that, owing to its Transformer structure, Poly-DETR can be naturally extended to recent DETR variants equipped with foundation-model priors.

Figures & tables

Figure 1: Comparison of three foreground representations in Transformers. Polygon detection offers stronger shape expressiveness by only linearly extending the box-detection network structure, while avoiding the additional pixel-wise branch required by mask segmentation.

Figure 2: Mismatched Supervision Reference. The starting point shifts from its origin ( green ) to the optimized one ( red ). The static reference yields a polygon that drifts away from the contour ( yellow ), while the dynamic reference keeps the reconstructed polygon aligned.

Figure 3: Incompatible Sampling Reference. Deformable Attention sampling locations ( white crosses ) and their density heatmaps show that box-oriented sampling concentrates around box cues, whereas polar-oriented sampling favors boundary regions.

Figure 4: Overview of the proposed Poly-DETR. The upper part illustrates our polygon detection pipeline, where Enclosing Pos. denotes positional embeddings constructed from the enclosing boxes. The bottom-left part shows the sampling mechanism of Polar Deformable Attention. The bottom-right part illustrates the Position-Aware Training Scheme using three color-coded object queries with different predicted starting points.

Comment	$\mathbf{mAP}$	$\mathrm{\mathbf{AP}}_{\mathbf{50}}$	$\mathrm{\mathbf{AP}}_{\mathbf{75}}$
Baseline	33.4	56.5	34.3
+DETR	34.3 (+0.9)	60.6	34.2
+Polar-DA	36.1 (+2.7)	61.5	36.4
+PATS	37.8 (+4.4)	62.2	39.1

Table 1: Ablation study on module components and their impact on accuracy.

Type	H/T/R	mAP	Params
Vanilla-DA	8/4/1.0	36.1	42.39
Vanilla-DA	32/4/1.0	36.2 (+0.1)	43.68
	32/4/4.0	38.1 (+2.0)	50.21
	32/4/1.0	37.8 (+1.7)	43.68
Polar-DA	32/2/1.0	37.5 (+1.4)	43.17

Table 2: Ablation study on attention design. H : attention heads; T : sampling points; R : projection ratio.

Dynamic	Enclosing	Inner	mAP
			36.1
✓			36.5 ( $+$ 0.4)
✓	✓		37.3 ( $+$ 1.2)
✓	✓	✓	37.8 ( $+$ 1.7)
	✓	✓	35.2 ( $-$ 0.9)

Table 3: Ablation study on Position-Aware Training Scheme. Dynamic , Enclosing , and Inner denote dynamic supervision, enclosing rectangle for positional embedding, and inner cost, respectively.

Type	Method	Backbone	Epoch	mAP	AP50	AP75	FPS	Params $\downarrow$	FLOPs $\downarrow$
Contour	DeepSnake	DLA34	160	30.3	–	–	24	–	–
	E2EC	DLA34	150	33.8	52.9	32.8	35	29.80	121
	PolySnake	DLA34	250	34.4	–	–	18	–	–
	DANCE	Res50	36	36.8	58.5	39.0	16	44.60	274
	BoundaryFormer	Res50	12	36.4	57.2	39.0	19	46.32	229
	ContourFormer	HGNetv2	72	37.6	58.1	39.7	21	–	–

Table 4: Comparison with contour/polar-based methods on MS COCO test-dev. “

\downarrow

” means smaller values are preferred.

Method	mAP	$\mathbf{AP}_{50}$	$\mathbf{AP}_{75}$	FPS	FLOPs $\downarrow$	Mem $\downarrow$
MS COCO val (800 $\times$ 1333)
Mask-DETR	39.8	62.9	43.1	24	264	643
Poly-DETR	37.8	63.1	39.5	32	208	521
Cityscapes (1024 $\times$ 2048)
Mask-DETR	35.3	59.7	34.1	10	533	1557
Poly-DETR	33.8	62.4	29.7	15	454	833

Table 5: Comparison between Poly-DETR and Mask-DETR across datasets. All results are evaluated using 12-epoch models. Poly-DETR and Mask-DETR contain 43.68M and 49.39M parameters, respectively.

Figure 5: High-resolution scalability on MS COCO. Poly-DETR shows lower inference latency and peak memory consumption than Mask-DETR as the input resolution increases.

Figure 6: Qualitative results of Poly-DETR on regular-shaped instance domains.

Figure 7: Qualitative results of Poly-DETR for structured junction topology prediction in autonomous driving tasks. Green solid lines denote stop lines, while white dashed lines indicate rays parameterized by learnable polar angles.

Methods	Epoch	mAP	FPS	Params $\downarrow$
RF-DETR-M-seg	12	44.6	37	36.20
RF-DETR-M-poly	12	43.2	42	33.90
Mask2Former-R50	50	42.9	23	44.03

Table 6: Comparison of RF-DETR-based and Mask2Former mask/polygon variants on MS COCO val.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Motivation of continuous starting-point prediction. Four adjacent grid locations produce polygons with different representation errors (RE), while the optimal starting point lies between these discrete locations and cannot be reached by grid-based selection.

Figure 9: Distribution of representation quality for starting points predicted by PolarNeXt and Poly-DETR. Each segment represents the proportion of instances whose reference polygons fall into the corresponding IoU interval. Poly-DETR achieves a higher proportion of instances in high-IoU intervals, indicating more accurate starting-point localization.

Figure 10: Architecture of Hybrid Supervision Strategy. An additional one-to-many classification branch ( $\mathrm{\mathbf{cls}_{1m}}$ ) is introduced alongside the original one-to-one branch ( $\mathrm{\mathbf{cls}_{11}}$ ), while the polygon regression branch ( $\mathrm{\mathbf{poly}_{11+1m}}$ ) is shared by both supervision paths.

Figure 11: Training curves of Distance Loss with and without Inner Cost. The upper plot compares the Distance Loss curves, and the lower plot presents the corresponding loss gap over training iterations.

Query Initialization	Enclosing Pos.	mAP
Polar Parameters	✓	37.3
Polar Parameters		36.5 (-0.8)
Box Parameters		37.4 (+0.1)

Appendix

Table 7: Effect of decoder query initialization strategies. Polar and Box parameters are predicted from the encoder features and used to initialize decoder queries. Enclosing Pos. denotes whether the enclosing rectangles of polygons are additionally provided for positional embedding.

RMask Resolution		Decoder Layers
RMask	mAP	Layers	mAP
$32\times 32$	37.8	6	37.8
$64\times 64$	37.9	9	37.7

Appendix

Table 8: Sensitivity analysis of RMask resolution and decoder layers.

Model	Top Subset
Model	50%	30%	10%
Mask-DETR	42.4	45.6	47.3
Poly-DETR	41.8	46.1	49.2
impr .	-0.6	+0.5	+1.9

Appendix

Table 9: Comparison on MS COCO subsets ranked by Polygon Approximability Score. Higher-ranked subsets contain instances that are easier to approximate by compact geometry.

Figure 12: Instance-level comparison between PolarNeXt and Poly-DETR.

Box Branch	mAP	Params	Mems	Epochs	GPU Time
w/o.	37.8	43.68	521	12	300h
w.	37.7	43.92	530	12	160h

Appendix

Table 10: Impact of retaining the box branch in the decoder.

Figure 14: TIDE Error Analysis. We adopt TIDE to compare the errors of PolarNeXt, Poly-DETR, and Mask-DETR. Cls : classification error; Loc : localization error; Miss : missing detections; Bkg : background detections; Dupe : duplicated detections; Both : Cls + Loc error.

Figure 15: Image-level results across four datasets.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection
Mar 6, 2026Zhengjian Kang, Jun Zhuang, Kangtong Mo +3Object Detection Detection Transformer
New York University · Boise State University · University of Illinois at Urbana-Champaign +3
Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement
Sep 17, 2026Yitong Xing, Yuhao Cheng, Yanping Li +1Detection Transformer Object Detection
School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China
Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers
Aug 2, 2026Robin Kim, Colin Samplawski, Benjamin M. MarlinDetection Transformer Representation Probing
UMass Amherst, Amherst, MA, USA · SRI International, Menlo Park, CA, USA

Mar 6, 2026J/K move · Enter open · S save

PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection

Zhengjian Kang, Jun Zhuang, Kangtong Mo, Qi Chen+2

New York University · Boise State University · University of Illinois at Urbana-Champaign+3

Object Detection Detection Transformer

Open paper AskPDF arXiv

Detection Transformer (DETR) has redefined object detection by casting it as a set prediction task within an end-to-end framework. Despite its elegance, DETR and its variants still rely on fixed learnable queries and suffer from severe query utilization imbalance, which limits adaptability and leaves the model capacity underused. We propose PaQ-DETR (Pattern and Quality-Aware DETR), a unified framework that enhances both query adaptivity and supervision balance. It learns a compact set of shared latent patterns capturing global semantics and dynamically generates image-specific queries through content-conditioned weighting. In parallel, a quality-aware one-to-many assignment strategy adaptively selects positive samples based on localizatio-classification consistency, enriching supervision and promoting balanced query optimization. Experiments on COCO, CityScapes, and other benchmarks show consistent gains of 1.5%-4.2% mAP across DETR backbones, including ResNet and Swin-Transformer. Beyond accuracy improvement, our method provides interpretable insights into how dynamic patterns cluster semantically across object categories.

Abstract

Object Detection

Explore similar work

PaQ-DETR: Learning Pattern and Quality-Aware Dynamic Queries for Object Detection

Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement

Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers