TopoDPO: Dense Preference Alignment for Structure-Sensitive Aerial Referring Segmentation
Authors: Huaiyuan Peng, Jisheng Dang, Wencan Zhang, Jingwen Zhao, Hong Peng, Wenjun Ma, Bin Hu, Qi Tian, Tat-Seng Chua
Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China. · School of Computing, National University of Singapore, Singapore 119077. · Huawei, Shenzhen 518129, China.
Preference optimization for multimodal segmentation faces a granularity mismatch: dense masks contain spatially heterogeneous preference signals, whereas image-level objectives assign one shared nonlinear weight to all spatial margins. We introduce TopoDPO, a topology-aware dense preference alignment framework for structure-sensitive aerial referring segmentation. Topo-Prior mines informative preference pairs using connected-component discrepancy and macro-corner recall, while the dense objective applies the preference penalty to each local margin before spatial aggregation against a frozen supervised reference. On Topo-RS-Hard, a 4590-instance structural stress set, TopoDPO improves intersection over union (IoU) from 50.91% to 51.24% over image-level direct preference optimization (DPO) and reduces Betti-0 error from 1.34 to 0.90; paired image-cluster bootstrap intervals against image-level DPO exclude zero for all five reported metrics. Matched supervised controls and a complete factorial analysis separate the effects of topology-aware pair mining, localized correction, and dense reference-relative preference weighting. An unfiltered stratification over all 39,244 validation records further shows the clearest structural gains on samples with moderate or severe pre-alignment component-count errors. The code is publicly available at: https://github.com/phy709/TopoDPO.
Figures & tables
Fig. 1: Image-level DPO compresses a spatial preference signal into one scalar, whereas TopoDPO applies the nonlinear preference penalty to the spatial margin before aggregation. The masks illustrate representative localized structural errors addressed by the proposed dense alignment objective.
Fig. 2: Overview of TopoDPO. Topo-Prior mines structurally informative mask preferences offline. During training, TopoDPO computes margins between the policy and reference branches on the mask grid and applies a pixel-wise preference penalty with a supervised segmentation anchor.
Method
Alignment/loss
IoU
F1
Precision
Δβ0
Corner recall
SFT baseline (2025) [ 3 ]
BCE+Dice
47.88
56.04
59.97
2.77
63.59
SFT + clDice (2021) [ 10 ]
Topology loss
48.57
56.76
60.54
2.37
64.51
SFT + SCNP (2026) [ 21 ]
Local structural loss
48.90
57.12
59.95
2.27
65.06
Topo-Prior continued SFT (ours)
BCE+Dice
49.63
57.92
60.08
2.12
65.38
Topo-Prior masked BCE (ours)
Disagreement pixels
50.72
59.03
60.18
1.18
67.21
Vanilla DPO (2023) [ 7 ]
Image level
50.91
59.16
59.62
1.34
67.58
TABLE I: Results on Topo-RS-Hard (4590 records). Topo-Prior supervised controls use the same 8762 samples and matched training schedules; masked BCE also matches the initial correction-gradient scale. Best values are bold.
Baseline
Metric
Mean difference
95% confidence interval
Vanilla DPO (2023) [ 7 ]
IoU
+0.33
[+0.24,+0.43]
F1
+0.30
[+0.20,+0.40]
Precision
+0.65
[+0.52,+0.79]
Δβ0
−0.44
[−0.50,−0.38]
Corner recall
+0.26
[+0.04,+0.48]
RN-DPO (2026) [ 16 ]
IoU
+2.12
[+1.86,+2.37]
TABLE II: Paired image-cluster bootstrap comparisons with preference-optimization baselines on Topo-RS-Hard. We resample 2581 unique images with replacement for 50 000 replicates, keeping multiple referring expressions from one image in the same cluster. Differences are TopoDPO minus the indicated baseline; positive values favor TopoDPO for IoU, F1, precision, and corner recall, while negative values favor TopoDPO for Betti-0 error. Differences and intervals for percentage metrics are expressed in percentage points; Betti-0 error is an absolute difference.
Fig. 3: Stratified Topo-RS-Hard analysis: (a) Betti-0 error by geometric complexity; (b) Betti-0 error by target category; (c) corner recall by target category. Shaded regions and error bars indicate 95% confidence intervals.
Fig. 4: Unfiltered full-validation-pool stratification by pre-alignment structural-error severity. All 39 244 Aerial-D and RISORS validation records are grouped by the frozen SFT model’s Betti-0 discrepancy, with no IoU-based inclusion filter. The vertical axis reports the reduction in Betti-0 error of TopoDPO relative to Vanilla DPO; positive values favor TopoDPO. Error bars are 95% image-cluster bootstrap confidence intervals.
ID
Pair selection
Objective/configuration
IoU
F1
Δβ0
Corner recall
A1
Random
Image-level DPO
49.18
58.31
2.06
64.72
A2
Random
Pixel-level DPO
49.52
58.78
1.85
65.12
A3
Topo-Prior
Image-level DPO
50.91
59.16
1.34
67.58
A4
Topo-Prior
Pixel-level DPO
51.24
59.45
0.90
67.83
B1
Topo-Prior
λsft=0
48.93
58.85
1.55
65.87
B2
Topo-Prior
λsft=0.1
51.24
59.45
0.90
67.83
TABLE III: Factorial ablations and anchor sensitivity on Topo-RS-Hard
Fig. 5: Qualitative comparison on structurally complex examples. Red baseline masks contain fragmentation, holes, or boundary smoothing; green TopoDPO masks are more coherent in the displayed cases.
Spoken language offers a natural, hands-free interface for specifying an arbitrary target in dense remote-sensing imagery, yet existing referring remote-sensing image segmentation benchmarks accept only written expressions. To bridge this gap, we introduce \dataset, a spoken-query benchmark derived from RISBench that adds accent- and voice-diverse speech while preserving the original image, mask, and data splits. Its hard evaluation sets combine rotor, wind, and mixed interference with three signal-to-noise levels. We also propose \model, an efficient bilateral network that combines a boundary-preserving visual path with token-preserving speech encoding, kernel linear cross-modal attention, and a resolution refinement head. The design conditions visual features at two scales without materializing a dense speech--visual affinity matrix, then restores fine boundaries using high-resolution visual features. On the clean test split, \model with Swin-Base achieves 62.09% mean intersection over union (mIoU) and 68.22% overall intersection over union (oIoU), outperforming the strongest audio-adapted remote-sensing baseline by 5.38 and 2.08 percentage points, respectively. It retains the best hard-set mIoU at 54.09%. To the best of our knowledge, this is the first benchmark and model study of full-sentence spoken-query referring segmentation for remote-sensing imagery. The code will be made publicly available.
Standard deep learning models for image segmentation cannot guarantee topology accuracy, failing to preserve the correct number of connected components or structures. This, in turn, affects the quality of the segmentations and compromises the reliability of the subsequent quantification analyses. Previous works have proposed to enhance topology accuracy with specialized frameworks, architectures, and loss functions. However, these methods are often cumbersome to integrate into existing training pipelines, they are computationally very expensive, or they are restricted to structures with tubular morphology. We present SCNP, an efficient method that improves topology accuracy by penalizing the logits with their poorest-classified neighbor, forcing the model to improve the prediction at the pixels' neighbors before allowing it to improve the pixels themselves. We show the effectiveness of SCNP across 13 datasets, covering different structure morphologies and image modalities, and integrate it into three frameworks for semantic and instance segmentation. Additionally, we show that SCNP can be integrated into several loss functions, making them improve topology accuracy. Our code can be found at https://jmlipman.github.io/SCNP-SameClassNeighborPenalization.
Juan Miguel Valverde, Dim P. Papadopoulos, Rasmus Larsen +1
Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise: images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training. We will release our code and models at: https://github.com/L-CodingSpace/semi-dpo