cs.CVApr 14, 2026

See&Say: Vision Language Guided Safe Zone Detection for Autonomous Package Delivery Drones

Authors: Mahyar Ghazanfari, Peng Wei

Organizations: George Washington University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomous drone delivery systems are rapidly advancing, but ensuring safe and reliable package drop-offs remains highly challenging in cluttered urban and suburban environments where accurately identifying suitable package drop zones is critical. Existing approaches typically rely on either geometry-based analysis or semantic segmentation alone, but these methods lack the integrated semantic reasoning required for robust decision-making. To address this gap, we propose See&Say, a novel framework that combines geometric safety cues with semantic perception, guided by a Vision-Language Model (VLM) for iterative refinement. The system fuses monocular depth gradients with open-vocabulary detection masks to produce safety maps, while the VLM dynamically adjusts object category prompts and refines hazard detection across time, enabling reliable reasoning under dynamic conditions during the final delivery phase. When the primary drop-pad is occupied or unsafe, the proposed See&Say also identifies alternative candidate zones for package delivery. We curated a dataset of urban delivery scenarios with moving objects and human activities to evaluate the approach. Experimental results show that See&Say outperforms all baselines, achieving the highest accuracy and IoU for safety map prediction as well as superior performance in alternative drop zone evaluation across multiple thresholds. These findings highlight the promise of VLM-guided segmentation-depth fusion for advancing safe and practical drone-based package delivery.

Figures & tables

Figure 1: Overview of the proposed See&Say framework. The system takes batches of five RGB frames and corresponding monocular depth maps as input. Depth gradients provide geometric cues for flatness and obstacle detection, while DINO-X produces open-vocabulary semantic hazard masks. These initial maps are fused to form a preliminary safety overlay. A VLM refines detection prompts using temporal RGB–depth context, improving semantic segmentation and updating the safety map. A second VLM agent evaluates the primary drop pad’s safety over time and ranks alternative candidate drop zones according to user-defined preferences. The final outputs include refined safety maps, primary drop-pad safety decisions, scene description, and a ranked list of safe alternative zones.
Figure 2: The VLM refines the object categories used by DINO-X based on the actual scene content, improving hazard detection and safety map accuracy.
Figure 3: The second VLM agent in See&Say incorporates human preferences into package drop zone selection, operating after the initial safety map is generated by the first VLM agent.
MethodSuccess rateReasoning
1 - DINO-X (VisDrone_category)0.483×\times
2 - DINO-X (Complete_category)0.916×\times
3 - See & Say (Ours, Single Frame)0.958✓
4 - See & Say (Ours, 5 Frames)0.975✓
Table 1: Success rate and reasoning status of different methods and frame settings.
Figure 4: Qualitative comparison of pipeline stages across three scenes. Columns: (a) RGB input, (b) monocular depth, (c) initial DINOx mask, (d) depth-gradient mask, (e) initial fusion (DINOx ∨\lor gradient), (f) refined DINOx (after VLM prompts), and (g) final fusion result.
 Overlap (unsafe)Per-pixel detection
MethodIoU (↑)Dice/F1 (↑)Precision (↑)Recall (↑)Specificity (↑)Accuracy (↑)Balanced Acc. (↑)
Gradient0.610 ±\pm 0.2060.736 ±\pm 0.1800.965 ±\pm 0.0470.618 ±\pm 0.2050.986 ±\pm 0.0190.813 ±\pm 0.1310.802 ±\pm 0.109
RTDETR+SAM20.082 ±\pm 0.0730.144 ±\pm 0.1160.993 ±\pm 0.0210.082 ±\pm 0.0731.000 ±\pm 0.0010.592 ±\pm 0.0990.541 ±\pm 0.037
RTDETR+SAM2+Gradient0.619 ±\pm 0.2000.744 ±\pm 0.1720.965 ±\pm 0.0440.628 ±\pm 0.1980.985 ±\pm 0.0190.817 ±\pm 0.1290.806 ±\pm 0.106
YOLOv8+SAM20.082 ±\pm 0.0800.143 ±\pm 0.1270.956 ±\pm 0.2040.082 ±\pm 0.0801.000 ±\pm 0.0000.593 ±\pm 0.0970.541 ±\pm 0.040
YOLOv8+SAM2+Gradient0.619 ±\pm 0.2010.744 ±\pm 0.1730.966 ±\pm 0.0450.627 ±\pm 0.1990.986 ±\pm 0.0190.817 ±\pm 0.1290.806 ±\pm 0.106
Table 2: Segmentation and per-pixel detection metrics (mean ±\pm std). Best results in each column are highlighted. Starred entries indicate methods evaluated five times per frame.
Methodη=95%\boldsymbol{\eta=95\%}η=90%\boldsymbol{\eta=90\%}η=85%\boldsymbol{\eta=85\%}η=80%\boldsymbol{\eta=80\%}
AP (↑)ROC (↑)MAE (↓)AP (↑)ROC (↑)MAE (↓)AP (↑)ROC (↑)MAE (↓)AP (↑)ROC (↑)MAE (↓)
Gradient0.741 ±\pm 0.2660.724 ±\pm 0.2240.151 ±\pm 0.2060.759 ±\pm 0.2530.795 ±\pm 0.1820.168 ±\pm 0.1780.835 ±\pm 0.2440.804 ±\pm 0.2540.174 ±\pm 0.1680.827 ±\pm 0.2390.778 ±\pm 0.2280.178 ±\pm 0.164
RTDETR+SAM20.347 ±\pm 0.1160.523 ±\pm 0.0860.410 ±\pm 0.1110.376 ±\pm 0.1280.529 ±\pm 0.0900.403 ±\pm 0.1020.444 ±\pm 0.1420.568 ±\pm 0.1070.391 ±\pm 0.1040.474 ±\pm 0.1340.538 ±\pm 0.0970.388 ±\pm 0.106
RTDETR+SAM2+Gradient0.741 ±\pm 0.2660.722 ±\pm 0.2260.151 ±\pm 0.2070.759 ±\pm 0.2530.794 ±\pm 0.1830.168 ±\pm 0.1800.835 ±\pm 0.2440.804 ±\pm 0.2540.173 ±\pm 0.1680.827 ±\pm 0.2390.778 ±\pm 0.2280.177 ±\pm 0.165
YOLOv8+SAM20.347 ±\pm 0.1160.534 ±\pm 0.0740.408 ±\pm 0.1070.376 ±\pm 0.1310.535 ±\pm 0.0670.401 ±\pm 0.0990.447 ±\pm 0.1390.572 ±\pm 0.0920.390 ±\pm 0.1020.476 ±\pm 0.1270.539 ±\pm 0.0940.387 ±\pm 0.104
DINOx (VisDrone category)*0.344 ±\pm 0.1170.522 ±\pm 0.0780.412 ±\pm 0.1120.372 ±\pm 0.1290.530 ±\pm 0.0760.402 ±\pm 0.1020.444 ±\pm 0.1410.567 ±\pm 0.1020.392 ±\pm 0.1030.471 ±\pm 0.1300.538 ±\pm 0.0810.387 ±\pm 0.106
Table 3: Drop-zone candidate evaluation across thresholds (mean ±\pm std over frames). Best per column is highlighted. Starred entries indicate methods evaluated five times per frame.
Figure 5: ROC curves (top row) and Precision–Recall curves (bottom row) shown across thresholds from left to right. True positive rate (TPR) is TPTP+FN\frac{\text{TP}}{\text{TP}+\text{FN}} , and false positive rate (FPR) is FPFP+TN\frac{\text{FP}}{\text{FP}+\text{TN}} .
ScenePreferenceAvg. R1Avg. R2κ\boldsymbol{\kappa}nn
Scene 1geometric: close to the parking garage door.2.252.120.8108
 semantic: avoid organic and vegetation zone.2.122.121.0008
Scene 2geometric: top-right corner of the scene.3.003.001.0008
 semantic: furthest from all detected people.2.882.750.6008
Scene 3geometric: closest flat zone to the designated H-pad.2.752.620.7148
 semantic: away from activity.2.752.750.3338
Table 4: Human preference evaluation scores (1–3) and inter-rater agreement (Cohen’s κ\kappa ) per scene and preference type.
Figure 6: Score distributions for human preference evaluation across three scenes. Each panel shows reviewer scores (1–3) for geometric (left) and semantic (right) preferences. Dashed lines indicate μR1\mu_{R_{1}} and μR2\mu_{R_{2}} .
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

ParameterValue
ModelDepth-Anything V2
Input resolutionOriginal frame size
NormalizationMin–max per frame
Numerical stability ϵ\epsilon10−610^{-6}
Appendix
Table B1 : Depth estimation parameters.
ParameterSymbolValue
Gaussian smoothing σ\sigmaσ\sigma1.0
Gradient thresholdτg\tau_{g}1.0
Local window sizew×ww\times w3×33\times 3
Flatness ratio thresholdγ\gamma0.4
Appendix
Table B2 : Gradient filter hyperparameters.
ParameterValue
Binarisation threshold θd\theta_{d}0.5
Initial prompt vocabularyVisDrone-10 categories ∗
Prompt update freq.Once per batch (Agent 1)
Appendix
Table B3 : DINO-X detection parameters.
ParameterSymbol / KeyValue
Circle radius (default)rr100 px
Circle radius (H-pad auto)rr12w2+h2\tfrac{1}{2}\sqrt{w^{2}+h^{2}}
Minimum safe ratioη\eta0.95
Top- kk candidates kepttop_k30
Top- nn zones returnedtop_n3
Hex grid column spacingΔx\Delta x2r2r
Appendix
Table B4 : Drop zone candidate selection parameters.
ParameterValue
Modelo3-2025-04-16
ProviderOpenAI API
Temperature1 (fixed, o3 )
Output formatStrict JSON
Stochastic runs/frame5
Agent 1 inputs
Appendix
Table B5 : VLM agent configuration (both agents).
SceneDescription
Scene 1Front yard with a paved driveway, grass area, garden hose, wooden deck, and 5 people moving unpredictably. The landing pad is on the driveway. Challenging due to dense human activity around and on the pad.
Scene 2Back yard with open grass, a wooden deck, rooftop visible in the background, and maximal human presence. The landing pad is placed on the grass.
Scene 3Back yard with mixed surface types (grass, pavement, flower beds), stairs, and periodic human, and dog activity. The landing pad is partially obstructed in several batches. Tests robustness to partial occlusion and cluttered geometry.
Appendix
Table D1 : Dataset scene summary.
CategoryMetricFormula
Safety MapIoUTPTP+FP+FN\tfrac{TP}{TP+FP+FN}
Dice/F12TP2TP+FP+FN\tfrac{2TP}{2TP+FP+FN}
PrecisionTPTP+FP\tfrac{TP}{TP+FP}
RecallTPTP+FN\tfrac{TP}{TP+FN}
SpecificityTNTN+FP\tfrac{TN}{TN+FP}
AccuracyTP+TNTP+TN+FP+FN\tfrac{TP+TN}{TP+TN+FP+FN}
Appendix
Table E1: Evaluation metrics for safety maps and drop zone selection.
SceneTypePreference string
Scene 1GeometricClose to the parking door
Scene 1SemanticAvoid organic and vegetation zones
Scene 2GeometricTop-right corner of the scene
Scene 2SemanticFurthest from all detected people
Scene 3GeometricClosest flat zone to the designated H-pad
Scene 3SemanticAway from activity
Appendix
Table F1 : Preferences used in the human evaluation study.
More in this topic

Robot Safety

Robot safety studies how to prevent harmful physical actions while enabling robots to complete tasks around people and in uncertain environments.

See all 229 papers

Explore similar work

CardsList

  1. SafeLand: Safe Autonomous Landing in Unknown Environments with Bayesian Semantic Mapping

    Mar 18, 2026Markus Gross, Andreas Greiner, Sai Bharadhwaj Matha +3Aerial RoboticsSafe Robot Navigation

    Fraunhofer Institute IVI · TU Munich · Munich Center for Machine Learning

  2. DroneGround: Open-Vocabulary Drone Payload Characterization Using Synthetic Data and Grounded Vision-Language Models

    Sep 7, 2026Ami Pandat, Rajasekhar Punna, Gopika Vinod +1Open-Vocabulary Object DetectionVision-Language Grounding

    Homi Bhabha National Institute, Mumbai, Maharashtra, India · Bhabha Atomic Research Centre, Mumbai, Maharashtra, India

  3. Synthetic-to-Real Pipeline for Safe Landing Zone Detection

    Jun 9, 2026Shrikant Banerjee, Reza FaieghiSynthetic-to-Real Domain AdaptationUAV Navigation

    Department of Aerospace Engineering Toronto Metropolitan University Toronto, Canada