Check Your Blind Spots: Decision-Guided Perception by Probing the Planner with Hypothetical Objects
Organizations: Hanoi University of Science and Technology · Victoria University of Wellington
Abstract
Not every perception error changes the planner's action. Yet adaptive-resolution perception chooses where to spend high-resolution computation from image cues, which can neither reliably point to an object that low resolution missed nor tell which objects matter to the planner. We observe that the planner can say where an object would matter before anything is seen: inserting a hypothetical object at a location tests whether its action would change. Decision-Guided Speculative Perception (DGSP) treats the low-resolution scene and its action as a draft and probes the planner in this way to obtain a decision-sensitivity field, the locations where an unseen object would change the action. Only this field and the support, the detections on which the current action rests, are verified at high resolution, repeatedly until the action is stable. For two planner classes, we prove that DGSP returns exactly the action of whole-frame high-resolution perception if crop detections match whole-frame ones and the crops cover the field, conditions that real images meet approximately. On three driving datasets, it reproduces this action on about 98% of frames with about a quarter of the pixels. At the pixel cost of uniform downscaling, it repairs more of the draft's disagreements with this action, brakes falsely less often, and recovers many more of the decision-relevant objects that the draft misses.
Figures & tables
| nuScenes | Argoverse 2 | Argoverse-HD | |||||||
| Method | Agr. | Rep. | px | Agr. | Rep. | px | Agr. | Rep. | px |
| u320 (draft) | 92.7 † | 0 † | 4.3 | 85.3 † | 0 † | 2.6 | 78.8 † | 0 † | 3.1 |
| u320 + | 96.5 † | 59.9 † | 4.3 | 94.7 † | 70.2 † | 2.6 | 91.4 † | 63.8 † | 3.1 |
| u512 | 97.7 † | 79.6 † | 10.2 | 96.4 † | 84.0 † | 6.7 | 94.5 † | 82.6 † | 7.1 |
| u640 | 98.2 † | 84.7 | 17.1 | 97.3 † | 88.3 † | 10.3 | 95.8 † | 87.4 † | 11.6 |
| u768 | 98.6 | 88.6 | 23.9 | 97.8 † | 90.4 | 14.7 | 96.1 † | 88.5 † | 16.0 |
| nuScenes | Argoverse 2 | Argoverse-HD | Mean | |||||||||||||||||
| Method | px | GF | P | R | F1 | px | GF | P | R | F1 | px | GF | P | R | F1 | px | GF | P | R | F1 |
| u320 (draft) | 4.3 | 4.3 | 31.4 † | 65.3 | 42.4 † | 2.6 | 5.8 | 29.3 † | 82.1 | 43.2 † | 3.1 | 5.0 | 10.9 † | 70.4 ∗ | 18.9 † | 3.3 | 5.0 | 23.9 † | 72.6 ∗ | 34.8 † |
| u320 + | 4.3 | 9.6 | 52.8 † | 57.1 † | 54.9 † | 2.6 | 19.0 | 62.9 † | 76.8 | 69.2 † | 3.1 | 20.6 | 25.5 † | 50.7 | 34.0 † | 3.3 | 16.4 | 47.1 † | 61.6 | 52.7 † |
| u512 | 10.2 | 17.8 | 58.8 † | 66.0 | 62.2 † | 6.7 | 33.1 | 64.0 † | 80.5 | 71.3 † | 7.1 | 32.6 | 34.0 † | 47.9 | 39.8 † | 8.0 | 27.8 | 52.3 † | 64.8 | 57.8 † |
| u640 | 17.1 | 26.1 | 60.2 † | 66.0 | 63.0 † | 10.3 | 42.9 | 71.8 † | 81.6 ∗ | 76.4 | 11.6 | 43.6 | 37.9 | 54.9 | 44.8 | 13.0 | 37.5 | 56.6 † | 67.5 | 61.4 † |
| u768 | 23.9 | 33.6 | 62.4 † | 66.7 | 64.5 † | 14.7 | 53.9 | 71.7 † | 82.6 ∗ | 76.8 | 16.0 | 52.5 | 33.3 † | 47.9 | 39.3 † | 18.2 | 46.7 | 55.8 † | 65.7 | 60.2 † |
| Decisive object | Ego speed | Range of decisive object | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | All | Seen | Missed | Low | High | 20 | 20–40 | 40–60 | |
| u512 | 97.7 † | 91.5 † | 74.5 † | 99.3 ∗ | 95.8 † | 92.5 | 88.1 † | 82.7 † | |
| u640 | 98.2 † | 93.6 | 81.6 | 99.5 ∗ | 97.0 † | 93.2 | 92.2 | 89.0 † | |
| nuSc. | DGSP | 98.7 | 95.9 | 87.9 | 98.8 | 98.7 | 94.8 | 95.1 | 95.3 |
| u512 | 96.4 † | 92.3 † | 81.1 † | 98.3 | 95.9 † | 92.8 † | 92.5 † | 88.8 † | |
| u640 | 97.3 † | 94.3 † | 85.4 † | 98.7 | 96.8 † | 95.0 | 93.4 † | 92.6 † | |
| nuScenes | Argoverse 2 | Argoverse-HD | |||||
|---|---|---|---|---|---|---|---|
| Variant | px | Rep. | F1 | Rep. | F1 | Rep. | F1 |
| DGSP (full) | 7.1 | 86.9 | 69.0 | 91.7 | 77.5 | 92.0 | 46.7 |
| Support only (w/o field) | 5.0 | 76.4 † | 61.9 † | 84.1 † | 75.3 | 86.5 † | 44.1 |
| Field only (w/o support) | 5.7 | 44.0 † | 48.6 † | 39.4 † | 44.6 † | 40.2 † | 19.0 † |
| Pivots instead of support | 6.9 | 85.6 † | 69.0 | 89.5 † | 74.7 † | 88.4 † | 32.5 † |
| Single round | 6.8 | 86.8 | 67.4 † | 91.5 | 77.0 | 91.6 | 44.2 |
| Curved path: Agr. | Curved path: F1 | Selector | |||||
| Regions | nuSc. | AV2 | A-HD | nuSc. | AV2 | A-HD | Agr. |
| Whole frame | 97.6 † | 96.2 † | 94.7 † | 55.4 | 69.2 † | 46.3 | 94.1 † |
| DGSP, braking | 97.1 † | 94.6 † | 94.7 † | 54.6 | 65.4 † | 38.4 † | 96.5 † |
| DGSP , own | 98.5 | 97.5 | 97.1 | 58.6 | 73.3 | 50.7 | 97.8 |
| Method | px | Miss. | Hard | Prev. |
|---|---|---|---|---|
| u320 (draft) | 4.3 | 0.44 † | 4.25 † | 1.5 |
| u512 | 10.2 | 0.35 † | 3.90 † | 1.7 † |
| u960 ( ) | 38.4 | 0.33 | 3.97 † | 1.1 |
| DGSP | 8.1 | 0.28 | 3.07 | 0.4 |
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Procedure or planner | Condition violated | Returned vs . | |
|---|---|---|---|
| (a) | field only | support (empty support not certifying) | vs . |
| (b) | support only | coverage of the field | vs . |
| (c) | pivots instead of the support | certification (two objects justify the action) | vs . |
| (d) | a single round | fixed point | vs . |
| (e) | additive demands | insertion stability | vs . |
| (f) | witness instead of the binding set | certification (bottleneck selector) | vs . |
| Condition | Implementation | Status | Where its effect is measured |
|---|---|---|---|
| Insertion stability | The braking controllers and the time-to-collision warning are max-composable; the selector on NAVSIM is a bottleneck selector | exact | Corollaries 1 , 2 ; joint effects in a general planner: Sec. A.7 |
| Certifying support | Closed forms of the two corollaries | exact | pivots in its place: Tab. 4 |
| Coverage | Field on a finite ground grid with one probe size and one closing speed; the margin enlarges it; the footprint bounds the sensitive probes; the stopping test accepts overlap | approx. | decisive object inside the verified crops: Tab. 30 ; other probes and grids: Tab. 34 |
| Locality | Crop detections differ from whole-frame ones through context, truncation, and temporal cues; grid-aligned crops, the truncation guard, and the same-scale history narrow the gap | approx. | without the guard: Tab. 4 , Tab. 26 ; agreement at changes of : Tab. 12 |
| Fixed point | At most three rounds | bounded | single round: Tab. 4 |
| (none) | Range refiner, applied after the loop | outside | without it: Tab. 4 |
| Dataset | Front camera | Development | Test | Frames | Episodes |
|---|---|---|---|---|---|
| nuScenes [ 3 ] | , keyframes at Hz | scenes | scenes | ||
| Argoverse 2 [ 44 ] | , annotated frames | training logs | val, logs | ||
| Argoverse-HD [ 27 ] | , Hz | – | val, sequences | ||
| held-out | – | training sequences | |||
| NAVSIM [ 7 ] | – | navtest | scenes | – |
| nuScenes | Argoverse 2 | Argoverse-HD | |
|---|---|---|---|
| DGSP u512, Agr. | [0.7, 1.6] ∘ | [1.4, 2.3] ∘ | [1.7, 4.9] ∘ |
| DGSP u512, Rep. | [2.4, 11.8] ∘ | [5.6, 10.0] ∘ | [4.7, 15.3] ∘ |
| Baseline | Implementation | Setting selected on development data |
|---|---|---|
| AdaScale [ 4 ] | Re-implementation (original: Caffe, R-FCN); ResNet-18 scale classifier over the whole-frame resolutions | Predicted scale raised by one step, resolutions up to u1280 |
| SAHI [ 1 ] | Official library | Slices of pixels, overlap |
| FOVEA [ 37 ] | Official grid generator, saliency from the draft detections | Canvas of u1280 |
| UFP [ 19 ] | Official foreground packing module, YOLOv8s on the mosaics | Merge scale , mosaics of pixels |
| Budget-Adaptive Routing (BAR) [ 12 ] | Official estimators, offloader, and arbiter | u1280 as the strong mode, half of the frames routed |
| Chanakya [ 13 ] | Objective of [ 13 ] implemented directly, since the official controller adapts detector settings that do not map to our actions | Cost-sensitive learner; pixel budget matched to DGSP |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
| Pipeline | Agr. | Rep. | Trans. | px | Agr. | Rep. | Trans. | px | Agr. | Rep. | Trans. | px |
| u320 (draft) | 92.7 † | 0 † | 65.3 † | 4.3 | 85.3 † | 0 † | 57.6 † | 2.6 | 78.8 † | 0 † | 45.5 † | 3.1 |
| u320 + | 96.5 † | 59.9 † | 71.8 † | 4.3 | 94.7 † | 70.2 † | 66.0 † | 2.6 | 91.4 † | 63.8 † | 54.0 † | 3.1 |
| u448 | 97.4 † | 76.7 † | 76.4 † | 8.0 | 96.0 † | 81.8 † | 72.2 † | 5.0 | 93.8 † | 79.5 † | 57.5 † | 5.6 |
| u512 | 97.7 † | 79.6 † | 76.7 † | 10.2 | 96.4 † | 84.0 † | 74.9 † | 6.7 | 94.5 † | 82.6 † | 54.0 † | 7.1 |
| u640 | 98.2 † | 84.7 | 81.6 | 17.1 | 97.3 † | 88.3 † | 77.0 † | 10.3 | 95.8 † | 87.4 † | 58.4 † | 11.6 |
| nuScenes | Argoverse 2 | Argoverse-HD | |||||||
| Method | Missed | False | False / 1k | Missed | False | False / 1k | Missed | False | False / 1k |
| u320 (draft) | 51 | 210 | 22.9 ∗ | 34 | 376 | 31.8 ∗ | 21 † | 408 | 27.2 ∗ |
| u320 + | 63 ∗ | 75 | 8.2 ∗ | 44 | 86 | 7.3 ∗ | 35 | 105 | 7.0 ∗ |
| u448 | 57 ∗ | 73 | 8.0 ∗ | 34 | 90 | 7.6 ∗ | 34 | 73 | 4.9 ∗ |
| u512 | 50 | 68 | 7.4 ∗ | 37 | 86 | 7.3 ∗ | 37 | 66 | 4.4 ∗ |
| u640 | 50 | 64 | 7.0 ∗ | 35 † | 61 | 5.2 ∗ | 32 | 64 | 4.3 ∗ |
| Ego speed (tercile) | Range of the decisive object (m) | ||||||||
| Pipeline | Low | Mid | High | 20 | 20–40 | 40–60 | 60 | none | |
| nuSc. | u512 | 99.3 ∗ | 97.6 † | 95.8 † | 92.5 | 88.1 † | 82.7 † | 58.8 | 98.6 † |
| u640 | 99.5 ∗ | 98.0 † | 97.0 † | 93.2 | 92.2 | 89.0 † | 70.6 | 98.9 | |
| DGSP | 98.8 | 98.8 | 98.7 | 94.8 | 95.1 | 95.3 | 82.4 | 99.2 | |
| frames | 3,330 | 2,992 | 2,859 | 439 | 345 | 127 | 17 | 8,253 | |
| AV2 | u512 | 98.3 | 95.0 † | 95.9 † | 92.8 † | 92.5 † | 88.8 † | 80.5 † | 98.1 † |
| Pipeline | , | |||
|---|---|---|---|---|
| nuSc. | u512 | 95.6 † | 94.5 † | 92.5 † |
| u640 | 96.7 † | 95.7 † | 94.2 † | |
| DGSP | 98.2 | 97.9 | 97.3 | |
| AV2 | u512 | 96.2 † | 96.1 † | 91.8 † |
| u640 | 97.1 † | 96.9 † | 93.9 † | |
| DGSP | 98.4 | 98.6 | 97.5 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Condition | Agr. | u512 | Rep. | u512 | Agr. | u512 | Rep. | u512 | Agr. | u512 | Rep. | u512 |
| All frames | 98.7 | 97.7 † | 86.9 | 79.6 † | 98.2 | 96.4 † | 91.7 | 84.0 † | 97.5 | 94.5 † | 92.0 | 82.6 † |
| Day | 98.9 | 98.0 † | 87.3 | 81.7 † | – | – | – | – | – | – | – | – |
| Night | 97.8 | 95.4 † | 84.8 | 68.6 † | – | – | – | – | – | – | – | – |
| Dry | 99.0 | 98.0 † | 88.8 | 81.4 † | – | – | – | – | – | – | – | – |
| Rain | 97.9 | 96.4 † | 81.5 | 74.6 | – | – | – | – | – | – | – | – |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pipeline | Dep. | Less | Row | Dep. | Less | Row | Dep. | Less | Row | |||
| u512 | 2.3 | 54 | 40 | 56 | 3.6 | 55 | 38 | 59 | 5.5 | 53 | 32 | 58 |
| u640 | 1.8 | 57 | 38 | 59 | 2.7 | 55 | 43 | 51 | 4.2 | 47 | 35 | 55 |
| u1280 | 1.7 | 54 | 56 | 40 | 2.4 | 47 | 52 | 43 | 3.8 | 61 | 53 | 37 |
| native | 2.3 | 70 | 57 | 38 | 4.0 | 57 | 49 | 45 | 6.3 | 65 | 56 | 35 |
| DGSP | 1.3 | 80 | 61 | 33 | 1.8 | 81 | 62 | 32 | 2.5 | 77 | 72 | 21 |
| Pipeline | px | Agr. | Rep. | P | R | F1 | False / 1k |
|---|---|---|---|---|---|---|---|
| u320 (draft) | 3.1 | 76.7 † | 0 † | 7.5 † | 81.8 ∗ | 13.8 † | 39.5 ∗ |
| u512 | 7.1 | 95.8 † | 87.1 † | 32.0 † | 62.3 | 42.3 † | 5.2 ∗ |
| u960 ( ) | 25.3 | 100 | 100 | 37.6 † | 63.0 | 47.1 † | 4.1 ∗ |
| DGSP-fast | 6.3 | 98.3 † | 94.5 † | 36.0 † | 62.3 | 45.6 † | 4.4 ∗ |
| DGSP | 6.7 | 98.5 | 95.3 | 43.0 | 61.7 | 50.7 | 3.2 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||
| Intervention (cumulative) | Agr. | Diff. | Agr. | Diff. | Agr. | Diff. |
| With the range refiner (in DGSP and in ) | ||||||
| DGSP | 98.75 | 115 | 98.25 | 207 | 97.52 | 372 |
| + crop detections of | 99.18 † | 75 | 98.63 † | 162 | 98.73 † | 191 |
| + states of for these objects | 99.38 † | 57 | 99.16 † | 99 | 98.99 † | 151 |
| + probe closing at m/s | 99.54 † | 42 | 99.37 † | 75 | 99.05 † | 143 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||
| Range refiner | with | without | with | without | with | without |
| Frames with an action | 115 | 145 | 207 | 240 | 372 | 381 |
| refiner only | 13 | – | 45 | – | 97 | – |
| coverage | 48 | 50 | 73 | 88 | 150 | 209 |
| locality | 36 | 60 | 70 | 106 | 83 | 103 |
| no object of raises the action | 18 | 35 | 19 | 46 | 42 | 69 |
| nuScenes | Argoverse 2 | Argoverse-HD | |||||||
| Pipeline | Agr. | Rep. | Trans. | Agr. | Rep. | Trans. | Agr. | Rep. | Trans. |
| u960, frame shifted by pixel | 99.2 ∗ | 93.6 ∗ | 90.0 ∗ | 98.8 ∗ | 95.2 ∗ | 90.4 ∗ | 98.7 ∗ | 96.4 ∗ | 77.1 ∗ |
| u960, half-pixel phase | 99.5 ∗ | 96.0 ∗ | 94.2 ∗ | 99.1 ∗ | 96.3 ∗ | 91.4 ∗ | 99.0 ∗ | 96.8 ∗ | 80.6 ∗ |
| u512 | 97.7 † | 79.6 † | 76.7 † | 96.4 † | 84.0 † | 74.9 † | 94.5 † | 82.6 † | 54.0 † |
| u768 | 98.6 | 88.6 | 84.7 | 97.8 † | 90.4 | 80.9 | 96.1 † | 88.5 † | 56.8 † |
| u1280 | 98.3 † | 87.8 | 84.2 | 97.6 † | 91.8 | 82.1 | 96.2 † | 90.0 | 60.5 † |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
| Pipeline | px | Agr. | Rep. | F1 | px | Agr. | Rep. | F1 | px | Agr. | Rep. | F1 |
| u320 (draft) | 4.3 | 93.1 † | 0 † | 47.3 † | 2.6 | 85.8 † | 0 † | 41.4 † | 3.1 | 77.8 † | 0 † | 20.2 † |
| u320 + | 4.3 | 96.7 † | 59.9 † | 60.3 † | 2.6 | 94.4 † | 67.5 † | 68.2 † | 3.1 | 92.0 † | 66.6 † | 36.2 † |
| u512 | 10.2 | 97.7 † | 77.3 † | 59.7 † | 6.7 | 96.7 † | 85.2 † | 72.3 † | 7.1 | 95.6 † | 86.8 | 35.2 † |
| u640 | 17.1 | 98.2 † | 83.8 | 62.0 † | 10.3 | 97.3 † | 89.1 † | 74.8 | 11.6 | 97.3 | 92.1 | 38.6 † |
| u960 ( ) | 38.4 | 100 | 100 | 65.0 † | 23.2 | 100 | 100 | 77.3 | 25.3 | 100 | 100 | 41.9 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||||||||
| Pipeline | px | Agr. | Rep. | P | R | F1 | px | Agr. | Rep. | P | R | F1 | px | Agr. | Rep. | P | R | F1 |
| u320 (draft) | 4.3 | 93.1 † | 0 † | 35.6 † | 75.5 | 48.4 † | 2.6 | 78.3 † | 0 † | 25.7 † | 91.1 ∗ | 40.1 † | 3.1 | 78.9 † | 0 † | 12.0 † | 77.5 ∗ | 20.8 † |
| u320 + | 4.3 | 96.2 † | 56.7 † | 50.0 † | 66.7 † | 57.1 † | 2.6 | 91.9 † | 69.5 † | 45.2 † | 86.3 | 59.3 † | 3.1 | 93.4 † | 73.5 † | 26.4 † | 54.9 | 35.6 † |
| u512 | 10.2 | 96.8 † | 71.2 † | 51.5 | 70.7 | 59.6 † | 6.7 | 92.9 † | 78.8 † | 46.3 † | 86.8 | 60.4 † | 7.1 | 94.6 † | 85.0 | 26.6 † | 57.7 | 36.4 |
| u640 | 17.1 | 97.4 | 77.6 | 48.0 † | 72.1 | 57.6 † | 10.3 | 93.5 † | 81.2 † | 44.7 † | 86.8 | 59.0 † | 11.6 | 95.9 | 87.3 | 28.6 † | 56.3 | 37.9 † |
| u768 | 23.9 | 97.7 | 80.2 | 49.3 † | 72.8 | 58.8 † | 14.7 | 94.0 † | 83.2 | 46.7 † | 86.3 | 60.6 † | 16.0 | 96.1 | 89.1 | 31.4 | 60.6 | 41.3 |
| nuScenes | Argoverse 2 | Argoverse-HD | |
| Frames | |||
| Frames with the draft only (%) | 35.5 | 28.7 | 24.3 |
| Detector-input pixels per frame (% of native) | |||
| Mean | 8.4 | 5.9 | 6.9 |
| Median | 7.4 | 4.7 | 5.3 |
| Mean pixels by source (% of native) | |||
| nuScenes | Argoverse 2 | Argoverse-HD | |||||||
| Pipeline | Det. | Lift | Total | Det. | Lift | Total | Det. | Lift | Total |
| u320 (draft) | 4.3 | 0.0 | 4.3 | 5.8 | 0.0 | 5.8 | 5.0 | 0.0 | 5.0 |
| u320 + | 4.3 | 5.3 | 9.6 | 5.8 | 13.2 | 19.0 | 5.0 | 15.6 | 20.6 |
| u512 | 10.4 | 7.4 | 17.8 | 15.0 | 18.1 | 33.1 | 11.5 | 21.1 | 32.6 |
| u640 | 17.3 | 8.8 | 26.1 | 23.0 | 19.8 | 42.9 | 18.7 | 24.9 | 43.6 |
| u768 | 24.2 | 9.4 | 33.6 | 32.8 | 21.1 | 53.9 | 25.9 | 26.6 | 52.5 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Variant | px | Agr. | P | R | px | Agr. | P | R | px | Agr. | P | R |
| DGSP (full) | 8.4 | 98.7 | 70.7 | 67.3 | 5.9 | 98.2 | 77.1 | 77.9 | 6.9 | 97.5 | 44.3 | 49.3 |
| Support only (w/o field) | 5.7 | 97.9 † | 74.3 | 53.1 † | 4.1 | 97.2 † | 79.5 ∗ | 71.6 † | 5.0 | 96.4 † | 46.2 | 42.3 † |
| Field only (w/o support) | 7.1 | 95.7 † | 36.7 † | 72.1 | 4.6 | 90.8 † | 30.4 † | 83.7 ∗ | 5.3 | 86.9 † | 11.0 † | 70.4 ∗ |
| Pivots instead of support | 8.3 | 98.6 † | 69.9 | 68.0 | 5.7 | 97.9 † | 71.8 † | 77.9 | 6.8 | 96.8 † | 23.6 † | 52.1 |
| Single round | 8.1 | 98.7 | 68.1 † | 66.7 | 5.6 | 98.2 | 75.6 | 78.4 | 6.5 | 97.4 † | 39.1 † | 50.7 |
| nuScenes | Argoverse 2 | Argoverse-HD | |||||||
| Pipeline | px | Natural | Deleted | px | Natural | Deleted | px | Natural | Deleted |
| u448 | 8.0 | 21.5 † | 98.5 ∗ | 5.0 | 32.3 † | 99.0 ∗ | 5.6 | 43.3 † | 96.0 |
| u512 | 10.2 | 23.3 † | 98.3 ∗ | 6.7 | 37.6 † | 99.3 ∗ | 7.1 | 52.0 † | 96.3 |
| u640 | 17.1 | 33.3 | 97.8 ∗ | 10.3 | 45.2 † | 99.6 ∗ | 11.6 | 64.2 † | 98.0 |
| u960 ( ) | 38.4 | 43.3 | 98.7 ∗ | 23.2 | 55.5 ∗ | 99.5 ∗ | 25.3 | 74.5 | 99.2 ∗ |
| u1280 | 65.4 | 48.1 ∗ | 99.0 ∗ | 40.0 | 63.3 ∗ | 99.6 ∗ | 44.4 | 76.3 | 99.6 ∗ |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||
| Pipeline | Natural | Deleted | Natural | Deleted | Natural | Deleted |
| u512 | 51.3 † | 98.3 ∗ | 62.9 † | 99.3 ∗ | 65.5 † | 96.3 |
| Uncertainty crops | 33.3 † | 46.9 † | 59.8 † | 62.4 † | 50.9 † | 72.5 † |
| CF + support | 25.6 † | 21.6 † | 40.7 † | 20.7 † | 21.4 † | 22.4 † |
| iterated | 25.6 † | 22.0 † | 41.9 † | 21.5 † | 22.0 † | 23.5 † |
| AnA + support | 76.1 † | 51.9 † | 76.5 † | 52.3 † | 82.2 † | 57.8 † |
| Natural, vehicles | Natural, VRU | Deleted, vehicles | Deleted, VRU | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pipeline | nuSc. | AV2 | A-HD | nuSc. | AV2 | A-HD | nuSc. | AV2 | A-HD | nuSc. | AV2 | A-HD |
| u512 | 28.4 † | 43.0 † | 60.4 † | 12.6 † | 20.8 † | 0.0 | 98.4 ∗ | 99.3 ∗ | 96.2 | 92.3 | 95.1 | 100.0 |
| u960 ( ) | 45.9 | 58.0 ∗ | 83.2 | 37.9 | 48.0 | 20.9 | 98.6 ∗ | 99.6 ∗ | 99.1 ∗ | 100.0 | 92.7 | 100.0 |
| AnA [ 5 ] | 7.1 † | 31.1 † | 21.1 † | 24.1 † | 8.7 † | 1.0 | 25.4 † | 26.3 † | 26.2 † | 30.8 † | 4.9 † | 23.8 † |
| CF [ 14 ] | 6.0 † | 26.3 † | 18.1 † | 21.8 † | 8.7 † | 1.0 | 22.2 † | 22.3 † | 21.5 † | 30.8 † | 7.3 † | 22.6 † |
| Support only (w/o field) | 19.1 † | 34.3 † | 53.2 † | 24.1 † | 7.5 † | 1.0 | 41.0 † | 58.8 † | 58.6 † | 23.1 † | 24.4 † | 23.8 † |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||
|---|---|---|---|---|---|---|
| Decisive object | ||||||
| In the draft | 84.8 | 71.6 | 90.2 | 84.4 | 70.4 | 66.7 |
| Box center in a verified or guard crop | 96.3 | 98.0 | 96.6 | 96.7 | 97.0 | 95.4 |
| Box center in the field footprint | 61.4 | 56.0 | 64.1 | 39.4 | 58.5 | 42.4 |
| In the final scene of DGSP | 92.3 | 88.7 | 96.5 | 94.6 | 96.3 | 92.5 |
| In the final scene, of those missing from the draft | 56.0 | 62.2 | 65.0 | 66.0 | 88.3 | 78.1 |
| nuScenes | Argoverse 2 | Argoverse-HD | |
| Frames with a requested field, by draft action (%) | |||
| keep | 59.8 | 59.7 | 65.6 |
| decelerate | 61.8 | 65.4 | 73.2 |
| hard-brake | 0.0 | 0.0 | 0.0 |
| Pixels of DGSP (% of native) as the ego speed is scaled | |||
| 8.4 | 5.9 | 6.9 | |
| nuScenes | Argoverse 2 | Argoverse-HD | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Regions | px | Agr. | Rep. | F1 | Rec. | px | Agr. | Rep. | F1 | Rec. | px | Agr. | Rep. | F1 | Rec. |
| u512 | 10.2 | 97.7 † | 79.6 † | 62.2 † | 23.3 † | 6.7 | 96.4 † | 84.0 † | 71.3 † | 37.6 † | 7.1 | 94.5 † | 82.6 † | 39.8 † | 52.0 † |
| Random crops, whole image | 9.2 | 94.6 † | 29.3 † | 44.9 † | 14.3 † | 6.3 | 88.8 † | 26.7 † | 43.7 † | 14.2 † | 8.0 | 83.9 † | 26.3 † | 17.4 † | 17.8 † |
| Random crops, lower half | 8.8 | 94.5 † | 27.5 † | 43.1 † | 13.7 † | 6.2 | 87.8 † | 19.8 † | 43.5 † | 8.1 † | 7.8 | 82.9 † | 21.3 † | 18.1 † | 10.8 † |
| Crops around relevant objects | 5.1 | 94.7 † | 29.6 † | 41.3 † | 43.0 | 3.5 | 90.5 † | 37.5 † | 39.9 † | 54.4 ∗ | 3.8 | 87.5 † | 43.2 † | 14.7 † | 75.1 |
| DGSP | 8.4 | 98.7 | 86.9 | 69.0 | 41.5 | 5.9 | 98.2 | 91.7 | 77.5 | 51.1 | 6.9 | 97.5 | 92.0 | 46.7 | 75.5 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
| Pipeline | px | Agr. | Rep. | F1 | px | Agr. | Rep. | F1 | px | Agr. | Rep. | F1 |
| u512 | 10.2 | 97.7 † | 79.6 † | 62.2 † | 6.7 | 96.4 † | 84.0 † | 71.3 † | 7.1 | 94.5 † | 82.6 † | 39.8 † |
| u960 ( ) | 38.4 | 100 | 100 | 65.0 † | 23.2 | 100 | 100 | 75.9 | 25.3 | 100 | 100 | 40.9 |
| Uncertainty crops | 9.6 | 97.7 † | 73.6 † | 60.7 † | 8.7 | 96.6 † | 81.2 † | 69.2 † | 11.5 | 94.1 † | 75.5 † | 34.0 † |
| DGSP, crops at native scale | 13.6 | 97.3 † | 77.3 † | 66.2 | 14.5 | 95.7 † | 84.3 † | 77.4 | 16.0 | 94.0 † | 83.3 † | 46.2 |
| DGSP | 8.4 | 98.7 | 86.9 | 69.0 | 5.9 | 98.2 | 91.7 | 77.5 | 6.9 | 97.5 | 92.0 | 46.7 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Field | px | Agr. | F1 | Diff. | px | Agr. | F1 | Diff. | px | Agr. | F1 | Diff. |
| Reported (vehicle, m/s) | 1 | 98.75 | 69.0 | 0 | 1 | 98.25 | 77.5 | 0 | 1 | 97.52 | 46.7 | 0 |
| Static probe ( m/s) | 1.00 | 98.75 | 69.0 | 0 | 1.00 | 98.25 | 77.5 | 0 | 1.00 | 97.52 | 46.7 | 0 |
| Probe closing at m/s | 1.20 | 98.92 ∗ | 68.3 | 24 | 1.25 | 98.51 ∗ | 76.4 | 59 | 1.15 | 97.61 | 44.0 | 40 |
| Lateral spacing | 1.00 | 98.75 | 69.0 | 0 | 1.00 | 98.25 | 77.5 | 0 | 1.00 | 97.52 | 46.7 | 0 |
| Lateral spacing | 0.85 | 98.71 | 68.3 | 29 | 0.89 | 98.20 | 77.1 | 42 | 0.87 | 97.36 | 45.6 | 72 |
| nuScenes | Argoverse 2 | Argoverse-HD | ||||||||||
| Regions | px | P | R | F1 | px | P | R | F1 | px | P | R | F1 |
| Curved-path braking controller | ||||||||||||
| u320 (draft) | 4.3 | 29.4 † | 63.5 | 40.2 † | 2.6 | 34.0 † | 78.9 ∗ | 47.6 † | 3.1 | 13.2 † | 64.1 ∗ | 21.9 † |
| u512 | 10.2 | 50.8 † | 61.0 | 55.4 | 6.7 | 62.3 † | 77.7 ∗ | 69.2 † | 7.1 | 42.1 † | 51.5 | 46.3 |
| u960 ( ) | 38.4 | 54.8 † | 64.8 ∗ | 59.4 | 23.2 | 65.9 † | 78.9 ∗ | 71.8 | 25.3 | 43.5 | 52.4 | 47.6 |
| native | 103.1 | 58.2 | 64.8 ∗ | 61.3 | 101.2 | 65.8 † | 79.4 ∗ | 71.9 | 101.3 | 43.8 | 55.3 ∗ | 48.9 |
| nuScenes | Argoverse 2 | Argoverse-HD | |||||||
| Regions | px | Agr. | Rep. | px | Agr. | Rep. | px | Agr. | Rep. |
| Curved-path braking controller | |||||||||
| u512 | 10.2 | 97.6 † | 81.6 | 6.7 | 96.2 † | 82.9 † | 7.1 | 94.7 † | 83.2 † |
| DGSP, braking regions | 8.4 | 97.1 † | 71.9 † | 5.9 | 94.6 † | 75.5 † | 6.9 | 94.7 † | 82.4 † |
| DGSP , own regions | 7.6 | 98.5 | 85.4 | 5.3 | 97.5 | 87.8 | 6.3 | 97.1 | 90.2 |
| Time-to-collision warning | |||||||||
| Pipeline | Miss. | Hard | Prev. | px |
|---|---|---|---|---|
| Privileged | 0.00 ∗ | 0.00 ∗ | 0.2 | – |
| u320 (draft) | 0.44 † | 4.25 † | 1.5 | 4.3 |
| u512 | 0.35 † | 3.90 † | 1.7 † | 10.2 |
| u960 ( ) | 0.33 | 3.97 † | 1.1 | 38.4 |
| native | 0.32 | 4.77 † | 1.3 | 103.1 |
| DGSP | 0.28 | 3.07 | 0.4 | 8.1 |
Active Visual Perception
Active visual perception studies how agents choose where, when, and how to look to gather task-relevant evidence efficiently.
See all 52 papers