Let LiDAR Decide Where to Look: Geometry-Measured Token Selection for Efficient Sparse Latent World Models
Organizations: University of Ottawa
Abstract
Vision-centric autonomous driving models encode thousands of image tokens per frame, yet most depict sky, buildings, or empty road---content irrelevant to planning. Worse, these tokens dilute the planner's attention: we show that randomly discarding 75% of tokens already outperforms the dense model, because the planner can concentrate on the remaining driving-critical content. Building on this finding, we present Geometry-Aware Token Selection (GATS), which uses LiDAR and radar point projections to select a small fraction of image tokens before the vision backbone runs. Because the selector operates on measured 3D geometry rather than visual features, the backbone encodes only the selected patches, cutting end-to-end FLOPs by up to . Each kept token receives a 3D anchor from its LiDAR point and is fed to a self-supervised latent world model that predicts future features and 3D displacements without box annotations. At 25% budget, GATS achieves a 3-sec L2 displacement error of 0.520 m on nuScenes and an EPDMS of 88.9 on NAVSIM---surpassing the dense baseline (0.573 m / 85.9) at fewer total FLOPs. At an extreme 10% budget (24 GFLOPs, reduction), it still beats the dense baseline on both nuScenes (0.537 m vs. 0.573 m L2) and NAVSIM (87.9 vs. 85.9 EPDMS). On 739 intent-critical scenarios, GATS covers 95.8% of hazardous vehicles and achieves a 0.09% collision rate---73.5% lower than the dense model (0.34%).
Figures & tables
| Method | Tok. | L2 | Col% | GFLOPs | |
|---|---|---|---|---|---|
| Published E2E (box+map sup., different backbone): | |||||
| UniAD [ 12 ] | 100% | — | 1.02 ‡ | 0.29 ‡ | — |
| VAD-Base [ 14 ] | 100% | — | 0.54 ‡ | 0.07 ‡ | — |
| SparseDrive [ 27 ] | 100% | — | 0.58 ‡ | 0.05 ‡ | — |
| Our models (DINOv2-S, self-supervised, no box/map ann.): | |||||
| Dense fusion | 100% | 6913 | 0.573 | 0.09 | 248 |
| Method | EPDMS | GFLOPs | |
| Published (different backbone / supervision): | |||
| WoTE [ 18 ] | 100% | 87.7 | — |
| DiffusionDrive [ 21 ] | 100% | 85.2 | — |
| DiffusionDrive-v2 [ 30 ] | 100% | 85.5 | — |
| Our models (DINOv2-S, self-supervised): | |||
| Dense fusion | 100% | 85.9 | 722 |
| Selector ( ) | Cov. | Col. |
|---|---|---|
| GATS | 95.8% | 0.09% |
| DINOv2-Attn | 93.9% | 0.13% |
| Random | 81.5% | 0.25% |
| Grid | 80.5% | 0.43% |
| Learned [ 13 ] | 70.0% | 0.61% |
| Distance (depth) [ 23 ] | 22.1% | 0.50% |
| Input | Entropy | ||
|---|---|---|---|
| Dense fusion | 3,456 | 7.0 | 11.8 |
| Random (25%) | 864 | 6.5 | 9.8 |
| GATS (25%) | 864 | 5.7 | 9.8 |
| Method | EPE @2s | Zero@2s | Rel.% | EPE @2s |
|---|---|---|---|---|
| GATS (25%) | 5.76 | 12.94 | 55.5% | 0.54 |
| Random (25%) | 6.81 | 11.93 | 42.9% | 0.58 |
| DINOv2-Attn (25%) | 6.65 | 11.55 | 42.4% | 0.56 |
| Dense fusion | 6.62 | 11.92 | 44.5% | 0.59 |
| Variant | L2 | Col% | |
|---|---|---|---|
| Full (GATS 25%) | 0.520 | – | 0.04 |
| w/o 3D anchoring | 0.958 | +84.2% | 0.43 |
| w/o road stream | 0.609 | +17.1% | 0.13 |
| w/o learned refine. | 0.606 | +16.5% | 0.09 |
| w/o world model | 0.574 | +10.4% | 0.14 |
| w/o past flow | 0.569 | +9.4% | 0.11 |
| Method | Clean L2 | Corrupt L2 | Rel. degrad. | Mot. blur |
|---|---|---|---|---|
| Dense camera | 0.668 | 0.924 | +38.3% | +130% |
| Dense fusion | 0.573 | 0.723 | +26.2% | +88% |
| GATS (25%) | 0.520 | 0.597 | +14.8% | +36% |
| Random (25%) | 0.550 | 0.599 | +8.8% | +20% |
| Grid (25%) | 0.604 | 0.678 | +12.2% | +32% |
| DINOv2-Attn (25%) | 0.584 | 0.690 | +18.0% | +44% |
| Configuration | BB | WM | Total |
|---|---|---|---|
| Dense (6 cams) | 185 | 248 | 433 |
| Sparse 25% (sparse BB) | 40 | 25.9 | 66 |
| Sparse 10% (sparse BB) | 16 | 7.6 | 24 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | Selector | L2 | Coverage |
|---|---|---|---|
| DINOv2-S/14 | GATS | 0.537 | 86.3% |
| DINOv2-S/14 | DINOv2-Attn | 0.628 | 84.7% |
| DINOv2-S/14 | Grid | 0.563 | 64.4% |
| DINOv2-S/14 | Random | 0.560 | 60.2% |
| DINOv3-S | GATS | 0.550 | 85.1% |
| DINOv3-S | Random | 0.568 | 60.2% |
| Selector ( ) | Cov. | Col. |
|---|---|---|
| GATS | 86.3% | 0.09% |
| DINOv2-Attn | 84.7% | 0.18% |
| Random | 60.2% | 0.38% |
| Grid | 64.4% | 0.26% |
| Learned [ 13 ] | 16.5% | 0.52% |
| Distance [ 23 ] | 10.7% | 0.64% |
| Method | EPE @2s | Zero@2s | Rel.% | EPE @2s |
|---|---|---|---|---|
| GATS (10%) | 5.10 | 13.29 | 61.6% | 0.48 |
| Random (10%) | 7.22 | 11.93 | 39.5% | 0.60 |
| Grid (10%) | 7.16 | 11.88 | 39.8% | 0.61 |
| Dense fusion | 6.62 | 11.92 | 44.5% | 0.59 |
| DINOv2-Attn (10%) | 5.46 | 11.40 | 52.1% | 0.50 |
| Distance (10%) | 6.18 | 12.22 | 49.4% | 0.59 |
| Variant | L2 | Col% | |
|---|---|---|---|
| Full (GATS 10%) | 0.537 | – | 0.09 |
| w/o radar velocity | 0.602 | +12.1% | 0.15 |
| w/o motion residuals | 0.558 | +3.9% | 0.14 |
| w/o 3D anchoring | 0.601 | +11.9% | 0.10 |
| w/o past flow | 0.592 | +10.2% | 0.10 |
| w/o learned refine. | 0.565 | +5.1% | 0.10 |
| Variant | PDMS | EPDMS | |
|---|---|---|---|
| Full (GATS 10%) | 88.9 | 87.9 | – |
| w/o motion residuals | 86.0 | 85.0 | 3.3% |
| w/o 3D anchoring | 84.6 | 83.4 | 5.1% |
| w/o past flow | 84.9 | 83.9 | 4.6% |
| w/o learned refine. | 84.5 | 83.5 | 5.0% |
| w/o road stream | 81.7 | 80.7 | 8.2% |
| Method | Clean L2 | Corrupt L2 | Rel. degrad. | Mot. blur |
|---|---|---|---|---|
| Dense camera | 0.668 | 0.924 | +38.3% | +130% |
| Dense fusion | 0.573 | 0.723 | +26.2% | +88% |
| GATS (10%) | 0.537 | 0.610 | +13.6% | +31% |
| Random (10%) | 0.560 | 0.624 | +11.4% | +27% |
| Grid (10%) | 0.563 | 0.637 | +13.2% | +29% |
| DINOv2-Attn (10%) | 0.628 | 0.717 | +14.2% | +31% |
End-to-End Autonomous Driving
End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.
See all 249 papers