Predicting the Needle in a Petabyte Scale Haystack: Open-Vocabulary Event Anticipation in Satellite Imagery
Organizations: Planet Labs · Cornell University
Abstract
Satellite constellations now image the Earth daily, capturing the lifecycle of events from deforestation to urban construction. Anticipating these events before completion enables timely intervention, yet existing systems cannot jointly identify where a change occurs, what it will become, and when it will finish across an open vocabulary. Building such a model requires diverse event data, but events are like needles in a haystack, making manual annotation at global scale infeasible. We address this problem by representing change as the vector difference between vision-language model embeddings at distinct time points. This arithmetic approach allows us to search for semantic transformations directly in the latent space, enabling (1) an automated data engine for global event discovery without manual annotation and (2) the largest global satellite event dataset to our knowledge, comprising 21,000+ locations across 64+ event types with ~2M images from PlanetScope and Sentinel-2 spanning six continents, to train (3) a novel global-scale event anticipation model. Our model detects change with a 93.5% max F1 score, outperforms baselines 3.6x on predicted event retrieval, and forecasts completion dates with a 3-day median error, even nine months before completion. Code: https://github.com/ShreelekhaR/predicting-the-needle
Figures & tables
| Seen | Unseen | Geo Seen | Geo Unseen | |||||||||||
| Method | Data | Embed | max F1 | mAP5 | mAP10 | max F1 | mAP5 | mAP10 | max F1 | mAP5 | mAP10 | max F1 | mAP5 | mAP10 |
| Oracle | S2 | RemoteCLIP | — | .500 | .481 | — | .500 | .463 | — | .306 | .286 | — | .429 | .420 |
| Oracle | S2 | RS-SigLIP | — | .489 | .452 | — | .700 | .641 | — | .333 | .313 | — | .714 | .673 |
| Oracle | PS | RemoteCLIP | — | .565 | .537 | — | .700 | .680 | — | .317 | .286 | — | .571 | .559 |
| Oracle | PS | RS-SigLIP | — | .715 | .677 | — | .800 | .802 | — | .656 | .595 | — | .714 | .717 |
| Static | S2 | RS-SigLIP | — | .154 | .125 | — | .333 | .275 | — | .096 | .079 | — | .071 | .042 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Change mAP@K | Intersection mAP@K | |||||
|---|---|---|---|---|---|---|
| Method | @5 | @10 | @20 | @5 | @10 | @20 |
| RemoteCLIP FT | 0.752 | 0.760 | 0.764 | 0.822 | 0.835 | 0.840 |
| RS-SigLIP Barzilai et al. [2025] | 0.688 | 0.709 | 0.710 | 0.691 | 0.716 | 0.720 |
| GIT-RS-CLIP Liu et al. [2025] | 0.400 | 0.418 | 0.431 | 0.422 | 0.459 | 0.478 |
| DOFA-CLIP-B Xiong et al. [2025] | 0.318 | 0.338 | 0.356 | 0.350 | 0.380 | 0.396 |
| RemoteCLIP Liu et al. [2024] | 0.262 | 0.287 | 0.294 | 0.327 | 0.365 | 0.380 |
| Method | max F1 | mAP@5 | mAP@10 | mAP@20 |
|---|---|---|---|---|
| Oracle (PS, RS-SigLIP) | – | .734 | .705 | .642 |
| Static (PS, RS-SigLIP) | – | .208 | .191 | .153 |
| TeoChat | .275 | .005 | .006 | .006 |
| SAT-JEPA-DIFF | – | .057 | .044 | .032 |
| Gemini 3 Flash | .435 | .155 | .126 | .100 |
| Ours (LSTM) | .766 | .361 | .315 | .270 |
Event Time Prediction
Event time prediction estimates when future events will occur from event histories, time series, and contextual information.
See all 52 papersForecasting Earth Surface Change in Embedding Space
Finding Change in Satellite Archives from Text: How to Combine Before-and-After Images Efficiently
find the image pairs where a new building appeared.'' This means searching an archive of before-and-after (bi-temporal) satellite image pairs and ranking each pair by how well it matches a natural-language description of the change. The component that performs this match, the fusion module that combines the before'' and ``after'' views, must be run at query time across many candidate pairs, so its speed largely sets the cost of every search. We present a controlled comparison of how to build that module. Using one fixed image encoder (a frozen CLIP model) and one training recipe for all variants, we evaluate eight designs drawn from three families: attention, state-space models (Mamba), and learned compression (our Temporal Bottleneck Fusion, TBF). Each design is tested on two benchmarks (LEVIR-CC and Dubai-CC) with ten random seeds, so the reported differences are statistically grounded. We outline three findings: first, a training-free two-stage search (a cheap difference model that shortlists candidates, followed by attention fusion that re-ranks them) matches or exceeds full-fusion recall on LEVIR-CC while cutting query cost -, with comparable R@1/R@5 on Dubai-CC; second, the linear-time scan of Mamba, attractive on paper, gives no speed benefit at the patch counts typical of vision transformers (): the scan is limited by memory bandwidth, whereas attention maps cleanly onto parallel hardware; and third, compressing the fused representation (TBF) reduces parameters by and latency by for a change-only BLEU-1 cost of , although more aggressive compression quietly discards change-relevant detail that aggregate metrics fail to reveal.