A Multi-Agent Feedback System for Detecting and Describing News Events in Satellite Imagery
Organizations: Massachusetts Institute of Technology · Planet Labs
Abstract
Changes in satellite imagery often occur over multiple time steps. Despite the emergence of bi-temporal change captioning datasets, there is a lack of multi-temporal event captioning datasets (at least two images per sequence) in remote sensing. This gap exists because (1) searching for visible events in satellite imagery and (2) labeling multi-temporal sequences require significant time and labor. To address these challenges, we present SkyScraper, an iterative multi-agent workflow that geocodes news articles and synthesizes captions for corresponding satellite image sequences. Our experiments show that SkyScraper successfully finds 5x more events than traditional geocoding methods, demonstrating that agentic feedback is an effective strategy for surfacing new multi-temporal events in satellite imagery. We apply our framework to a large database of global news articles, curating a new multi-temporal captioning dataset with 5,000 sequences. By automatically identifying imagery related to news events, our work also supports journalism and reporting efforts.
Figures & tables
| Method | Event Detections | Yield | Increase over Centroid Baseline |
|---|---|---|---|
| Weighted Centroid | 17 | 1.7% | – |
| GIPSY | 47 | 4.7% | 2.8 |
| SkyScraper (ours) | 84 | 8.4% | 4.9 |
| Imagery Source | Total Sequences | Confirmed Events | Average # Images Per Sequence |
|---|---|---|---|
| PlanetScope | 5,000 | 3,288 | 21 |
| Sentinel-2 | 4,463 | 2,931 | 8 |