EarthRecord: From Pixels to Land Change with Multimodal Agents over Four Decades of Satellite Imagery
Organizations: AIEyes Technologies (Hong Kong) Co., Limited, Hong Kong SAR, China · College of Future Technology, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511400, China · School of Physics and Optoelectronic Engineering, Guangdong University of Technology, Guangzhou 510006, China · Anhua Ocean Intelligent Equipment Co., Ltd., Shenzhen 518000, China ·
Abstract
While multimodal agents now carry out complex data analysis with tools, their ability to turn decades of satellite imagery into reliable land-change analysis, crucial for Earth observation science, remains unmeasured. Existing benchmarks fall short: they pose questions on curated images or give agents preprocessed inputs, leaving out the raw multi-sensor archives where such analysis begins. We introduce EarthRecord, a benchmark built on the raw record of 56 sites: four decades of Landsat, MODIS, VIIRS, and Sentinel-2 data, with cloud masking and cross-sensor harmonization left to the agent. It poses 238 land-change questions across ten analysis tasks, scored against withheld references from global products, agreement among methods, and visual interpretation. Two controls test what answers rest on: the closed book removes the record, and instrument pairs alter one satellite's data while nothing changes on the ground. Evaluations of twelve multimodal agents reveal that leading agents complete most analyses end to end but falter when an answer requires mapping from imagery. Further analysis shows that the record moves answers from prior knowledge to the pixels but not yet to the real change on the ground: when one satellite's data are altered, half of the right answers follow the alteration. EarthRecord thus provides a principled platform for measuring and advancing agents from pixels to land change.
Figures & tables
| Setting and data | Controls | |||||
| Group | Benchmark | Agent | Multi-decadal, multi-sensor | Raw record | Prior | Sensor / ground |
| Vision-language | MMStar [ 5 ] | ✓ | ||||
| NaturalBench [ 36 ] | ||||||
| OSI-Bench [ 64 ] | ✓ | |||||
| Remote sensing vision-language | CHOICE [ 2 ] | ✓ | ||||
| DVL-Bench [ 68 ] | ||||||
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Identifier | Effort |
|---|---|---|
| GPT-6 Sol | gpt-6-sol | Default |
| GPT-6 Astra | gpt-6-astra | Default |
| GPT-6 Luna | gpt-6-luna | Default |
| GPT-6.1 Sol | gpt-6.1-sol | Default |
| GPT-6.1 Terra | gpt-6.1-terra | Default |
| Claude Opus 5.5 | claude-opus-5-5 | Default |
| Class | Theme | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Computable | Recognition | Forest | Greening | Agriculture | Water | Urban | Fire | Snow | Coast | Regions |
| (153) | (73) | (29) | (20) | (23) | (26) | (30) | (24) | (19) | (24) | (31) | |
| Claude Opus 5.5 | 87.6 | 80.8 | 89.7 | 75.0 | 91.3 | 76.9 | 93.3 | 66.7 | 84.2 | 83.3 | 100.0 |
| GPT-6 Sol | 86.9 | 78.1 | 86.2 | 75.0 | 78.3 | 80.8 | 93.3 | 79.2 | 68.4 | 87.5 | 96.8 |
| GPT-6.1 Sol | 89.5 | 64.4 | 72.4 | 75.0 | 78.3 | 88.5 | 86.7 | 62.5 | 73.7 | 87.5 | 100.0 |
| Claude Sonnet 5.5 | 85.0 | 72.6 | 86.2 | 75.0 | 78.3 | 76.9 | 73.3 | 70.8 | 78.9 | 83.3 | 100.0 |
| Computable (153) | Recognition (73) | Insufficient (12) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Closed book | Record | Closed book | Record | Record | ||||||
| Model | One run | pass 3 | One run | pass 3 | One run | pass 3 | One run | pass 3 | One run | pass 3 |
| Claude Opus 5.5 | 68.6 | 62.1 | 91.1 | 87.6 | 68.9 | 65.8 | 84.0 | 80.8 | 97.2 | 91.7 |
| GPT-6 Sol | 57.1 | 49.0 | 91.1 | 86.9 | 63.9 | 57.5 | 83.6 | 78.1 | 97.2 | 91.7 |
| GPT-6.1 Sol | 46.6 | 37.9 | 91.5 | 89.5 | 65.3 | 60.3 | 68.0 | 64.4 | 100.0 | 100.0 |
| Claude Sonnet 5.5 | 59.7 | 54.2 | 90.2 | 85.0 | 61.6 | 53.4 | 79.9 | 72.6 | 83.3 | 75.0 |
| Model | Method | Record product | Mapping | Visual |
|---|---|---|---|---|
| (98) | (13) | (78) | (37) | |
| Claude Opus 5.5 | 91.8 | 100.0 | 71.8 | 91.9 |
| GPT-6 Sol | 89.8 | 92.3 | 70.5 | 94.6 |
| GPT-6.1 Sol | 91.8 | 92.3 | 62.8 | 89.2 |
| Claude Sonnet 5.5 | 91.8 | 84.6 | 65.4 | 83.8 |
| GPT-6 Astra | 93.9 | 92.3 | 51.3 | 81.1 |
| Term | Odds ratio | 95% interval | |
|---|---|---|---|
| Record product | 1.05 | 0.45–2.44 | 0.911 |
| Mapping | 0.31 | 0.14–0.66 | 0.003 |
| Visual | 0.65 | 0.18–2.32 | 0.510 |
| Two or more Landsat sensors | 0.59 | 0.33–1.06 | 0.078 |
| Period length, per decade | 0.94 | 0.71–1.24 | 0.662 |
| Passes right | |||||
|---|---|---|---|---|---|
| Model | 3 | 2 | 1 | 0 | 1 or 2 |
| Claude Opus 5.5 | 193 | 10 | 3 | 20 | 13 |
| GPT-6 Sol | 190 | 11 | 9 | 16 | 20 |
| GPT-6.1 Sol | 184 | 6 | 5 | 31 | 11 |
| Claude Sonnet 5.5 | 183 | 12 | 16 | 15 | 28 |
| GPT-6 Astra | 174 | 9 | 9 | 34 | 18 |
| Recognition | Comp. | Insufficient evid. | ||||
|---|---|---|---|---|---|---|
| Model | Abst. (%) | Tool calls | No image | Abst. (%) | Abst. runs | Right (of 12) |
| Claude Opus 5.5 | 4.2 | 12 | 11 | 0.0 | 35 | 11 |
| GPT-6 Sol | 4.6 | 11.5 | 80 | 0.0 | 35 | 11 |
| GPT-6.1 Sol | 29.7 | 3 | 86 | 0.4 | 36 | 12 |
| Claude Sonnet 5.5 | 5.0 | 13 | 82 | 0.4 | 30 | 9 |
| GPT-6 Astra | 38.8 | 2 | 95 | 0.7 | 36 | 12 |
| Mean | Computed (98) | Mapping (78) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | AUROC | Right | Wrong | Runs | Acc. | Runs | Acc. | Conf. | Acc. | Conf. | Acc. |
| Claude Opus 5.5 | 0.79 | 0.72 | 0.57 | 1.9 | 100.0 | 37.4 | 79.0 | 0.71 | 94.2 | 0.63 | 78.8 |
| GPT-6 Sol | 0.76 | 0.86 | 0.73 | 41.3 | 96.4 | 11.2 | 72.0 | 0.88 | 93.9 | 0.77 | 80.4 |
| GPT-6.1 Sol | 0.79 | 0.89 | 0.80 | 54.6 | 97.6 | 4.3 | 80.8 | 0.90 | 94.5 | 0.84 | 88.6 |
| Claude Sonnet 5.5 | 0.84 | 0.71 | 0.52 | 4.8 | 100.0 | 44.3 | 77.6 | 0.71 | 95.9 | 0.62 | 77.9 |
| GPT-6 Astra | 0.78 | 0.87 | 0.78 | 45.4 | 98.9 | 5.6 | 81.8 | 0.88 | 94.9 | 0.82 | 87.4 |
| Step | The run | Lead | Other |
|---|---|---|---|
| Data | counts a coarse land-cover map, uses the wrong product, misses data in the record, or joins sensors or products without making them consistent | 5 | 18 |
| Time sampling | fills gaps such as the polar night by assumption, or rests on few or seasonally uneven scenes | 16 | 7 |
| Question | computes a different quantity, such as winter as December to February | 15 | 6 |
| Analysis | maps change from the imagery with a different result, judges the type or cause of a change wrongly, or reaches its conclusion through the choice of test | 27 | 15 |
| Close call | uses a sound method whose result lies at a decision boundary | 7 | 4 |
| Unfinished | gives no answer, or one it says it could not compute | 2 | 19 |
| Computable (153) | Recognition (73) | |||||||
| Model | Wrong to right | Right to wrong | Right in both | Wrong in both | Wrong to right | Right to wrong | Right in both | Wrong in both |
| Claude Opus 5.5 | 47 | 8 | 87 | 11 | 13 | 2 | 46 | 12 |
| GPT-6 Sol | 65 | 7 | 68 | 13 | 17 | 2 | 40 | 14 |
| GPT-6.1 Sol | 83 | 4 | 54 | 12 | 10 | 7 | 37 | 19 |
| Claude Sonnet 5.5 | 56 | 9 | 74 | 14 | 19 | 5 | 34 | 15 |
| GPT-6 Astra | 84 | 2 | 50 | 17 | 9 | 12 | 31 | 21 |
| Model | With | Without | Closed book | Only with | Only without | |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 59 | 55 | 48 | 6 | 2 | 0.29 |
| GPT-6 Sol | 57 | 55 | 42 | 4 | 2 | 0.69 |
| GPT-6.1 Sol | 47 | 46 | 44 | 2 | 1 | 1.00 |
| Computable | Recognition | |||||
|---|---|---|---|---|---|---|
| Site | Closed | Record | Closed | Record | ||
| Famous areas | 20 | 34.2 | 77.1 | 17 | 55.9 | 50.0 |
| Stable areas | 17 | 32.4 | 77.5 | 5 | 26.7 | 46.7 |
| Obscure areas | 85 | 36.0 | 62.9 | 51 | 45.3 | 50.5 |
| Regions | 31 | 29.8 | 76.6 | – | – | – |
| Pair | Site | Right | Moved | Held: prior | Held: beyond | Shift in moved answers |
| Terra NDVI scaled down | ||||||
| a54-3 | cropland | 12 | 11 | 1 | 0 | no change decrease 19/33 |
| a56-1 | greening | 11 | 11 | 0 | 0 | increase decrease 16/33 |
| a09-1 | greening | 11 | 10 | 1 | 0 | increase no change 24/30 |
| a33-3 | fire | 9 | 9 | 0 | 0 | 2019 2018 25/27 |
| a48-3 | forest | 11 | 8 | 3 | 0 | no change decrease 19/24 |
| GPT-6 Astra, run 1 (moved) | DeepSeek Flash, run 1 (held beyond both) | |
|---|---|---|
| Data used | MODIS NDVI, Terra only | MODIS NDVI (Terra and Aqua), VIIRS, Landsat, Sentinel-2 |
| Work | 8 tool calls, 1.5 minutes, no image viewed | 66 tool calls, 20.1 minutes, 3 images viewed |
| Original | increase, +0.014 per decade (right) | increase, +0.014 per decade (right) |
| Altered | decrease, 0.091 per decade (wrong) | increase, +0.013 per decade (right) |
| From its report | “Used Terra MOD13Q1 throughout 2001–2025” … “annual mean NDVI is approximately 0.112 in 2001, 0.131 in 2017, -0.067 in 2018 and -0.056 in 2025” … “The cause of the break has not been independently verified.” | “MODIS Terra (MOD13Q1) is unusable after 2017 in this dataset: the AOI-mean NDVI steps down by 0.20 between 2017 and 2018” … “while Aqua shows no change” … “sub-period fits (Aqua 2003-2019/2020, Terra 2001-2017, VIIRS 2013-2019) all give +0.011 to +0.014/decade” |
Remote Sensing Benchmarks
Remote sensing benchmarks measure how well models interpret Earth-observation data across sensors, regions, time, tasks, and operating conditions.
See all 213 papers