cs.CVOct 13, 2026
SaveThe Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement
Organizations: The University of Tokyo · RIKEN · Stanford University
Abstract
Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a benchmark. We show that this loop can be handed to an agent. Any verifiable task can be cast as a harness and a verifier, where the harness defines what the agent may use and the verifier defines what counts as success. Every EO task can be made verifiable, through labeled benchmarks, independent sensors, physical models or the repeated observation of the same place. Given only these two components, a general-purpose coding agent builds the EO model and improves it recursively on its own outputs. Settings long treated as separate research problems, from learning with a handful of labels to adapting across domains, become choices of harness and verifier. On six EO benchmarks spanning optical, SAR and multispectral time-series imagery, with outputs that range from land-cover maps and ship instances to canopy heights and answers about large images, agent-built models surpass the state of the art on every one, and on land-cover mapping, they do so from 11% of the training labels. This establishes a new paradigm for EO, in which people design harnesses and verifiers and agents build the models, so EO models improve as agents do.
Figures & tables
| Benchmark | Task | Input | Classes | Metric | Train | Verifier | Test | Few-label task |
|---|---|---|---|---|---|---|---|---|
| OpenEarthMap | land-cover mapping | RGB, 0.25–0.5 m | 8 | mIoU | 2,700 | 300 | 500 | 6 tiles, one per continent (0.2%) |
| BRIGHT | building damage | optical + SAR pairs | 4 | mIoU | 2,918 | 450 | 878 | 14 pairs, one per event (0.5%) |
| HRSID | ship instances | SAR, 0.5–3 m | 1 | mask AP | 3,090 | 552 | 1,962 | 36 images from 36 scenes (1.2%) |
| PASTIS | crop-type mapping | Sentinel-2 time series | 19 | mIoU | 1,455 | 482 | 496 | 20 series, five per tile (1.4%) |
| Open-Canopy | canopy height | SPOT 6–7 RGB + NIR, 1.5 m | – | MAE (m) | 66,339 | 7,369 | 13,675 | 1,000 tiles, spread over France (1.5%) |
| LRS-GRO | large-image VQA, training-free | RGB, up to 27,328 px, + question | – | accuracy | 2,656 | 855 | 9,734 | 114 answers, six per question type (4.3%) |
Table 1: The six benchmarks. Training data form the harness, the verifier holds a validation split, and the test split stays hidden until the final evaluation. The few-label task keeps a few labeled samples per natural stratum and moves the other training samples to the unlabeled pool. On Open-Canopy, it keeps a geographically spread subset instead. Parentheses give the share of the training labels. BRIGHT and PASTIS count the background as a class. LRS-GRO counts questions and forbids any training, so its labels serve only as examples and for local validation.
| (a) BRIGHT building damage, pre-event optical + post-event SAR | |
|---|---|
| Method | mIoU |
| SiamAttnUNet ( Adriano et al., 2021 ) | 64.26 |
| DeepLabV3+ ( Chen et al., 2018 ) | 64.80 |
| UNet ( Ronneberger et al., 2015 ) | 64.94 |
| SiamCRNN ( Chen et al., 2020 ) | 65.73 |
| ChangeOS ( Zheng et al., 2021 ) | 65.98 |
Table 2: Agent-built models against published methods on the six benchmarks, each under its standard protocol. Rows are sorted from the weakest to the strongest result and the previous state of the art (SOTA) is underlined. Full-label RSI4EO rows give the gain over the SOTA, and few-label RSI4EO rows give the share of the training set in parentheses and, except for the MAE of Open-Canopy, the score as a percentage of the full-label score. Figure 2 and Table 4 give the few-label results of all six benchmarks. All agent rows come from GPT-6-Astra through Codex, within 12 hours on four GB200 GPUs. † Test-time augmentation. ‡ Ensemble of ChangeOS, DamageFormer and ChangeMamba with test-time augmentation and object-based voting. § Pretrained foundation model fine-tuned on PASTIS, as reported by Herzog et al. (2026) . ∥ Fold-I score from the released test logs of Cai (2023) , whose paper reports five-fold means of 66.8 and 67.9. PASTIS uses the official fold I split. The other baselines are those of Tarasiou et al. (2023) , who evaluate on crops of the same test fold. On HRSID, the four generic detectors are those of Wei et al. (2020) , and LFG-Net ⋆ is the stronger of the two configurations reported by its authors. OpenEarthMap reports 8-class mIoU on the official validation split, which serves as the test split because test labels are withheld. ∗ 8-class mean recomputed from the per-class IoUs of Wang et al. (2024) and Wang et al. (2025) . ¶ Encoder evaluated by Duc and Fukui (2026) . Open-Canopy reports the mean absolute error (MAE) of canopy height in meters on vegetated pixels below 60 m, where lower is better, with the baselines and the original LiDAR labels of Fogel et al. (2025) . LRS-GRO reports question-level accuracy on the 9,734 test questions under the official judge, with the baselines as reported by Zhang et al. (2026b) . ∘ Training-free, built on Qwen3-VL-8B.
Figure 2: Strong models from few labels. (a) The share of the training labels that the few-label task of each benchmark keeps (grey) and its score as a share of the full-label score (blue). For Open-Canopy, where a lower MAE is better, the score share is the full-label MAE divided by the few-label MAE. (b) Hidden-test mIoU on OpenEarthMap as the harness holds 6, 300, 600 or all 2,700 labeled tiles, which are 0.2, 11, 22 and 100% of the training set, on a log scale. The 300 and 600 tiles are selected without labels by k-center on frozen features and come without an unlabeled pool. In (b), the dashed line marks the previous state of the art and the dotted line a UNet, both trained on all labels. Tables 4 and 5 list the scores.
| Agent | Interface | Verifier | Test mIoU |
|---|---|---|---|
| Starter pipeline | – | – | 34.0 |
| Qwen3.8-27B | Claude Code | 40.8 | 39.5 +5.5 |
| Qwen3.8-Max | Claude Code | 53.3 | 51.9 +17.9 |
| Kimi-K3 | Claude Code | 57.3 | 55.5 +21.5 |
| Claude-Opus-5.5 | Claude Code | 58.1 | 56.7 +22.7 |
| GPT-6-Astra | Codex | 61.9 | 60.0 +26.0 |
Table 3: Five agents on six-label OpenEarthMap, each given the same harness, verifier and 12-hour budget. The verifier column gives the best verifier score of each run, and the gain over the starter is measured on the hidden test set.
Figure 3: Lineage of the six-label OpenEarthMap run. (a) In each generation, the best pipeline the verifier has seen labels the unlabeled pool, a student continues from the previous checkpoint on the six labeled tiles and these pseudo-labels, and the verifier decides whether it becomes the next teacher. In the later generations, controller scripts written by the agent select the teacher from the verifier log and start the next generation. (b) Verifier mIoU of the best pipeline after each generation. Generation 0 is the best model that the agent trains on the six tiles alone, the scripts alone run the last generation, and the star marks the hidden-test mIoU of the final three-network ensemble.
Figure 4: Inside the runs. (a) Share of the final verifier gain reached over time in the 14 runs of GPT-6-Astra in Table 6 , where the gain is measured from the first to the best verifier score of a run. A colored dot marks the end of each run, the black line is the median over runs, and the black dots mark the median times to half and 90% of the gain. (b) Hidden-test score against best verifier score, in the metric of each benchmark, for the runs of Table 6 scored in points and for the three further agents of Table 3 . The dashed line marks equal scores, the grey band a difference of at most 2 points, and the grey line joins the five agents on six-label OpenEarthMap from the weakest to the strongest. The two Open-Canopy runs, scored in meters of MAE, are listed in Table 6 . In both panels, blue marks few-label tasks and orange full-label or label-budget tasks.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Few labels | Full labels | |||||
|---|---|---|---|---|---|---|
| Benchmark | Metric | Starter | RSI4EO | Starter | RSI4EO | Published |
| OpenEarthMap | mIoU | 34.0 | 60.0 +26.0 | – | 71.6 | 67.5 |
| BRIGHT | mIoU | 34.8 | 49.4 +14.6 | 61.8 | 71.6 +9.8 | 68.9 |
| HRSID | mask AP | – | 50.2 | – | 68.7 | 63.9 |
| PASTIS | mIoU | 14.0 | 50.1 +36.2 | 53.0 | 69.7 +16.7 | 67.1 |
| Open-Canopy | MAE (m) | – | 2.72 | 4.00 | 2.44 1.56 | 2.52 |
Table 4: Hidden-test scores on all six benchmarks. The starter is the plain pipeline that ships with the harness and is run as provided. RSI4EO is the model that GPT-6-Astra builds from the same harness within 12 hours, and its gain over the starter is shown in blue. The last column gives the best published result under the same protocol ( Table 2 ). Lower is better for the MAE of Open-Canopy. A dash marks a score that is not available. ∘ Training-free, no parameter is trained.
| Labeled tiles | Share | Unlabeled tiles | GPT-6-Astra | Claude-Opus-5.5 |
| 6 | 0.2% | 4,194 | 60.0 | 56.7 |
| 300 | 11% | 0 | 64.3 | 67.8 |
| 600 | 22% | 0 | 69.4 | 68.6 |
| 2,700 | 100% | 1,500 | 71.6 | 70.8 |
| Previous SOTA Wang et al. [2025] | 67.5 | |||
Table 5: Hidden-test mIoU of two agents on OpenEarthMap as the harness holds more labeled tiles. The 300 and 600 tiles are selected without labels by k-center on frozen features, and these two tasks provide no unlabeled pool. The previous state of the art uses all training labels.
| Task | Agent | Local validation | Verifier | Test | Gap |
|---|---|---|---|---|---|
| With a local validation set | |||||
| PASTIS, full | GPT-6-Astra | 160 series, then five folds | 69.9 | 69.7 | 0.2 |
| OpenEarthMap, full | GPT-6-Astra | 170 tiles | 72.6 | 71.6 | 1.0 |
| OpenEarthMap, full | Claude-Opus-5.5 | 250 tiles | 72.1 | 70.8 | 1.3 |
| OpenEarthMap, 600 | GPT-6-Astra | 60 tiles | 70.6 | 69.4 | 1.2 |
| OpenEarthMap, 600 | Claude-Opus-5.5 | 98 tiles | 70.6 | 68.6 | 2.0 |
Table 6: Verifier and hidden-test scores of 18 runs, grouped by whether the agent built a local validation set from its labels. The gap is the best verifier score minus the hidden-test score in points and is computed before rounding. For Open-Canopy, the scores are the MAE in meters and the gap is the test MAE minus the verifier MAE. On full-label BRIGHT, three quarters of the gap come from the destroyed class, which covers 1.38% of the verifier pixels and 0.93% of the test pixels. Its test IoU of 61.5 is below the 65.4 that the first, untuned submission of the run scored on the verifier, so the gap exists before any selection.
More in this topic
Remote Sensing
Remote sensing uses satellite, aerial, and ground-based observations to measure and map Earth and other planetary surfaces.
See all 528 papers