Unreasonable Confidence from Simulator Misspecification: Directional Robustness on the SPEED+ Sim-to-Real Benchmark
Organizations: Independent Researcher
Abstract
Vision-based navigation systems for orbit must be trained almost entirely on synthetic imagery, because real labelled images of a given spacecraft in flight rarely exist. Benchmarks like SPEED+ are built for that situation: the training set comes from a simulator, and the test sets are real images of a satellite mockup under two kinds of lighting. That simulator approximates how light reflects off mylar and aluminium, and like every scientific simulator it is misspecified. Its error is radiometric, since it misses the specular highlights, hard shadows and sensor saturation common in real images. We probe the resulting synthetic-to-real gap, the central risk for such systems, with a deliberately simple monocular pose estimator. In a 2x2 study of backbone adaptation and photometric augmentation, trained on synthetic images and tested on both real hardware-in-the-loop domains, we find three things. (1) The model does not know it is failing: regions conformally calibrated to 90% on synthetic data cover only 9-15% of rotation errors on sunlamp and 21-38% on lightbox, so a wrong pose carries the confidence of a correct one. (2) Transfer is asymmetric: scaling data and features improved synthetic rotation error from 1.65 to 1.03 rad but moved the harshly lit sunlamp domain by only 0.05 rad, under twice its seed s.d., whereas adapting the backbone's last stage carried 63-83% of its extra synthetic gain to both real domains. (3) Once the backbone adapts, augmentation helps only in the direction it models: our photometric stack improved the diffusely lit lightbox domain from 1.17 to 0.99 rad on every seed but moved sunlamp no more than seed-to-seed variation, while with the backbone frozen it helped sunlamp more. Every post-review experiment was pre-registered, and the code, per-image predictions and pre-registration record are at https://github.com/lokindoo/spacecraft-pose-estimation.
Figures & tables
| # | Team | Ratio | PnP | ||
|---|---|---|---|---|---|
| 1 | UniAdelaide | 0.0094 | 0.3752 | yes | |
| 2 | EPFL_cvlab | 0.0215 | 0.1140 | yes | |
| 3 | pedro_fairspace | 0.0571 | 0.1555 | no | |
| 4 | Team Platypus | 0.0703 | 1.7201 | no | |
| 5 | motokimura1 | 0.0758 | 0.6011 | no | |
| 6 | Magpies | 0.1393 | 1.2659 | no |
| Component | Parameters | Arm A | Arm B |
|---|---|---|---|
| Brightness / contrast jitter | factor / | ✓ | ✓ |
| Gamma | log-uniform in | ✓ | ✓ |
| Sun-glare blob ( ) | clamped Gaussian, px, peak | ✓ | – |
| Gaussian blur ( ) | kernel 7, | – | – |
| Additive Gaussian noise ( ) | – | – | |
| Random erasing ( ) | area | ✓ | ✓ |
| Step | Synthetic (val) | Lightbox | Sunlamp |
|---|---|---|---|
| 0. 4k images, logits head, MSE on raw quaternions | – | – | – |
| 1. + labels canonicalised to one hemisphere | 1.666 | 2.002 | 2.073 |
| 2. + sign-invariant rotation loss | – | – | – |
| 3. + pooled 2048-d features | – | – | – |
| 4. + 28k images (published frozen arm, seed 42) | 1.053 | 1.752 | 2.048 |
| Step 0 recorded only its loss, and steps 2 and 3 have no saved score on this split. | |||
| Synthetic (val) | Lightbox | Sunlamp | |||||
| Backbone | Aug. | ||||||
| frozen † | – | ||||||
| frozen ‡ | ✓ | ||||||
| layer 4 | – | ||||||
| layer 4 | ✓ | ||||||
| † Seed 42 trained end to end, and seeds 43, 44 retrain the head on cached features (App. D ). Table 5 retrains all three, and the text quotes it. ‡ Seeds 43, 44 on an Apple M5. Per-seed rows: App. A . | |||||||
| Configuration | Synthetic (val) | Lightbox | Sunlamp |
|---|---|---|---|
| 4k images, 1000-d logits | |||
| 4k images, 2048-d pooled | |||
| 28k images, 1000-d logits | |||
| 28k images, 2048-d pooled (published config.) | |||
| 28k, pooled, tuned (lr , 60 ep.) | |||
| 28k, pooled, synthetic-BN backbone |
| vs. no augmentation | ||||||
|---|---|---|---|---|---|---|
| Seed | Synth. | Lightbox | Sunlamp | Synth. | Lightbox | Sunlamp |
| 42 | 1.180 | 1.718 | 1.864 | |||
| 43 | 1.205 | 1.734 | 1.883 | |||
| 44 | 1.169 | 1.673 | 1.881 | |||
| vs. full stack | Sunlamp | |||||||
| Arm | Seed | Synth. | Lightbox | Sunlamp | Synth. | Lightbox | Sunlamp | |
| full stack | 42 | 0.352 | 0.986 | 1.592 | – | – | – | 0.405 |
| 43 | 0.302 | 0.997 | 1.354 | – | – | – | 0.399 | |
| 44 | 0.311 | 0.977 | 1.459 | – | – | – | 0.261 | |
| A: no blur/noise | 42 | 0.320 | 1.070 | 1.408 | 0.311 | |||
| 43 | 0.328 | 1.001 | 1.360 | 0.325 | ||||
| Edge | Clipped | TV intensity | TV gradient | |||
|---|---|---|---|---|---|---|
| Pool | density (%) | (%) | lightbox | sunlamp | lightbox | sunlamp |
| lightbox | 6.6 | 0.08 | 0.00 | 0.28 | 0.00 | 0.22 |
| sunlamp | 2.0 | 2.78 | 0.28 | 0.00 | 0.22 | 0.00 |
| synthetic, no aug. | 10.9 | 0.07 | 0.68 | 0.83 | 0.07 | 0.26 |
| synthetic, full stack | 10.7 | 1.96 | 0.63 | 0.80 | 0.29 | 0.51 |
| synthetic, arm A | 9.6 | 2.62 | 0.65 | 0.81 | 0.10 | 0.33 |
| Aug. | Seed | Domain | Rotation coverage (%) | ||
|---|---|---|---|---|---|
| – | 42 | lightbox | 1.171 1.074 | 0.202 0.149 | 21.1 23.9 |
| sunlamp | 1.519 1.393 | 0.298 0.196 | 8.5 11.2 | ||
| 43 | lightbox | 1.199 1.110 | 0.153 0.130 | 26.0 28.0 | |
| sunlamp | 1.644 1.422 | 0.351 0.199 | 8.8 14.5 | ||
| 44 | lightbox | 1.160 1.057 | 0.156 0.132 | 25.6 29.3 | |
| sunlamp | 1.540 1.393 | 0.283 0.181 | 11.7 11.6 |
| With AdaBN | ||||||
|---|---|---|---|---|---|---|
| Arm | (rad) | In-domain | Lightbox | Sunlamp | Lightbox | Sunlamp |
| frozen, no aug. | 2.16–2.30 | 88.7–90.0 | 63.6–71.0 | 53.4–59.1 | – | – |
| frozen, full stack | 2.37–2.42 | 88.7–91.1 | 75.1–76.3 | 69.8–71.0 | – | – |
| layer 4, no aug. | 0.46–0.54 | 88.8–89.7 | 21.5–26.7 | 9.0–12.9 | 23.9–29.3 | 11.2–14.5 |
| layer 4, full stack | 0.48–0.58 | 91.1–91.4 | 32.6–38.0 | 12.6–15.4 | 35.5–40.4 | 18.6–19.6 |
| layer 4, arm A | 0.54–0.56 | 89.2–90.5 | 31.8–33.9 | 16.0–18.7 | – | – |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | Aug. | Seed | Trained on | Domain | ||
|---|---|---|---|---|---|---|
| frozen | – | 42 | Colab | synthetic (val) | 0.066 | 1.053 |
| frozen | – | 42 | Colab | lightbox | 0.151 | 1.752 |
| frozen | – | 42 | Colab | sunlamp | 0.253 | 2.048 |
| frozen | – | 43 | head only | synthetic (val) | 0.064 | 1.045 |
| frozen | – | 43 | head only | lightbox | 0.150 | 1.746 |
| frozen | – | 43 | head only | sunlamp | 0.263 | 1.952 |
| Translation (sphere) | Rotation (cone) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Backbone | Aug. | Seed | Domain | Cov. | Infl. | Cov. | Infl. | ||
| frozen | – | 42 | in-domain test | 87.7 (85.5–89.7) | 0.712 | 1.11 | 90.0 (88.0–91.8) | 2.299 | 1.01 |
| frozen | – | 42 | synthetic (val) | 88.0 (86.5–89.4) | 0.712 | 1.08 | 90.5 (89.1–91.7) | 2.299 | 0.99 |
| frozen | – | 42 | lightbox | 52.1 (50.9–53.3) | 0.712 | 2.48 | 71.0 (69.9–72.1) | 2.299 | 1.24 |
| frozen | – | 42 | sunlamp | 29.2 (27.5–30.9) | 0.712 | 4.76 | 58.2 (56.3–60.0) | 2.299 | 1.28 |
| frozen | – | 43 | in-domain test | 90.7 (88.7–92.4) | 0.751 | 0.96 | 88.9 (86.8–90.8) | 2.215 | 1.04 |
| Backbone | Aug. | Seed | Domain | Coverage | (relative) | Infl. |
|---|---|---|---|---|---|---|
| frozen | – | 42 | in-domain test | 89.4 (87.3–91.2) | 0.1189 | 1.01 |
| frozen | – | 42 | synthetic (val) | 89.3 (87.9–90.7) | 0.1189 | 1.03 |
| frozen | – | 42 | lightbox | 49.6 (48.4–50.8) | 0.1189 | 2.42 |
| frozen | – | 42 | sunlamp | 21.7 (20.2–23.3) | 0.1189 | 3.85 |
| frozen | – | 43 | in-domain test | 90.3 (88.3–92.1) | 0.1217 | 0.99 |
| frozen | – | 43 | synthetic (val) | 91.2 (89.9–92.4) | 0.1217 | 0.94 |
| Run | Seed | In-domain test | Synthetic (val) | Lightbox | Sunlamp | |
|---|---|---|---|---|---|---|
| A: no blur/noise | 42 | 90.5 | 91.1 | 33.2 | 16.0 | 0.558 |
| 43 | 89.2 | 88.8 | 33.9 | 16.8 | 0.538 | |
| 44 | 90.4 | 90.5 | 31.8 | 18.7 | 0.540 | |
| B: jitter, gamma, erasing | 42 | 89.7 | 89.0 | 30.7 | 14.5 | 0.478 |
| 43 | 91.4 | 90.0 | 36.1 | 14.4 | 0.539 | |
| 44 | 90.5 | 90.2 | 28.7 | 12.6 | 0.481 |
| Domain | Skill | |||
|---|---|---|---|---|
| synthetic (val) | 2,000 | 0.415 | 2.176 | 0.031 |
| lightbox | 6,740 | 0.367 | 2.203 | 0.005 |
| sunlamp | 2,791 | 0.372 | 2.199 | 0.009 |
| Run | Domain | Median | Near- | Cone, body | Cone, camera | |
|---|---|---|---|---|---|---|
| layer 4, no aug., 42 | lightbox | 0.92 | 0.051 | 344 | 0.105 | 0.122 |
| sunlamp | 1.31 | 0.086 | 240 | 0.083 | 0.071 | |
| layer 4, no aug., 43 | lightbox | 0.92 | 0.050 | 339 | 0.198 | 0.112 |
| sunlamp | 1.54 | 0.095 | 265 | 0.132 | 0.068 | |
| layer 4, no aug., 44 | lightbox | 0.90 | 0.044 | 299 | 0.100 | 0.094 |
| sunlamp | 1.37 | 0.082 | 229 | 0.140 | 0.118 |
| Arm | Domain | 3.8–5.3 | 5.3–6.7 | 6.7–8.3 | ||
|---|---|---|---|---|---|---|
| frozen, no aug. | in-domain | 97.5 | 95.7 | 90.2 | 82.9 | 75.2 |
| lightbox | 73.3 | 70.7 | 50.5 | 37.4 | 28.6 | |
| sunlamp | 63.6 | 53.0 | 22.2 | 6.6 | 1.4 | |
| frozen, full stack | in-domain | 97.4 | 95.0 | 92.1 | 90.9 | 75.6 |
| lightbox | 91.3 | 84.1 | 64.4 | 58.2 | 45.6 | |
| sunlamp | 81.7 | 39.4 | 11.6 | 3.0 | 0.1 |
| Arm | Domain | 3.8–5.3 | 5.3–6.7 | 6.7–8.3 | ||
|---|---|---|---|---|---|---|
| frozen, no aug. | in-domain | 77.3 | 93.5 | 89.0 | 90.7 | 95.7 |
| lightbox | 38.7 | 56.2 | 50.1 | 48.5 | 54.0 | |
| sunlamp | 31.8 | 39.6 | 21.9 | 9.7 | 5.3 | |
| frozen, full stack | in-domain | 74.5 | 88.5 | 88.6 | 94.3 | 92.9 |
| lightbox | 66.3 | 69.6 | 60.1 | 66.6 | 63.1 | |
| sunlamp | 45.0 | 22.2 | 9.8 | 4.0 | 0.6 |
| Arm | Domain | 3.8–5.3 | 5.3–6.7 | 6.7–8.3 | ||
|---|---|---|---|---|---|---|
| frozen, no aug. | in-domain | 96.1 | 91.4 | 93.6 | 87.3 | 83.3 |
| lightbox | 75.9 | 74.2 | 71.1 | 69.3 | 64.2 | |
| sunlamp | 64.2 | 56.5 | 61.4 | 56.2 | 52.5 | |
| frozen, full stack | in-domain | 92.0 | 92.8 | 92.9 | 90.1 | 84.9 |
| lightbox | 78.9 | 78.2 | 76.0 | 73.5 | 71.4 | |
| sunlamp | 74.2 | 70.3 | 70.0 | 70.0 | 67.1 |
| Experiment | Rule | Outcome |
| Revision (locked 28 Oct) | ||
| Arm A | Sunlamp, A full stack, 3/3 negative: keep “blur and noise work against sunlamp outright” | 2/3 ( , , ): softened to a hypothesis |
| Arm B | Sunlamp, B A, 3/3 negative: the glare sentence gains evidence | 1/3 ( , , ): softened |
| Control | Within 2 s.d. of published seed 42 on both real domains | Lightbox ( ), sunlamp ( ): inside; pairing holds |
| Frozen seeds | Sunlamp s.d. : “only 0.03 rad” becomes “no detectable change” | s.d. 0.031: reworded |
| Decomposition | One factor of the synthetic gain: name it; either factor moves sunlamp by frozen s.d.: split the claim | Data 0.33, features 0.29 (53%); largest sunlamp move 0.024 against 0.062: neither |
Sim-to-Real Transfer
Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.
See all 347 papers