Toward Planning Representations That Separate Task Structure from Visual Content
Organizations: Paul G. Allen School of Computer Science & Engineering University of Washington, Seattle, United States
Abstract
Effective planning requires representations that preserve task-relevant geometry and object identity while ignoring visual variation that does not alter the task. Existing evaluations of visual generalization often change several factors simultaneously, making the source of a failure difficult to identify. We introduce a controlled evaluation of object-goal navigation (ObjectNav), the task of reaching an instance of a named object category, in procedurally generated ProcTHOR houses. Training and evaluation houses share floor plans, object locations, goals, and start states, while visual factors are changed individually or in combination. Across model-free reinforcement learning (RL), model-based RL, and agents using frozen pretrained visual representations, our results show that appearance changes impair navigation, but their effects differ sharply by factor. Surface material changes account for most of the average change in success, whereas clutter and lighting have little effect. The magnitude of a global image statistics shift does not reliably predict performance: changing only the goal object's appearance can substantially impair an end-to-end agent despite a small image-level change. We found that a model based agent that plans over frozen features from the pretrained DINOv2 image encoder achieves the highest success (0.83) under the full shift, but remains sensitive to surface appearance. Our experiments reveal an important requirement for planning representations: they must separate invariant spatial task structure from appearance.
Figures & tables
| Agent | A | L1 | L2 | L3 |
|---|---|---|---|---|
| PPO | 0.93 0.02 | 0.52 0.04 | 0.29 0.04 | 0.29 0.04 |
| PPO+Aug | 0.93 0.04 | 0.52 0.08 | 0.35 0.07 | 0.34 0.08 |
| PPO+I-JEPA | 0.90 0.02 | 0.47 0.06 | 0.39 0.05 | 0.39 0.05 |
| PPO+MAE | 0.92 0.02 | 0.35 0.08 | 0.27 0.05 | 0.26 0.06 |
| PPO+DINOv2 | 0.92 0.04 | 0.72 0.09 | 0.65 0.06 | 0.66 0.07 |
| DreamerV3 | 0.76 0.10 | 0.61 0.16 | 0.60 0.10 | 0.62 0.09 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Pair | Target | Reachable cells | Clutter kept | Goal swapped | L1 path shift (m) | L2/L3 path shift (m) |
|---|---|---|---|---|---|---|
| P2 | Bed | 128 | 5 | no | 0.00 | 0.00 / 0.00 |
| P1 | Bed | 162 | 2 | yes | 0.00 | 0.25 / 0.25 |
| P0 | Fridge | 380 | 4 | yes | 0.00 | 0.49 / 0.49 |
| P3 | Bed | 460 | 8 | yes | 0.00 | 0.26 / 0.26 |
| P4 | Television | 681 | 8 | yes | 0.00 | 0.00 / 0.00 |
| Agent | Settings |
|---|---|
| PPO | Stable-Baselines3; Nature-CNN encoder [ 25 ] on RGB; rollout 1024 steps, minibatch 128, 4 epochs, learning rate , , generalized advantage estimation (GAE) , clip 0.2, entropy coefficient 0.01. |
| PPO+Aug | As PPO; brightness, contrast, and saturation jitter , hue , one draw per episode during training only. |
| PPO+frozen | As PPO, with the encoder replaced by frozen I-JEPA ViT-H/14, MAE ViT-H/14, or DINOv2 ViT-g/14; bilinear resize to , ImageNet normalization, mean patch tokens with class and register tokens removed, half-precision (fp16) inference, and a two-layer 64-unit MLP head. MAE runs without masking. |
| DreamerV3 | PyTorch reimplementation ( https://github.com/NM512/dreamerv3-torch ), default small model; observations; replay ratio 512; batch ; 2,500 random prefill steps. |
| TD-MPC2 | Official implementation ( https://github.com/nicklashansen/tdmpc2 ), 5M-parameter model; observations, three-frame stack; horizon 3; MPPI with 6 iterations, 512 samples, 64 elites; batch 256; one update per environment step; discrete action by argmax over a five-dimensional continuous output. |
| TD-MPC2+DINOv2 | As TD-MPC2, with the frozen 1536-dimensional DINOv2 ViT-g/14 feature used by PPO+DINOv2 as its one-frame observation, entering through TD-MPC2’s MLP state encoder. |
| Agent | House A | R1 | R2 | R3 | R4 |
|---|---|---|---|---|---|
| clutter | + lighting & sky | + object looks | + surfaces | ||
| Model-free, encoder trained from scratch | |||||
| PPO | 0.94 0.03 | 0.93 0.03 | 0.71 0.07 | 0.44 0.04 | 0.29 0.04 |
| PPO+Aug | 0.93 0.04 | 0.93 0.03 | 0.84 0.06 | 0.61 0.08 | 0.34 0.08 |
| Model-free, frozen pretrained encoder | |||||
| PPO+I-JEPA | 0.90 0.02 | 0.89 0.02 | 0.87 0.02 | 0.64 0.02 | 0.37 0.05 |
| Pair | Agent | A | R1 | R2 | R3 | R4 |
|---|---|---|---|---|---|---|
| P2 Bed | PPO | 0.91 0.03 | 0.91 0.03 | 0.89 0.03 | 0.79 0.07 | 0.79 0.03 |
| PPO+Aug | 0.95 0.05 | 0.95 0.05 | 0.88 0.03 | 0.82 0.07 | 0.83 0.07 | |
| PPO+I-JEPA | 0.95 0.07 | 0.96 0.07 | 0.91 0.07 | 0.92 0.08 | 0.68 0.20 | |
| PPO+MAE | 0.92 0.12 | 0.92 0.12 | 0.61 0.29 | 0.56 0.26 | 0.27 0.30 | |
| PPO+DINOv2 | 0.88 0.06 | 0.89 0.06 | 0.82 0.10 | 0.83 0.13 | 0.78 0.09 | |
| DreamerV3 | 0.74 0.39 | 0.76 0.38 | 0.84 0.27 | 0.86 0.20 | 0.89 0.12 |
| Pair | Agent | A | L1 | L2 | L3 |
|---|---|---|---|---|---|
| P2 Bed | PPO | 0.91 0.03 | 0.86 0.07 | 0.79 0.03 | 0.79 0.03 |
| PPO+Aug | 0.95 0.05 | 0.86 0.05 | 0.83 0.07 | 0.83 0.07 | |
| PPO+I-JEPA | 0.96 0.07 | 0.62 0.28 | 0.70 0.18 | 0.70 0.18 | |
| PPO+MAE | 0.92 0.12 | 0.32 0.38 | 0.26 0.29 | 0.27 0.30 | |
| PPO+DINOv2 | 0.90 0.02 | 0.77 0.09 | 0.76 0.10 | 0.78 0.09 | |
| DreamerV3 | 0.76 0.39 | 0.72 0.37 | 0.69 0.39 | 0.70 0.39 |
| All five houses | P3 | |||||
|---|---|---|---|---|---|---|
| Agent | Original | Single-change | Reordered | Original | Single-change | Reordered |
| PPO | 0.29 0.04 | 0.29 0.04 | 0.29 0.04 | 0.04 0.05 | 0.04 0.05 | 0.04 0.05 |
| PPO+Aug | 0.34 0.08 | 0.34 0.08 | 0.34 0.08 | 0.19 0.18 | 0.19 0.18 | 0.19 0.18 |
| PPO+I-JEPA | 0.39 0.05 | 0.38 0.05 | 0.37 0.05 | 0.10 0.14 | 0.11 0.14 | 0.10 0.14 |
| PPO+MAE | 0.26 0.06 | 0.25 0.06 | 0.25 0.06 | 0.14 0.20 | 0.14 0.20 | 0.14 0.20 |
| PPO+DINOv2 | 0.66 0.07 | 0.66 0.08 | 0.66 0.07 | 0.22 0.23 | 0.22 0.23 | 0.22 0.24 |
| Agent | House A | Clutter | Lighting | Sky | Other objects | Goal object | Surfaces |
|---|---|---|---|---|---|---|---|
| PPO | 0.94 0.03 | 0.93 0.03 | 0.93 0.01 | 0.75 0.08 | 0.86 0.04 | 0.58 0.14 | 0.59 0.11 |
| PPO+Aug | 0.93 0.04 | 0.93 0.03 | 0.93 0.04 | 0.87 0.07 | 0.88 0.05 | 0.78 0.14 | 0.59 0.10 |
| PPO+I-JEPA | 0.89 0.02 | 0.90 0.02 | 0.90 0.02 | 0.90 0.03 | 0.84 0.03 | 0.82 0.06 | 0.53 0.09 |
| PPO+MAE | 0.92 0.02 | 0.92 0.02 | 0.84 0.05 | 0.86 0.03 | 0.88 0.03 | 0.87 0.04 | 0.47 0.15 |
| PPO+DINOv2 | 0.92 0.04 | 0.92 0.04 | 0.91 0.04 | 0.91 0.04 | 0.87 0.04 | 0.90 0.06 | 0.72 0.08 |
| DreamerV3 | 0.91 0.05 | 0.92 0.02 | 0.90 0.03 | 0.91 0.04 | 0.89 0.06 | 0.98 0.01 | 0.71 0.09 |
| Pair | Agent | House A | Clutter | Lighting | Sky | Other objects | Goal object | Surfaces |
|---|---|---|---|---|---|---|---|---|
| P2 Bed | PPO | 0.91 0.03 | 0.91 0.03 | 0.87 0.07 | 0.90 0.05 | 0.84 0.05 | – | 0.86 0.09 |
| PPO+Aug | 0.95 0.05 | 0.95 0.05 | 0.97 0.03 | 0.94 0.05 | 0.90 0.05 | – | 0.85 0.07 | |
| PPO+I-JEPA | 0.95 0.07 | 0.96 0.07 | 0.97 0.03 | 0.94 0.08 | 0.96 0.04 | – | 0.86 0.12 | |
| PPO+MAE | 0.92 0.12 | 0.92 0.12 | 0.72 0.31 | 0.78 0.15 | 0.85 0.15 | – | 0.34 0.35 | |
| PPO+DINOv2 | 0.90 0.04 | 0.91 0.03 | 0.86 0.07 | 0.88 0.00 | 0.84 0.04 | – | 0.82 0.08 | |
| DreamerV3 | 0.90 0.20 | 0.93 0.14 | 0.90 0.20 | 0.95 0.10 | 0.92 0.16 | – | 0.94 0.10 |
| Agent | L2noT (goal unchanged) | L2 (goal changed) |
|---|---|---|
| PPO | 0.37 0.06 | 0.17 0.05 |
| PPO+Aug | 0.38 0.10 | 0.23 0.10 |
| PPO+I-JEPA | 0.37 0.12 | 0.32 0.07 |
| PPO+MAE | 0.35 0.05 | 0.27 0.04 |
| PPO+DINOv2 | 0.66 0.09 | 0.63 0.07 |
| DreamerV3 | 0.59 0.07 | 0.60 0.10 |
| Change | P2 | P1 | P0 | P3 | P4 |
|---|---|---|---|---|---|
| Clutter | 0.0 | 4.2 | 0.1 | 0.0 | 0.1 |
| Lighting | 6.7 | 4.0 | 3.0 | 6.3 | 8.1 |
| Sky | 5.6 | 1.6 | 9.3 | 5.2 | 5.5 |
| Other objects | 1.0 | 1.2 | 1.1 | 0.6 | 6.4 |
| Goal object | – | 1.0 | 1.0 | 1.2 | 0.1 |
| Surfaces | 23.6 | 70.7 | 8.3 | 26.8 | 8.5 |
| Change | Measure | P2 | P1 | P0 | P3 | P4 |
|---|---|---|---|---|---|---|
| Lighting | W1, red | 6.5 | 4.2 | 3.0 | 6.7 | 9.7 |
| W1, green | 6.4 | 4.2 | 2.8 | 6.4 | 8.5 | |
| W1, blue | 7.3 | 3.6 | 3.1 | 5.7 | 6.2 | |
| W1, mean of the three | 6.7 | 4.0 | 3.0 | 6.3 | 8.1 | |
| Per-pixel difference | 6.7 | 4.0 | 3.0 | 6.3 | 8.1 | |
| Sky | W1, red | 7.0 | 1.2 | 17.5 | 5.0 | 7.7 |
| Color shift (W1) | Per-pixel difference | ||||
|---|---|---|---|---|---|
| Agent | all changes | no repaint | repaint | all changes | no repaint |
| PPO | +0.49 ( 0.001) | +0.10 (0.522) | +0.00 (0.990) | +0.67 ( 0.001) | +0.40 (0.012) |
| PPO+Aug | +0.67 ( 0.001) | +0.23 (0.160) | +0.16 (0.490) | +0.82 ( 0.001) | +0.47 (0.003) |
| PPO+I-JEPA | +0.61 ( 0.001) | +0.08 (0.612) | +0.21 (0.352) | +0.75 ( 0.001) | +0.30 (0.059) |
| PPO+MAE | +0.76 ( 0.001) | +0.50 (0.001) | +0.11 (0.634) | +0.85 ( 0.001) | +0.57 ( 0.001) |
| PPO+DINOv2 | +0.65 ( 0.001) | +0.17 (0.296) | +0.13 (0.583) | +0.76 ( 0.001) | +0.24 (0.133) |
| 150k steps | 300k steps | |||
|---|---|---|---|---|
| Agent | A | L2 | A | L2 |
| PPO | 0.92 0.03 | 0.27 0.04 | 0.93 0.02 | 0.29 0.04 |
| PPO+Aug | 0.84 0.05 | 0.47 0.07 | 0.93 0.04 | 0.35 0.07 |
| DreamerV3 | 0.71 0.13 | 0.63 0.11 | 0.76 0.10 | 0.60 0.10 |
| TD-MPC2 | 0.99 0.01 | 0.55 0.09 | 0.99 0.00 | 0.54 0.09 |
| Agent 1 | Agent 2 | Difference (points) | ||
|---|---|---|---|---|
| All pairs, retained success at L2 (28 comparisons) | ||||
| PPO | PPO+Aug | +6.7 | 0.107 | 0.640 |
| PPO | PPO+I-JEPA | +10.9 | 0.040 | 0.319 |
| PPO | PPO+MAE | 3.4 | 0.642 | 1.000 |
| PPO | PPO+DINOv2 | +40.4 | 0.0001 | 0.0003 |
| PPO | DreamerV3 | +33.5 | 0.0001 | 0.002 |
Object-Goal Navigation
Object-goal navigation studies how robots find and approach requested objects in unfamiliar environments using visual, spatial, and language cues.
See all 93 papers