All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation
Organizations: State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · North University of China · Xi’an Jiaotong University
Abstract
Deploying vision-and-language navigation (VLN) agents requires adaptation across diverse scenes and environments, but fine-tuning on a specific scenario often causes catastrophic forgetting in others, which severely limits flexible long-term deployment. We formalize this challenge as the all-day multi-scenes lifelong VLN (AML-VLN) problem. Existing parameter-efficient adapters (e.g., LoRA and its variants) are limited by their two-dimensional matrix form, which fails to capture the multi-hierarchical navigation knowledge spanning multiple scenes and environments. To address this, we propose Tucker Adaptation (TuKA), which represents the multi-hierarchical navigation knowledge as a high-order tensor and leverages Tucker decomposition to decouple the knowledge into shared subspaces and scenario-specific experts. We further introduce a decoupled knowledge incremental learning strategy to consolidate shared subspaces while constraining specific experts for decoupled lifelong learning. Building on TuKA, we also develop a VLN agent named AlldayWalker, which continually learns across multiple navigation scenarios, achieving all-day multi-scenes navigation. Extensive experiments show that AlldayWalker consistently outperforms state-of-the-art baselines.
Figures & tables
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | T21 | T22 | T23 | T24 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Seq-FT | 0 | 0 | 0 | 0 | 0 | 0 | 5 | 0 | 10 | 5 | 4 | 10 | 7 | 6 | 7 | 14 | 7 | 8 | 4 | 24 | 28 | 24 | 37 | 64 | 11 |
| LwF-LoRA | 5 | 0 | 5 | 0 | 0 | 0 | 4 | 3 | 13 | 5 | 2 | 14 | 8 | 4 | 8 | 14 | 10 | 9 | 5 | 25 | 31 | 28 | 40 | 65 | 12 |
| EWC-LoRA | 10 | 7 | 6 | 8 | 3 | 4 | 10 | 4 | 15 | 14 | 5 | 14 | 10 | 6 | 12 | 19 | 16 | 14 | 8 | 24 | 28 | 29 | 38 | 67 | 15 |
| Dense MoLE | 14 | 8 | 14 | 21 | 13 | 12 | 18 | 7 | 19 | 14 | 17 | 19 | 13 | 10 | 18 | 20 | 23 | 17 | 10 | 29 | 32 | 33 | 41 | 66 | 20 |
| Sparse MoLE | 29 | 10 | 24 | 29 | 17 | 28 | 29 | 11 | 37 | 24 | 23 | 30 | 18 | 24 | 25 | 34 | 31 | 10 | 14 | 38 | 36 | 36 | 45 | 69 | 28 |
| MoLA | 34 | 13 | 32 | 34 | 22 | 21 | 37 | 13 | 48 | 28 | 29 | 33 | 24 | 32 | 29 | 39 | 37 | 22 | 14 | 43 | 41 | 39 | 49 | 68 | 33 |
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | T21 | T22 | T23 | T24 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Seq-FT | 100 | 100 | 100 | 100 | 100 | 100 | 93 | 100 | 88 | 93 | 94 | 83 | 91 | 92 | 92 | 83 | 87 | 89 | 94 | 73 | 84 | 81 | 78 | 0 | 87 |
| LwF-LoRA | 92 | 100 | 91 | 100 | 100 | 100 | 92 | 89 | 81 | 93 | 95 | 78 | 88 | 93 | 89 | 83 | 85 | 84 | 93 | 71 | 82 | 76 | 73 | 0 | 84 |
| EWC-LoRA | 85 | 92 | 89 | 90 | 94 | 91 | 86 | 87 | 82 | 82 | 92 | 78 | 84 | 78 | 86 | 77 | 75 | 75 | 88 | 72 | 77 | 68 | 66 | 0 | 79 |
| Dense MoLE | 80 | 91 | 82 | 73 | 78 | 78 | 81 | 86 | 77 | 81 | 78 | 72 | 73 | 65 | 79 | 72 | 63 | 69 | 85 | 63 | 74 | 64 | 61 | 0 | 72 |
| Sparse MoLE | 68 | 89 | 69 | 64 | 72 | 49 | 58 | 82 | 56 | 68 | 64 | 56 | 68 | 51 | 70 | 58 | 51 | 82 | 82 | 54 | 63 | 59 | 56 | 0 | 62 |
| MoLA | 62 | 79 | 59 | 58 | 63 | 62 | 46 | 78 | 43 | 63 | 55 | 42 | 65 | 48 | 68 | 54 | 46 | 60 | 79 | 44 | 55 | 51 | 43 | 0 | 55 |
| Sd- | Sd- | Sd- | SR | F-SR | SPL | F-SPL | OSR | F-OSR |
|---|---|---|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | 53 | 10 | 47 | 17 | 56 | 8 |
| ✗ | ✓ | ✓ | 55 | 10 | 49 | 17 | 57 | 9 |
| ✓ | ✗ | ✓ | 65 | 11 | 58 | 18 | 69 | 9 |
| ✓ | ✓ | ✗ | 62 | 11 | 54 | 18 | 66 | 9 |
| ✓ | ✗ | ✗ | 63 | 11 | 55 | 18 | 67 | 9 |
| ✓ | ✓ | ✓ | 65 | 11 | 58 | 18 | 68 | 9 |
| Tasks | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SR (24 tasks) | 79 | 23 | 71 | 81 | 33 | 50 | 87 | 38 | 79 | 75 | 79 | 67 | 50 | 86 | 71 |
| SR (30 tasks) | 77 | 23 | 71 | 70 | 33 | 50 | 86 | 38 | 79 | 75 | 80 | 66 | 50 | 85 | 70 |
| Task | Scene | Environment | Test Number | StreamVLN SR | BranchLoRA SR | SD-LoRA SR | AlldayWalker SR |
|---|---|---|---|---|---|---|---|
| G1 | JeFG25nYj2p | Normal | 105 | 45 | 52 | 53 | 65 |
| G2 | ur6pFq6Qu1A | Low-light | 120 | 41 | 44 | 45 | 63 |
| G3 | r47D5H71a5s | Scattering | 105 | 36 | 40 | 41 | 54 |
| G4 | Vvot9Ly1tCj | Overexposed | 108 | 31 | 42 | 39 | 51 |
| G5 | Real-World 4 | Normal | 100 | 36 | 38 | 37 | 55 |
| G6 | Real-World 5 | Low-light | 100 | 18 | 21 | 21 | 43 |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Notations | Definitions | Notations | Definitions |
|---|---|---|---|
| The user language instruction | The Hadamard product | ||
| Navigation Scene | The Fisher information weights for | ||
| Scene Environment | The Fisher information weights for | ||
| The backbone agent | The vision features for each scene | ||
| The video stream observation | The vision features for each environment | ||
| The CLIP vision encoder | The output navigation actions annotations |
| Hyperparameters Name | Value |
| Number of scene-specific experts | 7 |
| Number of environment-specific experts | 4 |
| 4D Tucker decomposition ranks | (8,8,64,64) |
| Elastic Weight Consolidation regularization weight | 0.2 |
| Stability constraint weight for previously trained experts | 0.2 |
| Orthogonal regularization weight for / | 0.1 |
| Pattern | Hyperparameters Name | Value |
|---|---|---|
| Low-night | Brightness factor | 0.15 |
| Exposure time | 0.15 | |
| Sensor gain / ISO | 8.0 | |
| Shot noise factor | 0.4 | |
| Read noise standard deviation | 3.0 | |
| Gamma correction value | 2.2 |
| Task ID | Scene Index Number | Scene ID | Environment ID | Train Number | Test Number |
| 1 | ac26ZMwG7aT | Scene-2 | Normal-1 | 455 | 120 |
| 2 | 5LpN3gDmAk7 | Scene-1 | Scattering-2 | 405 | 150 |
| 3 | S9hNv5qa7GM | Scene-4 | Normal-1 | 410 | 105 |
| 4 | mJXqzFtmKg4 | Scene-3 | Low-light-3 | 445 | 105 |
| 5 | mJXqzFtmKg4 | Scene-4 | Overexposure-4 | 410 | 105 |
| 6 | ULsKaCPVFJR | Scene-5 | Overexposure-4 | 390 | 120 |
| Rank | Rank | Scale-Type | SR | F-SR | SPL | F-SPL | OSR | F-OSR |
|---|---|---|---|---|---|---|---|---|
| 4,4 | 32,32 | all | 60 | 11 | 51 | 18 | 62 | 9 |
| 4,4 | 64,64 | sos | 64 | 11 | 57 | 18 | 68 | 9 |
| 8,8 | 32,32 | eos | 61 | 11 | 52 | 18 | 63 | 9 |
| 8,8 | 64,64 | equal | 65 | 11 | 58 | 18 | 68 | 9 |
| Rank | Rank | Scale-Type | SR | F-SR | SPL | F-SPL | OSR | F-OSR |
|---|---|---|---|---|---|---|---|---|
| 16,16 | 128,128 | all | 68 | 11 | 61 | 18 | 71 | 9 |
| 16,16 | 64,64 | sos | 65 | 11 | 60 | 18 | 68 | 9 |
| 8,8 | 128,128 | eos | 67 | 11 | 61 | 18 | 71 | 9 |
| 8,8 | 64,64 | equal | 65 | 11 | 58 | 18 | 68 | 9 |
| Rank | Rank | Scale-Type | SR | F-SR | SPL | F-SPL | OSR | F-OSR |
|---|---|---|---|---|---|---|---|---|
| 24,24 | 192,192 | all | 69 | 11 | 62 | 18 | 71 | 9 |
| 24,24 | 64,64 | sos | 66 | 11 | 60 | 18 | 68 | 9 |
| 8,8 | 192,192 | eos | 68 | 11 | 61 | 18 | 71 | 9 |
| 8,8 | 64,64 | equal | 65 | 11 | 58 | 18 | 68 | 9 |
| Rank | Rank | Scale-Type | SR | F-SR | SPL | F-SPL | OSR | F-OSR |
|---|---|---|---|---|---|---|---|---|
| 32,32 | 256,256 | all | 69 | 11 | 62 | 18 | 71 | 9 |
| 32,32 | 64,64 | sos | 66 | 11 | 60 | 18 | 68 | 9 |
| 8,8 | 256,256 | eos | 68 | 11 | 61 | 18 | 71 | 9 |
| 8,8 | 64,64 | equal | 65 | 11 | 58 | 18 | 68 | 9 |
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3-order SR | 65 | 21 | 49 | 61 | 30 | 46 | 70 | 30 | 70 | 71 | 60 | 61 | 41 | 73 | 54 | 71 | 48 | 49 | 35 | 68 | 54 |
| 4-order SR | 79 | 23 | 71 | 81 | 33 | 50 | 87 | 38 | 79 | 75 | 79 | 67 | 50 | 86 | 71 | 76 | 67 | 63 | 43 | 81 | 65 |
| 3-order F-SR | 13 | 36 | 33 | 12 | 35 | 15 | 6 | 35 | 11 | 2 | 5 | 10 | 36 | 2 | 33 | 16 | 28 | 16 | 36 | 0 | 19 |
| 4-order F-SR | 2 | 27 | 23 | 8 | 21 | 1 | 0 | 28 | 9 | 0 | 2 | 6 | 30 | 24 | 10 | 1 | 4 | 21 | 0 | 11 | |
| 3-order SPL | 60 | 16 | 51 | 60 | 28 | 43 | 70 | 26 | 60 | 65 | 50 | 51 | 38 | 61 | 48 | 62 | 48 | 42 | 35 | 51 | 48 |
| 4-order SPL | 70 | 20 | 67 | 73 | 25 | 32 | 80 | 34 | 73 | 71 | 68 | 56 | 48 | 75 | 65 | 70 | 62 | 60 | 41 | 67 | 58 |
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| O-LoRA SR | 67 | 19 | 47 | 58 | 31 | 42 | 67 | 27 | 67 | 62 | 58 | 49 | 38 | 68 | 52 | 62 | 46 | 52 | 34 | 71 | 51 |
| ABC-LoRA SR | 71 | 20 | 55 | 64 | 30 | 43 | 68 | 30 | 68 | 66 | 59 | 52 | 43 | 72 | 61 | 68 | 55 | 52 | 35 | 77 | 55 |
| AlldayWalker SR | 79 | 23 | 71 | 81 | 33 | 50 | 87 | 38 | 79 | 75 | 79 | 67 | 50 | 86 | 71 | 76 | 67 | 63 | 43 | 81 | 65 |
| O-LoRA F-SR | 19 | 41 | 41 | 28 | 38 | 24 | 4 | 38 | 15 | 17 | 11 | 13 | 43 | 2 | 40 | 24 | 32 | 5 | 35 | 0 | 24 |
| ABC-LoRA F-SR | 15 | 36 | 33 | 21 | 32 | 25 | 4 | 30 | 15 | 15 | 8 | 7 | 36 | 0 | 33 | 15 | 26 | 5 | 25 | 0 | 19 |
| AlldayWalker F-SR | 2 | 27 | 23 | 8 | 21 | 1 | 0 | 28 | 9 | 0 | 2 | 6 | 30 | -3 | 24 | 10 | 1 | 4 | 21 | 0 | 11 |
| Task Statistics | HydraLoRA | SD-LoRA | AlldayWalker | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Task ID | Scene ID | Environment | Instruction | SR | F-SR | SR | F-SR | SR | F-SR |
| 1 | ac26ZMwG7aT | Normal | VLN | 46 | 44 | 69 | 13 | 79 | 2 |
| 2 | 5LpN3gDmAk7 | Low-light | DUN | 29 | 60 | 41 | 35 | 51 | 29 |
| 3 | ac26ZMwG7aT | Overexposure | VLN | 30 | 68 | 29 | 34 | 39 | 29 |
| 4 | S9hNv5qa7GM | Scattering | DUN | 51 | 32 | 60 | 21 | 69 | 15 |
| 5 | 5LpN3gDmAk7 | Normal | VLN | 33 | 28 | 69 | 8 | 87 | 0 |
| Method | Avg SR | Avg F-SR | Avg SPL | Avg F-SPL | Avg OSR | Avg F-OSR | Parameter |
|---|---|---|---|---|---|---|---|
| Seq-FT | 11 | 87 | 4 | 95 | 6 | 91 | 14.68 M |
| LwF-LoRA | 12 | 84 | 6 | 92 | 9 | 86 | 14.68 M |
| EWC-LoRA | 15 | 79 | 9 | 87 | 13 | 81 | 14.68 M |
| Dense MoLE | 20 | 72 | 14 | 80 | 18 | 74 | 16.52 M |
| Sparse MoLE | 28 | 62 | 21 | 70 | 26 | 64 | 16.52 M |
| MoLA | 33 | 55 | 26 | 63 | 31 | 57 | 16.52 M |
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Seq-FT | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 0 | 9 | 0 | 2 | 8 | 5 | 3 | 4 | 12 | 3 | 6 | 2 | 18 | 4 |
| LwF-LoRA | 5 | 0 | 5 | 0 | 0 | 0 | 4 | 3 | 9 | 5 | 2 | 12 | 8 | 2 | 6 | 14 | 7 | 8 | 3 | 21 | 6 |
| EWC-LoRA | 5 | 6 | 6 | 8 | 2 | 4 | 7 | 4 | 10 | 12 | 5 | 14 | 8 | 4 | 10 | 15 | 14 | 11 | 5 | 23 | 9 |
| Dense MoLE | 9 | 5 | 14 | 16 | 11 | 8 | 16 | 6 | 16 | 14 | 15 | 15 | 10 | 7 | 17 | 18 | 22 | 16 | 9 | 27 | 14 |
| Sparse MoLE | 24 | 7 | 24 | 15 | 17 | 14 | 27 | 9 | 31 | 22 | 22 | 21 | 17 | 21 | 21 | 29 | 28 | 10 | 12 | 33 | 21 |
| MoLA | 29 | 8 | 32 | 32 | 18 | 21 | 36 | 13 | 45 | 27 | 25 | 26 | 22 | 29 | 24 | 36 | 31 | 17 | 14 | 42 | 26 |
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Seq-FT | 100 | 100 | 100 | 100 | 100 | 100 | 93 | 100 | 91 | 93 | 95 | 89 | 93 | 94 | 94 | 89 | 90 | 89 | 100 | 82 | 95 |
| LwF-LoRA | 94 | 100 | 94 | 100 | 100 | 100 | 92 | 93 | 90 | 94 | 95 | 86 | 88 | 85 | 92 | 87 | 88 | 85 | 100 | 79 | 92 |
| EWC-LoRA | 91 | 93 | 89 | 90 | 94 | 92 | 88 | 93 | 87 | 85 | 92 | 83 | 93 | 94 | 89 | 80 | 79 | 84 | 94 | 75 | 87 |
| Dense MoLE | 85 | 91 | 86 | 77 | 78 | 78 | 85 | 89 | 80 | 83 | 82 | 82 | 79 | 69 | 83 | 73 | 67 | 72 | 86 | 68 | 80 |
| Sparse MoLE | 74 | 90 | 74 | 68 | 76 | 64 | 63 | 85 | 58 | 71 | 67 | 57 | 75 | 53 | 75 | 67 | 58 | 84 | 85 | 56 | 70 |
| MoLA | 68 | 81 | 63 | 62 | 64 | 65 | 48 | 80 | 48 | 67 | 59 | 49 | 69 | 49 | 72 | 62 | 54 | 69 | 83 | 47 | 63 |
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Seq-FT | 0 | 5 | 0 | 0 | 0 | 3 | 5 | 0 | 10 | 5 | 4 | 10 | 8 | 12 | 7 | 14 | 7 | 8 | 6 | 24 | 6 |
| LwF-LoRA | 5 | 8 | 7 | 4 | 0 | 7 | 4 | 3 | 17 | 8 | 8 | 15 | 8 | 8 | 10 | 15 | 12 | 11 | 7 | 27 | 9 |
| EWC-LoRA | 14 | 10 | 6 | 13 | 3 | 12 | 12 | 4 | 23 | 14 | 13 | 19 | 11 | 13 | 14 | 19 | 16 | 17 | 9 | 24 | 13 |
| Dense MoLE | 19 | 11 | 14 | 21 | 15 | 16 | 18 | 8 | 34 | 17 | 17 | 19 | 13 | 14 | 18 | 20 | 25 | 17 | 10 | 33 | 18 |
| Sparse MoLE | 29 | 13 | 27 | 33 | 18 | 28 | 31 | 11 | 59 | 24 | 23 | 30 | 20 | 29 | 25 | 36 | 31 | 10 | 14 | 38 | 26 |
| MoLA | 34 | 12 | 33 | 38 | 22 | 24 | 37 | 13 | 52 | 32 | 29 | 34 | 26 | 38 | 31 | 39 | 39 | 22 | 14 | 43 | 31 |
| Comparisons | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 | T11 | T12 | T13 | T14 | T15 | T16 | T17 | T18 | T19 | T20 | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Seq-FT | 100 | 91 | 100 | 100 | 100 | 94 | 93 | 100 | 88 | 93 | 94 | 83 | 87 | 89 | 92 | 83 | 87 | 89 | 93 | 73 | 91 |
| LwF-LoRA | 92 | 92 | 88 | 89 | 100 | 90 | 92 | 89 | 77 | 91 | 91 | 77 | 88 | 78 | 87 | 79 | 82 | 80 | 94 | 69 | 86 |
| EWC-LoRA | 81 | 89 | 89 | 77 | 94 | 85 | 84 | 87 | 73 | 82 | 88 | 74 | 81 | 73 | 81 | 77 | 75 | 71 | 91 | 72 | 81 |
| Dense MoLE | 73 | 90 | 84 | 70 | 71 | 77 | 81 | 84 | 69 | 76 | 78 | 72 | 73 | 61 | 79 | 72 | 60 | 69 | 89 | 61 | 74 |
| Sparse MoLE | 68 | 84 | 67 | 50 | 72 | 49 | 53 | 82 | 55 | 68 | 64 | 56 | 67 | 46 | 70 | 57 | 51 | 82 | 82 | 54 | 64 |
| MoLA | 62 | 79 | 63 | 45 | 63 | 58 | 46 | 78 | 42 | 61 | 55 | 41 | 74 | 39 | 66 | 54 | 42 | 60 | 84 | 44 | 57 |
| Task | Scene Description | Environment | Train Number | Test Number |
|---|---|---|---|---|
| T25 | 8WUmhLawc2A | Normal Environment | 460 | 120 |
| T26 | 8WUmhLawc2A | Low-light Environment | 460 | 120 |
| T27 | 8WUmhLawc2A | Scattering Environment | 460 | 120 |
| T28 | 8WUmhLawc2A | Overexposed Environment | 460 | 120 |
| T29 | Real-World Scene 3 | Normal Environment | 400 | 100 |
| T30 | Real-World Scene 3 | Low-light Environment | 400 | 100 |