ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Grounding
Authors: Shuo Feng, Piji Li
Organizations: College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, 211106 · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, 211106, China.
Embodied Referring Expression Grounding is the task of enabling an agent to navigate in real environments and to localize a remote object based on natural language instructions. In this scenario, the agent needs to select one view for navigation at each step and identify a specific object among all candidate objects at the destination. However, most of the previous approaches fail to distinguish between views and objects, instead processing them using the vanilla vision encoder, which results in ambiguous representations of both views and objects. To address the above issues, we propose ViSMoE, which equips sparse Mixture-of-Experts with a visual-aware routing policy for the embodied agent. This framework processes different types of visual information specifically, resulting in discriminative visual representations for both views and objects. Experimental results on REVERIE and SOON datasets demonstrate that ViSMoE outperforms the previous state-of-the-art methods, showing the superiority of our proposed method.
Figures & tables
Figure 1: Illustration of navigation instruction, panorama, discrete views and candidate objects in SOON dataset. Discrete views provide a depiction of the scene layout, while candidate objects offer detailed information about specific items.
Figure 2: Overview of ViSMoE: The dense FFN in the vision encoder is replaced by SMoE layer. We use brown boxes to denote scene images and green boxes to represent object images. Unlike vanilla SMoE, which ignores image categories, ViSMoE’s routing input includes them, allowing the trained router to select the appropriate expert based on both the input token and its visual type. Therefore, it can be seen that while the vanilla MoE router balances the distribution of input tokens across experts, ViSMoE learns to route visual tokens to specific experts (e.g., expert-1 and expert-3 for view images, expert-2 and expert-N for object images).
Methods
Val Unseen
Test Unseen
Navigation
Grounding
Navigation
Grounding
TL↓
OSR↑
SR↑
SPL↑
RGS↑
RGSPL↑
TL↓
OSR↑
SR↑
SPL↑
RGS↑
RGSPL↑
CKR [ 6 ]
26.26
31.44
19.14
11.84
11.45
-
22.46
30.40
22.00
14.25
11.60
-
SIA [ 13 ]
41.53
44.67
31.53
16.28
22.41
11.56
48.61
44.56
30.80
14.85
19.02
9.20
HAMT [ 3 ]
14.08
36.84
32.95
30.20
18.92
17.28
13.62
33.41
30.40
26.67
14.88
13.08
TD-STP [ 23 ]
-
39.48
34.88
27.32
21.16
16.56
-
40.26
35.89
27.51
19.88
15.40
Table 1: Evaluation on REVERIE dataset.
Methods
Val Unseen
Test Unseen
TL↓
OSR↑
SR↑
SPL↑
RGSPL↑
TL↓
OSR↑
SR↑
SPL↑
RGSPL↑
GBE [ 24 ]
28.96
28.54
19.52
13.34
1.16
27.88
21.45
12.90
9.23
0.45
KERM [ 11 ]
35.83
51.62
38.05
23.16
4.04
-
-
-
-
-
GridMM [ 21 ]
38.92
53.39
37.46
24.81
3.91
46.20
48.02
36.27
21.25
4.15
DUET [ 4 ]
36.20
50.91
36.28
22.58
3.75
41.83
43.00
33.44
21.42
4.17
Ours
36.17
51.06
38.23
26.13
4.31
38.41
48.56
36.69
21.53
5.20
Table 2: Evaluation on SOON dataset.
Baseline
SMoE
Vis.
Aux.
SR↑
SPL↑
RGS↑
RGSPL↑
✓
46.98
33.73
32.15
23.03
✓
✓
48.37
34.09
32.57
22.94
✓
✓
✓
48.46
34.19
32.67
23.16
✓
✓
✓
49.92
35.01
33.42
23.74
✓
✓
✓
✓
50.18
35.35
33.71
23.87
Table 3: Ablation study on the val unseen split of REVERIE.