Dynamic Graph Neural Network with Adaptive Features Selection for RGB-D Based Indoor Scene Recognition
Authors: Qiong Liu, Ruofei Xiong, Xingzhen Chen, Muyao Peng, You Yang
Organizations: School of Electronic Information and Communications, Huazhong University of Science and Technology, No. 1037, Luoyu Rd., Wuhan, 430074, P. R. China
Multi-modality of color and depth, i.e., RGB-D, is of great importance in recent research of indoor scene recognition. In this kind of data representation, depth map is able to describe the 3D structure of scenes and geometric relations among objects. Previous works showed that local features of both modalities are vital for promotion of recognition accuracy. However, the problem of adaptive selection and effective exploitation on these key local features remains open in this field. In this paper, a dynamic graph model is proposed with adaptive node selection mechanism to solve the above problem. In this model, a dynamic graph is built up to model the relations among objects and scene, and a method of adaptive node selection is proposed to take key local features from both modalities of RGB and depth for graph modeling. After that, these nodes are grouped by three different levels, representing near or far relations among objects. Moreover, the graph model is updated dynamically according to attention weights. Finally, the updated and optimized features of RGB and depth modalities are fused together for indoor scene recognition. Experiments are performed on public datasets including SUN RGB-D and NYU Depth v2. Extensive results demonstrate that our method has superior performance when comparing to state-of-the-arts methods, and show that the proposed method is able to exploit crucial local features from both modalities of RGB and depth.
Figures & tables
Figure 1: The framework of the proposed dynamic graph. First, RGB and depth images are input to two CNNs for feature extraction. Then, ANS module is used to select nodes to construct the multi-modality dynamic graph. In the last step, global and local features learned by the dynamic graph model are combined together for the final scene recognition.
Figure 2: The selected key local features (i.e., nodes) for single modality (e.g., RGB) graph construction. The contribution of each node is different from the task of scene recognition, and it should be evaluated properly in graph modeling. In our dynamic graph model, the importance of each node is represented by its value in the computed attention map. By taking k = 16 as an example, nodes are organized by 3 levels, including 1 main-central (i.e., in red), 3 sub-central (i.e., in green) and 12 leaf nodes (i.e., in blue), in the dynamic graph model. The three sub-central nodes { R2 , R3 , R4 } are connected to the main-central node R1 and the rest of the leaf nodes are connected to the sub-central node by euclidean distance. Similar process is performed for depth dynamic graph construction.
Figure 3: Connection between two modalities. Considering the semantic gap between the two modalities, a sparse connection is chosen which means main-central nodes, R1 and D1 , sub-central nodes, { R2 , R3 , R4 } and { D2 , D3 , D4 } are connected separately.
Methods
Mean-class Accuracy ( % )
State-of-the-arts
Wang et al . Wang et al. (2016)
48.1 %
Song et al . Song et al. (2017a)
54.0 %
Li et al . Li et al. (2018b)
54.6 %
Yuan et al . Yuan et al. (2019)
55.1 %
Li et al . Li et al. (2019)
56.2 %
Du et al . Du et al. (2019)
56.7 %
Table 1: Performance comparison with the state-of-the-art methods on SUN RGB-D Dataset.
Figure 4: The classification confusion matrix of the proposed dynamic graph model on the SUN RGB-D Dataset. The vertical axis shows the ground-truth classes, the horizontal axis shows the predicted classes. The classes on the horizontal axis are in the same order as those on the vertical axis.
Methods
Mean-class Accuracy ( % )
State-of-the-arts
Wang et al . Wang et al. (2016)
63.9 %
Song et al . Song et al. (2017a)
66.9 %
Li et al . Li et al. (2018b)
65.4 %
Yuan et al . Yuan et al. (2019)
67.2 %
Li et al . Li et al. (2019)
67.7 %
Du et al . Du et al. (2019)
69.2 %
Table 2: Performance comparison with the state-of-the-art methods on NYU Depth v2 Dataset
Figure 5: The classification confusion matrix of proposed dynamic graph model on the NYU Depth v2 Dataset. The vertical axis shows the ground-truth classes. The horizontal axis shows the predicted classes.
Feature Types
Methods
Mean-class Accuracy ( % )
Single Modality without Dynamic Graph
Only RGB
45.2 %
Only depth
43.7 %
Single Modality with Dynamic Graph
RGB Dynamic Graph
49.7 %
depth Dynamic Graph
48.0 %
Multi-modality Fusion
Mross-modality Simple Fusion
51.3 %
Dynamic Graph (without GNN)
54.2 %
Table 3: Ablation study on SUN RGB-D Dataset.
Feature Types
Methods
Mean-class Accuracy ( % )
Single Modality without Dynamic Graph
Only RGB
58.3 %
Only depth
54.6 %
Single Modality with Dynamic Graph
RGB Dynamic Graph
62.4 %
depth Dynamic Graph
60.5 %
Multi-modality Fusion
Multi-modality Simple Fusion
63.5 %
Dynamic Graph (without GNN)
65.9 %
Table 4: Ablation study on NYU Depth v2 Dataset.
The Number of Nodes
Mean-class Accuracy ( % )
Dynamic Graph
k =4
53.6 %
k =9
55.5 %
k =16
57.7 %
k =25
56.1 %
k =36
54.8 %
Table 5: The effect of number of nodes on SUN RGB-D Dataset.