Dynamic Graph Neural Network with Adaptive Features Selection for RGB-D Based Indoor Scene Recognition
Authors: Qiong Liu, Ruofei Xiong, Xingzhen Chen, Muyao Peng, You Yang
Organizations: School of Electronic Information and Communications, Huazhong University of Science and Technology, No. 1037, Luoyu Rd., Wuhan, 430074, P. R. China
Multi-modality of color and depth, i.e., RGB-D, is of great importance in recent research of indoor scene recognition. In this kind of data representation, depth map is able to describe the 3D structure of scenes and geometric relations among objects. Previous works showed that local features of both modalities are vital for promotion of recognition accuracy. However, the problem of adaptive selection and effective exploitation on these key local features remains open in this field. In this paper, a dynamic graph model is proposed with adaptive node selection mechanism to solve the above problem. In this model, a dynamic graph is built up to model the relations among objects and scene, and a method of adaptive node selection is proposed to take key local features from both modalities of RGB and depth for graph modeling. After that, these nodes are grouped by three different levels, representing near or far relations among objects. Moreover, the graph model is updated dynamically according to attention weights. Finally, the updated and optimized features of RGB and depth modalities are fused together for indoor scene recognition. Experiments are performed on public datasets including SUN RGB-D and NYU Depth v2. Extensive results demonstrate that our method has superior performance when comparing to state-of-the-arts methods, and show that the proposed method is able to exploit crucial local features from both modalities of RGB and depth.
Figures & tables
Figure 1: The framework of the proposed dynamic graph. First, RGB and depth images are input to two CNNs for feature extraction. Then, ANS module is used to select nodes to construct the multi-modality dynamic graph. In the last step, global and local features learned by the dynamic graph model are combined together for the final scene recognition.
Figure 2: The selected key local features (i.e., nodes) for single modality (e.g., RGB) graph construction. The contribution of each node is different from the task of scene recognition, and it should be evaluated properly in graph modeling. In our dynamic graph model, the importance of each node is represented by its value in the computed attention map. By taking k = 16 as an example, nodes are organized by 3 levels, including 1 main-central (i.e., in red), 3 sub-central (i.e., in green) and 12 leaf nodes (i.e., in blue), in the dynamic graph model. The three sub-central nodes { R2 , R3 , R4 } are connected to the main-central node R1 and the rest of the leaf nodes are connected to the sub-central node by euclidean distance. Similar process is performed for depth dynamic graph construction.
Figure 3: Connection between two modalities. Considering the semantic gap between the two modalities, a sparse connection is chosen which means main-central nodes, R1 and D1 , sub-central nodes, { R2 , R3 , R4 } and { D2 , D3 , D4 } are connected separately.
Methods
Mean-class Accuracy ( % )
State-of-the-arts
Wang et al . Wang et al. (2016)
48.1 %
Song et al . Song et al. (2017a)
54.0 %
Li et al . Li et al. (2018b)
54.6 %
Yuan et al . Yuan et al. (2019)
55.1 %
Li et al . Li et al. (2019)
56.2 %
Du et al . Du et al. (2019)
56.7 %
Table 1: Performance comparison with the state-of-the-art methods on SUN RGB-D Dataset.
Figure 4: The classification confusion matrix of the proposed dynamic graph model on the SUN RGB-D Dataset. The vertical axis shows the ground-truth classes, the horizontal axis shows the predicted classes. The classes on the horizontal axis are in the same order as those on the vertical axis.
Methods
Mean-class Accuracy ( % )
State-of-the-arts
Wang et al . Wang et al. (2016)
63.9 %
Song et al . Song et al. (2017a)
66.9 %
Li et al . Li et al. (2018b)
65.4 %
Yuan et al . Yuan et al. (2019)
67.2 %
Li et al . Li et al. (2019)
67.7 %
Du et al . Du et al. (2019)
69.2 %
Table 2: Performance comparison with the state-of-the-art methods on NYU Depth v2 Dataset
Figure 5: The classification confusion matrix of proposed dynamic graph model on the NYU Depth v2 Dataset. The vertical axis shows the ground-truth classes. The horizontal axis shows the predicted classes.
Feature Types
Methods
Mean-class Accuracy ( % )
Single Modality without Dynamic Graph
Only RGB
45.2 %
Only depth
43.7 %
Single Modality with Dynamic Graph
RGB Dynamic Graph
49.7 %
depth Dynamic Graph
48.0 %
Multi-modality Fusion
Mross-modality Simple Fusion
51.3 %
Dynamic Graph (without GNN)
54.2 %
Table 3: Ablation study on SUN RGB-D Dataset.
Feature Types
Methods
Mean-class Accuracy ( % )
Single Modality without Dynamic Graph
Only RGB
58.3 %
Only depth
54.6 %
Single Modality with Dynamic Graph
RGB Dynamic Graph
62.4 %
depth Dynamic Graph
60.5 %
Multi-modality Fusion
Multi-modality Simple Fusion
63.5 %
Dynamic Graph (without GNN)
65.9 %
Table 4: Ablation study on NYU Depth v2 Dataset.
The Number of Nodes
Mean-class Accuracy ( % )
Dynamic Graph
k =4
53.6 %
k =9
55.5 %
k =16
57.7 %
k =25
56.1 %
k =36
54.8 %
Table 5: The effect of number of nodes on SUN RGB-D Dataset.
In indoor environments, object positions frequently change due to human activities or embodied-agent interactions, causing previously constructed scene graphs to become inconsistent with the current scene. To address this issue, we propose DSG, a dynamic 3D scene graph construction framework that detects object changes and performs spatial relationship reasoning. First, we construct a semantic-aware 3D Gaussian scene representation and develop a dual-view rendering-based object change detection method to enable reliable scene graph node updates. Second, we propose a spatial relationship reasoning method that incorporates multi-granularity visual context, enabling a large language model to identify a richer set of interobject spatial relationships. Furthermore, we introduce DynTHOR, a dynamic indoor scene graph benchmark built on the AI2-THOR simulation platform for evaluating scene graph construction in dynamic environments. Extensive experiments on Dyn-THOR, 3RScan, and real-world scenes demonstrate that DSG consistently outperforms existing methods in both object node construction and spatial relationship reasoning, significantly improving the accuracy of dynamic scene graph construction.
Ming Liao, Chao Ye, Jianing Fei +1
Research Institute of Intelligent Control and Systems, Harbin Institute of Technology, Harbin 150001, China
Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as LiDAR or RGB-D cameras, for metric 3D reconstruction. This limits deployment to specialized robotic platforms and excludes settings where only RGB cameras are available, such as fixed external infrastructure. Existing pipelines also typically operate on passively collected observation trajectories, rather than selecting viewpoints based on the partially built scene representation, and therefore fail to effectively exploit the semantic and spatial information encoded within the graph during exploration. This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only, addressing both limitations. The proposed approach unifies perception and planning around a shared structured representation that captures object semantics, 3D geometry, relational context, and information from multiple viewpoints. Because the framework is hardware-agnostic and relies only on RGB observations, it can incorporate inputs from both onboard robot cameras and fixed external cameras within the same representation. Experiments on the Replica dataset show that the RGB-only pipeline achieves F1-score parity with baselines using ground-truth depth. Active exploration experiments on ReplicaCAD further show that semantic-driven viewpoint selection detects more than twice as many objects as a geometric frontier-based baseline under the same exploration budget. Finally, the external-camera setting demonstrates that complementary RGB views can effectively bootstrap the scene graph and improve contextual understanding at no additional exploration cost.
Giorgia Modi, Davide Buoso, Giuseppe Averta +1
Mobile Robotics Group (MRG), University of Oxford, UK · Visual and Multimodal Applied Learning Lab (VANDAL), Politecnico di Torino, Italy
Recommender systems are essential components of modern online platforms which presents personalized content in various domain. The traditional collaborative filtering methods depends on static user-item interaction graphs and a limited subset of similarity measures which fail to capture the changing nature of preferences of an individual. Recent graph neural network (GNN) based approaches focus on user-item bipartite graphs which do not use explicit user-user relational modelling and dynamic graph evolution during training. To address these limitations, this paper proposes a Dynamic Graph SimilarityAware Attention Graph Neural Network (DG-SA-GNN) framework that integrates dynamic user similarity graph construction with multi-similarity propagation and attention-based aggregation. The proposed architecture constructs four parallel user similarity graphs using Cosine, Jaccard, Discounted Pearson Correlation Coefficient (Discount PCC), and IPIJ similarity functions, each processed by a dedicated UserGNN module. A Graph Transformer fuses the four graph views, and a CrossAttention module refines user embeddings through interaction with item embeddings. Crucially, the graphs are reconstructed at scheduled epochs during training, enabling the model to adapt to the learned embedding space constituting the dynamic graph component. Mini-batch training with hard negative sampling improves scalability and convergence. Experiments on the MovieLens100K benchmark demonstrate that DG-SA-GNN achieves a Recall@20 of 0.162 and NDCG@20 of 0.065 which is better than the LightGCN baseline in recall. The results validate that dynamic multi-similarity graph construction coupled with attention-based fusion which produce recommendation performance
Aadarsh Senapati, Neha Kujur, Vivek Yelleti
Department of Computer Science and Engineering, SRM University Andhra Pradesh, Mangalagiri, Andhra Pradesh 522502, India