Learning Coordinate-based Convolutional Kernels for Continuous SE(3) Equivariant and Efficient Point Cloud Analysis

Jaein Kim; Hee Bin Yoo; Dong-Sig Han; Byoung-Tak Zhang

Figure 1: On the left, random SE(3) transformations are applied on the identical object point cloud. The black dots are centroids and colored plus signs designate neighbor points, where they are identical points with different poses. If these point clouds are transformed as the reference points are met on the subgroup, or the SO(2) axis, it is guaranteed that points with the identical topology lie on the disjoint orbit, or double coset , around the axis. Therefore, the SE(3) equivariant operation is achievable by utilizing the unique parameter defining those orbits. We visualized this concept with an example drawn from the ModelNet40 [ 42 ] .

Figure 2: The implementation details in ECKConv. (a) The neighbor points within a radius

r

around each centroid are sub-sampled by ball query [ 26 ] . After they are aligned by the inverse of section map as Equation 6 ,

\bar{\beta}_{g}

,

\bar{r}_{g}

, and

\bar{z}_{g}

, which correspond to the double coset parameters, are acquirable as depicted. These neighbor points are actually computed from the ModelNet40 [ 42 ] object with normal vectors. (b) The computation of explicit kernel in ECKConv. First, it maps a double coset parameter

\bar{x}_{i}=[\bar{\beta}_{i},\bar{r}_{i},\bar{z}_{i}]

into Gaussian embedding

[\Psi(\nicefrac{{\bar{\beta}_{i}}}{{\pi}}),\Psi(\bar{r}_{i}),\Psi(\nicefrac{{(\bar{z}_{i}+1)}}{{2}})]

[ 48 , 47 ] . Then the embedding is projected by

F_{\theta}

to

\omega(\bar{x};\theta)

, a coefficient vector with the dimension

A

. The kernel value of

\kappa(\cdot)

is gained from the weighted summation of learnable bases

[\mathbf{W}_{j}]^{A}_{j=1}

by

\omega(\bar{x}_{i};\theta)

, i.e . ,

\sum^{A}_{j=1}\omega_{j}(\bar{x}_{i};\theta)\mathbf{W}_{j}

.

Figure 3: Architectures of ECKConv and its residual connection. We apply batch normalizations and activations (

\sigma

) in the order shown in the figure. As we only visualize abstract flow of the ECKConv block and its residual block, please refer to Section 7.1 in the supplementary material for more details.

Models	Classification Accuracy (%)
Models	$I/I$	$I/\text{SO(3)}$	$\text{SO(3)}/I$	$\text{SO(3)}/\text{SO(3)}$
PointNet++ [ 26 ] †	89.13	9.58	81.16	80.29
PointConv [ 39 ]	92.31	5.29	88.14	87.74
KPConv [ 33 ] †	91.25	14.56	84.84	83.39
DGCNN [ 37 ]	91.41	14.34	88.24	87.97
PCT [ 15 ] †	91.25	15.84	86.39	84.20

Table 1: Classification accuracy on the ModelNet40 dataset. Since PRLC [ 22 ] requires the prior that input has an aligned orientation, it is impossible to apply SO(3) augmentation during its training. † denotes the results reported from previous works [ 49 , 20 ] .

Models (Batch Size = 12)	Memory (GB) $\downarrow$		$\#\text{Batch}/\text{sec}$ $\uparrow$		#params
Models (Batch Size = 12)	Train	Eval	Train	Eval	#params
VN [ 9 ]	6.10	1.98	4.34	8.70	2.9M
EPN [ 4 ]	13.40	6.34	2.11	3.36	3.2M
E2PN [ 49 ]	3.94	2.41	8.94	18.97	2.7M
ECKConv	4.89	1.28	7.74	15.09	27.7M
CSEConv [ 20 ]	2.95	0.44	30.71	59.77	2.1M

Table 2: Cost table of the equivariant classification baselines.

Models	Mean ( ∘ )	Median ( ∘ )	Max ( ∘ )	tRMSE
KPConv [ 33 ]	$128.57\pm 0.97$	$123.22\pm 0.52$	$180.00\pm 0.00$	-
DGCNN + DCP-v1 [ 36 ]	$121.29\pm 0.90$	$137.19\pm 1.95$	$179.60\pm 0.00$	0.081
DGCNN + DCP-v2 [ 36 ]	$120.07\pm 1.28$	$132.54\pm 1.94$	$179.60\pm 0.00$	0.072
w/ SO(3) Aug	$121.61\pm 1.58$	$137.62\pm 1.85$	$179.60\pm 0.00$	0.075
EPN [ 4 ]	$1.62\pm 0.02$	$2.40\pm 0.08$	$110.14\pm 45.53$	-
E2PN [ 49 ]	$11.70\pm 0.25$	$13.71\pm 0.19$	$163.62\pm 15.70$	-

Table 3: Pose registration results on the ModelNet40. We reproduced the methods by Chen et al. [4] and Zhu et al. [49] for KPConv, EPN, and E2PN, which do not estimate the translation difference. Models were evaluated 10 times with different seeds.

Models	$I/I$		$I/\text{SO(3)}$		$\text{SO(3)}/\text{SO(3)}$
Models	mIoU	mcIoU	mIoU	mcIoU	mIoU	mcIoU
PointNet++ [ 26 ]	85.56	82.24	28.97	32.95	82.10	78.02
DGCNN [ 37 ]	85.13	85.41	31.48	32.19	77.60	78.24
VN [ 9 ]	81.74	82.06	81.73	82.05	81.76	82.10
LC [ 19 ]	81.04	81.43	81.07	81.45	81.05	81.53
PRLC [ 22 ]	79.38	79.90	79.39	79.91	-	-

Table 4: Part segmentation on the ShapeNet. mIoU denotes the IoU per object averaged among every instance, and mcIoU denotes that IoUs are averaged per part class before the total mean.

Figure 4: Visualization of the part segmentation in the ShapeNet by ECKConv (Ours), Vector Neurons [ 9 ] , and CSEConv [ 20 ] .

Figure 5: Semantic segmentation results by ECKConv on the Area 5 of S3DIS [ 1 ] . The ceiling and wall points that obscure the view point are omitted in the figures for the better visualization. (Left) The input, ground truth label, and prediction results of office 28 , lobby 1 , and conference room 2 in the Area 5. Various indoor objects, e.g . , bookcase, table, chair, and sofa, are successfully segmented from the scenes by our method. (Right) The simulation by randomly placing chair objects from ModelNet40 [ 42 ] in the hallway 8 of Area 5. Even though the chair objects are out-of-distribution from the S3DIS, our method segments every object from the simulated scenes.

Models	OA	mAcc	mIoU

DGCNN [ 37 ] w/ SO(3) Aug	74.78	45.06	35.78
KPConv [ 33 ] w/ SO(3) Aug	84.50	64.66	57.42
RIConv++ [ 44 ]	84.68	66.41	56.94
RI-MAE [ 32 ] †	-	-	60.3
ECKConv-Normal	86.38	70.38	61.80

Table 5: Semantic segmentation results on the Area 5 of S3DIS. † denotes the result reported in RI-MAE [ 32 ] .

Figure 6: Detailed architectures of ECKConv block. We substitute abstract variables from Figure 3 with actual computational variables.

	Models	Memory (GB) $\downarrow$	#Batch/sec $\uparrow$	batch size	#params
pose reg	CSEConv	39.09	3.37	16	10.8M
pose reg	ECKConv	5.37	7.39	16	4.1M
part seg	CSEConv	66.46	2.04	6	14.9M
	ECKConv	10.26	3.80	16	22.5M

Table 6: Memory and time costs during the training, compared between ECKConv and CSEConv baselines from the pose registration and part segmentation tasks.

Figure 7: Architectures of ECKCov models for ModelNet40 [ 42 ] experiments. Since point clouds only contain 3D coordinates, we initialize its point feature as one vector per every point. An arrow between ECKConv layers designates the number of points and the dimension of each feature on its above and below. We also specify ECKConv hyperparameters below, such as the number of anchor points (

A

), a ball query radius (

r

), the maximal number of sampled points (

k

), the standard deviation of Gaussian kernel (

\sigma

), and the basis dimension of Gaussian embedding (

\Psi

) [ 48 , 47 ] . Linear layers map every point feature with weight-sharing networks, and we omit batch normalization and activation next to these layers in the above figures.

Figure 8: Architectures of ECKConv models fors segmentation tasks in the ShapeNet and S3DIS. We applied U-Net [ 31 ] style architecture and up-sampled point features using Point Feature Propagation [ 26 ] module, which assigns coarse point features to its top-K closest neighbors skipped from previous layers, weighted by the reciprocal of the point distance in the coordinate space.

Figure 9: Qualitative results in S3DIS simulation experiments. (top) We diversified the objects from the chair category of ModelNet40. Our model consistently succeeded to discern chair objects with different shapes and poses. (bottom) We diversified the object category, adding the table object from ModelNet40. Our model showed consistent results on segmenting the table objects regardless of their numbers or poses, although noisy results on the bottom part of tables were also observed.

Ablation Studies	#Anchor Bases		$\text{Gau}(\cdot)$ Bandwidth		#Neighbors for Alg 1.
Ablation Studies	$A=1$	$A=64$	$\Psi\in\mathbb{R}^{4}$	$\Psi\in\mathbb{R}^{256}$	$K=8$	$K=128$
Acc (%)	89.22	90.19	88.86	90.32	90.32	89.95
Mem (GB) $\downarrow$	4.36	6.57	1.90	14.51	4.90	4.90
$\#\text{Batch}/\text{sec}$ $\uparrow$	10.20	5.62	10.65	2.69	7.58	6.83

Table 7: Ablation experiments on the hyperparamters of ECKConv in the ModelNet40 classification task, where models received randomly rotated inputs only during the evaluation (

I/\text{SO(3)}

). Default values are

A=22

,

\Psi\in\mathbb{R}^{64}

, and

K=32

.

Learning Coordinate-based Convolutional Kernels for Continuous SE(3) Equivariant and Efficient Point Cloud Analysis

Abstract

Figures & tables

Explore similar work

SHReg: Strictly Rotation-Equivariant Point Cloud Registration via Spherical Harmonics

Efficient RWKV-based Representation Learning for 3D Point Clouds

Bases of Steerable Kernels for Equivariant CNNs: From 2D Rotations to the Lorentz Group