Organize Primitives into Semantic Parts: Reinforcement Reasoning for 3D Segmentation

Xiaoming Gong; Ruoyu Wu; Zhenhong Sun; Chunlin Chen; Daoyi Dong; Huadong Mo; Zhi Wang; Hongdong Li

cs.CVOct 3, 2026

Organize Primitives into Semantic Parts: Reinforcement Reasoning for 3D Segmentation

Authors: Xiaoming Gong, Ruoyu Wu, Zhenhong Sun, Chunlin Chen, Daoyi Dong, Huadong Mo, Zhi Wang, Hongdong Li

Organizations: Nanjing University, China · University of New South Wales, Canberra, Australia · Australian National University, Australia · University of Technology Sydney, Australia

Abstract

Primitive-based 3D segmentation offers a compact and explicit alternative to dense surface prediction, naturally supporting structural abstraction and boundary localization. However, geometric decomposition alone does not determine how primitives should be organized into semantic parts: a single part may span multiple primitives, while geometrically similar or touching primitives may belong to different parts. We therefore introduce RePart (Reinforcement Part Reasoning), which formulates primitive-to-part organization as a finite-horizon Markov decision process and learns semantic organization through trajectory-level reinforcement reasoning. RePart constructs a Composable Primitive Workspace from fine-grained superquadrics and applies a merge-and-stop policy whose decisions are optimized by their downstream effects on the resulting partition rather than local primitive compatibility. The inferred part identities are then mapped back to the original mesh through Boundary-Aware Surface Labeling, preserving accurate surface boundaries beyond the primitive approximation. On PartNet, RePart achieves the strongest results across all four aggregate partition metrics; on 3DCoMPaT++, it obtains the highest RI and SC without target-dataset fine-tuning. These results demonstrate that reinforcement reasoning provides an effective mechanism for organizing geometric primitives into semantic parts while retaining dense segmentation accuracy. Code is available at https://github.com/EngineeringAI-LAB/RePart.

Figures & tables

Figure 1: Primitive segmentation exposes compact structural abstractions, while organizing primitives into semantic parts requires sequential decisions, making reinforcement learning a natural fit.

Figure 2: Overview of RePart. RePart addresses primitive-based 3D segmentation through three coupled stages that bridge dense geometry, part structure, and surface boundaries. (a) Composable Primitive Workspace abstracts the dense surface into compact, composable geometric units for structural reasoning. (b) Reinforcement-Learning Part Reasoning organizes these primitives into coherent part-level structure. (c) Boundary-Aware Surface Labeling localizes the inferred part identities back onto the original surface to recover dense, boundary-aware segmentation.

	PartNet			3DCoMPaT++
Method	Furn.	Cont.	Daily	Furn.	Cont.	Daily
MPS	0.1591	0.5313	0.2169	0.3193	0.1144	0.2008
PartField	0.4098	0.6418	0.4969	0.2438	0.4127	0.4018
P3-SAM	0.3243	0.5663	0.5411	$\mathbf{0.4147}$	0.5110	$\mathbf{0.6003}$
SAMPart3D	0.2520	0.5111	0.4899	0.3313	0.4544	0.4720
PriMAny	0.2255	0.2067	0.3032	0.2419	0.1964	0.3236

Table 1: Category-group mIoU on PartNet and 3DCoMPaT++. RePart uses joint fine-tuning; bold and underlined values denote the best and second-best results.

Table 4

Figure 3: Qualitative comparison on five PartNet shapes (top five rows) and five 3DCoMPaT++ shapes (bottom five rows). RePart reduces the fragmented MPS decomposition and produces coherent part partitions across both datasets.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: PPO training rewards for category-specific training and mixed-category training. Each point is averaged over a non-overlapping window of five epochs.

	PartNet				3DCoMPaT++
Policy	Furn.	Cont.	Daily	Avg.	Furn.	Cont.	Daily	Avg.
Direct mixed	0.4186	0.6314	0.5263	0.5308	0.3558	0.4733	0.4477	0.4236
Staged joint	$\mathbf{0.4247}$	$\mathbf{0.6456}$	$\mathbf{0.5432}$	$\mathbf{0.5460}$	$\mathbf{0.3703}$	$\mathbf{0.5149}$	$\mathbf{0.4607}$	$\mathbf{0.4476}$

Appendix

Table A.1: Category-level mIoU under direct mixed-category and staged joint training. Bold denotes the preferred result.

Parameter	Category-specific	Joint	Role
num_samples	300	900	Number of PartNet training shapes. The joint set is the union of the three category groups.
train_epochs	200	100	Number of outer training epochs. Each epoch collects fresh trajectories before PPO updates.
lr	$1\!\times\!10^{-4}$	$5\!\times\!10^{-5}$	Adam learning rate. The lower joint rate reduces changes to the averaged specialist initialization.
rollout_episodes_per_epoch	304	304	Number of complete merge-and-stop episodes collected in each outer epoch.
rollout_workers	16	16	Parallel workers used to collect trajectories. This changes throughput, not the objective.
ppo_epochs	4	4	Number of optimization passes over each collected rollout buffer.

Appendix

Table A.2: Data, schedule, and PPO settings for category-specific and joint training.

Parameter	Category-specific	Joint	Role
bootstrap_source	MPS	MPS	Uses the MPS decomposition as the initial over-segmented SQ proxy.
point_sample_size	512	512	Number of mesh-surface points sampled whenever the geometry observation is encoded.
merge_neighbor_k	4	4	Restricts each group to merge candidates among its four nearest groups.
sq_label_method	coverage	coverage	Assigns training-time SQ labels using geometric coverage rather than point-wise majority alone.
final_mesh_mapping_mode	atom	atom	Maps grouped SQs by atom voting, SQ volumetric rescue, and a final nearest-surface fallback.
aux_part_loss_w	0.5	0.5	Weight for the critic’s auxiliary prediction of terminal part mIoU.

Appendix

Table A.3: Representation, supervision, reward, and output settings shared by both training stages.

Figure A.2: PPO reward curves on the 300-shape furniture training set. Curves are averaged over every five epochs. The default cold-start PPO setting uses $\kappa=4$ and both group-count penalties.

	mIoU $\uparrow$		$\|G_{T}-K\|\downarrow$
Training setting	PartNet	3DCoMPaT++	PartNet	3DCoMPaT++
Cold-start PPO (default)	$\mathbf{0.4280}$	$\mathbf{0.3699}$	$\mathbf{4.1500}$	$\mathbf{5.9167}$
BC warm start $+$ PPO	0.4165	0.3566	–	–
No group-count penalties	0.4130	0.3558	9.9667	11.0833

Appendix

Table A.4: Furniture results for reward-shaping and behavior-cloning variants. Count error is the mean absolute difference between the terminal group count and the ground-truth part count; lower is better. All models use the same 300 PartNet furniture training shapes.

Seed	mIoU $\uparrow$	RI $\uparrow$	VoI $\downarrow$	SC $\uparrow$
42	0.4280	0.7804	1.2997	0.5751
91161	0.4266	0.7669	1.3310	0.5649
926	0.4311	0.7967	1.3198	0.5794
339	0.4247	0.7240	1.3500	0.5496
32746	0.4253	0.7551	1.3148	0.5646
Mean (variance)	0.4271 ( $6.513\times 10^{-6}$ )	0.7646 ( $7.560\times 10^{-4}$ )	1.3231 ( $3.533\times 10^{-4}$ )	0.5667 ( $1.330\times 10^{-4}$ )

Appendix

Table A.5: Random-seed sensitivity of RePart-S trained on 300 PartNet furniture shapes. The final row reports the mean, with sample variance in parentheses.

Feature group	Dim.	Components	Update and scaling
Point token	$P\!\times\!7$	Raw xyz, centered and radius-normalized xyz, and normalized radial distance.	Points are resampled per state; center and radius are recomputed by the encoder.
SQ token	$M\!\times\!11$	Two exponents, three axis lengths, three Euler angles, and three translations.	Fixed after MPS; inactive slots are zero padded and masked.
Group context	$M\!\times\!7$	Nearest-SQ distance, group-size ratio, group diagonal, group-center radius, group support, candidate degree, and nearest-group AABB gap.	Recomputed after each merge from all members of the current group.
Assignment statistics	$M\!\times\!4$	Normalized assignment entropy, top-1 confidence, low-margin boundary ratio, and top-1–top-2 margin.	Recomputed from soft point-to-SQ assignments.
Shape and fit	$M\!\times\!6$	PCA linearity, planarity, and scattering; SQ–PCA axis alignment; mean and 90th-percentile surface residuals.	Recomputed from assigned points; residuals are divided by scene radius.
Global progress	$4$	Step progress, primitive-slot occupancy, group-slot occupancy, and group count relative to initialization.	Recomputed after each action from normalized counts.

Appendix

Table B.6: Geometry-only state used by the RePart actor. Dimensions use

M

SQ slots,

P

sampled points, and

C

candidate pairs. Subgroups follow their order in the implementation.

State field	Dim.	Components	Use
privileged_prim_stats	$M\!\times\!6$	Point-support ratio, label purity, second-label share, unique-label ratio, normalized label entropy, and fragmentation $(1-\text{purity})$ .	Critic only; recomputed after each merge.
privileged_global_features	$4$	Target part-count ratio, signed and absolute group-count errors, and exact-count indicator.	Critic only; recomputed after each merge.
target_group_count_ratio	$1$	Ground-truth part count divided by $M$ .	Auxiliary target for the actor’s geometry-based count predictor.

Appendix

Table B.7: Label-derived quantities available only during training. They are excluded from the deployed actor observation.

Figure B.3: Grouped feature ablations on 100 PartNet furniture shapes. Curves show PPO reward averaged over every five epochs during the 100-epoch training runs. Each w/o variant zeros only the indicated feature group.

Figure B.4: Asymmetric and symmetric training on 300 PartNet furniture shapes for 200 epochs. Curves show PPO reward averaged over every five epochs. The asymmetric variant uses label-derived critic features and target-count supervision only during training; both actors use geometry-only inputs.

Category	Projection	mIoU $\uparrow$	SC $\uparrow$	VoI $\downarrow$	Final cov.	FB faces $\downarrow$
Daily-use	Facewise nearest SQ	0.4500	0.5574	1.2427	1.0000	–
Daily-use	$+$ Curvature atoms	0.5351	0.6597	0.8608	1.0000	3.7864%
Daily-use	$+$ SQ volumetric evidence	$\mathbf{0.5432}$	$\mathbf{0.6598}$	$\mathbf{0.8607}$	1.0000	$\mathbf{3.1309\%}$
Furniture	Facewise nearest SQ	0.3310	0.4645	1.8139	1.0000	–
Furniture	$+$ Curvature atoms	0.4203	$\mathbf{0.5536}$	$\mathbf{1.3358}$	1.0000	6.8709%
Furniture	$+$ SQ volumetric evidence	$\mathbf{0.4247}$	0.5522	1.3373	1.0000	$\mathbf{0.9656\%}$

Appendix

Table C.8: Ablation of Boundary-Aware Surface Labeling on PartNet. Partition metrics are evaluated on all valid faces after the common fallback. Final coverage is 100% for every variant; FB faces denotes the percentage assigned only by fallback and does not apply to direct facewise nearest assignment. Bold denotes the preferred result.

Method	mIoU $\uparrow$	RI $\uparrow$	VoI $\downarrow$	SC $\uparrow$
MPS	0.2103	0.4460	1.1156	0.4460
PartField	$\mathbf{0.4295}$	$\mathbf{0.7416}$	1.1886	0.5875
P3-SAM	0.3834	0.7112	1.1090	0.5851
SAMPart3D	0.2393	0.5380	1.3913	0.4574
PriMAny	0.2028	0.5793	2.7416	0.2379
RePart (Ours)	0.3992	0.7033	$\mathbf{0.9327}$	$\mathbf{0.5934}$

Appendix

Table D.9: Part segmentation on PartNet categories excluded from training. Results are averaged over the six unseen categories. Bold and underline denote the best and second-best results, respectively.

Category	MPS	PartField	P3-SAM	SAMPart3D	PriMAny	RePart
Bed	0.0789	$\mathbf{0.2908}$	0.2313	0.1526	0.2094	0.2661
Display	0.1980	0.4304	0.4301	0.2888	0.2000	$\mathbf{0.4693}$
Hat	0.4417	0.6135	$\mathbf{0.7073}$	0.3813	0.1860	0.6222
Laptop	0.1662	0.3291	0.3184	0.2097	0.2385	$\mathbf{0.3672}$
Refrigerator	0.1330	$\mathbf{0.3337}$	0.1930	0.1464	0.1289	0.2554
TrashCan	0.2439	$\mathbf{0.5791}$	0.4206	0.2530	0.2539	0.4153

Appendix

Table D.10: Category-level mIoU on PartNet categories excluded from training. Bold and underline denote the best and second-best results, respectively.

Figure D.5: Qualitative comparison on PartNet categories excluded from training.

Figure E.6: Additional qualitative comparisons on PartNet.

Figure E.7: Additional cross-dataset qualitative comparisons on 3DCoMPaT++.

3D Part Segmentation

3D part segmentation divides objects and scenes into meaningful components using geometry, appearance, language, and motion cues.

See all 36 papers

Point2Part: Unified 3D Partitioning from Point Prompts
Sep 29, 2026Hao-Tang Tsui, Yu-Rou Tuan, Xiaoxuan Ma +33D Part Segmentation
Carnegie Mellon University
FlexPart: Text-Controlled Semantics-Aware 3D Part Segmentation at Flexible Granularity
Oct 16, 2026Yiqun Zhao, Jiachen Xu, Yihao Zhi +53D Part Segmentation 3D Referring Segmentation
The University of Hong Kong · Tencent Hunyuan · The Chinese University of Hong Kong, Shenzhen +1
Hi-TOPS: Hierarchical Topology-aware Scoring Prior for 3D Part Decomposition
Aug 1, 2026Ruoyu Wu, Zhenhong Sun, Xiaoming Gong +53D Part Segmentation
University of New South Wales, Australia · Australian National University, Australia · Nanjing University, China +2

Sep 29, 2026J/K move · Enter open · S save

Point2Part: Unified 3D Partitioning from Point Prompts

Hao-Tang Tsui, Yu-Rou Tuan, Xiaoxuan Ma, Nicolas Ugrinovic+2

Carnegie Mellon University

3D Part Segmentation

Open paper AskPDF arXiv

Existing 3D part decomposition methods do not necessarily partition the original shape into non-overlapping parts that collectively cover the entire shape, allowing overlaps or gaps that hinder downstream part-level applications. We instead formulate part decomposition as a joint partitioning of the entire shape, where the predicted parts are non-overlapping and jointly recover the entire shape. Our key insight is that part decomposition should consider all desired parts jointly, rather than modeling each part independently. To this end, we develop a promptable model for 3D part decomposition from images or meshes. Users can specify desired parts through 3D point prompts for controllable decomposition. Given one point prompt per desired part, our model produces the corresponding parts as a complete partition of the entire shape. We build on a pretrained 3D generation model and first obtain a shape latent from either an input image or mesh. We then introduce a prompt encoder that maps each 3D point prompt to a part token while attending to the shape latent. To decode the desired parts, we propose a novel part decoder jointly scoring the entire shape against all part tokens in a coarse-to-fine manner, assigning every position within the shape volume to exactly one part. We perform part decomposition in this shared shape latent space, enabling a unified model for image-to-part generation, mesh-to-part generation, and part segmentation. Our method outperforms existing works on all part-quality metrics across all three tasks, and improves compatibility among parts by an order of magnitude over previous SOTA methods. Code and models will be released.

Abstract

3D Part Segmentation

Explore similar work

Point2Part: Unified 3D Partitioning from Point Prompts

FlexPart: Text-Controlled Semantics-Aware 3D Part Segmentation at Flexible Granularity

Hi-TOPS: Hierarchical Topology-aware Scoring Prior for 3D Part Decomposition