FrameSeg3D: Learned Local Frames for Segmenting Small Parts on High-Resolution 3D Scans
Organizations: Altum Technology · Purdue University · Rochester Institute of Technology
Abstract
Segmenting small parts on high-resolution 3D scans, such as nails on hand scans and teeth on intraoral scans, poses three challenges: 1) each part covers a tiny fraction of the scan, 2) its boundary is only a subtle change of height and curvature, and 3) it faces its own direction. Existing methods fall into two groups, and both fall short. The first group processes the whole scan at once and sees each part too coarsely. Point and mesh networks must shrink the scan to fit in memory, so a small part keeps only a few of its points. Multi-view methods render images of the whole object, so each part fills only a few pixels and a tilted part is seen from the side. The second group first finds each part and then segments it up close. But these methods find the parts with knowledge about one kind of part, for example the shape of the dental arch for teeth. Such knowledge does not carry over to other kinds of parts. We find that accuracy is limited by how a part is seen, from which direction and at what scale, not by how many points land on it. We therefore present FrameSeg3D. It first predicts a local frame for every part, its center and the direction it faces, from a low-resolution view of the whole scan. It then segments the part in views around its frame normal at the scanner's resolution, so covering the whole scan no longer costs resolution. Because every part arrives centered, facing the camera, and at a fixed scale, one segmentation network can serve different kinds of parts and learn from few labeled masks. We test FrameSeg3D on two kinds of parts, nails and teeth. It outperforms state-of-the-art methods on hand scans (mean Dice 0.970 on nails) and on the Teeth3DS+ test fold (per-tooth IoU 96.5), with the most accurate tooth boundaries (boundary IoU 70.5).
Figures & tables
| Method | Dice | Dice | ASSD | BF | Fail |
|---|---|---|---|---|---|
| Whole scan | |||||
| MeshSegNet [ 29 ] | 0.735 | 0.797 | 1.010 | 0.198 | 22% |
| ToothGroupNet. [ 3 ] | 0.797 | 0.874 | 0.633 | 0.374 | 18% |
| PTv3 [ 56 ] | 0.953 | 0.958 | 0.201 | 0.675 | 0.4% |
| Whole-hand render, 0.25 mm | 0.966 | 0.969 | 0.163 | 0.784 | 0% |
| Whole-hand render, 0.08 mm | 0.956 | 0.968 | 0.168 | 0.777 | 0.4% |
| Test fold | Challenge set | |||||
|---|---|---|---|---|---|---|
| Method | IoU | BIoU | TIR | IoU | BIoU | TIR |
| TSegNet [ 7 ] | 88.42 | 41.97 | 97.87 | 84.05 | 36.71 | 90.83 |
| ToothGroupNetwork [ 3 ] | 95.96 | 69.59 | 98.67 | 92.62 | 66.37 | 93.14 |
| TeethSplineSeg [ 55 ] | 94.86 | 60.26 | 98.26 | 92.32 | 59.60 | 90.72 |
| 3DTeethSAM [ 33 ] | 94.39 | 59.28 | 97.97 | 88.00 | 52.80 | 88.77 |
| ToothInstanceNet [ 53 ] | 95.90 | 67.37 | 98.63 | 96.23 | 68.96 | 95.36 |
| Views around | Centered on | mm/px | Pixels | Dice | BF |
|---|---|---|---|---|---|
| mat normal | whole hand | 0.25 | 1.2k | 0.965 | 0.769 |
| mat normal | whole hand | 0.08 | 11.6k | 0.956 | 0.777 |
| mat normal | frame center | 0.08 | 11.5k | 0.965 | 0.801 |
| frame normal (Ours) | frame center | 0.08 | 12.4k | 0.970 | 0.829 |
| Mean Dice by nail tilt from the mat | ||||||
|---|---|---|---|---|---|---|
| Views around | Centered on | mm/px | 0–15 ∘ (43) | 15–30 ∘ (24) | 30–45 ∘ (7) | (9) |
| mat normal | whole hand | 0.25 | 0.970 | 0.963 | 0.956 | 0.950 |
| mat normal | whole hand | 0.08 | 0.965 | 0.960 | 0.956 | 0.899 |
| mat normal | frame center | 0.08 | 0.972 | 0.966 | 0.964 | 0.927 |
| frame normal (Ours) | frame center | 0.08 | 0.973 | 0.967 | 0.967 | 0.965 |
| Input | mm/px | Dice | Dice | ASSD | BF |
|---|---|---|---|---|---|
| Frame, 1 view, GT | 0.32 | 0.964 | 0.967 | 0.173 | 0.757 |
| 0.16 | 0.968 | 0.971 | 0.150 | 0.812 | |
| 0.08 | 0.969 | 0.971 | 0.151 | 0.811 | |
| Frame, 1 view, predicted | 0.32 | 0.964 | 0.968 | 0.170 | 0.766 |
| 0.16 | 0.968 | 0.970 | 0.157 | 0.797 | |
| 0.08 | 0.969 | 0.971 | 0.152 | 0.807 |
| Nails | Teeth, test fold | Chall. | |||
|---|---|---|---|---|---|
| Segmentation network | Dice | ASSD | IoU | BIoU | IoU |
| Two dedicated networks | 0.970 | 0.142 | 96.46 | 70.34 | 95.73 |
| One network, 24k steps | 0.970 | 0.144 | 96.48 | 70.46 | 95.84 |
| Mean Dice on test nails | ||||
|---|---|---|---|---|
| Training people (of 30) | 1 | 3 | 8 | 30 |
| Ours, segmentation network on the subset | 0.949 | 0.966 | 0.968 | 0.970 |
| Ours, both stages on the subset | 0.772 | 0.943 | 0.967 | 0.970 |
| DGCNN in our frames, segmentation on the subset | 0.935 | 0.956 | 0.964 | 0.966 |
| DGCNN in our frames, both stages on the subset | 0.747 | 0.930 | 0.964 | 0.966 |
| Whole-hand render | 0.888 | 0.959 | 0.961 | 0.964 |
| Per-tooth IoU on the test fold | |||
|---|---|---|---|
| Training patients (of 420) | 5 | 42 | 420 |
| Ours, segmentation network on the subset | 96.38 | 96.41 | 96.48 |
| Ours, both stages on the subset | 96.16 | 96.36 | 96.48 |
| Whole-jaw render, 0.08 mm/px | 92.36 | 94.23 | 94.77 |
| ToothGroupNetwork | 87.18 | 89.40 | 95.96 |
| Nails | Teeth, test fold | |||
| Variant | Dice | BF | IoU | BIoU |
| FrameSeg3D (Ours) | 0.970 | 0.829 | 96.48 | 70.46 |
| No frame: views of the whole scan | 0.956 | 0.777 | 94.77 | 63.87 |
| Views around the scan’s up axis † | 0.965 | 0.801 | 96.38 | 69.67 |
| One view along the frame normal ‡ | 0.968 | 0.794 | 86.63 | 39.68 |
| No re-centering | 0.970 | 0.832 | 96.45 | 70.40 |
| Median / mean Dice | ||
|---|---|---|
| Ground-truth frame perturbed by | One pass | Re-centered |
| none | 0.973 / 0.969 | 0.972 / 0.969 |
| tilt | 0.972 / 0.968 | 0.973 / 0.969 |
| tilt | 0.968 / 0.959 | 0.973 / 0.969 |
| shift 4 mm | 0.972 / 0.969 | 0.973 / 0.969 |
| shift 8 mm | 0.946 / 0.837 | 0.972 / 0.913 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Mean Dice | Faces kept | Smooth noise (mm) | Hole radius (mm) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | Clean | 1/2 | 1/4 | 1/8 | 0.025 | 0.05 | 0.1 | 1 | 2 | 3 |
| FrameSeg3D (Ours) | 0.970 | 0.969 | 0.967 | 0.964 | 0.969 | 0.966 | 0.964 | 0.970 | 0.968 | 0.964 |
| DGCNN in the frame | 0.966 | 0.964 | 0.960 | 0.955 | 0.965 | 0.962 | 0.954 | 0.966 | 0.964 | 0.961 |
| Whole-hand render | 0.966 | 0.961 | 0.957 | 0.952 | 0.964 | 0.962 | 0.891 | 0.965 | 0.964 | 0.960 |
| PTv3, whole hand | 0.953 | 0.950 | 0.918 | 0.461 | 0.952 | 0.948 | 0.932 | 0.951 | 0.950 | 0.943 |
| Method | Time (s) | GPU memory (GB) | Parameters (M) |
|---|---|---|---|
| Hands, per hand (1.6–3.3 million vertices) | |||
| FrameSeg3D (Ours) | 15.5 | 1.3 / 2.6 | 97.8 |
| frame prediction | 2.5 | 0.7 | 73.3 |
| in-frame segmentation, two passes | 12.9 | 1.3 | 24.4 |
| FrameSeg3D (Ours), one view per frame | 9.3 | 0.7 / 1.5 | 97.8 |
| DGCNN in the frame | 6.6 | 7.4 / 28.3 | 74.3 |
| Their metrics | Per ground-truth tooth | |||||
|---|---|---|---|---|---|---|
| Method | IoU | F1 | TIR | IoU | BIoU | BIoU |
| PointNet | 78.38 | 85.89 | 92.63 | 81.12 | 29.26 | 38.32 |
| DGCNN (released checkpoint) | 83.27 | 89.08 | 94.02 | 86.24 | 33.91 | 43.16 |
| PointNet++ | 89.54 | 93.56 | 96.58 | 91.26 | 44.25 | 56.51 |
| TSegNet [ 7 ] | 90.77 | 94.83 | 97.87 | 88.42 | 41.97 | 54.02 |
| Point Transformer | 91.09 | 94.64 | 97.08 | 92.31 | 48.70 | 61.00 |
| Mean Dice | |||||
|---|---|---|---|---|---|
| Method | Thumb | Index | Middle | Ring | Pinky |
| FrameSeg3D (Ours) | 0.967 | 0.969 | 0.975 | 0.974 | 0.963 |
| DGCNN in the frame | 0.964 | 0.962 | 0.974 | 0.970 | 0.960 |
| PTv3 in the frame | 0.962 | 0.939 | 0.975 | 0.972 | 0.963 |
| Whole-hand render, 0.25 mm/px | 0.957 | 0.964 | 0.969 | 0.972 | 0.957 |
| Whole-hand render, 0.08 mm/px | 0.911 | 0.955 | 0.968 | 0.971 | 0.956 |
| Per-tooth IoU | Gain , IoU / BIoU | ||||
|---|---|---|---|---|---|
| Teeth | FrameSeg3D (Ours) | TGN | TGN | whole jaw | |
| tilt – | 735 | 96.77 | 96.37 | +0.40 / +0.6 | +1.72 / +6.4 |
| tilt – | 991 | 96.61 | 96.15 | +0.46 / +0.7 | +1.34 / +6.1 |
| tilt – | 452 | 96.15 | 95.41 | +0.74 / +1.5 | +1.93 / +6.7 |
| tilt | 196 | 95.54 | 94.80 | +0.73 / +1.4 | +2.97 / +9.8 |
| incisors | 719 | 96.00 | 95.71 | +0.28 / +0.6 | +1.84 / +7.5 |
| FrameSeg3D (Ours) | ToothInstanceNet | TGN | ||||||
|---|---|---|---|---|---|---|---|---|
| Teeth | IoU | BIoU | Fail | IoU | BIoU | Fail | Fail | |
| 1–6 | 7048 | 96.52 | 72.94 | 0.3 | 96.34 | 70.27 | 0.2 | 2.4 |
| 7 | 1074 | 95.00 | 65.98 | 2.0 | 96.00 | 62.81 | 0.5 | 10.2 |
| 8 | 152 | 70.00 | 46.22 | 27.6 | 92.67 | 51.79 | 1.3 | 65.8 |
| All | 8274 | 95.84 | 71.55 | 1.1 | 96.23 | 68.96 | 0.3 | 4.6 |
3D Part Segmentation
3D part segmentation divides objects and scenes into meaningful components using geometry, appearance, language, and motion cues.
See all 39 papers