Using Transformers to Generate 3-Manifold Triangulations
Organizations: School of Mathematics and Statistics F07, The University of Sydney, NSW 2006 Australia · Department of Computer Science, Technion, Haifa, 3200003, Israel
Abstract
Using dehydration signatures as an encoding, we train GPT-2-style transformers to generate triangulations of 3-dimensional manifolds. Our main runs each use approximately 3 exaFLOPs, spanning a 200 million parameter model trained on 60 million sphere triangulations to an 11 million parameter model trained on 1.1 billion sphere triangulations. The optimal model has 10 blocks, width 640 and generates dehydration signatures of closed manifolds of various topological types at a rate of 6.9%, 82% of which are spheres. To investigate how our method scales, we use the technique of IsoFLOP curves on 15-tetrahedron triangulations of the 3-sphere, over a compute range of 6 to 3000 petaFLOPs, and observe neural scaling behaviour. We extract a parameter scaling exponent of ( stat, syst), and an optimal ratio of 234 tokens per parameter for training. In addition, we present a number of smaller experiments, including evidence that transformers are able to learn to count degree-one edges near-perfectly, but get progressively worse when counting edges with higher degrees.
Figures & tables
| ( ) | |||
|---|---|---|---|
| Closed | Sphere | Sphere Percent | |
|---|---|---|---|
| 1 | |||
| 2 | |||
| 3 | |||
| 4 | |||
| 5 | |||
| 6 |
| Triangulation Type | Final Loss |
|---|---|
| 0.0129 | |
| Random manifold | 0.0755 |
| Near miss | 0.1331 |
| Random gluing | 0.0088 |
| 0.0033 | |
| 0.0753 |
| Degree | Optimal LR | Minimum loss (test) |
|---|---|---|
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 |
| Signature Length | Optimal LR | Acceptable (%) | Valid (%) | Closed (%) | Sphere (%) | |
|---|---|---|---|---|---|---|
| 10 | 29 | 99.55 | 99.53 | 9.95 | 8.08 | |
| 11 | 31 | 98.32 | 98.32 | 6.2 | 4.89 | |
| 12 | 33 | 96.02 | 96.01 | 4.03 | 3.08 | |
| 13 | 37 | 96.64 | 96.64 | 2.35 | 1.83 | |
| 14 | 39 | 95.87 | 95.86 | 1.80 | 1.25 | |
| 15 | 41 | 91.29 | 91.28 | 1.22 | 0.92 |
| Prime component | Occurrences | |
|---|---|---|
| , | 15,717 | |
| 9,080 | ||
| 15 | ||
| Prism manifold | 5 | |
| Prism manifold | 3 | |
| Prism manifold | 2 |
| Manifold | |
|---|---|
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Target petaFLOPs | |||||||||||
| 6 | 10 | 30 | 60 | 100 | 300 | 600 | 1 000 | 3 000 | |||
| 128 | 2 | 417 280 | 58 | 97 | – | – | – | – | – | – | – |
| 192 | 3 | 1 365 696 | 18 | 30 | 89 | 179 | – | – | – | – | – |
| 256 | 4 | 3 200 512 | 8 | 13 | 38 | 76 | 127 | 381 | – | – | – |
| 320 | 5 | 6 216 640 | 4 | 7 | 20 | 39 | 65 | 196 | 392 | 654 | – |
| 384 | 6 | 10 708 992 | 2 | 4 | 11 | 23 | 38 | 114 | 228 | 380 | 1139 |
| Width | Depth | Aspect Ratio | Parameter Count | Optimal learning rate | Final test loss |
|---|---|---|---|---|---|
| 224 | 10 | 22.4 | 6,086,528 | 1.473 | |
| 256 | 8 | 32 | 6,359,552 | 1.444 | |
| 288 | 6 | 48 | 6,041,088 | 1.386 | |
| 320 | 5 | 64 | 6,216,640 | 1.378 | |
| 352 | 4 | 88 | 6,022,720 | 1.382 | |
| 416 | 3 | 138.7 | 6,313,632 | 1.371 |
| Depth | Parameter count | Number of samples | Optimal learning rate |
|---|---|---|---|
| 2 | 1,620,992 | 1,572,864 | |
| 3 | 2,410,752 | 2,359,296 | |
| 4 | 3,200,512 | 3,145,728 | |
| 5 | 3,990,272 | 3,932,160 | |
| 6 | 4,780,032 | 4,718,592 | |
| 7 | 5,569,792 | 5,505,024 |
Neural Scaling Laws
Neural scaling laws describe how model performance changes with training data, compute, model capacity, and other resources.
See all 91 papers