Training-Free Acceleration of Multi-View 3D Transformers via Regime-Aware Attention Approximation
Organizations: Independent Researcher · Carnegie Mellon University
Abstract
Multi-view 3D transformers such as VGGT reconstruct a scene from a set of unposed images in a single forward pass, predicting camera poses, depth and 3D points, but the cost of their global attention grows quadratically with the num- ber of images. Accelerators borrowed from large language models assume that each token attends to only a few oth- ers; on these models they lose accuracy or gain little speed. We study the global attention of three multi-view transform- ers on seven datasets and find that every layer plays one of three roles: it averages over the whole scene, matches points across many images, or anchors on the first image, whose coordinates the outputs use. Building on this anal- ysis, we propose Regime-AWare ATTention Approximation (RAWATTA), a training-free method that runs attention in 8- bit precision and keeps 16-bit precision only in the layers where a worst-case calibration check requires it. Its fast set- ting, RAWATTA-Fast, also approximates attention in a few averaging and anchoring layers, while matching layers and the first image keep all their keys. On a 12 GB GPU at 50 im- ages, RAWATTA runs VGGT faster than the released code, mostly through our CUDA engine, and loses at most about one point of pose accuracy (AUC@3) on any dataset; RAWATTA-Fast is slightly faster and less accurate. Both fit 200 images, where the released code runs out of memory at 100.
Figures & tables
| ETH3D | CO3D | RE10K | 7-Sc. | NRGBD | TUM-dyn | worst | worst | vs Off. | cal. | |||||
| ms | official | s | exact | |||||||||||
| Exact engine (AUC@3) | 49.91 | 51.21 | 38.49 | 21.11 | 85.46 | 39.66 | 0 | 0 | – | 2237 | 1.61 | 20.74 | 1.00 | |
| RAWATTA (precision only) | 0.23 | 0.15 | 0.19 | 0.01 | 0.08 | 0.22 | 0.11 | 3/3 | 1734 | 2.08 | 12.51 | 1.66 | ||
| RAWATTA-Fast (+ schedule) | 0.34 | 0.07 | 0.22 | 0.56 | 0.49 | 0.12 | 0.17 | 2/3 | 1616 | 2.23 | 10.56 | 1.97 | ||
| threshold schedule | 0.17 | 0.42 | 0.24 | 0.22 | 0.36 | 0.84 | 0.54 | 0/3 | 1544 | 2.33 | 9.39 | 2.21 | ||
| greedy search, budget | 1.03 | 0.10 | 0.45 | 0.73 | 1.48 | 0.16 | 0.50 | – | 1547 | 2.33 | 9.51 | 2.18 | ||
| (ms) | (s) | |||||
| component | Exact | RAWATTA | Fast | Exact | RAWATTA | Fast |
| patch embedding (DINO) | 402 | 402 | 402 | 1.63 | 1.63 | 1.63 |
| frame attention | 28 | 28 | 28 | 0.11 | 0.11 | 0.11 |
| global attn., blocks 0–9, 22 | 400 | 197 | 100 | 6.61 | 3.17 | 1.55 |
| global attn., blocks 10–13, 15, 16 | 300 | 150 | 150 | 4.95 | 2.41 | 2.41 |
| global attn., blocks 17–21, 23 | 248 | 116 | 97 | 4.13 | 1.85 | 1.52 |
| format (accumulation) | worst | worst | mean | ms |
|---|---|---|---|---|
| bf16 (exact engine) | 0 | 0 | 0 | 2237 |
| FP8 (FP32), SageAttention2 | 2.14 | 0.85 | 0.91 | 1799 |
| FP8 (FP16), SageAttention2++ | 0.56 | 1.12 | 0.38 | 1713 |
| FP8 (FP32), row sum of rounded | 0.41 | 0.12 | 0.01 | 1811 |
| 16-bit (FP16) | 0.21 | 0.06 | 0.01 | 1943 |
| RAWATTA : FP8 (FP16), 16-bit on 4, 15 | 0.41 | 0.12 | 0.15 | 1738 |
| s ( ) | worst | worst | s ( ) | ||
| Official VGGT- | 3.61 | 1.00 | 0 | 0 | OOM |
| SparseVGGT † [ 37 ] , default | 3.08 | 1.17 | 1.04 0.26 | 0.51 | OOM |
| CDF 0.90 | 3.06 | 1.18 | 0.74 ∗ | 0.64 | OOM |
| GPA-VGGT † [ 35 ] , default | 2.77 | 1.30 | 3.14 1.08 | 0.89 | OOM |
| on 9–23 | 2.85 | 1.27 | 4.65 ∗ | 1.92 | OOM |
| FastVGGT † [ 31 ] | 2.81 | 1.28 | 21.19 ∗ | 9.17 | OOM |
| schedule | worst | mean | ms |
| RAWATTA (no schedule) | 0 | 0 | 1732 |
| RAWATTA-Fast : frame 0, 4, 5; merge 4 3, 19 | 0.19 | 1608 | |
| (a) merge instead of | 0.09 | 1620 | |
| (b) no bias | 0.17 | 1604 | |
| (c) frame 0 merged in block 19 | 0.00 | 1606 | |
| (d) same operators, random layers (5 draws) | to | 1621 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| paper name | configuration |
|---|---|
| Official VGGT- | official eval/model.py Predictor (fp32 weights, bf16 autocast) |
| Exact engine | dense@bf16 |
| RAWATTA | dense@sagepp~L4=sage.pvh16/L15=sage.pvh16 |
| RAWATTA-Fast (greedy search, one calibration sample, budget ) | L0:frame/L3:merge4/L4:frame/L5:frame/L19:merge4@sagepp~L15=sage.pvh16 |
| threshold schedule | L0:merge4/L1:merge4/L3:merge4/L4:merge4/L5:merge4/L7:merge4/L8:merge4/L19:merge4/L22:merge4@sagepp~L4=sage.pvh16/L15=sage.pvh16 |
| 3-sample search, first run ( Sec. Q.1 ) | L0:frame/L1:merge4/L4:merge4/L5:frame/L19:merge4@sagepp~L4=sage.pvh16/L15=sage.pvh16 |
| VGGT- | VGGT-1B | |||
|---|---|---|---|---|
| statistic (tolerant direction) | frame | merge 4 | frame | merge 4 |
| per-frame entropy (high) | 0.91 | 0.85 | 0.84 | 0.95 |
| frame-0 mass (low) | 0.91 | 0.80 | 0.81 | 0.85 |
| min. head entropy (high) | 0.80 | 0.91 | 0.78 | 0.97 |
| operator’s own output error (low) | 0.87 | 0.72 | 0.70 | 0.67 |
| perturbation | change of merged blocks | worst | ms |
|---|---|---|---|
| none (threshold schedule) | 0, 1, 3, 4, 5, 7, 8, 19, 22 | 0.27 | 1538 |
| 10% | 3, 4, 5, 7, 8 | 0.29 | 1656 |
| 20% | 3, 4, 5, 7, 8, 22 | 0.14 | 1679 |
| 20% | 10, 18 | 1.78 | 1502 |
| 20% | 17 | 0.56 | 1518 |
| 20% | 22 | 0.14 | 1559 |
| GPU | method | s | AUC@3 | |
|---|---|---|---|---|
| 4070 SUPER | Official | 50 / 70 / 90 | 4.3 / 7.0 / 10.2 | 34.7 / 34.2 / 34.9 |
| the exact engine, RAWATTA | 50 | 2.7, 2.0 | 34.6, 34.8 | |
| Blackwell | Official | 50 / 150 / 340 / 810 | 1.0 / 4.9 / 19.6 / 97.5 | 35.1 / 34.0 / 33.5 / 32.3 |
| RAWATTA | 50 / 210 / 460 / 1000 | 0.5 / 4.8 / 19.7 / 85.2 | 34.6 / 33.8 / 32.8 / 31.9 |
| ETH3D | CO3D | RE10K | 7-Sc. | NRGBD | TUM-dyn | worst | worst | |
| Exact engine | 49.91/91.05 | 52.85/91.33 | 38.09/77.79 | 22.63/86.08 | 85.93/98.08 | 40.65/84.88 | 0 | 0 |
| uniform SageAttention2 | 50.09/91.28 | 52.06/91.14 | 37.04/76.95 | 21.44/85.76 | 83.79/97.86 | 40.19/84.03 | ||
| uniform SageAttention2++ | 49.46/90.68 | 52.30/91.22 | 37.76/77.46 | 22.39/85.99 | 85.60/98.02 | 40.28/83.76 | ||
| uniform 16-bit | 50.43/91.11 | 52.74/91.33 | 37.88/77.81 | 22.63/86.10 | 85.76/98.09 | 40.56/84.82 | ||
| RAWATTA | 49.69/91.07 | 52.46/91.29 | 37.99/77.67 | 22.76/86.16 | 85.51/98.02 | 40.74/85.04 | ||
| 16-bit on 15 only | 50.20/91.03 | 52.98/91.37 | 37.74/77.41 | 23.05/86.15 | 85.76/98.09 | 41.20/85.15 |
| 7-Scenes | NRGBD | TUM-dyn | CO3D | RE10K | worst | ||
| 50 | Official VGGT- | 21.5 / 86.2 | 87.1 / 98.1 | 40.1 / 84.0 | 59.4 / 94.0 | 44.8 / 84.9 | 0.09 |
| Exact engine | 21.4 / 86.2 | 86.8 / 98.1 | 39.9 / 84.0 | 59.3 / 94.0 | 44.7 / 84.8 | 0 | |
| RAWATTA | 21.4 / 86.2 | 86.6 / 98.1 | 39.9 / 84.0 | 59.3 / 93.9 | 44.6 / 84.8 | 0.14 | |
| RAWATTA-Fast | 21.0 / 86.0 | 86.1 / 98.1 | 39.8 / 83.6 | 58.9 / 93.9 | 44.4 / 86.0 | 0.14 | |
| threshold schedule | 21.3 / 86.1 | 86.6 / 98.2 | 39.9 / 84.1 | 58.8 / 93.9 | 44.7 / 86.2 | 0.21 | |
| sensitivity-only greedy † | 21.2 / 86.1 | 86.9 / 98.2 | 39.9 / 83.9 | 58.9 / 93.9 | 44.0 / 86.0 | 0.43 |
| dataset | calibration (sequences frames) | evaluation (sequences frames, seeds 0–2) | disjoint |
|---|---|---|---|
| Mip-NeRF 360 | bicycle, garden, stump, kitchen ( ) | – (NVS only, Tab. 20 ) | – |
| CO3Dv2 | banana, donut ( ) | 40 sequences of 8 other categories ( ) | sequence + category |
| RealEstate10K | 30 sequences ( ) | 38 other sequences ( ) | sequence |
| 7-Scenes | first training sequence per scene ( ) | 10 test sequences ( ) | sequence |
| TUM-dynamic | fr3 walking xyz/halfsphere/rpy, sitting xyz validation ( ) | 8 other takes ( ) | sequence |
| ETH3D | – | 13 scenes, official frames | dataset |
| seed 0 (search) | seed 1 | seed 2 | |
|---|---|---|---|
| RAWATTA | / | / | / |
| RAWATTA-Fast | / | / (object, indoor) | / |
| threshold schedule | / (outdoor) | / | / (outdoor) |
| 3-sample, first run ∗ | / | / | / |
| 3-sample, seed-matched | / | / | / |
| calibration | evals | worst eval. | Jaccard |
|---|---|---|---|
| 1 scene 10 | 38 | 0.24 | |
| 2 scenes 10 | 38 | 0.27 | |
| 4 scenes 10 | 47 | 0.36 | |
| 8 seqs 10 (3 datasets) | 44 | 0.40 | |
| 1 scene 50 | 37 | 0.27 | |
| 2 scenes 50 | 41 | 0.31 |
| single | marginal | |||
| layer:op | Cal-out | eval. | Cal-out | eval. |
| 0:merge4 | ||||
| 1:merge4 | ||||
| 3:frame | ||||
| 4:frame | ||||
| 5:frame | ||||
| kernel | ||
|---|---|---|
| ours, INT8/FP8, FP16 acc. | 22.9 (203) | 376.6 (204) |
| ours, INT8/FP8, FP32 acc. | 29.5 (159) | 445.4 (173) |
| SageAttention2 (official, fp32+fp32) | 30.9 (157) | 474.5 (164) |
| SageAttention2++ (official, fp32+fp16) | 27.2 (179) | 414.0 (188) |
| SageAttention 1.0.6 | 37.2 (131) | 580.0 (134) |
| SDPA FlashAttention-2 (bf16) | 69.6 (70) | 1101.3 (71) |
| method | 50 | 100 | 200 | 500 | |
|---|---|---|---|---|---|
| Official VGGT- | 6.49 / 0.571 / 31.1 | 8.75 / 4.32 / 35.1 | OOM | OOM | OOM |
| SparseVGGT | 6.49 / 0.558 / 30.4 | 8.75 / 3.63 / 34.9 | OOM | OOM | OOM |
| GPA-VGGT (1B config) | 6.49 / 0.551 / 30.0 | 8.75 / 3.23 / 34.4 | OOM | OOM | OOM |
| FastVGGT | 6.49 / 0.565 / 28.5 | 8.75 / 3.29 / 33.5 | OOM | OOM | OOM |
| chunked VGGT- (60/30) | = Official | = Official | 9.55 / 17.0 / 29.0 | 9.70 / 34.0 / 28.4 | 10.14 / 90.6 / 24.2 |
| Exact engine | 3.23 / 0.339 / 31.5 | 4.82 / 2.71 / 35.2 | 6.91 / 7.98 / 33.2 | 10.63 / 26.1 / 32.8 | OOM (plan 20.8 GB) |
| 10 | 50 | 100 | 200 | 7S | 7S | 7S | ||
|---|---|---|---|---|---|---|---|---|
| Official | 33.4 (4601) | 112.9 | 826.6 (6923) | 2130.3 | 6274.9 (14550) | 35.14 / 997 | 32.92 / 7851 | 31.97 / 39322 |
| Exact engine | 14.6 (2470) | 67.0 | 585.2 (4614) | 1773.4 | 5953.6 (9830) | 34.73 / 731 | 32.73 / 7608 | 31.89 / 42573 |
| uniform SageAttention2 | 14.6 | 63.0 | 435.2 | 1152.0 | 3442.7 (10278) | 34.16 / 525 | – | – |
| RAWATTA | 14.7 | 62.5 | 443.4 (4742) | 1184.6 | 3563.4 (10406) | 34.56 / 534 | 32.60 / 4470 | 31.88 / 22928 |
| earlier schedule point | 14.6 | 60.8 | 398.5 (4806) | 1002.2 | 2846.0 (10630) | 33.88 / 476 | – | – |
| CO3D | RE10K | 7-Sc. | NRGBD | TUM-dyn | ETH3D | worst | s | ||
|---|---|---|---|---|---|---|---|---|---|
| VGGT-1B | dense bf16 | 59.99 | 15.87 | 11.69 | 78.71 | 28.15 | 30.14 | 0 | 5.85 |
| Sage only | 0.18 | 0.42 | 0.03 | 0.66 | 0.00 | 0.34 | 0.22 | 4.31 | |
| threshold schedule (VGGT- thresholds) | 6.24 | 3.69 | 0.24 | 7.60 | 7.19 | 6.21 | 0.85 | 3.70 | |
| searched | 0.50 | 1.50 | 0.19 | 3.79 | 0.31 | 2.50 | 0.76 | 3.82 | |
| dense bf16 | 61.37 | 43.64 | 11.85 | 79.37 | 20.31 | 36.58 | 0 | 4.88 | |
| Sage only | 0.34 | 0.12 | 0.16 | 0.52 | 0.03 | 0.23 | 0.44 | 3.72 |
| scene | COLMAP + trainer | Exact engine | RAWATTA | RAWATTA-Fast |
|---|---|---|---|---|
| bicycle | 25.49 | 25.43 (20.6) | 25.34 (16.1) | 25.27 (14.0) |
| bonsai | 32.40 | 32.48 (43.0) | 32.55 (29.6) | 32.39 (25.5) |
| counter | 29.13 | 29.00 (43.0) | 28.94 (29.6) | 28.97 (25.5) |
| flowers | 21.65 | 21.85 (17.0) | 21.99 (13.9) | 22.01 (12.4) |
| garden | 27.77 | 27.61 (19.3) | 27.59 (15.2) | 27.55 (15.7) |
| kitchen | 29.32 | 29.37 (43.0) | 28.74 (29.6) | 29.34 (25.6) |
| AUC@3 | AUC@30 | vs Exact | vs Off. | |
|---|---|---|---|---|
| Official VGGT- | 66.13 | 95.13 | 0 | |
| Exact engine | 65.56 | 95.05 | 0 | |
| RAWATTA | 65.62 | 95.06 | ||
| RAWATTA-Fast | 63.91 | 94.82 | ||
| threshold schedule | 65.75 | 95.08 | ||
| sensitivity-only greedy (post hoc) | 65.25 | 95.03 |
| AbsRel | Chamfer | |||
|---|---|---|---|---|
| ETH3D | Official VGGT- | 99.83 | 0.0115 | – |
| Exact engine | 99.83 | 0.0115 | 0.0929 | |
| RAWATTA | 99.83 | 0.0116 | 0.0930 | |
| RAWATTA-Fast | 99.84 | 0.0117 | 0.0979 | |
| 7-Scenes | Official VGGT- | 94.61 0.17 | 0.0520 0.0010 | – |
| Exact engine | 94.61 0.17 | 0.0520 0.0010 | 0.0395 0.0014 |
| session medians (ms) | mean | |
|---|---|---|
| Official VGGT- | 3605.0 | 3605 (1 session) |
| Exact engine | 2237.7, 2237.3, 2234.9 | |
| earlier | 2236.7, 2229.4 | |
| RAWATTA | 1734.9, 1732.7, 1735.0 | |
| earlier | 1735.3, 1729.5 | |
| RAWATTA-Fast | 1616.4, 1612.5, 1606.4 |
| peak MiB ( ) | ||||
|---|---|---|---|---|
| Official VGGT- | 90.5 (22.1) | 167.5 (23.9) | 482.2 (20.7) | 6179 / 6265 / 6571 |
| Exact engine | 52.0 (38.5) | 108.5 (36.9) | 285.4 (35.0) | 2478 / 2766 / 3182 |
| RAWATTA | 51.4 (38.9) | 105.4 (38.0) | 265.5 (37.7) | 2478 / 2766 / 3182 |
| RAWATTA-Fast | 51.3 (39.0) | 104.7 (38.2) | 260.7 (38.4) | 2478 / 2766 / 3182 |
| schedule (frame; merge 4 ) | worst | worst | ms |
| RAWATTA | 0.11 | 0.11 | 1732 |
| RAWATTA-Fast (0, 4, 5; 3, 19) | 0.17 | 0.00 | 1610 |
| non-matching #1 (0, 8, 23; 5, 18) | 2.48 | 1.06 | 1622 |
| non-matching #2 (5, 8, 17; 7, 23) | 1.81 | 2.40 | 1624 |
| non-matching #3 (4, 18, 19; 3, 5) | 1.35 | 0.00 | 1612 |
| non-matching #4 (1, 3, 19; 5, 18) | 1.85 | 0.00 | 1623 |
3D ViTs
3D Vision Transformers model spatial and volumetric data for tasks such as reconstruction, segmentation, detection, and pose estimation.
See all 155 papers