Organizations: EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · The Hong Kong Polytechnic University · Munich Center for Machine Learning, LMU Munich
Multi-view representations enable pretrained image and video vision-language models to understand 3D scenes, but their long visual token sequences make inference costly. Redundancy arises both from repeated observations of the same scene regions across views and from processing all visual tokens at every LLM layer, regardless of their relevance to the current computation. We propose GeoGR, a two-stage framework that jointly compresses scene representations and visual computation. At the projector level, GeoSemZip consolidates cross-view observations into 3D voxel tokens and combines semantic saliency with spatial coverage to construct a compact scene representation. Unselected voxel features are aggregated into contextual anchors to retain scene information that would otherwise be discarded. At the LLM level, we identify an effective visual computation window and observe that query-conditioned token relevance remains locally stable but changes over longer layer intervals. Guided by these findings, GroupRoute combines late entry and early exit with recoverable group-wise skipping. Low-relevance tokens bypass complete LLM layers while retaining their hidden states and can rejoin the active group at subsequent anchor layers selected under a computation budget. For compression-aware post-training, we construct a compact dataset of 67K samples by retaining 30% of the original training pool. With this protocol, GeoGR retains 95.2% and 96.8% of the dense post-trained models' performance on LLaVA-OneVision-7B and Video-3D LLM-7B, respectively, while accelerating LLM prefill by 6.8× and 6.1×. These results demonstrate the consistent effectiveness of GeoGR across 3D VLMs with and without explicit 3D spatial encoding, highlighting a favorable performance-efficiency trade-off. The code is available at https://github.com/EIT-NLP/GeoGR.
Figures & tables
Figure 1: Redundancy in multi-view 3D VLMs. (a) Overlapping views repeatedly observe the same 3D region (blue boxes). (b) For the same question, visual token importance remains locally stable but shifts across more distant layers. See Section 3 for quantitative analysis.
Figure 2: Cross-layer stability of visual token rankings. Ranking stability decreases with layer distance, while initially low-ranked tokens increasingly enter the top half and account for a growing share of its attention.
Source
Full Set
Selected
ScanRefer
36,665
11,000
Multi3DRefer
43,838
13,151
Scan2Cap
36,665
11,000
ScanQA
26,515
7,955
SQA3D
79,445
23,834
Total
223,128
66,940
Table 1: Our post-training dataset distilled by selecting 30% of samples from each source using PRISM (top), and performance retained relative to full-data post-training (bottom).
Figure 3: Overview of GeoGR . (1) Projector-level Compression: GeoSemZip consolidates cross-view observations into 3D voxel tokens using depth and camera poses. Saliency-based selection preserves dominant tokens, while Morton-order sampling selects contextual anchors that aggregate unselected voxel features by cosine similarity. The compressed visual tokens are combined with text tokens to form the LLM input. (2) LLM-level Compression: GroupRoute confines visual computation to an intermediate-layer window through Late Entry and Early Exit. Within this window, Recoverable Group-wise Skipping routes query-conditioned token groups through or around complete LLM layers. Skipped tokens retain their hidden states and can resume computation after regrouping at subsequent anchor layers. Text tokens remain active throughout the LLM.
Figure 4: Visual-window diagnosis for LLaVA-OneVision on 256 ScanQA examples: Late Entry (left) and Early Exit (right). Lines show mean next-token KL divergence relative to full visual computation; shading indicates ±1 standard deviation, with the lower bound clipped at zero.
Table 2: Performance on LLaVA-OneVision-7B and Video-3D LLM-7B at approximately 10% retained LLM-side visual FLOPs. LLaVA-OneVision is evaluated only on ScanQA and SQA3D, as it does not support the 3D localization outputs required by the other benchmarks. “Avg.” measures performance retention relative to the dense model. “Tokens” denotes the average number of active visual tokens per LLM layer. Unshaded baseline rows apply compression without recovery, while light-red rows use compression-aware post-training. Light-blue rows denote GeoGR. † marks baselines originally proposed without recovery and evaluated with it here. ∗ marks compression baselines originally designed for 3D inputs, while the remaining baselines were designed for 2D inputs. Bold and underlined values indicate the best and second-best results among compressed methods, respectively. GeoGR achieves the highest average performance retention on both 3D VLMs at comparable computational budgets.
Method
KV Cache (MB)
Latency (ms)
Avg. (%)
Projector
LLM Prefill
TTFT
LLaVA-OneVision-7B
Dense
359.7
53.3
1021.4 ( 1.0× )
1426.7 ( 1.0× )
100.0
SeGPruner †
36.7
100.8
103.0 ( 9.9× )
548.5 ( 2.6× )
91.7
DTC †
40.3
2025.0
147.2 ( 6.9× )
2522.2 ( 0.6× )
92.6
GeoGR
36.6
99.0
151.2 ( 6.8× )
605.0 ( 2.4× )
95.2
Table 3: Efficiency comparison on two 3D VLMs. Parentheses indicate speedups relative to the dense model. “Projector” covers all operations between the vision encoder and the LLM, including token compression. Encoder latency is similar across methods and is omitted from the breakdown but included in TTFT. GeoGR offers a favorable trade-off between performance retention and inference efficiency on both 3D VLMs.
Semantic
Spatial
Residual
ScanQA
SQA3D
Avg.
Saliency
Coverage
Merge
CIDEr
EM
EM
✓
85.97
26.16
55.24
93.53%
✓
92.04
26.50
56.81
96.99%
✓
✓
92.67
26.95
57.57
98.19%
✓
✓
✓
93.95
27.51
57.63
99.33%
Table 4: Component ablation of GeoSemZip. All variants retain 30% of the visual tokens and are post-trained independently from the same initialization. The semantic-only variant omits cross-view voxel consolidation. Residual merging requires contextual anchors from the spatial coverage branch.
Grouping
ScanQA
SQA3D
Avg.
Signal
CIDEr
EM
EM
Projector-derived
85.34
25.30
55.98
92.73%
Query-conditioned
88.25
26.20
56.61
95.18%
Table 5: Grouping signal with anchor layers 9 and 13 and κ=0.5 . Both variants share the same checkpoint, schedule, and decoding protocol and differ only in the ranking used to form the groups.
Method
Anchors
ScanQA
SQA3D
Avg.
CIDEr
EM
EM
Pruning
9
87.66
25.97
56.52
94.65%
13
87.20
26.07
57.23
95.03%
9, 13
87.82
25.88
56.75
94.73%
Skipping
9, 13
88.25
26.20
56.61
95.18%
Table 6: Skipping vs. pruning within the window [8,23] . All variants use the same window-adapted checkpoint without further training and average 638 active visual tokens per LLM layer.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Blocks
Active sequence
Event
1–7
Text only
Visual tokens not yet injected
8
All
Visual tokens enter
9
All
Score and group at anchor 9
10–12
Text, H9
L9 bypasses
13
All
Anchor 13; score and regroup
14–23
Text, H13
L13 bypasses
Appendix
Table 7: GroupRoute execution schedule on LLaVA-OneVision.
Component
LLaVA-OneVision-7B
Video-3D LLM-7B
Pretrained VLM family
LLaVA-OneVision
LLaVA-Video
Vision encoder
SigLIP SO400M/14
SigLIP SO400M/14
Encoder input resolution
384×384
384×384
Visual feature dimension
1,152
1,152
Multimodal projector
Two-layer MLP, GELU
Two-layer MLP, GELU
LLM
Qwen2-7B
Qwen2-7B
Appendix
Table 8: Architectures and scene inputs of the evaluated 3D VLMs. Token counts use the 32-view setting and exclude formatting tokens. Q/KV denotes query and key/value attention heads.
Table 9: Training sources and evaluation splits for five ScanNet-based benchmarks. Train counts examples before PRISM selection; Eval counts held-out examples. Evaluation and scene counts follow the Video-3D LLM annotations [ zheng2024video3dllm ] .
Training data
ScanQA CIDEr
ScanQA EM
SQA3D EM
Retention (%)
Full set (223,128)
100.27
29.07
59.53
100.00
PRISM 30% (66,940)
94.84
28.09
57.06
95.69
Appendix
Table 10: Dense post-training on full data and PRISM-selected data. PRISM retains approximately 30% from each source; retention averages the metric-wise ratios to the full-data post-trained checkpoint.
Budget
Vis. tokens
Avg. (%)
≈20%
1255.00
97.26
≈30%
1881.73
99.33
≈40%
2475.47
98.38
Appendix
Table 11: GeoSemZip budget selection on LLaVA-OneVision. Each budget uses a separately post-trained checkpoint. Vis. tokens is the ScanQA mean entering the LLM; Avg. is retention relative to Dense.
Voxel size s
K
Keep ratio
ScanQA
SQA3D
Avg.
CIDEr
EM
EM
0.05
4778.6
30%
92.10
27.40
57.26
98.34%
0.10
3574.0
30%
93.95
27.51
57.63
99.33%
0.15
2520.8
30%
91.31
27.42
57.63
98.30%
Appendix
Table 13: GeoSemZip voxel-size ablation at a 30% token budget. LLaVA-OneVision variants are post-trained independently; K is the mean number of occupied voxels per scene.
Aggregation
ScanQA
SQA3D
Avg. (%)
CIDEr
EM
EM
Mean
93.15
27.76
57.12
99.06
Max (default)
93.95
27.51
57.63
99.33
Appendix
Table 14: Voxel-score aggregation for GeoSemZip at a 30% token budget. LLaVA-OneVision variants are post-trained independently and differ only in the within-voxel score-pooling rule.
Dominant ratio ρ
ScanQA
SQA3D
Avg. (%)
CIDEr
EM
EM
0.65
91.46
27.40
57.46
98.23
0.75
92.89
27.66
57.37
98.99
0.85 (default)
93.95
27.51
57.63
99.33
0.95
92.36
27.61
57.00
98.52
Appendix
Table 15: GeoSemZip budget-split ablation at a 30% token budget. On LLaVA-OneVision, independently post-trained variants differ in ρ , the fraction assigned to dominant tokens; residual merging remains enabled.
Window
ScanQA
SQA3D
Avg.
CIDEr
EM
EM
Step 1: late entry (visual tokens through block 28)
[5,28]
87.60
25.56
56.27
93.99%
[6,28]
87.87
25.13
56.15
93.51%
[7,28]
85.37
25.41
55.61
92.65%
[8,28]
86.61
25.35
56.49
93.53%
Appendix
Table 16: Visual-computation window selection on LLaVA-OneVision. We first vary late entry with visual tokens retained through block 28, then vary early exit with entry fixed at block 8. Avg. is relative to the dense reference in Table 2 .
Keep ratios (9 / 13)
Visual tokens
ScanQA
SQA3D
Avg.
CIDEr
EM
EM
0.75 / 0.25
520.79
87.28
25.99
56.38
94.46%
0.50 / 0.50 (default)
638.38
88.25
26.20
56.61
95.18%
0.25 / 0.75
755.98
87.77
26.07
56.07
94.55%
Appendix
Table 17: Keep-ratio allocation at GroupRoute anchors 9 and 13. On LLaVA-OneVision with window [8,23] , “Visual tokens” is the mean active count per LLM layer; Avg. is relative to the dense model.
Figure 5: Layer-pair stability of visual-token relevance on ScanQA. For LLaVA-OneVision decoder layers 8–23, (a) gives Spearman correlation between complete visual-token rankings, and (b) gives the fraction shared by the two layers’ top-200 sets.
Configuration
Proj. keep
Vis. tokens
ScanQA
SQA3D
Avg. (%)
CIDEr
EM
EM
GeoSemZip only
10%
628
87.44
25.41
56.69
94.00
GroupRoute only
100%
2128
88.48
26.07
55.84
94.65
GeoGR (both)
30%
638
88.25
26.20
56.61
95.18
Appendix
Table 18: Projector-only, LLM-only, and two-stage compression on LLaVA-OneVision. Vis. tokens is the mean active count per decoder block; Avg. is retention relative to the dense model.
3D vision-language models (3D VLMs) enable spatial reasoning over multi-view scenes but suffer from substantial token redundancy due to duplicated observations and large uninformative regions, leading to high computational cost. Although visual token compression has shown promise in accelerating 2D VLMs, it fails to capture the structured nature of 3D scenes and leads to incomplete spatial coverage and loss of fine-grained details. In this paper, we propose HiSC, a training-free framework for hierarchical spatial clustering token compression in 3D VLMs. HiSC lifts token compression from token-level selection to cluster-level processing by organizing tokens into spatially grounded clusters using joint geometric and semantic cues. Specifically, we first introduce a spatial graph-based merging (SGraM) strategy that models cross-view redundancy as spatial connectivity and consolidates physically consistent regions, effectively merging extremely similar redundant tokens prior to LLM inference. We then propose a spatial clustering-based pruning (SCluP) paradigm within LLM inference, which performs hierarchical compression across clusters and within clusters, preserving object instance completeness while retaining fine-grained details for important regions. Extensive experiments on diverse 3D reasoning benchmarks show validate the effectiveness of HiSC, particularly under high visual token pruning ratios. Besides, HiSC achieves over 90% token reduction with minimal performance degradation. Code is accessible at https://github.com/elecreak/HiSC.
Multimodal large language models have demonstrated remarkable capabilities in 2D vision, motivating their extension to 3D scene understanding. Recent studies represent 3D scenes as 3D spatial videos composed of image sequences with depth and camera pose information, enabling pre-trained video-language models to perform 3D reasoning tasks. However, the large number of visual tokens in spatial videos remains a major bottleneck for efficient inference and context management. Existing pruning methods overlook the view consistency of spatial videos and the spatial diversity of the remaining tokens, which prevents them from effectively removing inter-frame redundancy and preserving scene completeness. In this paper, we propose Geo3DPruner, a Geometry-Guided 3D Visual Token Pruning framework. Geo3DPruner first models cross-frame relevance through geometry-aware global attention, and then performs a two-stage pruning process. The intra-voxel stage selects representative multi-view features within each voxel, while the inter-voxel stage preserves spatial diversity by selecting a globally distributed subset of voxels. Extensive experiments on multiple 3D scene understanding benchmarks demonstrate that Geo3DPruner retains over 90% of the original performance while pruning 90% of visual tokens, significantly outperforming existing text-guided and vision-guided pruning methods.
Recent 3D vision-language models (3D VLMs) construct geometry aware tokens by projecting 2D visual features into world coordinates, enabling spatial reasoning for tasks such as 3D question answering. However, this design generates thousands of tokens per scene, resulting in substantial computational and memory overhead. While token compression has been extensively studied in 2D VLMs, existing approaches rely on semantic relevance or attention-based selection that overlook the structured spatial nature of 3D tokens. Moreover, redundancy in 3D representations cannot be resolved by spatial proximity alone, as object-level token imbalance persists even after spatial aggregation. To address this, we propose 3DZip, a three-stage token compression framework that first applies coarse voxelization to remove point-level redundancy, then selects anchor tokens based on feature-space diversity via a Determinantal Point Process, and finally merges remaining tokens under spatial constraints to preserve geometric coherence. Experiments on three 3D question answering benchmarks demonstrate that 3DZip consistently outperforms existing compression methods, retaining 94.7% of the original performance with only 128 tokens, achieving a 1.92× faster inference speed.
Changwoo Baek, Kyeongbo Kong
Department of Electrical and Electronics Engineering, Pusan National University, Busan, Republic of Korea