Efficient Encoder-Free Fourier-based 3D Large Multimodal Model

Guofeng Mei; Wei Lin; Luigi Riz; Yujiao Wu; Yiming Wang; Fabio Poiesi

cs.CVFeb 26, 2026

Efficient Encoder-Free Fourier-based 3D Large Multimodal Model

Authors: Guofeng Mei, Wei Lin, Luigi Riz, Yujiao Wu, Yiming Wang, Fabio Poiesi

Organizations: Fondazione Bruno Kessler, Italy · JKU Linz, Austria · CSIRO, Australia

Abstract

Large Multimodal Models (LMMs) that process 3D data typically rely on heavy, pre-trained visual encoders to extract geometric features. While recent 2D LMMs have begun to eliminate such encoders for efficiency and scalability, extending this paradigm to 3D remains challenging due to the unordered and large-scale nature of point clouds. This leaves a critical unanswered question: How can we design an LMM that tokenizes unordered 3D data effectively and efficiently without a cumbersome encoder? We propose Fase3D, the first efficient encoder-free Fourier-based 3D scene LMM. Fase3D tackles the challenges of scalability and permutation invariance with a novel tokenizer that combines point cloud serialization and the Fast Fourier Transform (FFT) to approximate self-attention. This design enables an effective and computationally minimal architecture, built upon three key innovations: First, we represent large scenes compactly via structured superpoints. Second, our space-filling curve serialization followed by an FFT enables efficient global context modeling and graph-based token merging. Lastly, our Fourier-augmented LoRA adapters inject global frequency-aware interactions into the LLMs at a negligible cost. Fase3D achieves performance comparable to encoder-based 3D LMMs while being significantly more efficient in computation and parameters. Project website: https://tev-fbk.github.io/Fase3D.

Figures & tables

Figure 1: Fase3D’s contribution overview. Mainstream 3D LMMs are based on computationally-heavy scene encoders to extract geometric features before alignment with the LLM. In contrast, our method (Fase3D) employs a lightweight Fourier-based tokenizer to process raw point clouds directly and introduces Fourier-augmented LoRA adapters, which infuse global frequency-aware context into the LLM without additional computational overhead.

Figure 2: The Fase3D pipeline. A lightweight tokenizer ( $\bullet$ ) produces $M$ superpoint tokens, which are refined by an FFT-based context enhancer ( $\bullet$ ). A graph is then constructed, and a token-merging block ( $\bullet$ ) compresses the tokens into $T$ compact 3D tokens ( $T<M$ ). Finally, an LLM ( $\bullet$ ) with an FFT-based global filter ( $\bullet$ ) processes these tokens together with textual and user prompts.

Method	LLMs	#Params $\downarrow$	FLOP $\downarrow$	ScanQA (val)				SQA3D (test)
Method	LLMs	#Params $\downarrow$	FLOP $\downarrow$	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$	EM@1 $\uparrow$
Encoder-based 3D LMMs with point cloud as inputs
LL3DA [ 6 ]	OPT-1.3B [ 53 ]	118.87M	40.21	37.31	15.88	13.53	76.79	-
PerLA [ 34 ]	OPT-1.3B [ 53 ]	119.76M	163.38	39.60	17.44	14.49	78.13	-
3D-LLaVA [ 12 ]	Vicuna-1.5-7B [ 9 ]	58.26M	37.75	43.10	18.40	17.10	92.60	54.5
Encoder-free 3D LMMs with point cloud as inputs

Table 1: Question answering results on ScanQA [ 2 ] and SQA3D [ 33 ] . #Param/FLOP: number of activated parameters and Floating Point Operation count required for the encoding/tokenization stage. Best result is in bold . Second best result is underlined .

Figure 3: Qualitative results and comparisons between Fase3D, PerLA [ 34 ] , and LL3DA [ 6 ] on the ScanQA [ 2 ] dataset.

Method	Segmenter	#Params $\downarrow$	FLOP $\downarrow$	ScanRefer				Nr3D
Method	Segmenter	#Params $\downarrow$	FLOP $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$
Encoder-based 3D LMMs
LL3DA [ 6 ]	✓	118.87M	80.43	65.19	36.79	25.97	55.06	51.18	28.75	25.91	56.61
PerLA [ 34 ]	✓	119.76M	326.76	69.41	38.02	29.07	56.80	55.06	31.24	28.52	59.13
3D-LLaVA [ 12 ]	✓	58.26M	37.75	78.80	36.90	27.10	57.70	-	-	-	-
Encoder-free 3D LMMs

Table 2: Dense captioning results on ScanRefer and Nr3D. #Param/FLOP: number of activated parameters and Floating Point Operation count required for the encoding/tokenization stage. Best result in bold . Second best result is underlined .

Figure 4: Qualitative comparison between Fase3D, PerLA [ 34 ] , and LL3DA [ 6 ] on the ScanRefer [ 5 ] dataset.

Module			ScanQA (Validation)
Point	Superpoint	FFT	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
✓			37.03	15.43	13.14	76.04
✓	✓		37.18	16.38	13.96	79.70
✓		✓	39.56	17.03	15.11	82.97
✓	✓	✓	41.64	17.80	16.70	86.91
Full model with pretraining			42.56	18.24	18.02	90.11

Table 3: Ablation study of vision embedding modules. Point (downsampled raw point tokens), Superpoint (superpoint pooling), FFT (lightweight FFT-based context enhancer).

Module		ScanQA (Validation)
vision	text	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
LoRA	-	36.98	15.37	13.35	76.45
-	LoRA	36.49	15.40	13.19	76.21
sLoRA	sLoRA	37.34	16.04	12.83	78.24
dLoRA	dLoRA	39.96	17.23	15.09	82.53
dLoRA+FFT	dLoRA	41.64	17.80	16.70	86.91

Table 4: Ablation study of LoRA placement and Fourier residual on ScanQA (Validation) [ 2 ] . sLoRA: the branches share the same LoRA modules. dLoRA: the branches use separate (decoupled) LoRA modules. “+FFT” adds a parallel Fourier residual branch.

Method	Enc.	#Params $\downarrow$	FLOP $\downarrow$	ScanQA (val)
Method	Enc.	#Params $\downarrow$	FLOP $\downarrow$	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
OPT-1.3B [ 53 ] LLM
LL3DA [ 6 ]	✓	118.87M	40.21	37.31	15.88	13.53	76.79
PerLA [ 34 ]	✓	119.76M	163.38	39.60	17.44	14.49	78.13
Fase3D	✗	9.30M	2.01	40.34	17.63	15.96	86.24
Qwen2.5-3B [ 52 ] LLM

Table 5: Question answering results with different LLMs on ScanQA [ 2 ] . #Param/FLOP(G) denote the activated parameters and encoding/tokenization FLOPs. Best results in bold .

#Layers	ScanQA (Validation)
#Layers	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
0	35.93	14.94	13.39	74.03
4	38.27	16.30	13.76	80.04
8	41.64	17.80	16.70	86.91
10	41.51	17.14	16.67	87.05
12	40.47	16.56	15.78	86.40

Table A: Ablation study on the number of LLM’s LoRA layers.

$n_{C}$	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
1	37.42	15.96	14.23	79.06
2	40.49	16.78	15.96	85.64
4	41.64	17.80	16.70	86.91
6	41.72	17.93	17.01	87.14

Table B: Effect of the number of SFC curves

n_{C}

in multi-curve serialization on ScanQA (val).

#Token	ScanQA (Validation)
#Token	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
64	37.96	16.23	14.09	79.53
128	38.21	15.85	14.88	83.92
256	41.64	17.80	16.70	86.91
320	41.73	17.67	16.61	87.20

Table C: Ablation study on the number of LLM’s input tokens.

Method	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
Scratch	41.64	17.80	16.70	86.91
w/pretrain	43.37	18.61	16.87	91.74

Table D: Effect of two-stage training on ScanQA validation.

Selector	R $\uparrow$	M $\uparrow$	B-4 $\uparrow$	C $\uparrow$
Q-Former	37.52	15.87	14.06	77.41
FPS	39.67	17.10	15.18	82.54
Fase3D	41.64	17.80	16.70	86.91

Table E: Effect of different token selection / merging strategies on ScanQA validation performance. All models are trained from scratch on the ScanQA training split, without any pre-training.

Panel of Figure 15 — Figure A : Visualization of our SFC-based $k$ NN graph construction via window voting. We show two representative examples of curve-guided neighbor selection.

Panel of Figure 16 — Figure B : Comparison of space-filling curves (Hilbert, transposed Hilbert, Z-order, and transposed Z-order) under four views (columns). All four variants produce broadly similar locality-preserving paths, with Hilbert-type curves appearing smoother and Z-order variants exhibiting more axis-aligned segments. Using both the standard and transposed versions provides complementary directional coverage and reduces orientation bias, so in Fase3D we apply the FFT jointly over all four curves.

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 211 papers

Point Cloud as a Foreign Language for Multi-modal Large Language Model
Mar 10, 2026Sneha Paul, Zachary Patterson, Nizar Bouguila3D VLMs Point Cloud Learning
Concordia University, Canada
Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs
Apr 3, 2026Yuhui Lin, Siyue Yu, Yuxing Yang +2Efficient VLM Inference Visual Token Pruning
Xi’an Jiaotong-Liverpool University · University of Liverpool
Octrees as an Explicit 3D Language
Oct 1, 2026Ran Dan, Si-Tong Wei, Pengfei Xiong +33D Shape Generation Unified Multimodal Models
Peking University · Independent Researcher

Mar 10, 2026J/K move · Enter open · S save

Point Cloud as a Foreign Language for Multi-modal Large Language Model

Sneha Paul, Zachary Patterson, Nizar Bouguila

Concordia University, Canada

3D VLMs Point Cloud Learning 3D VQA

Open paper AskPDF arXiv

Multi-modal large language models (MLLMs) have shown remarkable progress in integrating visual and linguistic understanding. Recent efforts have extended these capabilities to 3D understanding through encoder-based architectures that rely on pre-trained 3D encoders to extract geometric features. However, such approaches suffer from semantic misalignment between geometric and linguistic spaces, resolution sensitivity, and substantial computational overhead. In this work, we present SAGE, the first end-to-end 3D MLLM that directly processes raw point clouds without relying on a pre-trained 3D encoder. Our approach introduces a lightweight 3D tokenizer that combines geometric sampling and neighbourhood aggregation with vector quantization to convert point clouds into discrete tokens--treating 3D data as a foreign language that naturally extends the LLM's vocabulary. Furthermore, to enhance the model's reasoning capability on complex 3D tasks, we propose a preference optimization training strategy with a semantic alignment-based reward, specifically designed for open-ended 3D question answering where responses are descriptive. Extensive experiments across diverse 3D understanding benchmarks demonstrate that our end-to-end approach outperforms existing encoder-based methods while offering significant advantages in computational efficiency, generalization across LLM backbones, and robustness to input resolution variations. Code is available at: github.com/snehaputul/SAGE3D.

Abstract

3D VLMs

Explore similar work

Point Cloud as a Foreign Language for Multi-modal Large Language Model

Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs

Octrees as an Explicit 3D Language