TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI

Hyunwoo Oh; Hanning Chen; Sanggeon Yun; Yang Ni; Suyeon Jang; Behnam Khaleghi; Fei Wen; Mohsen Imani

cs.ARMar 24, 2026

TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI

Authors: Hyunwoo Oh, Hanning Chen, Sanggeon Yun, Yang Ni, Suyeon Jang, Behnam Khaleghi, Fei Wen, Mohsen Imani

Organizations: University of California, Irvine · Purdue University Northwest · Qualcomm · Samsung

Abstract

Multimodal stacks that mix ViTs, CNNs, GNNs, and transformer NLP strain embedded platforms because their compute/memory patterns diverge and hard real-time targets leave little slack. TRINE is a single-bitstream FPGA accelerator and compiler that executes end-to-end multimodal inference without reconfiguration. Layers are unified as DDMM/SDDMM/SpMM and mapped to a mode-switchable engine that toggles at runtime among weight/output-stationary systolic, 1xCS SIMD, and a routable adder tree (RADT) on a shared PE array. A width-matched, two-stage top-k unit enables in-stream token pruning, while dependency-aware layer offloading (DALO) overlaps independent kernels across reconfigurable processing units to sustain utilization. Evaluated on Alveo U50 and ZCU104, TRINE reduces latency by up to 22.57x vs. RTX 4090 and 6.86x vs. Jetson Orin Nano at 20-21 W; token pruning alone yields up to 7.8x on ViT-heavy pipelines, and DALO contributes up to 79% throughput improvement. With int8 quantization, accuracy drops remain <2.5% across representative tasks, delivering state-of-the-art latency and energy efficiency for unified vision, language, and graph workloads-in one bitstream.

Figures & tables

Work	AI	Sparse	Token
Work	Workload	Kernel	Pruning
This work	ViT+GNN+CNN+NLP	Yes	Run-time
( Zhang et al., 2024b )	CNN+GNN or ViT	Yes	No
( Zhang et al., 2024a )	CNN+GNN	No	No
( Dong et al., 2023 )	ViT	No	Run-time ∗
( You et al., 2023 )	ViT	No	Compile-time ∗

Table 1. FPGA accelerators vs. supported workloads, sparse-kernel support, and token-pruning capability.

Figure 1. TRINE overview and mode-switchable engine (MSE). ( a ) Accelerator with an RPU grid and local inter-RPU buffers to localize traffic and pipeline inter-tile exchange. ( b ) Each RPU integrates a shared-datapath MSE, a width-matched two-stage top-

k

, compact nonlinear units, and a lightweight feed scheduler. ( c ) One PE array time-shares four dataflows via small interconnect muxes and per-PE op control: (1) Systolic (WS/OS) for dense DDMM; (2)

1{\times}C_{S}

SIMD for moderately sparse SDDMM/SpMM; (3) RADT for highly sparse/irregular reductions; (4) normal SIMD for element-wise ops. Selection policy (runtime): DDMM

\rightarrow

WS/OS (WS for small-token/high weight reuse; OS for wide feature maps/many tokens). SDDMM/SpMM

\rightarrow

1{\times}C_{S}

when active operands per row/col

\lesssim C_{S}

and fairly uniform; switch to RADT as sparsity grows or degree skews. Feed scheduling provides systolic delay insertion and sparsity-aware indexed reads without host-side packing.

Figure 2. Two-stage, width-matched top-

k

. A fixed-width (up to

C_{S}

) pipelined bitonic stage matches array width; a lightweight

C_{S}{\rightarrow}k

merge completes selection. Values/indices stream from MSE into a center buffer (CB) and sparse queue buffer (SQB), avoiding off-chip detours and scaling better than single large bitonic networks.

Figure 3. Feed scheduler. ( a ) Pipelined delay insertion aligns rows/columns for WS/OS systolic waves with no host-side data reshaping. ( b ) Sparsity-aware indexed reads via a small BRAM-backed sparse queue (SQB) and address generator feed only active pairs to

1{\times}C_{S}

SIMD and RADT, eliminating bubbles and DRAM detours after top-

k

pruning.

Figure 4. TRINE compile–run flow. The compiler parses a structured model, classifies layers (predictable vs. fuzzy), maps them to DDMM/SDDMM/SpMM, selects MSE modes via a sparsity/shape policy, and emits compact instruction blocks plus a dependency DAG. At runtime, an APU-backed controller fills fuzzy templates (e.g., token counts), configures top-

k

, and schedules ready blocks across RPUs using dependency-aware layer offloading (DALO). Pruning indices flow forward to shrink subsequent kernels.

	Alveo U50	ZCU104
RPU Grid	2×2	1×1
MSE PE	32×32	32×32
Top- $k$ Eng.	Bitonic, Max- $k$ =256	Dual-Layer, Max- $k$ =256
Clock Freq.	300 MHz	200 MHz
LUT	693,092 / 871,680 (79.51%)	163,112 / 230,400 (70.80%)
FFs	379,608 / 1,743,360 (21.77%)	84,064 / 460,800 (18.24%)

Table 2. Hardware configurations and resource utilization.

Model	Label	Workload1	Workload2	Param. Count (M)
TinyCLIP (ViT + NLP)	A	ViT-8M/16	TEXT-3M	11
	B	ViT-39M/16	TEXT-19M	58
	C	ViT-40M/32	TEXT-19M	59
	D	ViT-61M/32	TEXT-29M	90
MDETR (CNN + NLP)	E	ResNet-18	RoBERTa	152
	F	ResNet-50	RoBERTa	166

Table 3. AI model configurations used in evaluation.

Figure 5. End-to-end latency with/without DALO; labels refer to Table 3 . Token pruning disabled to isolate scheduling effects.

Figure 6. Speedup from ViT token pruning (DynamicViT); labels per Table 3 .

Figure 7. Accuracy of FP32 vs. int8 quantization; labels per Table 3 .

	Unit	AMD DPU ’22 ( Inc., 2021 )	VisionAgile TPDS’24 ( Zhang et al., 2024b )	GCV-Turbo FCCM’24 ( Zhang et al., 2024a )	HPTA FPL’23 ( Han and Liu, 2023 )	FTRANS ISLPED’20 ( Li et al., 2020 )	HeatViT HPCA’23 ( Dong et al., 2023 )	SWAT ASP-DAC’24 ( Dong et al., 2024 )	TRINE This Work
ML Workload	-	CNN, Transformer	CNN, ViT, GNN	CNN, GNN	Transformer	Transformer	ViT *	Swin	CNN, GNN, ViT, NLP
Scheduling Mechanism	-	Sequential	Sequential	Sequential	Sequential	Sequential	Sequential	Sequential	Inter-Model Parallel
Number Data Type	-	fixed-point8	-	float16	int8	fixed-point16	fixed-point8	int8	int8
FPGA Platform	-	ZCU 104	Alveo U250	Alveo U250	ZCU 102	VCU118	ZCU102	Alveo U50	Alveo U50
Memory Bandwidth	GB/s	19.2	77	77	19.2	-	19.2	316	316
Operating Frequency	MHz	600/300 **	600/300 **	600/300 **	200	-	150	200	300

Table 4. Comparison with other FPGA-based accelerators including CNN, GNN, ViT, and NLP transformers.

Workload	TinyCLIP		MDETR		MissionGNN		Workload	TinyCLIP
Cfg.	A	B	F	G	J	L	Cfg.	A	B
Alveo U50 ∗	1.6	6.5	18.7	23.0	56.9	7.4	ZCU104 ∗	4.4	20.8
RTX 4090 ∗	37.1	49.9	21.7	28.4	50.1	44.1	Jetson Orin Nano ∗	30	40.6
Speedup (×)	22.57	7.64	1.16	1.24	0.88	5.98	Speedup (×)	6.86	1.96
Labels refer to Table 3 . ∗ Numbers are latencies (ms).

Table 5. Latency comparison between TRINE and GPUs (ms); ViT uses

p{=}0.3

configuration.

Efficient Multimodal Inference

Efficient multimodal inference reduces the compute, memory, latency, and energy needed to process image, video, audio, text, and sensor inputs.

See all 227 papers

ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models
Oct 18, 2026Guangzong Si, Dong Wang, Yifan Yu +3Efficient VLM Inference Visual Token Compression
University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Tsinghua University +2
FTerViT: Fully Ternary Vision Transformer
May 20, 2026Szymon Ruciński, Pietro Bonazzi, Engin Türetken +3Ternary Quantization Vision Transformer
CSEM, Neuchâtel, Switzerland · ETH Zürich, Zurich, Switzerland
ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs
Mar 25, 2026An Yu, Ting Yu Tsai, Zhenfei Zhang +3Efficient VLM Inference Token Pruning
University at Albany, SUNY · Peking University

Oct 18, 2026J/K move · Enter open · S save

ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

Guangzong Si, Dong Wang, Yifan Yu, Miao Liu+2

University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Tsinghua University+2

Efficient VLM Inference Visual Token Compression

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) encode images into dense visual token sequences, creating substantial attention and KV-cache overhead. Existing visual token compression methods reduce this cost by pruning low-saliency tokens in shallow layers, but this practice couples temporary computational deactivation with irreversible information deletion. We show that this coupling breaks an intrinsic inference pattern in MLLMs. Through layer-wise attention analysis, we identify Visual Backtracking: visual pivots remain spatially stable, yet in middle layers they actively retrieve context from tokens that appeared unimportant earlier. Thus, early low-saliency tokens can be inactive for computation while still being useful for later reasoning. Motivated by this finding, we propose ReTrace, a backtracking-aware compression framework that introduces a dormant token state. ReTrace caches non-pivot tokens outside expensive layer-wise computation, restores them during the backtracking window, and prunes only after their information has been absorbed. Experiments on Qwen2.5-VL and InternVL3 demonstrate nearly 90% reductions in FLOPs and KV cache, a 2.83

\times

prefilling speedup, and strong performance retention across complex vision-language benchmarks.

Abstract

Efficient Multimodal Inference

Explore similar work

ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

FTerViT: Fully Ternary Vision Transformer

ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs