SSQA: Sibling-Selective Quadtree Attention for Hierarchical Modeling in Perception Tasks

Yufan Chen; Arnav Bali; Angela Liu; Laura Zheng; Ming C. Lin

cs.CVOct 28, 2026

SSQA: Sibling-Selective Quadtree Attention for Hierarchical Modeling in Perception Tasks

Authors: Yufan Chen, Arnav Bali, Angela Liu, Laura Zheng, Ming C. Lin

Organizations: Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA, USA. · South River High School, Edgewater, MD, USA. · Department of Computer Science, University of Maryland, College Park, MD, USA.

Abstract

Perception tasks for navigation in robotics, including aerial platforms such as drones and autonomous driving systems, are inherently structured. Drone-mounted cameras typically capture sky above, terrain below, and obstacles or man-made structures in between, while driving data often contains organized road layouts, lane markings, and surrounding agents. Motivated by these axis-aligned structural priors, we note that such information is typically more structured than in generic image tasks. We hypothesize that processing information in a quadtree-esque manner can not only model features effectively in a hierarchical manner, but also offers an efficient linear-time alternative to vanilla attention mechanisms, which run in quadratic time. In this paper, we propose Sibling-Selective Quadtree Attention (SSQA), which models image tokens hierarchically as a structured, full quadtree. We show analytical complexity analysis that guarantees linear-time feature modeling, in addition to empirical experiments comparing inference speeds with other popular modeling approaches, such as Mamba 2 and Quadtree Attention. Our results, benchmarked across several tasks, show that we achieve results at least as good, if not notably better, as others at a fraction of the computational costs.

Figures & tables

Fig. 1 : Mean Epoch Wall Times (s) across 50 epochs on IllusionAnimals classification (224 2 inputs) on A100 GPU. Despite other methods also modeling attention in linear time, a practical caveat is the lack of compute optimization in practice. As a result, some methods are even slower than vanilla ViT [ 6 ] out-of-the-box from official implementations. SSQA is both easy to integrate and requires no additional practical optimizations, our results show that SSQA maintains or even improves performance over other hierarchical modeling baselines.

Fig. 2 : Top-Down vs. Bottom-Up Gating; Squares indicate leaf nodes. Our work has a key difference from previous work in hierarchical quadtree-based attention approaches: bottom-up gating. Previous works (a) avoid subdivisions into tokens with smaller granularity if possible by selecting top-k tokens with highest saliency during training. As a result, tokens on the same level outside of top-k saliency neglect hierarchical reasoning, even if there agents present in those patches . For example, in a driving scene, one patch may have a higher density of pedestrians, but it shouldn’t imply less attention on the single agent falling outside of the same patch. In our approach, we model features from bottom-up, ensuring that features are modeled hierarchically across the entire image . Doing so not only improves performance on various benchmarked tasks, but also maintains linear complexity with respect to number of tokens (shown in § III-G ).

Method	Params (M) $\downarrow$	FLOPs (G) $\downarrow$	Acc $\uparrow$	Avg Time (s) $\downarrow$
Baseline-ViT [ 28 ]	90.52	33.70	98.25%	53.17
ViT-LinAtt [ 29 ]	90.49	33.74	98.25%	54.65
Mamba 2 [ 30 ]	107.43	28.77	98.41%	67.70
SSQA (ours)	36.18	4.29	98.65%	26.19

TABLE I : Compute vs. Validation performance for image classification: upto 3x fewer parameters, 8x fewer FLOPs, 2.08x mean epoch time.

Panel of Figure 4 — Fig. 3 : Training Dynamics . SSQA (a) converges faster and (b) reaches target accuracy earlier than the ViT baseline.

Method	Params (M)	FLOPs (G) $\downarrow$	Acc
Baseline-ViT [ 28 ]	90.52	33.70	98.25%
ViT-LinAtt [ 29 ]	90.51	33.74	98.25%
Mamba 2 [ 31 ]	107.43	28.77	98.41%
Swin TRXFM [ 1 ]	32.24	8.75	99.05%
MLP-Mixer [ 32 ]	63.84	25.24	98.02%
ConvNeXT V2 [ 33 ]	32.54	8.92	98.33%

TABLE II : Thorough Compute vs. Accuracy for Classification on IllusionAnimals (50 epochs,

224^{2}

Panel of Figure 6 — Fig. 4 : Model Efficiency . SSQA is substantially lighter by reducing the number of parameters by nearly 3x and more compute-efficient by up to a factor of 8x for classification test.

Variant	Params (M)	FLOPs (G)	Acc	$\Delta$ Acc
SSQA (full; $k{=}3$ )	36.18	4.29	98.65%	+0.00
w/o CIF (OFF)	36.18	4.29	98.22%	$-0.43$
NO-GATE (no CIF gate)	36.18	4.29	98.53%	$-0.12$
w/o local FFN	36.18	1.52	98.29%	$-0.36$
$k{=}2$	36.18	4.29	98.40%	$-0.25$
$k{=}4$ (no top- $k$ )	36.18	4.29	98.37%	$-0.28$

TABLE III : Ablation on SSQA components (IllusionAnimals, 50 epochs,

224^{2}

). We report best validation accuracy for one run (seed=42) and include compute for reference; key ablations additionally report mean

\pm

std over 3 seeds (42/43/44) when available.

Method	Params (M)	FLOPs (G) $\downarrow$	mIoU
Baseline-ViT [ 28 ]	24.94	49.63	0.3349
ViT-LinAtt [ 29 ]	23.98	43.22	0.2932
Mamba 2 [ 31 ]	14.11	48.22	0.3518
Swin TRXFM [ 1 ]	35.49	53.60	0.3307
MLP-Mixer [ 32 ]	74.72	148.15	0.3432
ConvNeXT V2 [ 33 ]	33.13	24.64	0.2803

TABLE IV : Compute vs. mIoU for Segmentation on LoveDA (

512^{2}

)

Fig. 5 : Segmentation Model Performance Comparison . SSQA has the lowest FLOPs 1.54x lower than the second lowest flops model and still be able to achieve the second best accuracy with only 0.16 percentage apart.

Fig. 6 : CARLA Evaluation Score . Benchmark evaluation score across 14 scenarios (0–13) for Baseline [ 27 ] vs. Quadtreen ATTN [ 24 ] vs. SSQA (higher is better).

Efficient ViTs

Efficient Vision Transformers reduce the compute, memory, energy, or latency needed to process images and video while retaining useful accuracy.

See all 243 papers

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
Apr 8, 2026Yuheng Shi, Xiaohuan Pei, Linfeng Wen +2Efficient VLM Inference Large Vision-Language Models
University of Sydney, Australia · Sun Yat-sen University, China · City University of Hong Kong, China
QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices
Sep 24, 2026Nicholas Foley, Devin Marinelli, Donny Moore +2Attention Mechanisms Vision Transformer
Department of Computer Science, University of Texas at San Antonio
DroneWAM: Efficient World Action Model for Drone Visual Navigation
Sep 27, 2026Liang Yao, Fan Liu, Hongbo Lu +5UAV Navigation World Models for Robotics
Hohai University · Key Laboratory of Water Big Data Technology of Ministry of Water Resources · Shanghai Jiao Tong University +4

Apr 8, 2026J/K move · Enter open · S save

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

Yuheng Shi, Xiaohuan Pei, Linfeng Wen, Minjing Dong+1

University of Sydney, Australia · Sun Yat-sen University, China · City University of Hong Kong, China

Efficient VLM Inference Large Vision-Language Models

Open paper AskPDF arXiv

MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indiscriminately flood the quadratic self-attention mechanism with visually redundant tokens, severely bottlenecking inference throughput while ignoring spatial sparsity and query intent. To overcome this, we propose Q-Zoom, a query-aware adaptive high-resolution perception framework that operates in an efficient coarse-to-fine manner. First, a lightweight Dynamic Gating Network safely bypasses high-resolution processing when coarse global features suffice. Second, for queries demanding fine-grained perception, a Self-Distilled Region Proposal Network (SD-RPN) precisely localizes the task-relevant Region-of-Interest (RoI) directly from intermediate feature spaces. To optimize these modules efficiently, the gating network uses a consistency-aware generation strategy to derive deterministic routing labels, while the SD-RPN employs a fully self-supervised distillation paradigm. A continuous spatio-temporal alignment scheme and targeted fine-tuning then seamlessly fuse the dense local RoI with the coarse global layout. Extensive experiments demonstrate that Q-Zoom establishes a dominant Pareto frontier. Using Qwen2.5-VL-7B as a primary testbed, Q-Zoom accelerates inference by 2.52 times on Document & OCR benchmarks and 4.39 times in High-Resolution scenarios while matching the baseline's peak accuracy. Furthermore, when configured for maximum perceptual fidelity, Q-Zoom surpasses the baseline's peak performance by 1.1% and 8.1% on these respective benchmarks. These robust improvements transfer seamlessly to Qwen3-VL, LLaVA, and emerging RL-based thinking-with-image models. Project page is available at https://yuhengsss.github.io/Q-Zoom/.

Abstract

Efficient ViTs

Explore similar work

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices

DroneWAM: Efficient World Action Model for Drone Visual Navigation