Efficiency Follows Global-Local Decoupling

Zhenyu Yang; Gensheng Pei; Tao Chen; Yichao Zhou; Tianfei Zhou; Yazhou Yao; Fumin Shen

cs.CVMar 20, 2026

Efficiency Follows Global-Local Decoupling

Authors: Zhenyu Yang, Gensheng Pei, Tao Chen, Yichao Zhou, Tianfei Zhou, Yazhou Yao, Fumin Shen

Organizations: Nanjing University of Science and Technology · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery · Department of Electrical and Computer Engineering, Sungkyunkwan University · Beijing Institute of Technology · University of Electronic Science and Technology of China

Abstract

Modern vision models must capture image-level context without sacrificing local detail while remaining computationally affordable. We revisit this tradeoff and advance a simple principle: decouple the roles of global reasoning and local representation. To operationalize this principle, we introduce ConvNeur, a two-branch architecture in which a lightweight neural memory branch aggregates global context on a compact set of tokens, and a locality-preserving branch extracts fine structure. A learned gate lets global cues modulate local features without entangling their objectives. This separation yields subquadratic scaling with image size, retains inductive priors associated with local processing, and reduces overhead relative to fully global attention. On standard classification, detection, and segmentation benchmarks, ConvNeur matches or surpasses comparable alternatives at similar or lower compute and offers favorable accuracy versus latency trade-offs at similar budgets. These results support the view that efficiency follows global-local decoupling.

Figures & tables

Figure 1: Efficiency v.s. accuracy on the ImageNet-1K benchmark. The proposed ConvNeur family establishes a new state-of-the-art frontier, delivering superior TOP-1 Acc while demanding significantly fewer FLOPs and parameters than existing methods.

Figure 2: The overall architecture of ConvNeur. On the left, a locality-preserving convolutional branch extracts fine-grained features. In parallel, a compressed global branch performs chunked neural memory aggregation, and produces a gating map to modulate the local features. On the right, we show the neural-memory module: each chunk is linearly mapped to

\{q_{t},k_{t},v_{t}\}

q_{t}

reads from the previous memory state

\mathcal{M}_{t-1}

, while

k_{t}

and

v_{t}

compute a surprise loss to update the memory to

\mathcal{M}_{t}

Model	Depths	Dims	Memory Dim $C_{m}$	Chunk Size $L$	FLOPs (G)
ConvNeur-M1	[2, 2, 6, 2]	[40, 80, 160, 320]	80	196	0.71
ConvNeur-M2	[2, 2, 6, 2]	[48, 96, 192, 384]	96	196	1.01
ConvNeur-M3	[2, 2, 6, 2]	[64, 128, 256, 512]	128	196	1.77
ConvNeur-M4	[2, 2, 8, 2]	[80, 160, 320, 640]	160	196	3.06

Table 1: ConvNeur variants used in our experiments. Depths and dims are listed from high to low resolution. Memory dimension is matched to the stage width, and the chunk size is shared.

Method	Venue	Params (M) $\downarrow$	FLOPs (G) $\downarrow$	Top-1 Acc $\uparrow$
RepVGG-A0 [ 18 ]	CVPR 2021	8.3	1.4	72.4
DeiT-T [ 61 ]	ICML 2021	5.7	1.1	72.2
ConViT-Ti [ 21 ]	ICML 2021	5.7	1.0	73.1
CrossViT-Ti [ 8 ]	ICCV 2021	6.9	1.6	73.4
PVT-T [ 66 ]	ICCV 2021	13.2	1.9	75.1
GFNet-Ti [ 58 ]	NeurIPS 2021	7.0	1.3	74.6

Table 2: Comparison results of vision backbone performance on the ImageNet-1K [ 15 ] dataset pre-training.

Method	AP $\uparrow$	$\text{AP}_{50}$ $\uparrow$	$\text{AP}_{75}$ $\uparrow$
ResNet50 [ 34 ]	38.0	58.6	41.4
PVT-T [ 66 ]	36.7	59.2	39.3
PVTv2-B0 [ 67 ]	38.2	60.5	40.7
BiViT [ 35 ]	40.8	59.2	44.1
ConvNeur-M2 (ours)	41.2	60.9	44.5
ResNet101 [ 34 ]	40.4	61.1	44.2

Table 3: Comparison results of object detection and instance segmentation performance on the COCO 2017 [ 42 ] validation set using Cascade Mask R-CNN [ 5 ] under the 1x schedule. All methods share the same detector, training recipe, and data pipeline.

Figure 3: Qualitative results on the ADE20K [ 81 ] dataset. All examples are from the validation set. In the figure, B represents Boundary F1 score, T represents Trimap-based mIoU, and H represents Hausdorff distance.

Head	Method	FLOPs(G) $\downarrow$	mIoU $\uparrow$	mAcc $\uparrow$
	ResNet50 [ 34 ]	183	36.59	46.85
	PVT-T [ 66 ]	158	36.57	46.72
	FLatten-PVT-T [ 28 ]	169	37.21	48.95
	ConvNeur-M2(ours)	106	39.17	49.85
S-FPN [ 40 ]	ConvNeur-M3(ours)	123	41.42	52.35
	ResNet50 [ 34 ]	953	42.10	52.91

Table 4: Comparison results of semantic segmentation performance on the ADE20K [ 81 ] validation set. The FLOPs are computed with an input image at the resolution of

512\times 2048

. In the table, S-FPN is short for the Semantic FPN [ 40 ] model.

Experiment	Varient #	Method	FLOPs (G) $\downarrow$	Top-1 Acc $\uparrow$
Component
	❶	Local-only	1.4	78.2
	-	Per-stage (ours)	1.8	80.0
Global Memory	❷	Per-layer	2.3	80.4
	❸	CBAM [ 70 ]	1.5	78.4
	❹	Non-local [ 68 ]	3.0	79.1

Table 5: Ablation studies on the ImageNet-1K [ 15 ] dataset. Rows highlighted in purple indicate the default configuration. The highest value is highlighted in bold .

Figure 4: Stage-wise visualization of global to local modulation. Columns correspond to the four stages from shallow to deep. Row 1 shows local-only features $F_{\text{loc}}$ produced by the locality-preserving branch without the global path. Row 2 shows the local features in ConvNeur before gating. Row 3 shows the gated features $F_{\text{out}}$ . Green arrows mark the gating step.

Explore similar work

On Locality and Length Generalization in Visual Reasoning
Jul 10, 2026Pulkit Madan, Sanjay Haresh, Reza Ebrahimi +3Visual Reasoning Length Generalization
Qualcomm AI Research*
Visual Jev: Accurate and Efficient Decisions from Shared Visual Context
Sep 22, 2026Guanxu Yu, Yuhang YaoVisual Question Answering Efficient Inference
Independent Research · Carnegie Mellon University
Unifying Convolution and Attention via Convolutional Nearest Neighbors
Nov 18, 2025Mingi Kang, Jeová Farias Sales Rocha NetoHybrid CNN-Transformer Architectures Self-Attention
Bowdoin College

Jul 10, 2026J/K move · Enter open · S save

On Locality and Length Generalization in Visual Reasoning

Pulkit Madan, Sanjay Haresh, Reza Ebrahimi, Sunny Panchal+2

Qualcomm AI Research*

Visual Reasoning Length Generalization Local Attention Attention Mechanisms Compositional Generalization

Open paper AskPDF arXiv

A striking feature of the human visual system is that it ingests visual information through a series of local foveated glimpses, rather than a single global computation. This makes human vision distinctly different from most popular computer vision models in use today, which input images globally and in a single shot. A natural question therefore is whether local, sequential vision models may provide any fundamental computational benefits in addition to being biologically more plausible than global models. In this work, we investigate this question from the perspective of visual state tracking and length generalization. Inspired by recent studies of length generalization in language models, we study the behavior of vision models trained on simple vision tasks that require the aggregation of local information across an image. Our experiments reveal that, similar to language models, vision models can learn to exploit global shortcuts and thereby fail to generalize over task length or complexity. We also show that recurrent vision policies based on strictly local perception can mitigate these failures, thereby allowing models to generalize on these tasks. Our results show that local attention may be an essential overlooked requirement for robust compositional generalization.

Abstract

Explore similar work

On Locality and Length Generalization in Visual Reasoning

Visual Jev: Accurate and Efficient Decisions from Shared Visual Context

Unifying Convolution and Attention via Convolutional Nearest Neighbors