HiCI: Hierarchical Construction-Integration for Long-Context Attention

Xiangyu Zeng; Qi Xu; Yunke Wang; Chang Xu

cs.CLMar 21, 2026

HiCI: Hierarchical Construction-Integration for Long-Context Attention

Authors: Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu

Organizations: School of Computer Science, University of Sydney, Sydney, Australia · 2Westlake University, Hangzhou, China.

Abstract

Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches. Drawing on cognitive theories of discourse comprehension, we propose HiCI (Hierarchical Construction--Integration), a hierarchical attention module that constructs segment-level representations, integrates them into a shared global context, and broadcasts both to condition segment-level attention. We validate HiCI through parameter-efficient adaptation of LLaMA-2 with only <5.5% additional parameters, extending context from 4K to 100K tokens (7B) and 64K tokens (13B). Across language modeling, retrieval, and instruction-following benchmarks, HiCI yields consistent improvements over strong baselines, including matching proprietary models on topic retrieval and surpassing GPT-3.5-Turbo-16K on code comprehension. These results demonstrate the effectiveness of explicit hierarchical structuring as an inductive bias for long-context modeling.

Figures & tables

Figure 1: Overview of HiCI. Left: HiCI integrated into a Transformer block; trainable components are highlighted. Right: HiCI constructs hierarchical context through three stages. (1) Local Construction: the input sequence is partitioned into $N$ segments, and cross-attention with $M$ learnable query slots extracts a local representation $L_{i}$ from each segment. (2) Global Integration: local representations $\{L_{i}\}_{i=1}^{N}$ are aggregated into a shared global context $G$ via multi-view statistical pooling and attention-based weighting. (3) Top-down Broadcast: $G$ and $L_{i}$ are prepended to each segment’s key–value sequence, conditioning attention on hierarchical context while preserving parallelism across segments. At inference, HiCI is optionally applied during prefill, while autoregressive decoding uses standard attention.

PG-19

Proof-pile

Base Model

Train

Method

16K

32K

64K

100K

16K

32K

64K

100K

LLaMA-2-7B

7.49

7.15

>\!10^{2}

>\!10^{2}

>\!10^{2}

>\!10^{2}

>\!10^{2}

3.21

2.91

>\!10^{2}

>\!10^{2}

>\!10^{2}

>\!10^{2}

>\!10^{2}

ChunkLLaMA

7.49

7.15

6.98

6.96

7.08

15.15

–

3.21

2.91

2.75

2.69

2.70

2.75

–

LongLoRA

7.70

7.35

7.14

–

3.20

2.91

2.72

–

HiCI

7.27

7.01

6.93

–

3.07

2.82

2.65

–

16K

LongLoRA

7.65

7.28

7.02

6.86

–

3.17

2.87

2.66

2.51

–

Table 1: Perplexity (

\downarrow

) on PG-19 and Proof-pile test sets for LLaMA-2-7B/13B and Qwen3-8B continually pre-trained on RedPajama for 1k training steps. LLaMA-2 models are trained with context lengths ranging from 8K to 100K and evaluated up to 100K; Qwen3-8B extends its original 32K context window to 48K and is evaluated up to 48K, with an additional 500-step variant reported.

Model	3K	6K	10K	13K	16K
GPT-4o-mini-128K	1.00	1.00	1.00	1.00	1.00
Claude-3.5-Sonnet-200K	1.00	1.00	1.00	1.00	1.00
MPT-30B-Chat-8K	0.96	1.00	0.76	–	–
ChatGLM2-6B-8K	0.88	0.46	0.02	0.02	0.02
MPT-7B-StoryWriter-65K	0.46	0.46	0.28	0.34	0.36
LongChat-13B-16K	1.00	1.00	1.00	0.98	0.90

Table 2: Topic retrieval accuracy on LongChat ( Li et al., 2023 ) . We compare HiCI against both proprietary models and open-source long-context LLMs across 3K–16K context lengths. HiCI-13B-16K matches proprietary model performance up to 13K and outperforms all open-source baselines at 16K.

Figure 2: Passkey retrieval accuracy for LongLoRA-7B, HiCI-7B (both fine-tuned at 32K), and base LLaMA-2-7B. HiCI achieves 100% accuracy within the training length and extrapolates more gracefully to 56K via position interpolation without additional fine-tuning.

Model	Single-Doc QA	Multi-Doc QA	Summ	Few-shot	Synthetic	Code	Overall
Model	Single-Doc QA	Multi-Doc QA	Summ	Few-shot	Synthetic	Code	EN	ZH	All
GPT-3.5-Turbo-16k	45.1	36.2	23.9	57.6	51.0	54.1	44.0	44.5	44.7
Llama2-7B-chat-4k	21.7	18.2	18.5	49.9	4.1	48.1	31.0	14.3	26.8
LongChat-7B-32k	28.8	20.3	22.5	50.8	13.0	54.1	34.3	23.9	31.6
Vicuna-v1.5-7B-16k	31.8	18.8	23.2	56.8	5.3	47.3	31.9	26.4	30.5
LongLoRA-7B-16k	23.7	25.0	20.9	54.2	12.0	55.8	36.8	10.9	30.6

Table 3: Results (%) on LongBench ( Bai et al., 2024b ) benchmark. Best in bold , second underlined .

\uparrow

indicates improvement over LongLoRA-7B-16K (direct baseline) and

\uparrow

highlights top-2 gains for each HiCI variant.

Variant	L	G	B	PG19		Proof-pile
Variant	L	G	B	4K	8K	4K	8K
HiCI	✓	✓	✓	7.01	6.93	2.82	2.65
w/o G	✓	✗	✓	7.25	7.04	2.95	2.78
w/o L	✗	✓	✓	7.13	6.99	2.86	2.69
Only Group	✗	✗	✗	8.01	7.54	3.26	2.97
$M=5,\ K=3$	✓	✓	✓	7.15	6.98	2.85	2.68

Table 4: Component and cardinality ablation for HiCI fine-tuned on LLaMA-2-7B at 8K context for 1,000 steps.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$M$	Local cardinality (queries per segment)
$K$	Global cardinality (context vectors)
$d_{s}$	Intermediate compression dimension
$d_{b}$	Bottleneck dimension for attention

Appendix

Table 5: Summary of notation.

	2K	4K	8K
LongLoRA	7.70	7.35	7.14
Shared $G$	7.27	7.01	6.93
Causal $G_{i}$	7.28	7.00	6.94

Appendix

Table 6: Perplexity (

\downarrow

) on the PG-19 test set for LLaMA-2-7B at evaluation context lengths of

{2\text{K},4\text{K},8\text{K}}

. All models are fine-tuned for 1000 steps using HiCI with segment size

S{=}2048

, and evaluated under standard full attention, consistent with Table 1 .

Statistic	Captured Information
$\boldsymbol{\mu}$ (mean)	Central tendency
$\boldsymbol{\sigma}$ (std)	Dispersion
$\boldsymbol{\mu}^{+},\boldsymbol{\mu}^{-}$ (max, min)	Extremal activations
$\hat{\boldsymbol{\mu}}$ (normalized mean)	Directional structure

Appendix

Table 7: Statistical summaries computed in global integration.

Method	Time	Space	Cross-Segment
Standard Attention	$O(T^{2}d)$	$O(T^{2})$	Full
Linear Attention	$O(Td^{2})$	$O(Td)$	Approximated
Segmented Attention	$O(TSd)$	$O(S^{2})$	None
LongLoRA	$O(TSd)$	$O(S^{2})$	Shifted windows
HiCI	$O(TSd)$	$O(S^{2})$	Hierarchical

Appendix

Table 8: Computational complexity comparison.

T

: sequence length,

d

: hidden dimension,

S

: segment/window size.

	LLaMA-2			Qwen3
Hyperparameter	7B (PT)	13B (PT)	7B (SFT)	8B (PT)
Optimization
Optimizer	AdamW	AdamW	AdamW	AdamW
Backbone learning rate	$2\times 10^{-5}$	$2\times 10^{-5}$	$2\times 10^{-5}$	$2\times 10^{-5}$
HiCI learning rate	$2\times 10^{-4}$	$2\times 10^{-4}$	$2\times 10^{-4}$	$2\times 10^{-4}$
Weight decay	0	0	0	0

Appendix

Table 9: Hyperparameters for HiCI training. PT denotes continued pretraining on RedPajama, and SFT denotes supervised fine-tuning on LongAlpaca-12k.

Context	Method	Attn	Proj	FFN	Others	LC+GI	Total
8K	Full Attn	35.2	35.2	70.9	2.1	–	143.4
	S 2 -Attn	8.8	35.2	70.9	2.1	–	117.0
	HiCI	9.4	35.2	70.9	2.1	2.2	119.9
16K	Full Attn	140.7	70.4	141.8	4.3	–	357.2
	S 2 -Attn	35.2	70.4	141.8	4.3	–	251.7
	HiCI	36.4	70.4	141.8	4.3	4.4	257.3

Appendix

Table 10: Per-layer FLOPs (TFLOPs) for LLaMA-2-7B across context lengths. LC+GI denotes the combined overhead of the Local Construction and Global Integration stages; dashes indicate stages absent in that method.

Figure 3: Peak GPU memory (left) and wall-clock training time (right) for HiCI and LongLoRA (LLaMA-2-7B, 8

\times

H100-80GB, 1,000 steps; Stage-2 for 8K–64K, Stage-3 for 100K). The three-stage HiCI pipeline raises memory by 3.5–9.9%, which necessitates finer partitioning at long contexts (

N{=}10

at 100K vs. LongLoRA’s

N{=}4

); the resulting quadratic reduction in per-segment attention cost yields a 19.3% wall-clock speedup.

Figure 4: Training loss comparison between HiCI and LongLoRA on LLaMA-2-7B continual pre-training (RedPajama, 2,000 steps). Both methods are trained at 8K and 16K context with

S\in\{1024,2048\}

. HiCI with

S{=}1024

sustains optimization throughout, while HiCI with

S{=}2048

and all LongLoRA variants plateau beyond step 1,000.

Module	Component	Per Layer	Total (32L)
Local Construction	Memory slots ( $M{=}8$ )	32.8K	1.0M
	Cross-attention (Q/K/V/O)	8.4M	268.4M
	Subtotal	8.4M	269.5M
Global Integration	Shared compression ( $d_{s}{=}128$ )	591.1K	18.9M
	Global queries ( $K{=}4$ )	2.0K	0.1M
	Lightweight attention (Q/K/V/O)	1.0M	33.6M

Appendix

Table 11: Parameter overhead of HiCI on LLaMA-2-7B. We use

d{=}4096

, bottleneck dimension

d_{b}{=}512

, shared compression dimension

d_{s}{=}128

M{=}8

local slots, and

K{=}4

global slots across 32 layers.

Figure 5: Layer-wise attention allocated to global slots during evaluation on PG-19. Background shading denotes depth groups: early (L0–7, blue), middle (L8–23, white), and deep (L24–31, red). (a) Comparison of segment sizes

S{=}1024

and

S{=}2048

under matched conditions (8K evaluation, 2K steps): finer segmentation yields substantially higher attention to global slots, with the final layer (L31) reaching 40.4% versus 12.7%. (b) Robustness at

S{=}2048

: varying evaluation length (8K vs. 4K) and training steps (2K vs. 1K) yields nearly identical layer-wise allocation patterns, with per-layer deviations within 1 percentage point. In both panels, attention to global slots increases toward deeper layers.

Long-Context Language Modeling

Long-context language modeling studies how language models can use, retain, and efficiently process information across very long sequences.

See all 182 papers

Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Long-Context Language Model Inference Long-Context Language Modeling
Tencent HY Team · ShanghaiTech University · The Hong Kong University of Science and Technology +1
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
May 31, 2026Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1LLM Compression LLM Fine-Tuning
SambaNova Systems, Inc. San Jose, CA, USA

Mar 18, 2026J/K move · Enter open · S save

Learning When to Attend: Conditional Memory Access for Long-Context LLMs

Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato, Elvis Nunez+3

Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA

Self-Attention Long-Context Language Modeling LLM Inference Acceleration Sparse Attention Attention Mechanisms Long-Context Reasoning in Language Models

Open paper AskPDF arXiv

Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval. Continued pretraining on long-context data can help but is expensive due to the quadratic scaling of Attention. We observe that most tokens do not require (Global) Attention over the entire sequence and can rely on local context. Based on this, we propose L2A (Learning To Attend), a layer that enables conditional (token-wise) long-range memory access by deciding when to invoke global attention. We evaluate L2A on Qwen 2.5 and Qwen 3 models, extending their effective context length from 32K to 128K tokens. L2A matches the performance of standard long-context training to within 3% while skipping Global Attention for

\sim

80% of tokens, outperforming prior baselines. We also design custom Triton kernels to efficiently implement this token-wise conditional Attention on GPUs, achieving up to

\sim

\times

improvements in training throughput and time-to-first-token over FlashAttention. Moreover, L2A enables post-training pruning of highly sparse Global Attention layers, reducing KV cache memory by up to 50% with negligible performance loss. Our code is released under Apache 2.0 at https://github.com/awslabs/hybrid-model-factory/tree/main/examples/research/L2A.

Abstract

Long-Context Language Modeling

Explore similar work

Learning When to Attend: Conditional Memory Access for Long-Context LLMs

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning