An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention

Madhusudan Ghosh; Rishabh Gupta

cs.SEFeb 25, 2026

An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention

Authors: Madhusudan Ghosh, Rishabh Gupta

Organizations: Indian Association for the Cultivation of Science, Kolkata, India · Bosch Research and Technology Centre, Bangalore, India

Abstract

The rapid advancement of large language models (LLMs) has led to a significant increase in automated tools in the software engineering, capable of performing various code-related tasks such as code generation, completion, and translation. Despite these advancements, its effectiveness is constrained by fixed context lengths, limiting its ability to generalize across long, domain-specific code sequences. To address this challenge, we investigate zero-shot, inference-only methods aimed at improving position encodings and optimizing attention mechanisms. Our goal is to provide a thorough analysis of current approaches that facilitate context length extrapolation in code, particularly in the context of long code completion tasks.

Figures & tables

Figure 1. Comparison of length extrapolation techniques for code completion, categorized into Positional Encoding Based (e.g., RoPE, ReRoPE) and Efficient Attention Based methods (e.g., StreamingLLM, Paged Attention, Flash Attention). These approaches address the challenges of handling long code sequences in Transformer models.

	LLaMA-2	ShearedLLaMA	TinyLLaMA	Vicuna
Para.	7B	1.3B	1.1B	7B
$\mathbf{L}_{\textbf{pretrain}}$	4096	4096	2048	2048
Vocab Size	32000	32000	32000	32000
Hidden Size	4096	2048	2048	4096

Table 2. Model statistics for various investigated Large Language Models (LLMs) used in code completion tasks. The table compares LLaMA-2, ShearedLLaMA, TinyLLaMA, and Vicuna across several parameters, including the number of parameters (Para.), the maximum pretraining sequence length (

L_{pretrain}

), vocabulary size, and hidden size.

Model		Python		Csharp		Java
Model		EM	Edit Sim	EM	Edit Sim	EM	Edit Sim
TinyLlama	RoPE	0.000	8.886	0.025	10.533	0.000	11.141
	ReRoPE	0.000	19.271	0.000	20.241	0.012	18.821
	StreamingLLM	0.000	12.656	0.000	11.769	0.000	11.693
	Paged Attn	0.034	2.278	0.087	4.194	0.024	3.019
	Flash Attn	0.000	11.031	0.000	11.629	0.000	11.672

Table 3. A Comparison of different models and their variants on code completion tasks across three programming languages: Python, Csharp, and Java. Each model’s performance is evaluated using two metrics: Exact Match (EM) and Edit Similarity (Edit Sim). EM represents the percentage of code completions that exactly match the ground truth, while Edit Sim measures the similarity between the predicted and actual code, accounting for minor variations. It is to be noted that the highest scores for each metric across all model variants for a particular language are highlighted in bold. Bold-faced values indicate the best-performing variants for a specific metric and programming language.

Long-Context Modeling

Long-context modeling studies how AI systems process, retrieve, and reason over sequences that exceed ordinary context windows.

See all 133 papers

The Limits of Long-Context Reasoning in Automated Bug Fixing
Feb 17, 2026Ravi Raju, Mengmeng Ji, Shubhangi Upasani +2Long-Context Reasoning in Language Models Long-Context Language Model Evaluation
SambaNova Systems San Jose, CA 95134, USA
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
May 31, 2026Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1LLM Compression LLM Fine-Tuning
SambaNova Systems, Inc. San Jose, CA, USA
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
May 14, 2026Han Tian, Luxuan Chen, Xinran Chen +10Long-Context Language Modeling LLM Fine-Tuning
1Nankai University · 2Baidu Inc. · 3Shanghai Jiao Tong University +1

Feb 17, 2026J/K move · Enter open · S save

The Limits of Long-Context Reasoning in Automated Bug Fixing

Ravi Raju, Mengmeng Ji, Shubhangi Upasani, Bo Li+1

SambaNova Systems San Jose, CA 95134, USA

Long-Context Reasoning in Language Models Long-Context Language Model Evaluation Automated Program Repair Agentic Code Generation

Open paper AskPDF arXiv

Rapidly increasing context lengths have led to the assumption that large language models (LLMs) can directly reason over entire codebases. Concurrently, recent advances in LLMs have enabled strong performance on software engineering benchmarks, particularly when paired with agentic workflows. In this work, we systematically evaluate whether current LLMs can reliably perform long-context code debugging and patch generation. Using SWE-bench Verified as a controlled experimental setting, we first evaluate state-of-the-art models within an agentic harness (mini-SWE-agent), where performance improves substantially: GPT-5-nano achieves up to a 31% resolve rate on 100 samples, and open-source models such as Deepseek-R1-0528 obtain competitive results. However, token-level analysis shows that successful agentic trajectories typically remain under 20k-30k tokens, and that longer accumulated contexts correlate with lower success rates, indicating that agentic success primarily arises from task decomposition into short-context steps rather than effective long-context reasoning. To directly test long-context capability, we construct a data pipeline where we artificially inflate the context length of the input by placing the relevant files into the context (ensuring perfect retrieval recall); we then study single-shot patch generation under genuinely long contexts (64k tokens). Despite this setup, performance degrades sharply: Qwen3-Coder-30B-A3B achieves only a 7% resolve rate at 64k context, while GPT-5-nano solves none of the tasks. Qualitative analysis reveals systematic failure modes, including hallucinated diffs, incorrect file targets, and malformed patch headers. Overall, our findings highlight a significant gap between nominal context length and usable context capacity in current LLMs, and suggest that existing agentic coding benchmarks do not meaningfully evaluate long-context reasoning.

Abstract

Long-Context Modeling

Explore similar work

The Limits of Long-Context Reasoning in Automated Bug Fixing

LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning

EndPrompt: Efficient Long-Context Extension via Terminal Anchoring