Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Aman Behera; Namit Solanki; Mehul Anand

cs.CLOct 22, 2026

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Authors: Aman Behera, Namit Solanki, Mehul Anand

Organizations: IIT Roorkee · AISSMS IOIT, Pune · Independent Researcher

Abstract

Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.

Figures & tables

Figure 1: Paired position intervention. $A_{p}$ is accuracy when the gold document is at position $p$ . Complete ten-position question bundles are resampled together.

System or contrast	Primacy	95% CI	Recency	95% CI	Floor	Ceiling
Transformer	$+0.052$	$[0.032,0.072]$	$+0.075$	$[0.054,0.097]$	$0.091$	$0.640$
State-space v1	$-0.001$	$[-0.016,0.014]$	$+0.077$	$[0.058,0.097]$	$0.115$	$0.615$
State-space v2	$-0.018$	$[-0.034,-0.003]$	$+0.105$	$[0.086,0.124]$	$0.128$	$0.619$
Hybrid minus pure	$+0.019$	$[-0.006,0.044]$	$-0.028$	$[-0.055,-0.001]$	–	–

Table 1: Primary edge effects. The final row is the paired hybrid-minus-pure interaction; other rows are within-model edges. For that paired row, Holm

p=0.144

for primacy and

p=0.089

for recency, so neither effect survives correction.

Panel of Figure 2 — Figure 2: Accuracy as the gold document moves through ten positions on 800 exploratory questions. Shaded bands are 95% question-bootstrap confidence intervals.

Figure 3: Primacy across five approximate size pairs. The family gap appears only among systems that can answer the task, making this a capability-and-family trend rather than a causal scale result.

Figure 4: Exploratory evidence on attention concentration, position decodability, and prompt sensitivity. The sink and probe measurements are correlational, and the prompt variants change multiple prompt properties rather than isolating one causal mechanism.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Phase	Executed comparison	Committed source	Interpretation
1	200 QA; 50 key-value	artifacts/phase1	Calibration and control
2	3 systems, $n=800$ each	artifacts/phase2	Primary family interaction
3	pure and hybrid 8B, $n=800$	artifacts/phase3	Matched, uncertain contrast
4	5 size pairs, $n=800$ each	artifacts/phase4	Capability and scale trend
5	2 corpora, $n=800$ each	artifacts/phase5	Corpus scope check
6	3 systems, 50 at two lengths	artifacts/phase6	Synthetic task check

Appendix

Table 2: Executed evidence by phase. Counts are questions unless otherwise stated.

Figure 5: Calibration of the primary Transformer setting. The key-value task confirms that the pipeline recovers an imposed position signal, and the QA anchors separate guessing, the multi-document conditions, and answerability.

Figure 6: Primacy across five approximate Transformer-state-space size pairs. The family gap is visible only among models that can answer the task, so the trend combines capability and family.

Figure 7: Fixed architecture with two training corpora. The corpus swap changes overall accuracy without a detectable edge-shape change.

Figure 8: Primacy and recency on multi-document QA and synthetic retrieval. The synthetic task reproduces Transformer primacy but leaves the state-space systems at ceiling.

Figure 9: Descriptive position curves for three additional Transformer and hybrid systems. All show positive primacy, but cross-system differences do not isolate architecture.

Phases	Single-GPU node	Scope
1 and 5	A10G (23 GB)	Calibration and corpus check
2–4	A40 (46 GB)	Primary, matched, and scale comparisons
6	T4 (15 GB) and L40S (46 GB)	Synthetic retrieval
7	L40S (46 GB) and T4 (15 GB)	Prompt, sink, and probe studies
8	L40S (46 GB)	Additional systems

Appendix

Table 3: Execution environments recorded by the phase artifacts.

Paper key	Repository identifier	Revision
Transformer 2.8B	EleutherAI/pythia-2.8b	2a259cdd96a4
State-space v1 2.8B	state-spaces/mamba-2.8b-hf	96c48e0292b6
State-space v2 2.7B	AntonV/mamba2-2.7b-hf	ef542707386f
Pure state-space 8B	nvidia/mamba2-8b-3t-4k	b915550c63ba
Hybrid state-space 8B	nvidia/mamba2-hybrid-8b-3t-4k	35e8852e2240
SlimPajama state-space 2.8B	state-spaces/mamba-2.8b-slimpj	a7bdd41af90c

Appendix

Table 4: Selected checkpoint repositories and pinned revisions.

State Space Language Models

See all 19 papers

EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering
Jun 5, 2026Xiaopeng Yuan, Zebin Wang, Suwen Wang +3Long-Context Retrieval Long-Context QA
University of Illinois Urbana-Champaign · 2Harvard University · 3Brion, ASML US LP +1
Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction
Oct 25, 2026Zhe Xu, Tianhao Liao, Zhenqing Ling +3Long-Context Retrieval Information Retrieval
Sun Yat-sen University · Alibaba Group
Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks
May 22, 2026Chuyifei Zhang, Hongyu Cui, Xiaowen Huang +1LLM Evaluation Long-Context Language Model Evaluation
Beijing Jiaotong University · Central South University of Forestry and Technology

Jun 5, 2026J/K move · Enter open · S save

EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering

Xiaopeng Yuan, Zebin Wang, Suwen Wang, Zongxin Yang+2

University of Illinois Urbana-Champaign · 2Harvard University · 3Brion, ASML US LP+1

Long-Context Retrieval Long-Context QA Long-Context Modeling Test-Time Training

Open paper AskPDF arXiv

Long-context question answering (QA) remains challenging for smaller language models even when answer-bearing evidence is already present in the input. Existing within-context retrieval methods localize and expose candidate evidence chunks for the question, but they stop at input-level evidence exposure rather than adapting the query-side attention parameters that control how the model allocates attention over full-context positions. In contrast, lightweight test-time adaptation methods, such as query-only test-time training (qTTT), leave evidence localization unresolved because their generic span-level self-supervised objectives do not identify which context positions support the current answer. In this paper, we propose Evidence-Aligned SElective Test-Time Training (EASE-TTT), a within-context retrieval-augmented test-time training framework that converts selected evidence chunks into a soft attention supervision target over their token positions. Instead of replacing the full context with retrieved chunks, EASE-TTT uses the resulting attention target to guide query-side adaptation, with the adapted model generating the final answer from the original full context. Experiments on six LongBench QA tasks and three small decoder-only language models show that EASE-TTT achieves the strongest macro-average performance among full-context inference, retrieval-only baselines, and qTTT, supporting evidence-aligned test-time adaptation in long-context QA.

Abstract

State Space Language Models

Explore similar work

EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering

Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction

Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks