Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

Vakul Raghavan; Utkarsh Grover; Mingyang Mao; Chengwen Wang; Xiaomin Lin

cs.LGOct 9, 2026

Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

Authors: Vakul Raghavan, Utkarsh Grover, Mingyang Mao, Chengwen Wang, Xiaomin Lin

Organizations: Department of Mathematics and Computer Science, Rutgers University, Newark, NJ, USA · Embodied Robotics and Automation Lab, University of South Florida, Tampa, FL, USA

Abstract

Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the $0.10$ chance floor; adding binding raises it to $0.92$ . Trained only on $32\,\mathrm{kb}$ genomic contexts, DT reaches $0.927$ AUROC at $131\,\mathrm{kb}$ and $0.856$ near $1\,\mathrm{Mb}$ , while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.

Figures & tables

Figure 1: Double-Take architecture. Each block combines depthwise local convolution, sliding-window attention, and a persistent fast-weight memory. Local attention handles nearby interactions, while DT carries information across chunks through a fixed-size memory updated during the forward pass.

Figure 2: Capacity and binding are separate requirements for usable memory. a , Increasing associative load exposes the capacity limit of the stock and renormalized writes, while DT remains effective up to the memory-head dimension. b , Greater capacity alone does not enable cross-position retrieval; adding binding raises recall from chance to

0.92

. c , Once formed, the relationship remains available over distance through the persistent memory, while the local attention path degrades sharply.

Figure 3: TAD-boundary context scaling. AUROC of frozen encoder features under a shared logistic-probe protocol from

2\,\mathrm{kb}

to approximately

1\,\mathrm{Mb}

. Matched controls use the same encoder architecture, while HyenaDNA and Caduceus serve as external long-context references. “cap” denotes contexts beyond a model’s native limit. Values are means over three seeds (

0.014<\mathrm{std}<0.043

Figure 4: Selective transport of structural contrast. a , Boundary deletion changes the readout. b , Associative transport is not selective. c , The change-point write increases selectivity on controlled and real gnomAD variants. d , It also preferentially propagates rare constrained deletions.

Model	CCF	WB	Thy	SNSE	SSEL	Musc	Nerv	Art	Adip	Avg
Task-specific expert	$.639$	$.689$	$.612$	$.710$	$.700$	$.621$	$.683$	$\mathbf{.741}$	$\mathbf{.736}$	$.681$
CNN	$.547$	$.577$	$.487$	$.499$	$.499$	$.502$	$.516$	$.576$	$.551$	$.528$
HyenaDNA (fine-tuned)	$.584$	$.512$	$.529$	$.471$	$.544$	$.487$	$.511$	$.479$	$.513$	$.514$
Caduceus-Ph (fine-tuned)	$.597$	$.594$	$.527$	$.586$	$.574$	$.538$	$.588$	$.547$	$.541$	$.565$
Caduceus-PS (fine-tuned)	$.549$	$.542$	$.547$	$.529$	$.541$	$.523$	$.552$	$.536$	$.519$	$.537$
DT (ours, frozen)	$\mathbf{.745}$	$\mathbf{.747}$	$\mathbf{.638}$	$\mathbf{.780}$	$\mathbf{.829}$	$\mathbf{.729}$	$\mathbf{.763}$	$.682$	$.640$	$\mathbf{.728}$

Table 1: eQTL variant-effect prediction across nine GTEx tissues. AUROC across tissue-specific tasks. DT uses a frozen encoder with a logistic adapter; published baselines use their reported fine-tuning protocols. Bold denotes the best result in each column.

	Encoder	2k	8k	32k	131k	262k	524k	1M
Human	DT, memory on	.693	.760	.847	.927	.903	.870	.856
	Same encoder, memory off	.640	.641	.617	.623	.658	.625	.610
	Caduceus-PS	.668	.739	.782	.827	cap	cap	cap
	HyenaDNA-1m	.633	.701	.640	.660	.776	.712	.784
Mouse	DT, memory on	.677	.699	.786	.785	.761	.783	.796
	Same encoder, memory off	.653	.583	.602	.639	.630	.648	.713

Table 2: Zero-shot cross-species transfer. TAD-boundary AUROC from frozen human-pretrained encoders under a matched logistic-probe protocol. Human results use held-out chromosomes 8/9; mouse results use mm10 chromosomes 1-5 without adaptation. “cap” denotes context beyond a model’s native limit.

	Human	Chimp	Macaque	Mouse	Chicken
	$(0)$	$(6.4M)$	$(29M)$	$(87M)$	$(319M)$
Frozen	$.000$	$.090$	$.133$	$.286$	$.112$
Adapted	$.000$	$.097$	$.136$	$.271$	$.120$

Table 3: Cross-species representation alignment. Normalized centroid distance from each species mean window embedding to the human embedding centroid, before and after test-time adaptation. Lower is closer; parentheses denote evolutionary distance from human in millions of years(M).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Configuration
Parameters	$\approx 87$ M
Blocks	$12$ pre-LN blocks
Embedding width	$768$
Attention heads	$24\times 32$
Memory heads	$24\times 32$
Attention window	$\pm 128$ nt per block

Appendix

Table 4: Encoder architecture. Architecture shared by the matched DT and stock-memory models.

Configuration	MS	FR	LM	CP	Role
Stock memory					Matched control
Capacity repair	✓	✓			Capacity test
Capacity + binding	✓	✓	✓		Binding test
DT, no gain	✓	✓	✓		No readout gain
DT + gain 4	✓	✓	✓		$4\times$ memory gain at inference
Selective DT	✓	✓	✓	✓	Selectivity experiments

Appendix

Table 5: Memory configurations. Matched variants share the encoder and differ only in the marked operations. MS: multi-step; FR: free retention; LM: local mixing; CP: change-point write.

Model	Params.	Context	Protocol	Purpose
DT	$87$ M	$32$ kb	frozen	Full repaired fast-weight memory.
Stock memory	$87$ M	$32$ kb	frozen	Architecture-matched memory control.
Stock + distal	$87$ M	$32$ kb	frozen	Tests distal supervision alone.
Stock + span	$87$ M	$32$ kb	frozen	Tests span masking alone.
HyenaDNA-medium	$\sim 7$ M	$160$ kb	public	Long-context convolutional reference.
HyenaDNA-large	$\sim 7$ M	$1$ Mb	public	Megabase-context reference.

Appendix

Table 6: Comparison models. Matched controls preserve the DT architecture; public models serve as external long-context references.

Experiment	Encoder	Readout	Metric
Associative capacity	isolated memory	direct retrieval	recall
Cross-position binding	isolated memory	direct retrieval	recall
TAD boundary	frozen	pooled logistic probe	AUROC
eQTL variant effect	frozen	logistic / linear adapter	AUROC
Enhancer–target gene	frozen	matched frozen adapter	AUROC
Structural perturbation	frozen	boundary readout	probability change

Appendix

Table 7: Evaluation interfaces. The encoder remains frozen in the genomic downstream evaluations.

Method	$\mathbf{1}$	$\mathbf{2}$	$\mathbf{4}$	$\mathbf{8}$	$\mathbf{16}$	$\mathbf{32}$	$\mathbf{64}$
Stock	1.00	.60	.50	.23	.09	.04	.02
5-step + renorm	1.00	1.00	1.00	.82	.34	.07	.03
DT	1.00	1.00	1.00	1.00	.99	.93	.74

Appendix

Table 8: Associative capacity. Recall as the number of simultaneously stored key–value pairs increases. The memory-head dimension is

d=32

Configuration	Recall
Stock	.10
Capacity only	.10
Capacity + Binding	.92

Appendix

Table 9: Cross-position binding. End-to-end recall when the associated key and value occur at different positions. Chance performance is

0.10

Method	Near	Far
DT	.859	.827
Sliding-window attention	.142	.019

Appendix

Table 10: Association transport over distance. Recall for locally created associations evaluated at near and far separations.

Repaired memory
Encoder	2k	8k	32k	65k	131k	262k	524k	999k
DT + gain 4	.693	.760	.847	.897	.927	.903	.870	.856
DT, no gain	.723	.771	.788	.802	.834	.830	.846	.722
Matched architecture / objective controls
Stock memory	.716	.758	.721	.749	.742	.752	.750	.741
Span objective	.680	.618	.703	.725	.721	.762	.740	.715

Appendix

Table 11: Full TAD-boundary context sweep. AUROC of frozen encoder features under the common logistic-probe protocol. “cap” denotes contexts beyond a model’s native maximum length.

Distance	16 kb	32 kb	65 kb
Boundary / random	.99	1.07	1.07

Appendix

Table 12: Selectivity of associative transport. Boundary-to-random effect ratio as perturbation distance increases.

Write	Controlled	gnomAD-SV
Associative	1.06	1.01
Change-point	1.15	1.09

Appendix

Table 13: Selectivity introduced by the change-point write. Boundary-to-random effect ratios on controlled boundaries and real structural variants.

Write	$\mathbf{P(\mathrm{rare}>\mathrm{common})}$
Associative	.42
Change-point	.67

Appendix

Table 14: Selective transport under functional constraint. Probability that a rare constrained deletion produces a larger distal effect than a common tolerated variant.

	Encoder	2k	8k	32k	131k	262k	524k	1M
Human	DT, memory on	.693	.760	.847	.927	.903	.870	.856
	Same encoder, memory off	.640	.641	.617	.623	.658	.625	.610
	Caduceus-PS	.668	.739	.782	.827	cap	cap	cap
	HyenaDNA-1m	.633	.701	.640	.660	.776	.712	.784
Mouse	DT, memory on	.677	.699	.786	.785	.761	.783	.796
	Same encoder, memory off	.653	.583	.602	.639	.630	.648	.713

Appendix

Table 15: Zero-shot human-to-mouse transfer. TAD-boundary AUROC from human-pretrained frozen encoders under the matched logistic-probe protocol. “cap” denotes context beyond a model’s native maximum length.

Long-Context Modeling

Long-context modeling studies how AI systems process, retrieve, and reason over sequences that exceed ordinary context windows.

See all 133 papers

MMLA: Memory-Mediated Learning Architecture for Predictive Dual-State Adaptation
Jun 27, 2026Junyi Zou, Avrova DonzContinual Learning for LLMs Test-Time Adaptation
1MMLA-org · 2Communication University of China (CUC), No. 1 Dingfuzhuang East Street, Chaoyang District, Beijing 100024, China
WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems
May 23, 2026Jiangnan Yu, Kisson Songqi Lin, Jilong WuLong-Context Retrieval Memory-Augmented Language Models
MemDuet: Learning Dual-Process Memory for Long-Context Reasoning
Oct 14, 2026Jiangcheng Song, Haodong Jing, Yulin Mu +7Dual-Process Reasoning Memory-Augmented Language Model Agents
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University · Imperial College London

Jun 27, 2026J/K move · Enter open · S save

MMLA: Memory-Mediated Learning Architecture for Predictive Dual-State Adaptation

Junyi Zou, Avrova Donz

1MMLA-org · 2Communication University of China (CUC), No. 1 Dingfuzhuang East Street, Chaoyang District, Beijing 100024, China

Continual Learning for LLMs Test-Time Adaptation Memory-Augmented Language Models LLM Memory

Open paper AskPDF arXiv

Memory-Mediated Learning Architecture (MMLA) separates slow base parameters theta, a bounded numerical policy carrier Phi, and a bounded authoritative memory M. Predictive Dual-State Adaptation (PDSA) lets feedback update Phi while one problem remains active and lets a trusted lifecycle atomically commit one typed row or exact NULL. Later reasoning may read both states, but their writers, resets, rollback domains, and ledgers remain distinct. Realized futures supervise values only during training; deployment is causal and future-blind. We give conditional theory and falsifiable contracts for reasoning-time updates, completed-segment consolidation, predictive admission, authoritative memory, and dual-state attribution. Assumptions, counterexamples, capacity and cost ledgers, recovery duties, and identifying experiments are explicit; these are not implementation guarantees. Controlled studies show exact lifecycle execution on 300/300 held-out records for each of three seeds, calibrated retrieval gains over frozen-hidden dense and BM25 baselines, and exact typed anchor-filler transport on 240/240 held-out records per seed. These validate restricted components, not natural-language memory management or complete PDSA. Post-training studies retain positive and negative evidence. Later protocols obtain restricted readout progress, but a nine-trajectory comparison finds that matched latent readout, a full-width bridge, and bridge plus frozen text-teacher alignment all fail continuous-event qualification across both task families. A separately adapted text reference and restoration checks pass. At the September 13, 2026 evidence cutoff, no strict policy-only reasoning-time-training effect, predictive-admission oracle margin, learned future-blind admission policy, or policy-by-memory factorial advantage is established.

Abstract

Long-Context Modeling

Explore similar work

MMLA: Memory-Mediated Learning Architecture for Predictive Dual-State Adaptation

WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems

MemDuet: Learning Dual-Process Memory for Long-Context Reasoning