AMemOps: Evaluating Memory Updates in Long Audio Streams

Peiwei Chang; Li Zhang; Mark D. Plumbley; Philip J. B. Jackson; Wenwu Wang

cs.SDOct 13, 2026

AMemOps: Evaluating Memory Updates in Long Audio Streams

Authors: Peiwei Chang, Li Zhang, Mark D. Plumbley, Philip J. B. Jackson, Wenwu Wang

Organizations: Centre for Vision, Speech and Signal Processing, University of Surrey, UK · Department of Computer Science, Royal Holloway, University of London, UK · Department of Informatics, King’s College London, UK

Abstract

Audio memory must recognise recurring sources across acoustic changes while keeping their attributes up to date. Overall accuracy does not reveal how these demands interact. We introduce Audio Memory Operations (AMemOps), a benchmark for latest-attribute retrieval, preservation of unchanged attributes and absent-source rejection in long audio streams. Paired controls vary recordings, training identities and earlier events to examine their effects on updating. Training on other recordings of the test instruments increases a reference delta-rule memory's update advantage over matched attention by 13.4 percentage points. At retrieval delays of 120,s, matched histories separate benefits of consistent repetition from costs of conflicting earlier values, while additive-memory controls identify benefits of erasure.

Figures & tables

Figure 1: AMemOps overview. (a) Latest gain or pan queries; source labels are illustrative, not inputs. (b) Decay, erasure, writing and readout, comparing Gated DeltaNet (GDN) [ 21 ] , Kimi Delta Attention (KDA) [ 22 ] , Gated DeltaNet-2 (GDN2) [ 23 ] and attention. (c) Source-key state edits test selective access against norm-matched controls.

	Attention
Mixer component	Standard	Matched	Recurrent
Heads $\times$ dimension	$4\times 32$	$2\times 64$	$2\times 64$
Q/K/V processing	None	Conv-4 + SiLU	Conv-4 + SiLU
Q–K normalisation	None	$L_{2}$	$L_{2}$
Output norm. and gate	None	RMSNorm + SiLU	RMSNorm + SiLU
RoPE	Optional	Optional	None

Table 1: Model components: two pre-normalised blocks, width

d{=}128

. Conv-4 is four-frame depthwise causal convolution, and output normalisation is per head. SiLU: sigmoid linear unit; RMSNorm: root mean square layer normalisation; RoPE: rotary position embeddings [ 24 ] .

	all queries		query type, 24k
model	12k	24k	absent	updated
ESC-50 environmental sounds
GDN2	68.8	69.1	60.1	67.5
KDA	67.3	67.8	58.0	65.8
GDN	65.3	67.0	49.7	65.4
no-erase	63.6	65.3	49.3	63.2

Table 2: Query-weighted accuracy (%) on 48 AMemOps streams after 12,000 (12k) and 24,000 (24k) training steps, averaged over three seeds.

Seen identities (Set 1)
	Initial		Updated		Retained
Model	O	F	O	F	O	F
GDN2	92.2	99.4	89.2	98.8	94.1	99.5
GDN	93.6	98.7	92.0	98.8	95.6	99.2
no-erase	88.9	98.6	80.9	95.3	92.7	99.1
matched + RoPE	91.8	99.6	77.1	98.7	94.9	99.7

Table 3: Mean per-stream NSynth accuracy (%) on original (O) and fixed (F) recordings: 192 held-out streams per set, 24k models from Table 2 , three seeds.

Figure 2: NSynth (24k, three seeds). (a) Pitch distance from query to latest same-source event. Dashed: matched attention without convolution (9–48 streams/bin, at least two queries/stream). (b) Left: update accuracy on identical test streams for identities seen/unseen in training. Right: change in the GDN2–attention gap (Eq. 3 ). Attention uses RoPE; bars show 95% CIs.

Figure 3: NSynth Set 1, Table 2 models at 24k, three seeds. (a) Accuracy: consistent (N), earlier target absent (B), updated (U), retained (R). Load counts distractor sources. (b) Repetition benefit N

-

B and net conflict cost B

-

U, with marginal 95% CIs. Each condition has 192 paired queries. Attention has no RoPE.

Figure 4: (a) GDN2’s update advantage over matched attention (NSynth, 24k, 929 queries in 192 streams). (b,c) Rank-3 edits (ESC-50, 12k): 52 target and 148 other-source queries in 26 streams before target reappearance. (b) Net target accuracy loss. (c) Correct-to-wrong transitions divided by all eligible other-source queries. Stream-weighted means, three seeds, 95% CIs.

Audio Understanding

Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.

See all 215 papers

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models
Sep 26, 2026Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden +1Audio-Language Model Evaluation Long-Term Conversational Memory
University of Melbourne · University of New South Wales
Why Can't They Remember? Uncovering Representation and Retrieval Bottlenecks in Multi-Turn Acoustic Memory
May 26, 2026Yang Xiao, Siyi Wang, Han Yin +4Audio-Language Model Evaluation Memorization in Language Models
1The University of Melbourne · 2KAIST · 3The University of Auckland +1
Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Models
Sep 29, 2026Zhenhong Zhou, Xuanyue Zhao, Youji Liu +4Audio-Language Models Speech Enhancement
Nanyang Technological University · Beijing University of Posts and Telecommunications

Sep 26, 2026J/K move · Enter open · S save

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden, Ting Dang

University of Melbourne · University of New South Wales

Audio-Language Model Evaluation Long-Term Conversational Memory Audio-Language Models Long-Context Language Model Evaluation

Open paper AskPDF arXiv

Spoken conversational systems must recover information from prior interactions (i.e., memory), yet relevant information in speech extends beyond what was said to who said it, how it was spoken, and what was audible, information that exists only in the audio signal and cannot be recovered from a transcript. Beyond what to remember, memory also demands diverse operations: retrieving a single fact, integrating evidence across turns, tracking an evolving state. Real interactions further unfold across sessions, meaning information accumulates across distinct episodes rather than a single continuous recording. Existing benchmarks fall short on all three dimensions: they focus primarily on lexical content, adopt limited and ad hoc memory operations, and treat memory as a single-session problem. We argue that principled memory evaluation requires jointly characterizing the acoustic evidence to be retained and the operations applied to it, and introduce a taxonomy along these two axes. Building on this taxonomy, we present VoxMem: 3,196 evaluation instances over 34,743 spoken sessions (177 hours) crossing four acoustic evidence types (speech semantics, speaker identity, paralinguistic cues, environmental sound) with four memory operations (information extraction, multi-session reasoning, temporal tracking, and answer refusal), grounded in multi-session histories and stratified across context budgets from 8K to 64K tokens. Evaluating 15 LALMs, no model exceeds 40% at 32K. Models retain what was said far better than who said it, how, or what was audible, a gap that widens for complex operations, grows with history length, and manifests as qualitatively distinct failure modes across evidence types. VoxMem aims to provide a foundation to measure and drive progress on the full scope of spoken conversational memory.

Abstract

Audio Understanding

Explore similar work

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

Why Can't They Remember? Uncovering Representation and Retrieval Bottlenecks in Multi-Turn Acoustic Memory

Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Models