From Voiceprints to User Entities: Online Speaker-Memory Construction for Multi-User Spoken Agents

Yucong Zhang; Cancan Li; Bo Xiao; Ke Zeng; Ming Li

cs.SDOct 27, 2026

From Voiceprints to User Entities: Online Speaker-Memory Construction for Multi-User Spoken Agents

Authors: Yucong Zhang, Cancan Li, Bo Xiao, Ke Zeng, Ming Li

Organizations: School of Computer Science, Wuhan University · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen · Meituan

Abstract

Personalized spoken agents must identify speakers before accessing user-specific memory, yet memory benchmarks typically assume correctly attributed histories. We argue that voiceprints are observations of identity, not identities themselves, and formulate online speaker-memory construction to maintain persistent user entities as acoustic evidence evolves. We introduce EvoSpeakerMem, a training-free layer for causal speaker routing, selective acoustic-state updates, and identity-partitioned memory writes and retrieval through a replaceable text-memory backend. We also construct LoCoMo-ID, a spoken-memory benchmark with hidden speaker identities, speaker-sensitive queries, simulated recording-condition shifts, and four-speaker identity-confusion tests. On longitudinal SMIIP-TV, genuine-probe rejection under a frozen near-enrollment threshold rises from 29.8% to 55.6% over approximately one year. On LoCoMo-ID speaker-sensitive queries, gated updating improves identity-gated QA over fixed representations by 3.86 and 11.62 percentage points under clean and shifted conditions, respectively, with no wrong-user authorizations. The advantage persists in the four-speaker test. Yet reducing fragmentation and improving QA can coexist with incorrect cross-user links, revealing a fragmentation--contamination trade-off that QA alone cannot capture.

Figures & tables

Figure 1: Overview of EvoSpeakerMem as a speech support layer for replaceable textual-memory backends. Acoustic routing assigns each turn to a persistent user entity before its content is written to memory.

Condition	Acoustic evidence	Metric	Fixed	Update	Gated update	All-exemplar KNN
Clean	ECAPA-only	$F_{B^{3}}$ $\uparrow$	97.04	98.41	99.54	98.51
		FNUR $\downarrow$	0.77	0.27	0.31	0.21
		Clusters/Gold	3.217	1.767	1.883	1.583
	ECAPA + diarization	$F_{B^{3}}$ $\uparrow$	76.96	77.06	77.06	77.09
		FNUR $\downarrow$	0.08	0.07	0.07	0.07
		Clusters/Gold	1.150	1.117	1.117	1.117

Table 1: Online speaker-memory construction results on LoCoMo-ID clean speech and recording-shifted condition. Acoustic evidence distinguishes ECAPA-only routing from routing augmented with session-level diarization. We report partition quality (

F_{B^{3}}

) and fragmentation-related indicators (FNUR and Clusters/Gold). All rates are percentages except Clusters/Gold.

Gap (days)	EER	Below- $\tau$	Imp. accept	Gen. sim.
0–1 days	9.7	29.8	2.0	0.538
60–100 days	11.2	40.7	1.8	0.495
365–450 days	14.0	55.6	1.4	0.446

Table 2: Fixed-template verification across SMIIP-TV time gaps. Error rates are percentages. “Imp. accept” is the proportion of impostor trials exceeding the frozen threshold; “Gen. sim.” is mean genuine cosine similarity.

Acoustic evidence / policy	Clean	Condition-shifted
ECAPA / Fixed	72.31	69.27
ECAPA / Update	73.30	72.55
ECAPA / Gated update	73.38	72.04
ECAPA / All-exemplar KNN	73.38	72.79
Parakeet+ECAPA / Fixed	73.43	72.96
Parakeet+ECAPA / Update	73.50	73.16

Table 3: Matched all-memory QA (%) for eight policies at merge threshold 0.489. Results are micro-pooled judge-correct answers over speaker-sensitive queries without query-speaker filtering or privacy denial.

Condition	ECAPA policy	C/W/D	QA	QA $\mid$ Correct
Clean	Fixed	4084/0/29	71.41	71.91
	Update	3993/115/5	73.43	75.63
	Gated update	4110/0/3	75.27	75.33
	KNN	3993/116/4	73.64	75.86
Condition-shifted	Fixed	4079/1/33	60.15	60.65
	Update	3993/115/5	70.24	72.35

Table 4: Top-1 private-access results on 4,113 speaker-sensitive query audios under clean and shifted condition. C/W/D denotes correct, wrong, and denied authorizations; QA requires correct authorization and a judge-correct answer (%). KNN denotes All-exemplar KNN.

Acoustic evidence / policy	Online ID	$F_{B^{3}}$	C / W / D	Targeted QA ( $n=19$ )	Full QA ( $n=1{,}371$ )
ECAPA / Fixed	98.84	96.54	1342 / 18 / 11	78.95	67.83
ECAPA / Update	93.35	95.45	1335 / 32 / 4	78.95	72.06
ECAPA / Gated update	96.25	97.77	1368 / 0 / 3	78.95	73.60
ECAPA / All-exemplar KNN	94.85	97.20	1335 / 33 / 3	78.95	71.99
Parakeet+ECAPA / Fixed	74.03	72.86	1335 / 32 / 4	68.42	71.48
Parakeet+ECAPA / Update	71.96	72.57	1333 / 32 / 6	73.68	72.28

Table 5: Four-speaker interleaved stress test: ten clean collections, 11,764 history turns, and 40 gold speakers. Online ID excludes missing predictions. C/W/D counts correct/wrong/denied authorizations on 1,371 queries. Both QA columns require correct authorization and a judge-correct answer; targeted QA uses 19 gated items among 20 diagnostics. Rates are percentages. This differs from the standard 4,113-query condition.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Model	SR	SV
Qwen2.5-Omni Xu et al. (2025a)	19.00	24.00
AudioFlamingo-3 Goel et al. (2025)	23.00	29.00
Qwen3-Omni Xu et al. (2025b)	49.00	40.00
Kimi-Audio KimiTeam et al. (2025)	46.00	24.00
StepAudio2 Wu et al. (2025a)	34.00	43.00
MiMoAudio Xiaomi LLM-Core Team et al. (2025)	38.00	48.00

Appendix

Table 6: MSU-Bench speaker retrieval (SR) and verification (SV) accuracy (%). Unified speaker state uses deterministic tool readout.

	0–1 days		60–100 days		365–450 days
Encoder	EER	Below- $\tau$	EER	Below- $\tau$	EER	Below- $\tau$
ECAPA-TDNN	9.71	29.82	11.18	40.70	14.09	55.54
ReDimNet2-B6	3.34	5.19	4.82	8.78	6.35	12.96

Appendix

Table 7: Matched-set fixed-template verification at representative enrollment–probe gaps. EER and Below-

\tau

are percentages. The comparison uses the 205,765 text-independent utterances successfully processed by both encoders, with identical enrollment utterances, genuine probes, impostor trials, and time buckets. Each encoder uses its own threshold frozen after near-enrollment calibration at target FAR=2% (

\tau_{\mathrm{ECAPA}}=0.4752

;

\tau_{\mathrm{ReDimNet2}}=0.6131

), so threshold values are not compared directly. ReDimNet2-B6 lowers absolute error, but both encoders exhibit higher EER and a larger below-threshold genuine rate at longer gaps.

Figure 2: Matched-set short-term fixed-template verification for ECAPA and ReDimNet2-B6. Both encoders use identical utterances and time buckets, while each uses an encoder-specific threshold frozen after near-enrollment FAR=2% calibration. ReDimNet2-B6 lowers absolute EER and below-threshold genuine rate, but both measures increase as the enrollment–probe gap grows.

Figure 3: Matched-set long-term fixed-template verification for ECAPA and ReDimNet2-B6, including the approximately one-year follow-up. Although ReDimNet2-B6 substantially improves absolute separability, its EER and below-threshold genuine rate still worsen at longer enrollment–probe gaps.

	ECAPA-TDNN			ReDimNet2-B6
Large-span condition	$F_{B^{3}}$ $\uparrow$	FNUR $\downarrow$	Frag. $\downarrow$	$F_{B^{3}}$ $\uparrow$	FNUR $\downarrow$	Frag. $\downarrow$
Intra-day TOD	63.08	18.09	25.52	62.91	6.75	12.33
Inter-day	66.58	1.21	12.20	70.56	0.22	3.08
Long-term cross-batch	67.60	3.60	20.79	81.49	0.91	4.51
Environment	67.05	5.00	23.53	73.43	1.16	7.84
Speaker state	58.88	6.78	27.63	66.90	1.14	11.45

Appendix

Table 8: Matched-session large-span causal replay for the all-exemplar KNN policy. ECAPA-TDNN and ReDimNet2-B6 replay identical common-utterance session streams under encoder-specific thresholds calibrated with the same near-enrollment FAR=2% protocol. Frag. denotes speaker fragmentation (%). ReDimNet2-B6 improves absolute online entity maintenance in most conditions, particularly for long-term and context-variable replay; the near-equal intra-day B 3 F1 should be interpreted together with its lower FNUR and fragmentation.

Figure 4: Fixed-template verification over the full observed time range. The same low-drift-calibrated threshold is frozen for all buckets; the final bucket uses the approximately one-year follow-up collection.

Figure 5: Below-

\tau

and EER across time gaps in the unrestricted data and in the controlled observational subset. The controlled comparison reduces several measured context differences but does not establish a causal effect of time.

Figure 6: Speaker-level Below-

\tau

in the nearest and farthest within-release time buckets. Lines connect observations from the same speaker where both buckets are available.

Panel of Figure 7 — Figure 7: Fixed-template Below- $\tau$ and EER across recording environment, speaker state, and time of day. Probes are within 14 days of enrollment, and displayed categories contain at least 300 probes. Complete category breakdowns follow below.

Figure 8: Complete near-enrollment breakdown by recording environment. Sample sizes are shown in the plot; categories with small support should be interpreted cautiously.

Figure 9: Complete near-enrollment breakdown by reported speaker state. Lower Below-

\tau

in a category does not necessarily imply uniformly easier routing because impostor acceptance (shown as FMR in the figure) and sample support can differ.

Figure 10: Complete near-enrollment breakdown by time of day. All five periods have broad speaker coverage, but the comparison remains observational.

Dimension	Small span (control)	Large span (higher drift)	Purpose
Intra-day TOD	Same day and same time-of-day bucket as enrollment	Same day, at least two time-of-day buckets from enrollment	Isolate within-day change
Inter-day	0–3 days after enrollment	30–100 days after enrollment	Short-term temporal drift
Long-term cross-batch	Same-release SMIIP-TV probe	Follow-up probe approximately one year later	Long-term return
Environment	Within 14 days, same environment as enrollment	Within 14 days, different environment	Approximate environment contrast
Speaker state	Within 14 days, normal state	Within 14 days, non-normal state (e.g., sleepy, eating, hoarse, or exercise)	Approximate state contrast
Long-term $\times$ hist.-env.	Follow-up probes from speakers with low historical environment entropy	Follow-up probes from speakers with high historical environment entropy	Long-term return by prior variability

Appendix

Table 9: Small- and large-span definitions for the six SMIIP-TV online replay dimensions. Hist.-env. uses variability in the original SMIIP-TV histories.

Policy	$P_{\mathrm{create}}$ $\uparrow$	$R_{\mathrm{create}}$ $\uparrow$	$P_{\mathrm{link}}$ $\uparrow$	$R_{\mathrm{known}}$ $\uparrow$	$F_{B^{3}}$ $\uparrow$
First exemplar	21.77	52.90	66.25	62.90	46.82
Enrollment mean	27.23	42.83	57.18	56.51	53.58
Latest exemplar	23.95	54.14	34.60	33.10	32.20
Online mean	31.24	38.48	46.54	46.65	55.99
EMA	31.07	42.01	41.36	41.35	52.03
All-exemplar KNN	31.06	56.24	52.56	51.30	65.31

Appendix

Table 10: Macro-averaged large-span decision and B 3 results across six SMIIP-TV conditions (five seeds; %; higher is better).

Figure 11: Large-span B 3 F1 (%), averaged over five seeds. Bold marks the best policy per condition. TOD denotes time of day; Hist.-env. denotes historical environment variability.

Figure 12: Large-span CREATE/LINK decision metrics across six policies and six conditions (five-seed means; %; higher is better).

Acoustic evidence / policy	Clean	Condition-shifted
ECAPA / Fixed	65.53	61.09
ECAPA / Update	66.63	65.54
ECAPA / Gated update	66.87	64.99
ECAPA / All-exemplar KNN	66.65	66.25
Parakeet+ECAPA / Fixed	66.30	65.36
Parakeet+ECAPA / Update	65.93	65.74

Appendix

Table 11: Full-set all-memory QA (%) for eight policies on clean and condition-shifted speech. Each condition comprises 30 collections and 5,958 query audios. Scores are micro-pooled judge-correct rates without speaker-based retrieval filtering.

Condition	Acoustic evidence / policy	Turns	Entities	$P_{\mathrm{create}}$	$R_{\mathrm{create}}$	$P_{\mathrm{link}}$	$R_{\mathrm{known}}$	FNUR
Clean	ECAPA / Fixed	17643	193	30.05	96.67	99.98	99.22	0.77
	ECAPA / Update	17643	106	54.72	96.67	97.49	97.23	0.27
	ECAPA / Gated update	17643	113	51.33	96.67	99.95	99.65	0.31
	ECAPA / All-exemplar KNN	17643	95	61.05	96.67	97.48	97.29	0.21
	Parakeet+ECAPA / Fixed	17644	69	79.71	91.67	81.41	81.36	0.08
	Parakeet+ECAPA / Update	17644	67	82.09	91.67	81.40	81.37	0.07

Appendix

Table 12: History turn-micro causal routing results for all eight policies under clean speech and condition shifted at merge threshold 0.489. Values are percentages except Turns and Entities.

Condition	Acoustic evidence / policy	C	W	D	Safe auth.	QA	QA $\mid$ Correct
Clean	ECAPA / Fixed	4084	0	29	99.29	71.41	71.91
	ECAPA / Update	3993	115	5	97.08	73.43	75.63
	ECAPA / Gated update	4110	0	3	99.93	75.27	75.33
	ECAPA / All-exemplar KNN	3993	116	4	97.08	73.64	75.86
	Parakeet+ECAPA / Fixed	3833	273	7	93.19	68.08	73.05
	Parakeet+ECAPA / Update	3833	272	8	93.19	68.25	73.23

Appendix

Table 13: Speaker-sensitive Top-1 authorization and end-to-end answer utility over 4,113 questions per condition (clean and condition-shifted). C/W/D are counts, other values are percentages.

Policy	Online ID	$F_{B^{3}}$	C / W / D	Targeted strict (19)	Targeted raw (20)	Full strict (1,371)	QA $\mid$ Correct
ECAPA
Fixed	98.84	96.54	1342/18/11	15/19 (78.95)	15/20 (75.00)	930/1371 (67.83)	69.30
Update	93.35	95.45	1335/32/4	15/19 (78.95)	16/20 (80.00)	988/1371 (72.06)	74.01
Gated update	96.25	97.77	1368/0/3	15/19 (78.95)	15/20 (75.00)	1009/1371 (73.60)	73.76
KNN	94.85	97.20	1335/33/3	15/19 (78.95)	16/20 (80.00)	987/1371 (71.99)	73.93
Parakeet+ECAPA

Appendix

Table 14: Complete four-speaker results on ten clean collections. Targeted strict QA (19) and full strict QA (1,371) require correct authorization and a judge-correct answer; targeted raw judge (20) requires only a judge-correct answer and includes the non-gated item. C/W/D are full-set authorization counts. QA

\mid

Correct conditions full strict QA on C. All rates are percentages; KNN denotes All-exemplar KNN.

$\tau_{\mathrm{update}}$	Entities	Online (%)	B 3 F1 (%)	Query assigned / 5958	W / 4113	QA (%)
0.50	106	97.23	98.41	5953	115	73.33
0.54	110	98.65	99.11	5954	32	74.81
0.58	113	99.64	99.54	5955	0	75.27
0.62	114	99.64	99.54	5955	0	75.27
0.66	114	99.64	99.55	5955	0	75.47
0.70	118	99.61	99.45	5955	0	75.44

Appendix

Table 15: Clean ECAPA gated-update sweep, fixed merge threshold 0.489. Routing/cluster scores and query assigned are from the upstream replay; W and QA are from the separate complete downstream threshold-sweep runs, not the default-run main table.

Condition	ECAPA policy	$<1$ s: correct / $n$	$1$ – $<2$ s: correct / $n$	$\geq 2$ s: correct / $n$
Clean	Gated update	48 / 59 (81.36%)	375 / 382 (98.17%)	17157 / 17202 (99.74%)
	All-exemplar KNN	50 / 59 (84.75%)	372 / 382 (97.38%)	16742 / 17202 (97.33%)
Condition-shifted	Gated update	2 / 13 (15.38%)	195 / 252 (77.38%)	17223 / 17379 (99.10%)
	All-exemplar KNN	3 / 13 (23.08%)	210 / 252 (83.33%)	16894 / 17379 (97.21%)

Appendix

Table 16: Post-hoc duration-bucket first-seen-binding accuracy on evaluated history turns. Each policy/condition denominator sums to 17,643 (clean) or 17,644 (condition-shifted); the short bins are very small in the augmented condition.

Stage	Model	Decoding configuration
History and query transcription	Qwen3-Omni-30B-A3B-Instruct	Sampling: temperature 0.7, top- $p$ 0.8, top- $k$ 20; maximum 512 new tokens.
LightMem factual extraction	Qwen3-Omni-30B-A3B-Instruct	Sampling: temperature 0.1, top- $p$ 0.1; maximum 16,000 new tokens. The configured top- $k=1$ is not forwarded to the decoder.
Answer generation	Qwen3-Omni-30B-A3B-Instruct	Greedy decoding ( do_sample=False ); maximum 300 new tokens.
LLM-as-a-judge	Qwen3-Omni-30B-A3B-Instruct	Greedy decoding ( do_sample=False ); maximum 200 new tokens.
Text embedding	bge-small-en-v1.5	384-dimensional embedding; no generative decoding.

Appendix

Table 17: Models and generation settings for the reported LoCoMo-ID experiments.

Conversational Memory

Conversational memory equips AI assistants to retain, update, and retrieve useful information from past interactions for coherent, personalized dialogue.

See all 71 papers

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs
Sep 29, 2026Runqiu Xu, Zhisheng Zheng, David HarwathSpoken Dialogue Systems
The University of Texas at Austin, Austin, TX, USA
PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue
Oct 6, 2026Achira Lin, Siyuan Hou, Wenyi Yu +7Conversational Memory Persistent Memory for Language Models
Tsinghua University · Huawei Technologies Ltd.
Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations
Sep 26, 2026Wenxu Jia, Xize Cheng, Zihan Zhang +5Conversational Memory Multimodal Memory
Zhejiang University · Meituan

Sep 29, 2026J/K move · Enter open · S save

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs

Runqiu Xu, Zhisheng Zheng, David Harwath

The University of Texas at Austin, Austin, TX, USA

Spoken Dialogue Systems

Open paper AskPDF arXiv

Multi-user voice agents must track who said what across dialogue sessions. Text LLMs are attractive backbones for such agents, but transcripts alone do not expose acoustic speaker identity, leaving the model without a persistent reference for linking information to speakers across sessions. We address this gap by introducing Speaker Handles, soft-token representations that expose acoustic speaker identity to a frozen text LLM for cross-session speaker-dependent reasoning. A three-stage curriculum trains a lightweight projector, with fewer than 0.1% of the backbone's parameters, to map speaker embeddings into these handles. Establishing whether the resulting handles truly support cross-session speaker-dependent reasoning is challenging with existing benchmarks because textual cues can partially reveal fact ownership. We therefore present SpeakerBind, a controlled shared-agent benchmark in which overlapping facts across users require correct cross-session speaker attribution. Speaker Handles achieve 97.40-98.36% accuracy on VoxCeleb1 and 70.40% on SpeakerBind, close to the 71.88% topline. These results show that the proposed Speaker Handles provide an efficient way to integrate acoustic speaker identity into frozen text LLMs for speaker-content reasoning.

Abstract

Conversational Memory

Explore similar work

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs

PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue

Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations