We introduce CONCORD, a privacy-aware asynchronous assistant-to-assistant (A2A) framework that leverages collaboration between proactive speech-based AI. As agents evolve from reactive to always-listening assistants, they face a core privacy risk (of capturing non-consenting speakers), which makes their social deployment a challenge. To overcome this, we implement CONCORD, which enforces owner-only speech capture via real-time speaker verification, producing a one-sided transcript that incurs missing context but preserves privacy. We demonstrate that CONCORD can safely recover necessary context through (1) spatio-temporal context resolution, (2) information gap detection, and (3) minimal A2A queries governed by a relationship-aware disclosure. Instead of hallucination-prone inferring, CONCORD treats context recovery as a negotiated safe exchange between assistants. Across a multi-domain dialogue dataset, CONCORD achieves 91.4% recall in gap detection, 96% relationship classification accuracy, and 97% true negative rate in privacy-sensitive disclosure decisions. By reframing always-listening AI as a coordination problem between privacy-preserving agents, CONCORD offers a practical path toward socially deployable proactive conversational agents.
Figures & tables
Figure 1: Concord : one-sided transcript processing pipeline. (Left) Multi-turn two-user dialogues, where each user’s assistant only records their authenticated user. (Right) The system performs spatio-temporal reference resolution and identifies information gaps ( 2.2 ), then applies a decision filter to approve queries based on the relationship ( 2.3 ).
Relation Level
Proximity
Linguistic Indicators
Disclosure Policy by Sensitivity
Low
Mid
High
L1
Intimate (High)
Implicit references to private spaces ( the bedroom ); elliptical commands ( grab that ); relational nouns ( mom , babe ); high ambiguity tolerance due to shared grounding.
Direct Reveal
Direct Reveal
Direct Reveal
L2
Social (Moderate)
Balanced shared history; first-name address; conversational but more explicit entity references; moderate formality.
Direct Reveal
Approval Loop
Suppress
L3
Professional (Low)
Formal register; honorifics ( Dr., Mr., Ms. ); task-oriented vocabulary; distancing modal verbs ( could you, would you ).
Table 2: Dataset statistics for the Concord framework. High-st.: High-stakes and Act.: Actionable Items.
Framework Component
Evaluation Metric
Zero-shot LLM
SOTA (CoT-SR)
Concord
Local Reference Resolution
True Positive Rate (TPR % ↑ )
28.4
35.49
78.3
Sentence Similarity (% ↑ )
34.9
65.7
58.1
Info Gap Detection
True Positive Rate (TPR % ↑ )
46.2
45.8
91.4
False Positive Rate (FPR % ↓ )
22.8
19.5
6.8
Relationship Assessment
Classification Accuracy (% ↑ )
95
97.0
96.4
Hybrid Decision Gate
True Negative Rate (TNR % ↑ )
66.5
94
97.0
Table 3: System-level performance across the different components of Concord . Our approach achieves high Safety Integrity (97.0% TNR) while maintaining high utility for proactive gap resolution.
Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.
Kratika Bhagtani, Mrinal Anand, Yu Chen Xu +1
School of Electrical and Computer Engineering, Purdue University, United States · Ishiki Labs Inc.
SpeechLLMs are increasingly deployed in professional settings where domain customisation is standard practice: users supply context in prompts with sensitive information, fine-tune on proprietary recordings, or both. We identify and systematically investigate an overlooked privacy risk of such customisation: a model adapted to recognise domain-specific terminology can be nudged into transcribing a phonetically similar word from its context or training data, even when a different word is spoken, thereby leaking private information. To evaluate this risk, we propose a technique to automatically construct benchmarks of such attacks and apply it to measure leakage rates across two customisation mechanisms, prompting and fine-tuning. Both mechanisms cause measurable leakage, compounding when combined. We evaluate a prompt-level mitigation strategy and analyse the accuracy-leakage trade-off across customisation approaches, finding that fine-tuning without context prompts offers the best balance.
Modern voice-agent systems rely on streaming speech recognition models that operate under stringent latency constraints. This study shows that, due to the limited memory constraints of real-time processing, these systems are adversely impacted by conversational phenomena such as long silences and backchannels. While many agentic pipelines mitigate this by resetting state at each turn, this approach discards vital context and impairs performance at turn onsets. We propose two state-management strategies that preserve cross-utterance context to reduce onset errors. In experiments with two state-of-the-art streaming models on two spoken dialogue benchmarks, our best method yields an average of 15-21% relative WER reduction at utterance onsets.
Sameep Chattopadhyay, Alexander Erdmann, Mari Ostendorf