cs.AIOct 19, 2026

When Should AI Chime In? Benchmarking Timely Contribution of Proactive Agents in Multi-Party Chat

Authors: Ruotong Wang, Brian S. Yu, Xinyi Zhou, Joseph Chee Chang, Pao Siangliulue, Jonathan Bragg, Liwei Jiang, Amy X Zhang

Organizations: University of Washington · Boise State University · Allen Institute for Artificial Intelligence

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Proactive agents can support group collaboration by surfacing useful information without waiting to be asked, but even relevant contributions can be disruptive if poorly timed. In this work, we isolate the timing question: given a fixed candidate contribution such as a clarification or relevant fact, when should an AI agent chime in? We introduce PITCH (Proactive Intervention Timing in CHat), to our knowledge the first benchmark with large-scale human judgments of proactive agent timing collected from participants in the same unscripted multi-party conversations they took part in. To collect high-quality judgments, we design a novel interface in which 2-4 people chat synchronously, author candidate AI contribution messages, place their own and others' contributions into the chat history, and then deliberate as a group about the best placement for each message. PITCH contains 384 candidate contributions and 1,091 judgments, each marking an ideal turn, an acceptable window, or that a contribution should not appear, from 89 participants engaged in 35 conversations. Evaluating 15 models across 28 configurations, we find that while models can place contributions well when given the entire conversation history, they systematically chime in too early when asked to decide live as the conversation unfolds: best-turn accuracy falls from 57.8% to 19.0%, with nearly half of placements occurring before every participant's acceptable window. In comparison, lay humans deciding live are more accurate (23.7% vs 19.0%), and fewer of their placements are premature (39.7% vs 46.4%). Model reasoning traces show that 93% of intervention justifications cite relevance, while almost none consider waiting. By providing a human-grounded benchmark, our work reveals gaps in agents' ability to integrate into group settings and points towards proactive agents that can reason explicitly about appropriate timing.

Figures & tables

Figure 1: Pitch ( P roactive I ntervention T iming in CH at) is the first benchmark for proactive intervention timing built from unscripted human conversations and participants’ own judgments of when proactive interjections should occur. We collected the benchmark data via a four-phrase protocol: Conversation, Message Contribution, Timing Judgments, and Deliberation. We then evaluated 15 models across 28 configurations against the benchmark in three task setups that differ in their task difficulty and realism for real-world deployment of agents in group conversations.
Figure 2: Left: spread between participants’ ideal turns. Right: participants scored against the rest of their own group.
Figure 3: Share of the annotator group a single placement satisfies, against the human reference (dashed). Pick reaches the human line; both replay task setups fall far short of it.
Figure 4: Where predictions land. Online, the single dominant outcome is firing before any annotator’s window opened; with hindsight it nearly vanishes and the residual error becomes lateness.
Figure 5: Where models place messages against where people do. The distributions barely overlap online but then nearly coincide with hindsight.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Message contribution interface. The full transcript sits in a scrollable pane on the left and the candidate message and placement controls on the right.
Figure 7: Timing judgment interface. Users navigate the transcript in the left pane while making annotation decisions in the right pane for each message contributed by other participants in the group.
Figure 8: Deliberation interface. Users are presented with the conversation transcript once again, alongside timing judgments made on a contribution message by all other participants. The right pane allows users to optionally revise and update their judgments.
 Human (Lay Users)Models
MetricScholarlyCollaborativeAggregate 
Anyone’s ideal turn22.4%24.6%23.7%19.0%
Within anyone’s window55.2%52.6%53.7%50.5%
Within unanimous window34.3%32.5%33.2%24.2%
Earlier than anyone’s window37.6%41.1%39.7%46.4%
Mean timing shift (turns)−4.3-4.3−11.8-11.8−8.7-8.7−11.1-11.1
Appendix
Table 4: Lay-user and model performance on intervention timing. Human performance is reported separately for scholarly and collaborative scenarios, as well as in aggregate (N=331 placements).
Figure 9: Timing drift by framing. Averaging the online and hindsight distributions would report a mild early bias that describes neither, which is why the framings are never pooled.
Figure 10: Every configuration against the human reference (dashed). In pick, three arms clear the human best-turn bar and eighteen clear the window bar; in both online framings, none do. (R) means reasoning mode and (S) means standard model configuration
Figure 11: Anyone’s ideal turn for all 28 configurations. The gap between the pick bar and the replay bar is wider than the entire spread across models within either framing, and it is positive for every configuration. (R) indicates models in native reasoning mode with max effort; (S) indicates models in lowest reasoning effort.
More in this topic

Proactive Agents

See all 34 papers

Explore similar work

CardsList
Oct 5, 2026

Jarvis: A Proactive Speech Agent for Multi-Party Conversations

Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.

Seunghyun Oh, Hirotaka Hiraki, Shuyue Stella Li +2

Paul G. Allen School of Computer Science and Engineering, University of Washington, WA, USA · Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Bunkyo, Tokyo, Japan

Conversational AgentsProactive AgentsProactive AssistanceSpoken Dialogue SystemsAutomated Fact-Checking
AskPDF
Jul 4, 2026

ProACT: Towards Breakdown-Aware Proactive Agent in Multi-User Collaboration

Conversational agents are increasingly embedded in human collaborative work, yet they remain fundamentally passive and reactive: they respond to explicit user requests rather than proactively recognizing moments when a team would benefit from timely intervention as human collaborators often do. This reactive design substantially limits the use of agents as active participants in multi-user collaboration, where disagreements, ambiguous goals, forgotten constraints, underspecified plans, discussion loops, and imbalanced participation can gradually undermine group progress. To move agents from passive assistants toward active participants in multi-user collaboration, we introduce ProACT, a breakdown-aware agent framework grounded in theories of common ground, collaborative planning, and coordination work. ProACT observes the speaker-attributed conversation history, determines whether the current turn contains a collaboration breakdown requiring intervention, decides whether the agent should stay silent or speak, and, when speaking is needed, routes the case to a targeted collaboration skill. We further introduce the first multi-user collaboration benchmark for evaluating proactive agents across project planning, product design, research collaboration, logistics, education, and resource-constrained decision making. Across 3,244 turn-level examples and five LLM backbones, ProACT consistently improves collaborative appropriateness, non-interruptiveness, conciseness, and judged intervention quality over direct chat.

Shu Yang, Difei Xu, Jiaxin Pei +1

1PRADA Lab, King Abdullah University of Science and Technology · 2Stanford University

LLM Agent EvaluationProactive AssistanceConversational AgentsHuman-AI CollaborationProactive Agents
AskPDF
Jul 20, 2026

ProEvent: An Event-centric Benchmark for Proactive Agents

Proactive agents are expected to anticipate user needs and provide autonomous assistance by perceiving environmental context without explicit instructions. A fundamental capability of such agents is to identify and track users' upcoming events, enabling continuous and event-specific assistance. For example, by recording the time and location of a planned hike, an agent can deliver weather reminders in advance or provide navigation support before departure. However, existing works on proactive agents largely overlook event-centric assistance, and the open-ended nature of proactive assistance poses challenges for reliable evaluation. To bridge these gaps, we introduce ProEvent, the first event-centric benchmark designed to assess an agent's ability to proactively maintain a user's timetable based on ongoing instant messaging chats. ProEvent provides synthesized yet realistic chats that consider the dynamic interaction among users, concurrent chat threads, and noise in the real world, and evaluates proactive agents on response timing, single-step response correctness, and multi-step response correctness. Experiments on eight LLMs and pipelines reveal that current agents frequently overact and struggle with event cancellation. Notably, even GPT-5.1 only reacts correctly in 26.7% of scenarios. Further qualitative analysis reveals fundamental limitations of current LLMs as proactive agents, particularly in detecting implicit events and reasoning from the user's first-person perspective.

Guanzhen Li, Liangming Pan, Leye Wang

School of Computing, Peking University

Proactive AgentsLLM Agent EvaluationProactive AssistanceAgentic AI
AskPDF