Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement

Fengrui Liu; Xiao He; Tieying Zhang

cs.AIFeb 25, 2026

Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement

Authors: Fengrui Liu, Xiao He, Tieying Zhang

Organizations: ByteDance Beijing, China · ByteDance Hangzhou, China · ByteDance San Jose, United States

Abstract

In large-scale cloud service platforms, thousands of customer tickets are generated daily and are typically handled through on-call dialogues. This high volume of on-call interactions imposes a substantial workload on human support analysts. Recent studies have explored reactive agents that leverage large language models as a first line of support to interact with customers directly and resolve issues. However, when issues remain unresolved and are escalated to human support, these agents are typically disengaged. As a result, they cannot assist with follow-up inquiries, track resolution progress, or learn from the cases they fail to address. In this paper, we introduce Vigil, a novel proactive agent system designed to operate throughout the entire on-call life-cycle. Unlike reactive agents, Vigil focuses on providing assistance during the phase in which human support is already involved. It integrates into the dialogue between the customer and the analyst, proactively offering assistance without explicit user invocation. Moreover, Vigil incorporates a continuous self-improvement mechanism that extracts knowledge from human-resolved cases to autonomously update its capabilities. Vigil has been deployed on Volcano Engine, ByteDance's cloud platform, for over ten months, and comprehensive evaluations based on this deployment demonstrate its effectiveness and practicality. The open source version of this work is publicly available at https://github.com/volcengine/veaiops.

Figures & tables

Panel of Figure 1 — Figure 1. Comparison of reactive and proactive agent paradigms in the on-call support process. (a) Reactive agents disengage upon escalation, whereas (b) proactive agents extend the action-cycle to collaborate with human support analysts.

Figure 2. Framework of Vigil . Features with two primary functions: (1) Online Proactive Response; (2) Continuous Self-Improvement.

Figure 3. Proactive response card of Vigil . Including a distinct layout for distinguishing agent from human, explicit citations for verifiability and an Accept button for collecting feedback.

Figure 4. Continuous self-improvement framework. Vigil learns from answered questions, unanswered questions, and external documents shared in the on-call dialogue.

Panel of Figure 5 — Figure 5. Volcano Engine On-Call Statistics, June 23 – July 22, 2025

Foundation	$Precision_{w}$			$Recall_{w}$ /Accuracy			$F1_{w}$
Model	Baseline	Vigil	$\Delta$ (%)	Baseline	Vigil	$\Delta$ (%)	Baseline	Vigil	$\Delta$ (%)
GPT-5	0.789	0.870	10.3%	0.620	0.850	37.1%	0.660	0.857	29.8%
GPT-5-mini	0.849	0.875	3.0%	0.307	0.793	158.7%	0.268	0.813	203.2%
Gemini-2.5-flash	0.811	0.859	6.0%	0.700	0.737	5.2%	0.730	0.764	4.6%
Gemini-2.5-pro	0.784	0.827	5.5%	0.757	0.830	9.7%	0.768	0.828	7.9%
QWen-VL-72b	0.848	0.827	-2.5%	0.287	0.677	136.0%	0.235	0.711	202.0%

Table 1. Evaluation of question identification of Vigil across various foundation models.

Foundation	$w/o$	$w/o$	$w/$
Model	self-impr.	answer review	Vigil
GPT-5	0.389	0.537 (37.9% $\uparrow$ )	0.624 (60.3% $\uparrow$ )
GPT-5-mini	0.178	0.322 (81.1% $\uparrow$ )	0.493 (177.4% $\uparrow$ )
Gemini-2.5-flash	0.161	0.399 (147.9% $\uparrow$ )	0.423 (162.5% $\uparrow$ )
Gemini-2.5-pro	0.252	0.430 (70.7% $\uparrow$ )	0.456 (81.3% $\uparrow$ )
Qwen-VL-72b	0.225	0.322 (43.3% $\uparrow$ )	0.396 (76.1% $\uparrow$ )

Table 2. Ablation Study on the Answer Accuracy of Vigil

$\theta$	$Precision$	$Recall$	$Precision_{w}$	$Recall_{w}$	$F1_{w}$
0.00 a	1.000	0.396	0.864	0.509	0.542
0.20	0.996	0.398	0.861	0.510	0.543
0.40	0.886	0.585	0.771	0.602	0.646
0.60	0.826	0.921	0.730	0.778	0.747
0.70	0.822	0.982	0.760	0.812	0.752
0.80	0.817	0.994	0.770	0.814	0.740

Table 3. Evaluation metrics of answer deduplication under different threshold

\theta

Figure 6. Vigil operates different on-call dialogues for the same critical issue (simplified and anonymized).

Figure 7. Vigil reviews the unaccepted answer and refines its knowledge base (simplified and anonymized).

Proactive Assistance

Proactive assistance studies how AI systems can anticipate users’ needs and decide what help to offer, when to offer it, or whether to stay silent.

See all 46 papers

Self-evolving Agentic Customer Support System at LinkedIn
Aug 10, 2026Chih Hui Wang, Mengdie Tu, Qianyun Zhang +4Customer Support Automation LLM Agent Evaluation
LinkedIn
VIGIL: Towards Edge-Extended Agentic AI for Enterprise IT Support
Mar 17, 2026Sarthak Ahuja, Neda Kordjazi, Evren Yortucboylu +7Agentic AI Human-Centered AI
Amazon Engine, AI Center of Excellence
ProEvent: An Event-centric Benchmark for Proactive Agents
Jul 20, 2026Guanzhen Li, Liangming Pan, Leye WangProactive Agents LLM Agent Evaluation
School of Computing, Peking University

Aug 10, 2026J/K move · Enter open · S save

Self-evolving Agentic Customer Support System at LinkedIn

Chih Hui Wang, Mengdie Tu, Qianyun Zhang, Wei Wu+3

Customer Support Automation LLM Agent Evaluation Agentic RAG Self-Evolving Agents

Open paper AskPDF arXiv

Enterprise support agents operate in rapidly changing environments where policies, product capabilities, and knowledge bases evolve continuously, making static assistants brittle and costly to maintain. We present LinkedIn's self-evolving agentic support system, which integrates retrieval-augmented generation with evolutionary auto-prompting and a modular, production-aligned evaluation framework to enable safe, continuous improvement without retraining foundation models. The system treats prompts, retrieval, and evaluation as a closed-loop, versioned workflow with operational guardrails. Offline simulations and ablations show clear quality gains over vanilla RAG and baseline agents, including reduced hallucinations and improved response completeness. In a two-week user-randomized A/B test on LinkedIn's production support traffic, the integrated self-evolved workflow increased QA self-serve by 9.0 percentage points, cancellation self-serve by 4.8 points, and routing accuracy by 30.6 points. These results demonstrate a practical path to scalable, self-evolving AI agents in real-world enterprise settings.

Abstract

Proactive Assistance

Explore similar work

Self-evolving Agentic Customer Support System at LinkedIn

VIGIL: Towards Edge-Extended Agentic AI for Enterprise IT Support

ProEvent: An Event-centric Benchmark for Proactive Agents