In large-scale cloud service platforms, thousands of customer tickets are generated daily and are typically handled through on-call dialogues. This high volume of on-call interactions imposes a substantial workload on human support analysts. Recent studies have explored reactive agents that leverage large language models as a first line of support to interact with customers directly and resolve issues. However, when issues remain unresolved and are escalated to human support, these agents are typically disengaged. As a result, they cannot assist with follow-up inquiries, track resolution progress, or learn from the cases they fail to address. In this paper, we introduce Vigil, a novel proactive agent system designed to operate throughout the entire on-call life-cycle. Unlike reactive agents, Vigil focuses on providing assistance during the phase in which human support is already involved. It integrates into the dialogue between the customer and the analyst, proactively offering assistance without explicit user invocation. Moreover, Vigil incorporates a continuous self-improvement mechanism that extracts knowledge from human-resolved cases to autonomously update its capabilities. Vigil has been deployed on Volcano Engine, ByteDance's cloud platform, for over ten months, and comprehensive evaluations based on this deployment demonstrate its effectiveness and practicality. The open source version of this work is publicly available at https://github.com/volcengine/veaiops.
Figures & tables
Figure 1. Comparison of reactive and proactive agent paradigms in the on-call support process. (a) Reactive agents disengage upon escalation, whereas (b) proactive agents extend the action-cycle to collaborate with human support analysts.
Figure 2. Framework of Vigil . Features with two primary functions: (1) Online Proactive Response; (2) Continuous Self-Improvement.
Figure 3. Proactive response card of Vigil . Including a distinct layout for distinguishing agent from human, explicit citations for verifiability and an Accept button for collecting feedback.
Figure 4. Continuous self-improvement framework. Vigil learns from answered questions, unanswered questions, and external documents shared in the on-call dialogue.
Figure 5. Volcano Engine On-Call Statistics, June 23 – July 22, 2025
Foundation
Precisionw
Recallw /Accuracy
F1w
Model
Baseline
Vigil
Δ (%)
Baseline
Vigil
Δ (%)
Baseline
Vigil
Δ (%)
GPT-5
0.789
0.870
10.3%
0.620
0.850
37.1%
0.660
0.857
29.8%
GPT-5-mini
0.849
0.875
3.0%
0.307
0.793
158.7%
0.268
0.813
203.2%
Gemini-2.5-flash
0.811
0.859
6.0%
0.700
0.737
5.2%
0.730
0.764
4.6%
Gemini-2.5-pro
0.784
0.827
5.5%
0.757
0.830
9.7%
0.768
0.828
7.9%
QWen-VL-72b
0.848
0.827
-2.5%
0.287
0.677
136.0%
0.235
0.711
202.0%
Table 1. Evaluation of question identification of Vigil across various foundation models.
Foundation
w/o
w/o
w/
Model
self-impr.
answer review
Vigil
GPT-5
0.389
0.537 (37.9% ↑ )
0.624 (60.3% ↑ )
GPT-5-mini
0.178
0.322 (81.1% ↑ )
0.493 (177.4% ↑ )
Gemini-2.5-flash
0.161
0.399 (147.9% ↑ )
0.423 (162.5% ↑ )
Gemini-2.5-pro
0.252
0.430 (70.7% ↑ )
0.456 (81.3% ↑ )
Qwen-VL-72b
0.225
0.322 (43.3% ↑ )
0.396 (76.1% ↑ )
Table 2. Ablation Study on the Answer Accuracy of Vigil
θ
Precision
Recall
Precisionw
Recallw
F1w
0.00 a
1.000
0.396
0.864
0.509
0.542
0.20
0.996
0.398
0.861
0.510
0.543
0.40
0.886
0.585
0.771
0.602
0.646
0.60
0.826
0.921
0.730
0.778
0.747
0.70
0.822
0.982
0.760
0.812
0.752
0.80
0.817
0.994
0.770
0.814
0.740
Table 3. Evaluation metrics of answer deduplication under different threshold θ .
Figure 6. Vigil operates different on-call dialogues for the same critical issue (simplified and anonymized).
Figure 7. Vigil reviews the unaccepted answer and refines its knowledge base (simplified and anonymized).
Enterprise support agents operate in rapidly changing environments where policies, product capabilities, and knowledge bases evolve continuously, making static assistants brittle and costly to maintain. We present LinkedIn's self-evolving agentic support system, which integrates retrieval-augmented generation with evolutionary auto-prompting and a modular, production-aligned evaluation framework to enable safe, continuous improvement without retraining foundation models. The system treats prompts, retrieval, and evaluation as a closed-loop, versioned workflow with operational guardrails. Offline simulations and ablations show clear quality gains over vanilla RAG and baseline agents, including reduced hallucinations and improved response completeness. In a two-week user-randomized A/B test on LinkedIn's production support traffic, the integrated self-evolved workflow increased QA self-serve by 9.0 percentage points, cancellation self-serve by 4.8 points, and routing accuracy by 30.6 points. These results demonstrate a practical path to scalable, self-evolving AI agents in real-world enterprise settings.
Enterprise IT support is constrained by heterogeneous devices, evolving policies, and long-tail failure modes that are difficult to resolve centrally. We present VIGIL, an edge-extended agentic AI system that deploys desktop-resident agents to perform situated diagnosis, retrieval over enterprise knowledge, and policy-governed remediation directly on user devices with explicit consent and end-to-end observability. In a 10-week pilot of VIGIL's operational loop on 100 resource-constrained endpoints, VIGIL reduces interaction rounds by 39%, achieves at least 4 times faster diagnosis, and supports self-service resolution in 82% of matched cases. Users report excellent usability, high trust, and low cognitive workload across four validated instruments, with qualitative feedback highlighting transparency as critical for trust. Notably, users rated the system higher when no historical matches were available, suggesting on-device diagnosis provides value independent of knowledge base coverage. This pilot establishes safety and observability foundations for fleet-wide continuous improvement.
Proactive agents are expected to anticipate user needs and provide autonomous assistance by perceiving environmental context without explicit instructions. A fundamental capability of such agents is to identify and track users' upcoming events, enabling continuous and event-specific assistance. For example, by recording the time and location of a planned hike, an agent can deliver weather reminders in advance or provide navigation support before departure. However, existing works on proactive agents largely overlook event-centric assistance, and the open-ended nature of proactive assistance poses challenges for reliable evaluation. To bridge these gaps, we introduce ProEvent, the first event-centric benchmark designed to assess an agent's ability to proactively maintain a user's timetable based on ongoing instant messaging chats. ProEvent provides synthesized yet realistic chats that consider the dynamic interaction among users, concurrent chat threads, and noise in the real world, and evaluates proactive agents on response timing, single-step response correctness, and multi-step response correctness. Experiments on eight LLMs and pipelines reveal that current agents frequently overact and struggle with event cancellation. Notably, even GPT-5.1 only reacts correctly in 26.7% of scenarios. Further qualitative analysis reveals fundamental limitations of current LLMs as proactive agents, particularly in detecting implicit events and reasoning from the user's first-person perspective.