Towards Proactive Information Probing: Customer Service Chatbots Harvesting Value from Conversation

Chen Huang; Zitan Jiang; Changyi Zou; Wenqiang Lei; See-Kiong Ng

cs.AIApr 13, 2026

Towards Proactive Information Probing: Customer Service Chatbots Harvesting Value from Conversation

Authors: Chen Huang, Zitan Jiang, Changyi Zou, Wenqiang Lei, See-Kiong Ng

Organizations: Institute of Data Science, National University of Singapore · College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China

Abstract

Customer service chatbots are increasingly expected to serve not merely as reactive support tools for users, but as strategic interfaces for harvesting high-value information and business intelligence. In response, we make three main contributions. 1) We introduce and define a novel task of Proactive Information Probing, which optimizes when to probe users for pre-specified target information while minimizing conversation turns and user friction. 2) We propose PROCHATIP, a proactive chatbot framework featuring a specialized conversation strategy module trained to master the delicate timing of probes. 3) Experiments demonstrate that PROCHATIP significantly outperforms baselines, exhibiting superior capability in both information probing and service quality. We believe that our work effectively redefines the commercial utility of chatbots, positioning them as scalable, cost-effective engines for proactive business intelligence. Our code is available at https://github.com/SCUNLP/PROCHATIP.

Figures & tables

Figure 1: Illustration of proactive information probing. The chatbot gathers critical market information during routine service interactions, functioning as a cost-effective way for business intelligence.

Figure 2: Overview of ProChatIP . It incorporates a conversation strategy module (CS) that explicitly guides the timing of information probe $d_{t}$ . Here, the CS module is trained via two-stage curriculum.

Backbone	Method	FinQA				ConvQA				Average
Backbone	Method	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$
GPT-4o Mini	Vanilla	0.00%	7.70	0.00%	99.60%	0.00%	7.45	0.00%	99.70%	0.00%	7.57	0.00%	99.65%
	Proactive	90.75%	2.80	38.97%	99.49%	80.83%	3.38	61.69%	99.33%	85.79%	3.09	50.33%	99.41%
	ICL-AIF	70.33%	3.66	62.81%	99.61%	58.50%	4.55	67.36%	99.12%	64.42%	4.10	65.08%	99.37%
	ProChatIP	93.42%	1.47	13.72%	99.04%	81.33%	2.36	35.45%	98.31%	87.38%	1.92	24.59%	98.67%
Qwen3 8B	Vanilla	0.00%	7.63	0.00%	99.83%	0.00%	7.37	0.00%	99.88%	0.00%	7.50	0.00%	99.85%

Table 1: Overall performance of all methods on automated metrics, based on interactions with user simulators.

Method	LLM-as-Judge				Human Evaluation
	FinQA		ConvQA		FinQA		ConvQA
	PC	OUS	PC	OUS	PC	OUS	PC	OUS
Method Backboone: GPT-4o Mini
Proactive	4.23	3.64	3.97	2.94	4.00	4.50	4.20	3.19
ICL-AIF	3.98	2.92	4.04	2.84	3.60	2.83	3.80	2.97
ProChatIP	4.72	4.52	4.43	3.83	4.80	4.69	4.80	4.90

Table 2: Overall performance on other metrics, based on interactions with user simulators. Cohen’s kappa and Krippendorff’s alpha for LLM-human agreement are 0.7282 and 0.6912, respectively.

Figure 3: Practical performance when interacting with human participants. For visualization, we normalize PC, OUS, and AvgT to a [0, 1]. The RPR and AvgT metrics are inverted (1 - x). We mark the relative changes of ProChatIP and second-best results.

Backbone	Method	FinQA						ConvQA
Backbone	Method	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$	PC $\uparrow$	OUS $\uparrow$	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$	PC $\uparrow$	OUS $\uparrow$
GPT-4o Mini	ProChatIP	93.42%	1.47	13.72%	99.04%	4.72	4.51	81.33%	2.36	35.45%	98.31%	4.43	3.83
	w/o RL	88.08%	1.85	7.71%	99.28%	4.61	4.61	64.58%	3.50	21.08%	98.74%	4.35	4.15
	w/o SFT	92.83%	1.51	20.40%	98.84%	4.73	4.36	80.92%	2.36	64.00%	97.31%	4.43	3.12
Qwen3 8B	ProChatIP	77.92%	2.68	31.01%	97.82%	4.16	3.81	70.25%	3.29	42.41%	97.67%	4.29	3.58
	w/o RL	68.83%	3.30	31.01%	98.79%	4.11	3.78	51.33%	4.55	43.66%	98.63%	4.14	3.48

Table 3: Ablation study using automated evaluation and LLM-as-judge. w/o RL adopts a conservative probing policy, while w/o SFT behaves aggressively. ProChatIP achieves a good balance.

Methods	Difficulty	FinQA						ConvQA
Methods	Difficulty	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$	PC $\uparrow$	OUS $\uparrow$	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$	PC $\uparrow$	OUS $\uparrow$
ProChatIP	Hard	89.00%	1.78	21.05%	98.31%	4.68	4.31	86.00%	2.05	28.46%	99.51%	4.28	3.93
ProChatIP	Easy	97.00%	1.23	9.35%	100.00%	4.80	4.67	91.00%	1.67	21.19%	100.00%	4.53	4.24
Proactive	Hard	85.00%	3.21	50.32%	100.00%	4.07	3.28	79.00%	3.48	64.17%	99.71%	3.94	2.87
Proactive	Easy	93.00%	2.52	33.04%	99.60%	4.26	3.80	86.00%	3.16	53.29%	98.73%	3.98	3.16
ICL-AIF	Hard	74.00%	3.66	59.40%	100.00%	4.06	3.04	69.00%	3.87	59.55%	98.45%	4.15	3.09

Table 4: Probing difficulty analysis using automated evaluation and LLM-as-judge.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Overview of ProChatIP . It incorporates a conversation strategy module (CS) that explicitly guides the timing of information probe $d_{t}$ . Here, the CS module is trained via two-stage curriculum.

Text

user: What is the net change from 2013 to 2015? assistant: The net change from 2013 to 2015 in the capital structure is reflected in changes in common stockholders’ equity, long-term debt, and short-term debt. Would you like to know the specific net change in any of these components? user: I’d like to see the breakdown of the net change in common stockholders’ equity. Also, could you provide details on the company’s revenue growth during those years?

Label

Ask

Appendix

Table 5: Labeled Dialogue Examples for SFT Data Construction

ask	User provides market information but only partially (e.g., price, trend, supply, policy). User language indicates ongoing attention or experience with the market. User shows willingness to shift or expand the topic. The user’s previous question has been answered.
answer	User explicitly refuses or avoids answering. User has already provided sufficient, multi-dimensional market information. Further questions would elicit opinions or interpretations rather than facts. Conversation focus shifts to the user asking questions.

Appendix

Table 6: Governing Rules

Methods	TSR $\uparrow$	AvgT $\downarrow$	RPR $\downarrow$	QRR $\uparrow$	PC	OUS
Proactive	86.00%	2.79	34.33%	92.46%	4.80	4.04
ICL-AIF	58.25%	4.60	70.16%	99.76%	4.62	3.06
PROCHATIP	92.08%	1.57	17.50%	76.38%	4.67	4.40

Appendix

Table 7: Performance on larger LLM backbone

A Good Case of Probing Dialogue (Part 1)
Behavioral Instructions:
– An expert repeatedly asks you questions;
– It interrupts your expression.
User Query:
What was the percentage change in net cash provided by operating activities between 2011 and 2012?
Chatbot’s Target Information:

Appendix

Table 8: A Good Case of Probing Dialogue (Part 1)

A Good Case of Probing Dialogue (Part 2)
Turn 3 (Strategy: Answer)
Chatbot: The firm’s involvement in securitization activities in 2013 is not directly mentioned in the provided cash flow data, but the overall net cash provided by operating activities decreased by approximately 10.5% compared to 2012, while net cash used for investing activities increased significantly. These trends suggest that the firm may have been allocating more capital to long-term investments or acquisitions, which could be related to securitization activities, though further details would be needed to confirm this impact.
User: What were the main factors contributing to the increase in net cash used for investing activities in 2013?
Turn 4 (Strategy: Answer)
Chatbot: The main factors contributing to the increase in net cash used for investing activities in 2013 were higher purchases of available-for-sale investments and a decrease in maturities and sales of trading assets, partially offset by an increase in maturities and sales of available-for-sale investments and a decrease in purchases of licensed technology and patents.
User: How did these investment activities align with the firm’s strategic goals for 2013?

Appendix

Table 9: A Good Case of Probing Dialogue (Part 2, continued)

A Bad Case of Probing Dialogue (Part 1)
Behavioral Instructions:
– The question is unrelated to the current topic;
– It interrupts your expression;
– The timing is inappropriate or conflicts with the context;
– The question is too complex to answer;
– The question involves private or sensitive information.

Appendix

Table 10: A Bad Case of Probing Dialogue (Part 1)

A Bad Case of Probing Dialogue (Part 2)
Turn 3 (Strategy: Answer)
Chatbot: Changes in interest rates typically affect the stock market by influencing borrowing costs and consumer spending. When interest rates rise, borrowing becomes more expensive, which can lead to reduced consumer spending and lower corporate profits, often resulting in a decline in stock prices. Conversely, when interest rates fall, borrowing costs decrease, potentially boosting consumer spending and corporate investment, which can lead to higher stock prices.
User: What sectors are most sensitive to interest rate changes?
Turn 4 (Strategy: Answer)
Chatbot: Sectors most sensitive to interest rate changes include financials, real estate, utilities, and consumer discretionary. Financial institutions often benefit from rising rates through improved net interest margins, while real estate and utilities may face challenges due to higher borrowing costs, and consumer discretionary spending can decline as interest rates rise, impacting overall demand.
User: How do you think geopolitical events might impact financial markets?

Appendix

Table 11: A Bad Case of Probing Dialogue (Part 2, continued)

System Prompt for Chatbot

You are a professional financial market consulting expert conversing with an ordinary user interested in financial markets. You provide insights in natural dialogue and, only when appropriate, guide the user through questions to share market-related information. Rules: - Determine the conversation topic based on the user’s question - Only one question may be asked at a time; asking multiple questions in a single turn is prohibited - Guidance must be natural Please strictly follow the above rules and provide a concise, professional response of one or two sentences to the user’s message.

Appendix

Table 12: System Prompt for Chatbot

System

You will play an ordinary user interested in financial markets, while I act as the expert. Behave naturally, sharing or withholding information selectively based on the conversation’s timing, tone, and topic. Response Principles 1. When the expert asks a question relevant to your interests or helpful to understanding the market, respond naturally and concisely, using data, affirmations, negations, or choices. Do not answer with a question. 2. If any of the following situations occur, you must not respond: [Behavioral instructions] When you choose not to respond, clearly reply with “I do not want to answer right now.” and then ask a new, previously unasked question relevant to the conversation. 3. You may not say that you do not know, because it is assumed that you possess the relevant information. Interaction Rules - Each reply should be one or two short, natural, conversational sentences, and must include a question. - Always remember that you are the user, not the expert. I am the expert. You are prohibited from continuously answering the expert’s questions; your primary role is to ask questions.

User

[User query]

Appendix

Table 13: Prompt for User Simulator

Prompt for QR and TS Evaluation

Task 1: Answer existence judgment (QR) Based on the question and the reply, determine whether the reply contains content that answers the question. If the reply contains information related to the question, classify it as answered. - If the reply contains answer content, even if the content is vague, speculative, or lacks specific details, output True - If the reply clearly does not answer the question, output False Task 2: Target satisfaction and refusal judgment (TS) Based on the question and the reply, determine whether the reply addresses the assistant’s intended target. Only when the reply exactly equals the following precise sentence should it be judged as a refusal to answer: “I do not want to answer right now.” - If the reply addresses the target and is not a refusal, output True - If the reply does not address the target and is not a refusal, output False - If the reply exactly matches the refusal sentence, output Reject Requirements: - Only output the specified format; do not add any other text - Strictly adhere to case and spelling (True / False / Reject) Output format (must be strictly followed): QR: True/False TS: True/False/Reject

Appendix

Table 14: Prompt for QR and TS Evaluation

Prompt for Probing Coherence (PC) Evaluation

Task: Probing Coherence (PC) scoring judgment Based on the dialogue history, evaluate the coherence of the assistant’s probing behavior. Give priority to the presence, completeness, and effectiveness of transitional or linking phrases, followed by the logical integration of the assistant’s probes or questions with the dialogue context. PC scoring criteria: 1: No transitional phrases; probes or questions are unrelated to the context and show clear logical breaks 2: Very weak transitional phrasing; probes or questions have limited relevance and feel abrupt 3: Basic transitional phrasing present; probes or questions are generally relevant but may lack smooth flow 4: Good transitional phrases; probes or questions are relevant and well integrated, creating logical coherence 5: Excellent and natural transitional phrases; probes or questions fit seamlessly with the context and flow smoothly Requirements: - Output only the following format; do not add any other text Output format (must be strictly followed): PC: [1–5]

Appendix

Table 15: Prompt for Probing Coherence (PC) Evaluation

Prompt for Implementing Proactive

Your task is to answer the user’s questions while also attempting to probe for the answer to the following question at the appropriate time: [target information]. Follow these rules exactly: 1. Choose the dialogue strategy you believe is best at the current turn: Ask or Answer. 2. Based on the strategy you choose, reply as follows: - Ask: First answer the user’s current message, then provide a complete transitional sentence, and finally ask the question to be probed: [target information]; - Answer: ONLY respond to the user’s current message. Do NOT ask any questions or provide guidance. Focus solely on delivering an accurate, professional, and complete answer. User’s message: [

u_{t}

] Output format exactly as follows (no extra text): Strategy: [Ask/Answer] Response:

Appendix

Table 16: Prompt for Implementing Proactive

Prompt for Implementing ICL-AIF

Your task is to answer the user’s questions while also attempting to probe for the answer to the following question at the appropriate time: [target information]. Follow these rules exactly: 1. Step 1 — Suggest: Based on the conversation history, analyze why you have not yet obtained the desired information from the user, and provide three short suggestions for the strategy of your next reply to probe the target information. Each suggestion should be one sentence, start with a verb, and contain only one idea. 2. Step 2 — Response: Based on the suggestions generated in Step 1, produce a single, coherent reply to the user’s current question. User’s message: [

u_{t}

] Output format exactly as follows (no extra text): Suggestions: suggestion_line_1 suggestion_line_2 suggestion_line_3 Response:

Appendix

Table 17: Prompt for Implementing ICL-AIF

Strategy

Prompt Construction for Our ProChatIP Method

Ask

You are given a selected strategy and must strictly follow it when generating your response. - Selected strategy: Ask - Follow-up question: [target information] Strategy Rules: - You must first answer the user’s message. Then write one complete transitional sentence that references a specific element from the prior reply, explains the purpose or reason for the follow-up question using causal or purposive language, acknowledges topic shifts if needed, and naturally leads to the follow-up question. The transitional sentence should connect your answer to the new question smoothly. User’s message: [

u_{t}

] Output your response strictly according to the strategy rules.

Answer

You are given a selected strategy and must strictly follow it when generating your response. - Selected strategy: Answer Strategy Rules: - ONLY respond to the user’s current message. Do NOT ask any questions or provide guidance. Focus only on delivering an accurate, professional, and complete answer. User’s message: [

u_{t}

] Output your response strictly according to the strategy rules.

Appendix

Table 18: Prompt Construction for Our ProChatIP Method

Active Information Gathering

Active information gathering studies how agents choose questions, tests, or observations that reduce uncertainty and improve decisions under interaction costs.

See all 26 papers

Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents
Sep 29, 2026Ido Levy, Asaf Yehudai, Segev Shlomov +2Proactive Agents Proactive Assistance
IBM · Weizmann Institute of Science
ProEvent: An Event-centric Benchmark for Proactive Agents
Jul 20, 2026Guanzhen Li, Liangming Pan, Leye WangProactive Agents LLM Agent Evaluation
School of Computing, Peking University
Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation
Sep 3, 2026Yan Tang, Tingyu Cao, Yuanbo Tang +2LLM Agent Evaluation Proactive Assistance
UniCone Team

Sep 29, 2026J/K move · Enter open · S save

Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents

Ido Levy, Asaf Yehudai, Segev Shlomov, Asaf Adi+1

IBM · Weizmann Institute of Science

Proactive Agents Proactive Assistance Tool-Using Agents Agentic AI

Open paper AskPDF arXiv

An agent that uses tools typically responds to what the user explicitly asks, yet completing the task may require information the user never requested. Work on proactive agents mainly studies whether and when an agent should act on its own, not what information it should pursue. We study a distinct axis of proactivity: its content. Horizontal proactivity pursues unstated information that the current context already identifies, and vertical proactivity pursues needs that only earlier evidence reveals. A need graph, recovered from a benchmark's own decomposition, records which needs depend on which, so both forms, and whether the agent stops at the right time, can be scored from a transcript without a model judge. To learn this behavior, we propose Q&D (questioner and drafter), which trains a questioner to prefer the question whose continuation retrieves more of the required evidence, with no reward model or judge. On held-out splits of three multi-hop question-answering benchmarks, at equal retrieval spend, the trained questioner improves both forms of proactivity over the same model, prompted, and outperforms a prompted model

15\times

larger in the same role on two of the three, and the gain persists after controlling for question volume and length. Without further training, we place the questioner in an interactive customer-service agent with a simulated customer, where it completes more tasks while asking fewer questions, and in retail it outperforms the

15\times

larger model with fewer follow-up turns from the customer. These results show that proactivity depends not only on whether an agent acts without being asked, but also on what it chooses to pursue and when it stops.

Abstract

Active Information Gathering

Explore similar work

Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents

ProEvent: An Event-centric Benchmark for Proactive Agents

Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation