HALO: Human-Grounded Granular Evaluation of Proactive Agents

Wenqi Pei; Xinyu Wang; Jinrui Liu; Cynthia Zhuge; Jie Sheng; Zihan Wang; Tenghao Huang; Muhao Chen

cs.CLOct 29, 2026

HALO: Human-Grounded Granular Evaluation of Proactive Agents

Authors: Wenqi Pei, Xinyu Wang, Jinrui Liu, Cynthia Zhuge, Jie Sheng, Zihan Wang, Tenghao Huang, Muhao Chen

Organizations: University of California, Davis · 2077AI · University of Southern California

Abstract

Proactive agents must decide whether, when, and how far to assist without an explicit request. Existing evaluations often rely on automated scores or binary verdicts, offering limited human diagnosis of evidence use and intervention quality. To address these gaps, we introduce HALO, Human-Grounded GrAnular EvaLuation of PrOactive Agents. HALO-Arena covers 4 proactive need types, Dependency, Blockage, Risk, and Opportunity, across 10 everyday domains and 50 personal environments. Scenarios require agents to retrieve evidence from structured databases or unstructured documents before judging whether assistance is warranted and choosing its scope. On HALO-Arena, we construct HALO-8K, a dataset of 8,000 responses from 10 models across 400 scenarios, annotated by 20 experts for action level, acceptability verdicts, and human diagnosis. For rejected responses, human diagnosis covers timing, intensity, and other issues. Recorded reasoning tokens and retrieval rounds measure the effort behind these responses. The best proprietary and open-source models achieve 72.88% and 56.63% acceptability, respectively. Our central finding is that over-proactivity is the dominant observed failure mode, with Premature timing or Excessive intensity accounting for 61.20% of rejected responses. Further analyses yield complementary findings on the balance between intervention and restraint, evidence acquisition, and the calibration of timing and intensity.

Figures & tables

Figure 1: Overview of HALO. HALO-Arena supplies personal apps and local evidence. Agents retrieve evidence and decide whether, when, and how far to intervene. HALO-8K records human verdicts, timing and intensity diagnosis, and resource consumption. The circled choices illustrate a premature and excessive L3 intervention.

Proprietary Models
Model	Action Levels				Inact. $\downarrow$	Timing Errors $\downarrow$		Intensity Errors $\downarrow$		Consumption $\downarrow$		Acc (%) $\uparrow$
	L0	L1	L2	L3		Prem.	Bel.	Insuff.	Excess.	Reas. (k)	Rounds
Gemini3.8-Flash	30.13	57.50	2.13	10.25	7.38	10.50	3.75	2.63	7.00	14.99	4.19	72.88
Claude-Opus5.5	31.75	48.00	5.13	15.13	6.75	10.88	3.88	2.00	8.38	3.48	2.27	69.75
Grok4.7	3.63	76.50	2.00	17.88	0.88	21.38	6.88	3.25	14.63	5.04	2.70	65.25
Gemini3.1-Pro	3.88	58.63	7.63	29.88	0.13	27.63	6.13	4.63	19.25	3.80	1.85	55.75

Table 1: Model performance and resource consumption. Action shares and error rates are percentages of all responses. Inact., Prem., Bel., Insuff., and Excess. denote Inaction, Premature, Belated, Insufficient, and Excessive. Shading marks best , second , and third .

Figure 2: HALO-Arena construction. Validator and Reviewer return revision feedback before background records and human audit finalize the dataset.

Proactive Need Types
Category	Action Levels				Inact. $\downarrow$	Timing Errors $\downarrow$		Intensity Errors $\downarrow$		Acc (%) $\uparrow$
	L0	L1	L2	L3		Prem.	Bel.	Insuff.	Excess.
Blockage	12.05	66.35	4.35	17.25	3.15	21.50	7.95	8.70	13.85	54.35
Dependency	7.20	71.45	4.95	16.40	1.95	21.05	7.80	7.10	16.05	55.05
Opportunity	13.60	65.80	4.75	15.85	2.95	24.00	6.60	5.80	15.30	55.85
Risk	9.30	73.20	2.65	14.85	2.85	21.60	7.80	7.30	13.65	56.00

Table 2: Results by proactive need type and domain. All values are percentages of category responses. Error abbreviations follow Table 1 . Shading marks best , second , and third values within each group.

Figure 3: Over-proactivity among rejected responses.

Figure 4: Paired intervention rates and acceptability.

Figure 5: L0 choices by model on the left and accepted action and restraint contributions on the right.

Figure 6: Mean retrieval rounds and acceptability across models (left), and L1 response proportions with and without local content access (right).

Figure 7: Intensity errors among appropriately timed but rejected interventions.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Environment			Evaluation
	Func. Apps	DB	Docs	Human Verdicts	Human Diagnosis	Consumption
PROBE ( Pasternak et al., 2025 )	–	$\checkmark$	$\checkmark$	$\checkmark$	–	–
ProAgentBench ( Tang et al., 2026 )	–	–	–	–	–	$\checkmark$
PARE-Bench ( Nathani et al., 2026 )	$\checkmark$	–	$\checkmark$	–	–	$\checkmark$
$\pi$ -Bench ( Zhang et al., 2026 )	$\checkmark$	–	$\checkmark$	–	–	$\checkmark$
Proactivity-Gym ( Oh et al., 2026 )	$\checkmark$	–	$\checkmark$	$\checkmark$	–	–

Appendix

Table 3: Selected benchmark capabilities. Checks denote explicit coverage. Dashes denote features not reported in the evaluated protocol.

App	Interaction commands (29)	Action commands (19)
Email	get_email , list_emails , search_emails , list_drafts	flag_email L1 , create_draft L2 , send_email L3
Calendar	get_event , list_events , find_conflicts , get_event_draft , list_event_drafts , get_reminder , list_reminders	create_reminder L1 , create_event_draft L2 , create_event L3 , update_event L3 , delete_event L3
Chat	get_conversation , list_conversations , get_message , list_messages , search_messages , list_drafts	pin_conversation L1 , create_draft L2 , send_message L3
Contacts	get_contact , list_contacts , search_contacts	flag_contact L1 , create_contact L2 , update_contact L2
Wallet	get_wallet , get_transaction , list_transactions , list_drafts	flag_transaction L1 , create_draft L2 , pay L3 , refund_transaction L3
Database	get_schema , execute	–

Appendix

Table 4: Agent functions by app and category. Superscripts indicate automatically inferred intensity levels, provided to guide human verification and reduce verification time.

Stage	Pass	Total	Pass (%)	Fail (%)
Validator	509	568	89.61	10.39
Reviewer	400	509	78.59	21.41
Overall	400	568	70.42	29.58

Appendix

Table 5: Initial pipeline pass and fail rates. Reviewer rates are conditional on Validator acceptance.

Audit Votes
Record	Count	Rate (%)
Approve	593	74.13
Needs Revision	175	21.88
Reject	32	4.00
Review Pairs
Approve + Approve	214	53.50

Appendix

Table 6: Human audit and adjudication. Review pairs are unordered.

Events	Scenarios	Checkpoints
4	14	56
5	212	1,060
6	174	1,044
Total	400	2,160

Appendix

Table 7: Checkpoint distribution by scenario length.

Model	Selected positions (%)
	1	2	3	4	5	6
Gemini3.8-Flash	15.75	19.25	19.75	21.63	17.25	6.38
Claude-Opus5.5	17.63	19.50	18.88	19.50	17.25	7.25
Grok4.7	17.75	16.63	19.25	20.13	19.00	7.25
Gemini3.1-Pro	15.75	20.00	21.38	19.00	17.63	6.25
GPT5.6-Luna	19.38	19.38	16.88	18.75	17.25	8.38

Appendix

Table 8: Selected event positions and selection diagnostics. pp denotes percentage points.

Model	Blockage	Dependency	Opportunity	Risk
Gemini3.8-Flash	73.0	70.0	73.5	75.0
Claude-Opus5.5	69.5	68.5	70.0	71.0
Grok4.7	63.5	65.5	64.0	68.0
DeepSeekV4-Pro	53.5	54.0	57.5	61.5
Gemini3.1-Pro	54.0	57.0	54.0	58.0
Qwen3.8-27B	54.0	59.5	52.5	54.5

Appendix

Table 9: Acceptability (%) by model and proactive need type.

Figure 8: Acceptability by model and domain. Outlines mark domain leaders.

Figure 9: Acceptability by need and domain.

Category	Count	Within (%)	Rejected (%)
Unsupported Evidence	503	50.30	14.07
Irrelevant Action	291	29.10	8.14
Factual Contradiction	128	12.80	3.58
Others	78	7.80	2.18

Appendix

Table 10: Other Issues distribution. Within and Rejected are percentages of the 1,000 responses with Other Issues and all 3,575 rejected responses, respectively.

Measure	$N$	Original	LoRA	Change
Qwen3.5-9B
Human verdict	400	48.50	56.00	$+7.50$
Timing	400	61.25	57.25	$-4.00$
Intensity	389	57.07	41.90	$-15.17$
Joint	400	33.75	22.75	$-11.00$
Qwen3.5-4B

Appendix

Table 11: Qwen3.5 judge accuracy (%).

Panel of Figure 10 — Figure 10: Annotation workspaces for scenario audit (upper) and response evaluation (lower).

Figure 11: Human diagnosis form for a rejected intervention, with independent timing and intensity questions and optional other issues.

Proactive Agents

See all 37 papers

Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation
Sep 3, 2026Yan Tang, Tingyu Cao, Yuanbo Tang +2LLM Agent Evaluation Proactive Assistance
UniCone Team
Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym
Sep 29, 2026Jio Oh, Seunghyun Do, Young-Jun Lee +2LLM Agent Evaluation Long-Horizon Agent Evaluation
KAIST · University of Minnesota
Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents
Sep 29, 2026Ido Levy, Asaf Yehudai, Segev Shlomov +2Proactive Agents Proactive Assistance
IBM · Weizmann Institute of Science

Sep 3, 2026J/K move · Enter open · S save

Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation

Yan Tang, Tingyu Cao, Yuanbo Tang, Huaze Tang+1

UniCone Team

LLM Agent Evaluation Proactive Assistance Proactive Agents

Open paper AskPDF arXiv

Large language model agents can plan, invoke tools, and modify external states, yet most systems still take an explicit user instruction as a fixed starting point. Proactive service moves the decision upstream: an agent must infer service opportunities from incomplete environmental and user signals, choose among remaining silent, asking, assisting, and acting, and account for interruption, misunderstanding, overreach, and privacy costs. This survey gives an operational definition centered on initiative and formulates the problem as a partially observable sequential decision process constrained by authorization and risk. The formulation represents timing, content, and delivery within one structured action, while making explicit the option value of waiting, the decision value of questions, and feedback-induced state changes. On this basis, we organize existing methods along one decision pipeline (state and need estimation, intervention gating, action construction, and feedback adaptation) and describe prescribed, predictive, model based, and return optimizing mechanisms as nonexclusive policy-construction components. We further normalize decision units and three-axis evidence descriptors across streaming dialogue, screen, video, software-engineering, and human-agent collaboration resources, and formalize metrics for triggering, timing, calibration, user burden, safety, and policy value. The synthesis shows why offline classification performance alone does not predict deployment benefit and why long-term memory is not a defining condition of proactivity. Reliable proactive service instead requires calibrated incremental intervention value, verifiable authorization, recoverable execution, and counterfactual evidence.

Abstract

Proactive Agents

Explore similar work

Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation

Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym

Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents