PRISM: Festina Lente Proactivity -- Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents

Yuxuan Fu; Xiaoyu Tan; Teqi Hao; Chen Zhan; Xihe Qiu

cs.AIFeb 2, 2026

PRISM: Festina Lente Proactivity -- Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents

Authors: Yuxuan Fu, Xiaoyu Tan, Teqi Hao, Chen Zhan, Xihe Qiu

Organizations: Shanghai University of Engineering Science · National University of Singapore

Abstract

Proactive agents must decide not only what to say but also whether and when to intervene. Many current systems rely on brittle heuristics or indiscriminate long reasoning, which offers little control over the benefit-burden tradeoff. We formulate the problem as cost-sensitive selective intervention and present PRISM, a novel framework that couples a decision-theoretic gate with a dual-process reasoning architecture. At inference time, the agent intervenes only when a calibrated probability of user acceptance exceeds a threshold derived from asymmetric costs of missed help and false alarms. Inspired by festina lente (Latin: "make haste slowly"), we gate by an acceptance-calibrated, cost-derived threshold and invoke a resource-intensive Slow mode with counterfactual checks only near the decision boundary, concentrating computation on ambiguous and high-stakes cases. Training uses gate-aligned, schema-locked distillation: a teacher running the full PRISM pipeline provides dense, executable supervision on unlabeled interaction traces, while the student learns a response policy that is explicitly decoupled from the intervention gate to enable tunable and auditable control. On ProactiveBench, PRISM reduces false alarms by 22.78% and improves F1 by 20.14% over strong baselines. These results show that principled decision-theoretic gating, paired with selective slow reasoning and aligned distillation, yields proactive agents that are precise, computationally efficient, and controllable. To facilitate reproducibility, we release our code, models, and resources at https://prism-festinalente.github.io/; all experiments use the open-source ProactiveBench benchmark.

Figures & tables

Figure 1: PRISM reduces false alarms by estimating confidence before speaking. The agent first estimates calibrated $(\hat{p}_{\mathrm{need}},\hat{p}_{\mathrm{accept}})$ , compares $\hat{p}_{\mathrm{accept}}$ with an adaptive threshold $\tau\left(\hat{p}_{\mathrm{need}};C_{\mathrm{FA}},C_{\mathrm{FN}}\right)$ , and triggers a single pass slow stage only within a narrow region near the threshold. In benign edit events, low $\hat{p}_{\mathrm{need}}$ raises $\tau$ , the gate remains silent , and the user continues without interruption.

Figure 2: PRISM Pipeline. A model first estimates two calibrated probabilities from the context $X_{t}$ : the user’s need for assistance ( $p_{\mathrm{need}}$ ) and the likely acceptance of an offer ( $p_{\mathrm{accept}}$ ). A cost-sensitive gate then decides to intervene only if the acceptance probability meets a dynamic threshold $\tau(p_{\mathrm{need}})$ that accounts for the relative costs of false alarms ( $C_{\mathrm{FA}}$ ) and missed opportunities ( $C_{\mathrm{FN}}$ ). To balance accuracy and efficiency, a resource-intensive slow reasoning pass is triggered only when the initial prediction is ambiguous and falls within a narrow margin $\delta_{\mathrm{slow}}$ of the decision boundary.

Automatic Evaluation (LLM-as-Judge)
Model	Recall $\uparrow$	Precision $\uparrow$	Accuracy $\uparrow$	False-Alarm $\downarrow$	F1-Score $\uparrow$
Claude-3.5-Sonnet	97.89%	45.37%	49.78%	54.63%	62.00%
GPT-4o-mini	100.00%	35.28%	36.12%	64.73%	52.15%
GPT-4o	98.11%	48.15%	49.78%	51.85%	64.60%
LLaMA-3.1-8B-Proactive	99.06%	49.76%	52.86%	50.24%	66.25%
Qwen2-7B-Proactive	100.00%	49.78%	50.66%	50.22%	66.47%

Table 1: Main comparison on ProactiveBench (held-out test set). We report performance under two protocols: Automatic Evaluation (using our validated LLM-ensemble) and Human Expert Evaluation (bottom rows). PRISM (our RDC-distilled student with dynamic gating) consistently outperforms strong proprietary and open-source baselines across both regimes. Notably, it surpasses its teacher ( DeepSeek-R1 ) in Precision and False-Alarm rate while maintaining near-perfect Recall, validated by human experts.

Policy	Recall $\uparrow$	Precision $\uparrow$	Accuracy $\uparrow$	False-Alarm $\downarrow$	F1-Score $\uparrow$
Fixed $\tau{=}0.5$	92.42%	71.96%	76.03%	28.23%	80.74%
Static-cost $\tau(C_{\mathrm{FA}},C_{\mathrm{FN}})$	38.54%	59.67%	63.94%	40.32%	46.83%
Dynamic $\tau(p_{\text{need}})$	72.41%	68.29%	69.52%	31.70%	70.29%

Table 2: Ablation A: thresholding only (no gating). Fixed threshold dominates when

p_{\text{need}}

is uncalibrated.

Strategy	Recall $\uparrow$	Precision $\uparrow$	False-Alarm $\downarrow$	F1-Score $\uparrow$	AUDBC $\uparrow$	Tokens	Lat (P95)
Fast-only	100%	71.08%	28.92%	83.09%	79.26%	510	176ms
Slow-only	100%	75.17%	24.83%	86.79%	81.43%	693	312ms
Slow-on-margin	100%	78.81%	21.19%	88.15%	82.72%	541	196ms

Table 3: Ablation B: Efficiency. Slow-on-margin (

\delta=0.1

) achieves the best performance with negligible latency overhead compared to Fast-only.

Figure 3: Efficiency-Quality Pareto Frontier. We plot AUDBC against P95 Latency across various cost ratios ( $C_{\mathrm{FA}}:C_{\mathrm{FN}}$ ) and slow-margins ( $\delta$ ). The blue line traces the optimal frontier, showing that PRISM (particularly at $\delta=0.1$ ) achieves maximal decision benefit with minimal latency overhead ( $\sim$ 20ms). Grey points represent suboptimal configurations (e.g., higher $\delta$ or uncalibrated baselines) that incur higher latency without corresponding utility gains.

Training Pathway	Recall $\uparrow$	Precision $\uparrow$	Accuracy $\uparrow$	False-Alarm $\downarrow$	F1-Score $\uparrow$
SFT	100.00%	72.43%	69.52%	27.56%	76.09%
DFT	80.14%	72.43%	69.52%	27.56%	76.09%
Weighted-SFT	93.70%	70.68%	72.10%	29.31%	80.59%
RDC-SFT(ours)	98.88%	77.05%	76.39%	22.49%	86.61%

Table 4: Ablation C: training paradigms on Qwen3-8B. RDC-SFT (ours) trains on RDC-filtered supervision with explicit need/accept targets; DFT and Weighted-SFT are second-stage finetunes starting from the RDC-SFT checkpoint.

Configuration	Recall $\uparrow$	Precision $\uparrow$	False-Alarm $\downarrow$	F1-Score $\uparrow$	AUDBC $\uparrow$
$p_{\text{accept}}$ only $(p_{\text{need}}{=}1)$	99.95%	46.20%	62.50%	63.19%	58.40%
$p_{\text{need}}$ only ( $p_{\text{acc}}{=}\tau(p_{\text{need}})$ )	99.15%	69.50%	29.10%	81.72%	82.45%
Uncalibrated $(p_{\text{need}},p_{\text{accept}})$	98.80%	74.77%	25.23%	85.12%	86.43%
Calibrated (Ours)	98.88%	77.05%	22.94%	86.61%	88.52%

Table 5: Ablation D: Gating Signals & Calibration. Using both signals is crucial for reducing false alarms. Post-hoc calibration further aligns probabilities with risk, boosting Precision and AUDBC.

Figure 4: Structured trace motivating PRISM . A ProactiveBench coding event (top) yields a structured output from a reasoning-capable teacher (DeepSeek-R1; middle). PRISM (bottom) abstracts the teacher’s implicit hints into calibrated signals $p_{\text{need}}$ and $p_{\text{accept}}$ , uses a lightweight gate to trigger interventions when $p_{\text{need}}$ is confident, and invokes slow-mode deliberation only near uncertain margins; the final offer is regulated by $p_{\text{accept}}$ . Only structured fields are analyzed; no free-form chain-of-thought is stored or surfaced.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Main Protocol Verification
Comparison Setting	Support	Agreement $\uparrow$	Cohen’s $\kappa$ $\uparrow$	MCC $\uparrow$
Our Ensemble Judge vs. Human	229	89.1%	0.71	0.71
Cross-Model Consistency Checks
Pool A (Frontier) vs. Human	229	90.8%	0.75	0.76
Pool B (Open) vs. Human	229	89.1%	0.71	0.71
Pool A vs. Pool B	229	91.2%	0.76	0.77

Appendix

Table 6: Comprehensive Human-Model and Inter-Model Agreement Analysis. ”Our Ensemble Judge” refers to the majority vote protocol (GPT-4o, Claude-3.5-Sonnet, DeepSeek-R1) used for the main results. The high agreement scores (

>0.70\kappa

) across all settings confirm the reliability of the automated evaluation protocol.

Judge Pool	Method	F1-Score $\uparrow$	$\Delta$ F1 (Gain) $\uparrow$
Pool A (Frontier)	DeepSeek-R1	82.92%	–
Pool A (Frontier)	PRISM (Ours)	86.05%	+3.13%
Pool B (Open)	DeepSeek-R1	83.28%	–
Pool B (Open)	PRISM (Ours)	86.61%	+3.33%
Human (Majority)	DeepSeek-R1	82.05%	–
Human (Majority)	PRISM (Ours)	84.85%	+2.80%

Appendix

Table 7: Performance comparison across diverse judges. PRISM maintains a consistent advantage over the DeepSeek-R1 baseline regardless of whether the evaluator is a proprietary model ensemble, an open-source ensemble, or human experts.

Metric	DeepSeek-R1	PRISM (Ours)	Imp.	95% CI	P-value
Recall $\uparrow$	98.12%	98.88%	+0.76%	$[-0.18\%,+1.75\%]$	0.115
Precision $\uparrow$	72.35%	77.05%	+4.70%	$[+2.35\%,+7.05\%]$	$<0.001$
Accuracy $\uparrow$	72.69%	76.39%	+3.43%	$[+1.10\%,+5.76\%]$	0.004
False-Alarm $\downarrow$	27.64%	22.94%	-4.70%	$[-7.12\%,-2.28\%]$	$<0.001$
F1-Score $\uparrow$	83.28%	86.61%	+3.33%	$[+1.52\%,+5.15\%]$	0.002

Appendix

Table 8: Statistical significance of PRISM vs. DeepSeek-R1. The results confirm that the improvements in Precision, Accuracy, False-Alarm rate, and F1 are statistically significant (

p<0.05

), while Recall remains statistically equivalent.

Margin $\delta$	Rec $\uparrow$	Pre $\uparrow$	Acc $\uparrow$	FA $\downarrow$	F1 $\uparrow$	AUDBC $\uparrow$	Tokens	Lat
0 (Fast)	100%	71.08%	71.07%	28.92%	83.09%	79.26%	510	176ms
0.05	100%	74.11%	74.10%	25.89%	85.12%	81.19%	518	178ms
0.10	100%	78.81%	79.33%	21.19%	88.15%	82.72%	541	196ms
0.15	100%	74.57%	75.20%	25.43%	85.43%	80.58%	563	230ms

Appendix

Table 9: Sensitivity sweep for slow-margin

\delta

. A margin of

\delta=0.1

strikes the optimal balance, achieving the highest F1 and AUDBC. Increasing

\delta

further to

0.15

yields diminishing returns in accuracy while significantly increasing latency and token consumption.

Panel of Figure 5 — Figure 5: Impact of Slow Margin. Increasing the margin initially boosts performance by correcting hard examples, but latency grows linearly. $\delta=0.1$ is the “sweet spot”.

Cost Ratio	Margin	Rec $\uparrow$	Pre $\uparrow$	Acc $\uparrow$	FA $\downarrow$	F1 $\uparrow$	AUDBC	Tokens	Lat
( $C_{\mathrm{FA}}:C_{\mathrm{FN}}$ )	( $\delta$ )
$1:4$	0	100%	67.24%	78.93%	32.76%	83.64%	79.97%	532	198ms
	0.05	100%	70.56%	81.24%	29.44%	82.13%	81.28%	547	210ms
	0.1	100%	70.41%	80.22%	29.59%	83.63%	81.57%	562	216ms
	0.15	100%	69.72%	81.57%	30.28%	82.96%	80.92%	590	244ms
$1:2$	0	100%	71.08%	71.07%	28.92%	83.09%	79.26%	510	176ms

Appendix

Table 10: Robustness Grid Sweep. Performance metrics across varying user cost profiles (

C_{\mathrm{FA}}:C_{\mathrm{FN}}

) and slow margins. PRISM consistently benefits from the margin-based gating (

\delta>0

) across diverse cost settings, maintaining high F1 and AUDBC scores.

$\mathbf{C_{\mathrm{FA}}:C_{\mathrm{FN}}}$	Rec $\uparrow$	Pre $\uparrow$	Acc $\uparrow$	FA $\downarrow$	F1 $\uparrow$	AUDBC $\uparrow$	Slow	Tokens	Lat
$1:4$	100%	70.41%	80.22%	29.59%	83.63%	81.57%	15.45%	562.43	216.16
$1:2$	100%	73.07%	81.36%	26.93%	84.44%	82.22%	13.47%	538.73	191.82
$1:1$	98.88%	72.36%	80.35%	23.64%	83.96%	81.97%	11.15%	533.79	182.90
$1.2:1$	89.62%	76.83%	79.59%	23.17%	83.74%	79.34%	9.96%	528.34	177.67

Appendix

Table 11: Impact of Cost Ratios. PRISM adapts to stricter penalties for false alarms (

1.2:1

) by naturally increasing precision and reducing chatter, maintaining high F1 scores across the spectrum.

Domain	Stage	Recall $\uparrow$	Precision $\uparrow$	False-Alarm $\downarrow$	F1-Score $\uparrow$	AUDBC $\uparrow$
In-Domain	Raw	98.80%	74.77%	25.23%	85.12%	86.43%
(Coding)	After-T	98.80%	74.77%	25.23%	85.12%	86.43%
Out-of-Domain	Raw	95.83%	71.39%	28.61%	81.37%	82.15%
(Writing)	After-T	98.37%	73.37%	26.63%	82.63%	84.66%

Appendix

Table 12: Domain Generalization & Calibration Effect. While the raw model shows a performance dip on the unseen Writing domain, post-hoc calibration (”After-T”) effectively recovers Recall (

+2.54\%

) and AUDBC (

+2.51\%

), narrowing the gap between in-domain and out-of-domain performance.

Figure 6: Reliability Diagrams. Visualization of calibration performance for Need (left) and Accept (right) probabilities. The green curves (After Scaling) align significantly closer to the perfect diagonal than the orange curves (Before Scaling), visually confirming the ECE improvements.

	Temperature		ECE $\downarrow$		Brier $\downarrow$		Task Metrics
Setting	$T_{\text{need}}$	$T_{\text{acc}}$	Need	Acc	Need	Acc	F1-Score $\uparrow$	False Alarm $\downarrow$
Before Scaling	1.0	1.0	21.53%	13.19%	17.95%	18.58%	84.85%	25.99%
After Scaling	0.5	0.7	12.16%	7.42%	14.71%	18.20%	85.70%	25.12%

Appendix

Table 13: Calibration Quality Metrics. Post-hoc scaling significantly reduces Expected Calibration Error (ECE) and Brier scores for both gating signals, creating a more reliable foundation for decision-making.

Moderate Drift (Robustness Region)
Scenario	$T$	$\epsilon$	Recall $\uparrow$	Pre $\uparrow$	FA $\downarrow$	F1 $\uparrow$	AUDBC $\uparrow$	Flip
Baseline	1.0	0	98.80%	74.77%	25.23%	85.12%	86.43%	0%
Moderate	0.75	0	98.95%	74.41%	25.89%	84.78%	84.47%	6.25%
Moderate	1.25	0	97.89%	75.58%	24.42%	85.56%	87.02%	4.46%
Moderate	1.0	+0.15	99.98%	71.87%	28.13%	83.34%	81.57%	12.08%
Moderate	1.0	-0.15	96.67%	76.22%	23.78%	85.30%	87.34%	14.83%

Appendix

Table 14: Sensitivity Analysis under Parameter Drift. The model maintains high F1-Score (

>82\%

) across a wide grid of calibration parameters. Flip denotes the percentage of decisions changed relative to the baseline decision set.

Proactive Agents

See all 37 papers

Sep 3, 2026

Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation

Large language model agents can plan, invoke tools, and modify external states, yet most systems still take an explicit user instruction as a fixed starting point. Proactive service moves the decision upstream: an agent must infer service opportunities from incomplete environmental and user signals, choose among remaining silent, asking, assisting, and acting, and account for interruption, misunderstanding, overreach, and privacy costs. This survey gives an operational definition centered on initiative and formulates the problem as a partially observable sequential decision process constrained by authorization and risk. The formulation represents timing, content, and delivery within one structured action, while making explicit the option value of waiting, the decision value of questions, and feedback-induced state changes. On this basis, we organize existing methods along one decision pipeline (state and need estimation, intervention gating, action construction, and feedback adaptation) and describe prescribed, predictive, model based, and return optimizing mechanisms as nonexclusive policy-construction components. We further normalize decision units and three-axis evidence descriptors across streaming dialogue, screen, video, software-engineering, and human-agent collaboration resources, and formalize metrics for triggering, timing, calibration, user burden, safety, and policy value. The synthesis shows why offline classification performance alone does not predict deployment benefit and why long-term memory is not a defining condition of proactivity. Reliable proactive service instead requires calibrated incremental intervention value, verifiable authorization, recoverable execution, and counterfactual evidence.

Yan Tang, Tingyu Cao, Yuanbo Tang +2

UniCone Team

LLM Agent Evaluation Proactive Assistance Proactive Agents

Ask PDF

Jun 2, 2026

Perceive Before Reasoning: A Pre-Reasoning Perception Framework for Efficient and Reliable Proactive Mobile Agents

Multimodal large language models (MLLMs) have substantially advanced mobile agents, yet proactive mobile assistance remains challenging because agents must decide when to intervene before determining how to assist. Existing systems often implement these two decisions within a unified MLLM-based pipeline, leading to goal misalignment between conservative intervention filtering and comprehensive assistance generation, as well as redundant inference when the agent should remain silent. To address these limitations, we propose the Pre-Reasoning Perception Framework (PRPF), a two-stage framework built on perceiving before reasoning. PRPF introduces a lightweight Multimodal Proactive Perceptor (MPP) for intervention gating and context compression, and activates the Proactive Agent Reasoner (PAR) only when intervention is warranted. Experiments on the ProactiveMobile benchmark show that PRPF substantially reduces false trigger rates (FTR) while improving success rates (SR) and inference efficiency over the ProactiveMobile baseline.

Zhijie Ding, Weinan Hong, Zicheng Zhu +6

HyperAI Team, Xiaomi Corporation · Zhongnan University of Economics and Law · 1HyperAI Team, Xiaomi Corporation +5

Efficient Multimodal Inference Proactive Assistance Multimodal Agents LLM Agent Reliability Proactive Agents

Ask PDF

Oct 29, 2026

HALO: Human-Grounded Granular Evaluation of Proactive Agents

Proactive agents must decide whether, when, and how far to assist without an explicit request. Existing evaluations often rely on automated scores or binary verdicts, offering limited human diagnosis of evidence use and intervention quality. To address these gaps, we introduce HALO, Human-Grounded GrAnular EvaLuation of PrOactive Agents. HALO-Arena covers 4 proactive need types, Dependency, Blockage, Risk, and Opportunity, across 10 everyday domains and 50 personal environments. Scenarios require agents to retrieve evidence from structured databases or unstructured documents before judging whether assistance is warranted and choosing its scope. On HALO-Arena, we construct HALO-8K, a dataset of 8,000 responses from 10 models across 400 scenarios, annotated by 20 experts for action level, acceptability verdicts, and human diagnosis. For rejected responses, human diagnosis covers timing, intensity, and other issues. Recorded reasoning tokens and retrieval rounds measure the effort behind these responses. The best proprietary and open-source models achieve 72.88% and 56.63% acceptability, respectively. Our central finding is that over-proactivity is the dominant observed failure mode, with Premature timing or Excessive intensity accounting for 61.20% of rejected responses. Further analyses yield complementary findings on the balance between intervention and restraint, evidence acquisition, and the calibration of timing and intensity.

Wenqi Pei, Xinyu Wang, Jinrui Liu +5

University of California, Davis · 2077AI · University of Southern California

Proactive Agents Proactive Assistance Human-in-the-Loop Evaluation

Ask PDF

Abstract

Proactive Agents

Explore similar work

Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation

Perceive Before Reasoning: A Pre-Reasoning Perception Framework for Efficient and Reliable Proactive Mobile Agents

HALO: Human-Grounded Granular Evaluation of Proactive Agents