cs.LGOct 5, 2026

Improving Proactive AI Assistance with Hierarchical Procedural Understanding

Authors: Jin-Seop Lee, TaeYeon Won, SeongJun Jung, JungHoon Kim, Boyang Albert Li, Jin-Young Park, Jaehong Yoon, Jee-Hyong Lee

Organizations: Department of Artificial Intelligence, Sungkyunkwan University, Republic of Korea · College of Computing and Data Science, Nanyang Technological University, Singapore

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Proactive AI assistants continuously observe a user's activity and decide whether to provide new guidance or remain silent. They should provide appropriate guidance for the task, determine when to provide the next guidance based on task progress, and adjust the guidance level to the user's expertise and needs. Supporting these capabilities requires training and evaluation data that reflect procedural structure and capture how guidance should adapt to task progress and user needs. However, existing datasets either focus on detection-based proactive understanding or provide procedural guidance at a fixed granularity. Fixed-granularity guidance provides limited information about fine-grained progress and broader procedural context, making it difficult to determine completion and adapt guidance granularity. To address these limitations, we introduce the ProactiveCoach suite, comprising ProactiveCoach-Instruct for training, ProactiveCoachBench for evaluation, and fine-tuned VLMs with an adaptive guidance system. ProactiveCoach-Instruct provides hierarchically structured guidance at the phase, step, and action levels for learning task progress and procedural context. ProactiveCoachBench evaluates whether models provide appropriate guidance at the right time across different guidance levels and adapt when the requested level changes. We fine-tune pretrained VLMs on ProactiveCoach-Instruct and demonstrate its effectiveness across backbones. Compared with fixed-granularity supervision, hierarchical supervision improves overall performance across backbones by up to 9.6%p. We further build an adaptive guidance system by combining our fine-tuned model with a lightweight guidance router. Without additional fine-tuning, our system outperforms the in-context adaptation baseline by 57.1%p across four guidance-level transitions. Our project page is available at https://jinsuby.github.io/ProactiveCoach/.

Figures & tables

Figure 1: Overview of ProactiveCoach suite. It tracks progress and completion across phase, step, and action levels and adapts the guidance level to the user’s request.
Figure 2: Overview of hierarchically structured guidance in our proposed dataset.
Figure 3
Figure 5: Construction pipeline for our proposed dataset.
Figure 6: Guidance-level adaptation with a lightweight router. ProactiveCoach generates multi-level guidance, while the router selects the level requested by the user.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Sample duration distribution.
Figure 8: Prompt for Procedural Video Selection.
Figure 9: Prompt for Video-grounded Recaptioning.
Figure 10: Prompt for Goal-Relevant Annotation Filtering.
Figure 11: Prompt for Hierarchical Procedure Construction.
Figure 12: Prompt for Hierarchy Checking and Refinement.
Figure 13: Prompt for Level-Specific Guidance Generation.
Figure 14: ProactiveCoachBench example-1 (goal: clean and lubricate a bicycle chain).
Figure 16: ProactiveCoachBench example-3 (goal: replace a toner cartridge and load paper).
Figure 18: ProactiveCoachBench example-5 (goal: upgrade computer RAM and reconnect cables).
Figure 20: Human quality review interface. Reviewers inspect the task video and its hierarchical annotations at the phase, step, and action levels and evaluate them using four criteria: video alignment, hierarchical coverage, goal relevance, and guidance quality.
Figure 21: Common inputs for evaluation.
Figure 22: Shared evaluation prompt for frontier models and multimodal baselines, including Qwen.
Figure 23: Prompts for streaming and proactive assistance models.
Figure 24: Evaluation prompt for our model.
Figure 25: Qualitative comparison on ProactiveCoachBench-1 (goal: make a peanut butter and jelly tortilla roll).
Figure 27: Qualitative comparison on ProactiveCoachBench-3 (goal: remove dry sticks from a fence).
Figure 29: Qualitative comparison on EgoProactive-5 (goal: clean ceiling fan blades).

Explore similar work

CardsList
Jun 3, 2026cs.CV

Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance

We envision a proactive multi-modal assistant system which gives users real-time step-by-step guidance on a procedural task, autonomously deciding \textit{when} to interrupt, and \textit{how} to coach. However, progress is limited by the absence of large-scale, cross-domain benchmarks that reflect realistic conditions, particularly the common case in which users deviate from the expected step sequence. We address this gap with four contributions: \textbf{(1)}we release \textbf{EgoProactive}, a large-scale wearable-egocentric dataset for proactive procedural assistance with explicit Out-of-Plan (OOP) annotations and recovery steps; \textbf{(2)}we augment five established benchmarks (Ego4D, EPIC-KITCHENS, EgoExo4D, HoloAssist, HowTo100M) into \textbf{Pro\textsuperscript{2}Bench} under a unified proactive-guidance schema; \textbf{(3)}we propose a \textbf{decoupled planner--interaction architecture} specialized for procedural state, visual cues, and recovery injection; \textbf{(4)}we introduce a post-training recipe that transfers across model families, validated by cross-backbone replication on Llama4 and Qwen-3.6-VL. In extensive experiments, our trained Llama-4 system substantially improves objective intervention quality over strong proprietary baselines (Claude Opus4.6, Gemini3.1Pro, GPT5.2) and open-weight baselines (Qwen3VL~235B) baselines across all six datasets. Oracle-plan experiments further show that, when plan quality is controlled, the trained duplex model produces high-quality guidance and large gains on Out-of-Plan recovery.

Kaustav Kundu, Ritvik Shrivastava, Maxim Arap +13

1Meta Reality Labs · 2Meta Superintelligence Labs

ProactiveArtificial Intelligence AssistantsEgocentric Dataset
AskPDF
May 5, 2026cs.AI

Pro2^2Assist: Continuous Step-Aware Proactive Assistance with Multimodal Egocentric Perception for Long-Horizon Procedural Tasks

Procedural tasks with multiple ordered steps are ubiquitous in daily life. Recent advances in multimodal large language models (MLLMs) have enabled personal assistants that support daily activities. However, existing systems primarily provide reactive guidance triggered by user queries, or limited proactive assistance for isolated short-term events rather than long-horizon procedural tasks. In this work, we introduce Pro2^2Assist, a step-aware proactive assistant that continuously tracks fine-grained task progress and reasons over the user's evolving state to provide timely assistance throughout tasks. Pro2^2Assist leverages multimodal data from augmented reality (AR) glasses to achieve motion-based perception. It then extracts step-oriented procedural context from multi-scale temporal dynamics and task-specific expert knowledge. Based on both sensory input and procedural context, Pro2^2Assist performs continuous reasoning to infer user needs and display timely assistance on AR glasses. We evaluate Pro2^2Assist using a dataset curated from public sources and a real-world dataset collected on our testbed with AR glasses. Extensive evaluations show that Pro2^2Assist outperforms the best-performing baselines by over 21% in procedural action understanding accuracy, and it achieves up to 2.29x the proactive timing accuracy of baselines. A user study with 20 participants further shows that 90% find Pro2^2Assist useful, indicating its effectiveness for real-world procedural assistance.

Lilin Xu, Bufang Yang, Siyang Jiang +6

Columbia University, United States · The Chinese University of Hong Kong, China

Multimodal AgentsArtificial Intelligence AssistantsFine-Grained ActionsAugmented RealityEgocentric VisionLong-Horizon Task Planning+1
AskPDF
May 14, 2026cs.AI

ππ-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce ππ-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, ππ-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.

Haoran Zhang, Luxin Xu, Zhilin Wang +11

Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +7

Agentic BenchmarksArtificial Intelligence Assistants
AskPDF