Learning Next Action Predictors from Human-Computer Interaction

Omar Shaikh; Valentin Teutschbein; Kanishk Gandhi; Yikun Chi; Nick Haber; Thomas Robinson; Nilam Ram; Byron Reeves

cs.CLMar 6, 2026

Learning Next Action Predictors from Human-Computer Interaction

Authors: Omar Shaikh, Valentin Teutschbein, Kanishk Gandhi, Yikun Chi, Nick Haber, Thomas Robinson, Nilam Ram, Byron Reeves,

Organizations: Stanford University · Hasso Plattner Institute · New York University

Abstract

Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.

Figures & tables

Figure 1: Long-Context Next Action Predictors (LongNAPs) draw from the entirety of a user’s multimodal context (e.g. screenshots)—retrieving over an unbounded history—to predict what they will do next. We train LongNAP end-to-end on data from 20 users over a month, spanning 1.9M screenshots or 1,800 hours of screen on-time. Predictions are rewarded based on LLM-judged similarity to a set of ground truth future actions.

Figure 2: LongNAP significantly outperforms all baselines by at least 39.4% relative to the strongest baseline . We evaluate with LLM-judge, which outputs similarity to ground truth future actions (0-1 score). Performance is averaged across 20 models trained on individual users.

Figure 3: NAPsack enables passive collection of human-computer interaction data . It ingests screenshots and input events, compresses them to retain only meaningful frames, and annotates with action descriptions.

	Ground-Truth	Predicted	Predicted	Predicted
	Score = 1.0	Score = 0.72	Score = 0.52	Score = 0.15
1	Switched to “Australia vs Zimbabwe” tab, entered fullscreen	Switched to “Australia vs Zimbabwe” tab, entered fullscreen	Switched to “Australia vs Zimbabwe” tab, clicked player	Clicked on YouTube home page, browsed recommended videos
2	Clicked thumbnail in “Key Moments” section	Clicked on video player timeline to skip forward	Pressed “f” to enter fullscreen on cricket video	Clicked video titled “Best Cricket Catches of 2025”
3	Pressed “esc” to exit fullscreen	Pressed “esc” to exit fullscreen	Pressed “esc” to exit fullscreen	Pressed “esc” to exit fullscreen
4	Switched to W&B workspace, clicked “Runs” icon	Switched to W&B workspace, scrolled through dashboard	Opened new tab, typed “wandb.ai” into address bar	Typed “australia cricket schedule” into Google search bar
5	Switched to “Australia vs Zimbabwe,” entered fullscreen	Switched to “Australia vs Zimbabwe” tab, entered fullscreen	Scrolled through W&B charts for training metrics	Clicked ESPN Cricinfo link in search results

Table 1: Qualitative examples of candidate trajectories

\hat{\mathcal{E}}

at decreasing LLM judge similarity scores (left to right), all evaluated against the same ground-truth

\mathcal{H}^{*}

human annotated trajectory. For readability, we highlight in red actions with substantially wrong intent.

Method	Judge Score ( $\uparrow$ )	Human WR % ( $\uparrow$ )	Size ( $\downarrow$ )
naive	0.48 $\pm$ 0.03	19.2 $\pm$ 6.7	$295\text{\,}\mathrm{MB}$
+ split	0.57 $\pm$ 0.03	45.4 $\pm$ 8.7	$295\text{\,}\mathrm{MB}$
+ split + compress	0.60 $\pm$ 0.03	49.6 $\pm$ 8.9	$76\text{\,}\mathrm{MB}$
+ split + compress + IO	0.70 $\pm$ 0.03	85.8 $\pm$ 6.0	$76\text{\,}\mathrm{MB}$

Table 2: NAPsack reduces the amount of data we have to save for effective captioning by 75% without compromising quality . Event-driven compression (where frames are saved only when a user interacts with their computer) yields improved efficiency; and I/O signals further improve performance. Performance is measured across both LLM-judge scores ([0, 1]) and human eval (win rate). 95% conf. intervals are computed through bootstrapping.

Figure 4: Predictions from LongNAP are generated in a two-phase process. In the first phase, LongNAP Reasons to Retrieve : conditioned on what the user sees right now (e.g. a set of paper reviews), LongNAP generates a reasoning trace and uses it to retrieve past traces. Using retrieved traces, LongNAP Reasons to Predict : generating a final reasoning trace, adding it back to memory, and then predicting the next steps a user might take. The predicted trajectory is compared against a ground truth (with an LLM-as-a-judge similarity score), and then LongNAP is optimized with GRPO ( Shao et al., 2024b ) .

User	Gemini		Qwen-2.5-VL-7B
$u_{i}$	Zero-shot	RAG	Zero-shot	RAG	SFT	LongNAP
$u_{1}$	0.28	0.29	0.21	0.23	0.23	+0.12 0.41
$u_{2}$	0.21	0.22	0.15	0.17	0.18	+0.12 0.34
$u_{3}$	0.23	0.23	0.19	0.18	0.17	+0.10 0.33
$u_{4}$	0.24	0.23	0.14	0.16	0.17	+0.09 0.32
$u_{5}$	0.23	0.24	0.18	0.16	0.16	+0.10 0.34

Table 3: Training on individual users substantially outperforms baseline methods, improving average performance by 39.4% ( +0.11 ) relative to the strongest baseline (Gemini Few-shot RAG). We report the similarity score (0-1) to ground truth future actions, as determined by our LLM judge ( §3.2 ).

u_{i}

denotes a LongNAP instance trained on a single user from our dataset of 20 users, and

u_{\mu}

represents the mean performance across all 20 individually trained models.

Figure 5: Some users are more predictable than others. When LongNAP is trained on a single user (in our generalizing over time experiments, §5.1 ), LLM-as-a-judge evals vary substantially from one user’s LongNAP to another user’s. In the above figure, we re-evaluate across checkpoints from training epochs for each user, highlighting variance.

Model	Method	Win Rate (%)
Qwen 2.5 VL	LongNAP	79.0 $\pm$ 6.6
	SFT	$29.5\pm 6.9$
	Few-shot RAG	$45.0\pm 7.6$
	Zero-shot	$33.0\pm 7.2$
Gemini	Few-shot RAG	$58.5\pm 6.8$
	Zero-shot	$55.0\pm 7.7$

Table 4: Human evaluation shows significantly higher win rates for LongNAP. We show pairwise human evaluation win rates across each method (combined across 2 annotators, 300 total comparisons, 95% bootstrapped CIs).

Model	Method	$u_{\mu}$	$u_{2}$	$u_{6}$	$u_{12}$	$u_{14}$	$u_{16}$
Gemini	Zero-shot	0.22	0.27	0.22	0.20	0.20	0.21
Gemini	Few-shot RAG	0.23	0.26	0.23	0.22	0.21	0.22
Qwen VL	Zero-shot	0.16	0.22	0.13	0.14	0.16	0.14
	Few-shot RAG	0.17	0.23	0.15	0.18	0.14	0.18
	SFT	0.17	0.22	0.14	0.16	0.15	0.17
	LongNAP	0.26	0.31	0.28	0.22	0.21	0.26

Table 5: When trained jointly on many users, LongNAP exhibits modest generalization to unseen users, improving average performance by 13.0% relative to the strongest baseline (again, Gemini Few-Shot RAG). We report similarity to ground truth future actions, as determined by our LLM judge ( §3.2 ). To test generalization to unseen users, we train randomly on 10 users; validate on 5; and then test on 5 (reported above).

u_{\mu}

denotes the mean performance across the evaluated users, and

u_{i}

represents performance on user

i

Model	$u_{\mu}$	$u_{2}$	$u_{6}$	$u_{12}$	$u_{14}$	$u_{16}$
LongNAP	$0.38$	$0.34$	$0.37$	$0.40$	$0.36$	$0.41$
Ablations
$\rightarrow$ Remove reasoning	(-0.07) $0.30$	(-0.09) $0.25$	(-0.06) $0.30$	(-0.06) $0.35$	(-0.09) $0.27$	(-0.06) $0.35$
$\rightarrow$ Remove retriever	(-0.06) $0.32$	(-0.07) $0.27$	(-0.07) $0.30$	(-0.01) $0.39$	(-0.08) $0.29$	(+0.01) $0.42$
$\rightarrow$ Shuffle dataset	(-0.04) $0.34$	(-0.05) $0.29$	(-0.04) $0.32$	(+0.00) $0.40$	(-0.05) $0.32$	(-0.00) $0.41$

Table 6: Both reasoning and retrieval are critical to LongNAP’s performance. We pick a random subset of users to perform ablations on. Removing the reasoning component leads to the largest drop in average performance (

-0.07

absolute), while removing the retriever reduces performance by

-0.06

on average. Shuffling the dataset also degrades results (

-0.04

), suggesting that preserving temporal structure is important.

u_{\mu}

denotes mean performance across the evaluated users, and

u_{i}

denotes performance on user

i

. We perform ablations on a random subset of 5 users.

Figure 6: At a given time (top), LongNAP is likely to retrieve over a substantial part of its past context (bottom) to predict what a user will do next. The visualization above (for a random user) shows what context from the past is retrieved for a query at the current time.

Figure 7: LongNAP learns diverse reasoning strategies when trained on different users. Different colors correspond to different users. We embed (using the all-MiniLM-L6-v2 model; Wang et al. (2020) ; Reimers and Gurevych (2019) ) and visualize traces from the reasoning to predict phase across a subset of users with tSNE. For some users (e.g. $u_{7}$ , blue, almost exclusively takes online surveys), reasoning strategies are homogenous. For others, LongNAP generates a library of reasoning patterns for specific contexts ( $u_{8}$ ; in orange).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Event type	GAP threshold (s)	MAX duration (s)
Click	0.2	0.3
Move	0.5	4.0
Scroll	0.5	3.0
Key	0.5	6.0

Appendix

Table 7: Event burst thresholds used by NAPsack.

Figure 8: Reasoning traces grow shorter across model training. Traces for the retrieve phase grow far shorter than queries for the predict phase (avg. 10.11 retrieval tokens v.s. 85.34 prediction). Qualitatively, we find that reasoning for the retrieve phase resembles queries to a retriever; while reasoning for the prediction phase resembles higher order descriptions of a user’s behavior.

Hyperparameter	LongNAP	SFT
Effective Batch Size	16	16
Learning Rate	3e-5	1e-4
Group Size	4	-
Epochs	10	10
Actions in history	16	16
Actions to predict	8	8

Appendix

Table 8: Hyperparameter settings. We the best checkpoint based on our validation performance, early stopping if appropriate. We only sweep learning rate on baselines (with SFT) due to budget constraints. We train on 8 B200 GPUs.

Figure 9: Pass@k scores for LongNAP. To count as a “pass,” we selected an LLM-judge threshold of 0.5; trajectories that get this score are often well aligned with the actual intent of the user, but miss minor details or skip a few actions (see §5.2 for an example). At this threshold, LongNAP achieves 17.1% at pass@1 across users, rising to 36.3% at pass@20 .

Figure 10: Empirical calibration of LongNAP. Confidence percentiles are computed per-user from intra-cluster variance of 20 sampled trajectories. Higher confidence prompts yield substantially higher pass@1 accuracy (25.9% vs. 10.3%).

Score	Ground Truth	Predicted Actions
0.05	1. Typed “icc world cup” into search bar 2. Clicked “STANDINGS” tab in ICC T20 World Cup widget 3. Scrolled through standings table 4. Clicked “SL” (Sri Lanka) in standings 5. Switched to W&B dashboard tab 6. Scrolled through W&B charts 7. Clicked “train/entropy” chart in W&B 8. Clicked chat with “Omar Shaikh” in Slack	1. Typed “x” into address bar, pressed Enter 2. Clicked Twitter logo in top left 3. Typed “arch” into Twitter search bar 4. Clicked search result for “Arch Labor” 5. Clicked “Home” button in Twitter sidebar 6. Scrolled through Twitter feed 7. Clicked search button on Twitter 8. Typed “forms” into Twitter search bar
0.12	1. Clicked “rouge-score” on PyPI, copied install command 2. Ran uv pip install rouge-score in terminal 3. Scrolled through “rouge-score” PyPI page 4. Ran accelerate launch training command 5. Scrolled through “rouge-score” PyPI page 6. Switched between W&B, Google Doc, and terminal 7. Typed training command, cancelled with Ctrl+C 8. Ran cd sim-v2 and git pull	1. Scrolled through search results for “install rouge score” 2. Clicked on a search result for “install rouge score” 3. Scrolled down “crag” repository on GitHub 4. Clicked “hub/parouza” link in GitHub repo 5. Clicked “eval” directory in file explorer 6. Pressed Enter to run evaluate_qm9.py 7. Switched to Google Chrome window 8. Copied pip install command from GitHub
0.20	1. Clicked on a purple property pin on the map 2. Clicked on the “ $3.5k+” property pin 3. Clicked on the “$ 2.8k” property pin 4. Clicked on the “ $3.5k+” property pin 5. Clicked on the “$ 3.3k” property pin 6. Clicked on the “ $3.9k” property pin 7. Clicked on the “$ 5.3k” property pin 8. Clicked on the “$2.8k” property pin	1. Scrolled through search results and map on Zillow 2. Clicked on a property marker on the Zillow map 3. Clicked “Down” arrow to expand price filter 4. Clicked “OK” to apply price filter settings 5. Scrolled through filtered search results 6. Selected a property listing card from results 7. Clicked “Show on map” on the listing page 8. Scrolled through rental listings on Zillow map
0.35	1. Scrolled through W&B dashboard 2. Clicked “kanishkgandhi” project link in W&B 3. Clicked on a run in W&B sidebar 4. Clicked “kanishkgandhi’s workspace” tab 5. Scrolled through Claude conversation 6. Clicked “New Tab” button in Chrome 7. Switched to W&B tab 8. Scrolled through W&B for “context-distillation-goyal”	1. Scrolled through W&B charts on “NAP Time” dashboard 2. Closed W&B dashboard tab in Chrome 3. Switched to Claude.ai tab 4. Closed current tab in Chrome 5. Switched to Claude tab 6. Inspected text in Claude chat 7. Re-fetched view selection content from Claude 8. Re-fetched view selection content from Claude
0.42	1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked thumbnail in “Key Moments” section 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, clicked “Runs” icon 5. Switched to “Australia vs Zimbabwe,” entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, scrolled through charts 8. Scrolled through W&B dashboard	1. Switched to “Australia vs Zimbabwe” tab 2. Clicked full screen button on cricket video 3. Clicked History channel logo at bottom of player bar 4. Pressed “escape” in Chrome 5. Clicked on the video player 6. Clicked History channel logo at bottom of player bar 7. Clicked History channel logo, pressed “space” 8. Pressed “space” in Chrome

Appendix

Table 9: Qualitative examples of predicted vs. ground-truth actions at different accuracy scores assigned by the Gemini judge (Part 1: low scores). The first five rows are from model predictions.

Score	Ground Truth	Predicted Actions
0.52	1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked thumbnail in “Key Moments” section 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, clicked “Runs” icon 5. Switched to “Australia vs Zimbabwe,” entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, scrolled through charts 8. Scrolled through W&B dashboard	1. Switched to “Australia vs Zimbabwe” tab, clicked player 2. Pressed “f” to enter fullscreen on cricket video 3. Pressed “esc” to exit fullscreen 4. Opened new tab, typed “wandb.ai” into address bar 5. Scrolled through W&B charts for training metrics 6. Clicked “Table” view in W&B dashboard 7. Switched to YouTube, clicked recommended video 8. Scrolled through comments section on YouTube
0.72	1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked thumbnail in “Key Moments” section 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, clicked “Runs” icon 5. Switched to “Australia vs Zimbabwe,” entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, scrolled through charts 8. Scrolled through W&B dashboard	1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked on video player timeline to skip forward 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, scrolled through dashboard 5. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, clicked on a chart 8. Clicked “Table” view in W&B workspace
0.78	1. Clicked on a purple property pin on the map 2. Clicked on the “ $3.5k+” property pin 3. Clicked on the “$ 2.8k” property pin 4. Clicked on the “ $3.5k+” property pin 5. Clicked on the “$ 3.3k” property pin 6. Clicked on the “ $3.9k” property pin 7. Clicked on the “$ 5.3k” property pin 8. Clicked on the “$2.8k” property pin	1. Clicked on a purple property pin on the map 2. Clicked on the “ $3.5k+” property pin 3. Clicked on the “$ 2.8k” property pin 4. Clicked on the “ $3.5k+” property pin 5. Clicked on the “$ 3.3k” property pin 6. Clicked on the “ $3.9k” property pin 7. Clicked on the “$ 4.2k” property pin 8. Clicked on the “$3.1k” property pin
0.82	1. Typed “icc world cup” into search bar 2. Clicked “STANDINGS” tab in ICC T20 World Cup widget 3. Scrolled through standings table 4. Clicked “SL” (Sri Lanka) in standings 5. Switched to W&B dashboard tab 6. Scrolled through W&B charts 7. Clicked “train/entropy” chart in W&B 8. Clicked chat with “Omar Shaikh” in Slack	1. Typed “icc world cup” into search bar 2. Clicked “STANDINGS” tab in ICC T20 World Cup widget 3. Scrolled through standings table 4. Clicked “IND” (India) in standings 5. Switched to W&B dashboard tab 6. Scrolled through W&B charts 7. Clicked “train/loss” chart in W&B 8. Clicked chat with “Omar Shaikh” in Slack
0.98	1. Clicked “rouge-score” on PyPI, copied install command 2. Ran uv pip install rouge-score in terminal 3. Scrolled through “rouge-score” PyPI page 4. Ran accelerate launch training command 5. Scrolled through “rouge-score” PyPI page 6. Switched between W&B, Google Doc, and terminal 7. Typed training command, cancelled with Ctrl+C 8. Ran cd sim-v2 and git pull	1. Clicked “rouge-score” on PyPI, copied install command 2. Ran uv pip install rouge-score in terminal 3. Scrolled through “rouge-score” PyPI page 4. Ran accelerate launch with dataset arguments 5. Scrolled through “rouge-score” PyPI page 6. Switched between W&B, Google Doc, and terminal 7. Typed CUDA training command, cancelled with Ctrl+C 8. Ran cd sim-v2 and git pull

Appendix

Table 10: Qualitative examples continued (Part 2: high scores). These examples are manually constructed to illustrate what the upper score range represents.

Human-Computer Interaction

See all 26 papers

Oct 1, 2026

NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video

We often plan ambitiously yet act habitually and wonder, in retrospect, whether we would have planned differently had we known what we would actually do. Hindsight offers a valuable perspective on past decisions, although we often wish we could have simulated hindsight at the moment of choosing. If a system could generate plausible trajectories from one's personal history, such previews might help people formulate more realistic plans and make better informed decisions. We introduce NextMe-800, an approximately 800-hour first-person dataset from one volunteer over 126 days with 1 Hz images, gaze, and audio, captioned at five hierarchical abstraction levels from atomic actions to major activities. We formulate personalized action anticipation as open-vocabulary K-step sequence prediction and construct NextAct, a 1,500-point benchmark combining NextMe-800 with the multi-person EgoLife dataset. Using an embedding-based soft edit distance as the metric, we evaluate how well different models can anticipate personal behavior across abstraction levels and prediction horizons. NextMe-800 and NextAct provide a months-long resource and evaluation framework for studying how far ahead personal behavior can be anticipated from egocentric observation.

Zhaoxu Meng, Yiming Sun, Mingyuan Gao +5

The University of Hong Kong · Tongji University

Next-Action Prediction Egocentric Video Understanding Human Behavior Prediction Egocentric Vision Next-Event Prediction

Ask PDF

May 22, 2026

OnePred: Next-Query Prediction via Recursive Intent Memory in Multi-Turn Conversations

Although large language model (LLM) conversational systems process millions of multi-turn dialogues daily, they remain fundamentally reactive: they respond only after the user types a query. A key step toward proactive interaction is next-query prediction, which anticipates the user's subsequent query based solely on the preceding dialogue. Progress on this task is hindered by the lack of dedicated benchmarks and a fundamental efficiency--quality trade-off: naively concatenating full dialogue history incurs linearly growing token consumption, while truncating to the latest turn discards crucial cross-turn context. Our key insight is that accurate prediction does not require re-reading raw history; it suffices to track the user's evolving intent trajectory across topics, unresolved needs, and interest shifts. We propose OnePred, which maintains a recursively updated memory as its sole cross-turn context, bounding the per-turn cost independently of conversation length. We train the model via a two-stage reinforcement learning pipeline that first teaches what to predict, then what to compress, shaping the memory into a prediction-oriented intent chain. To establish a rigorous testbed, we introduce NQP-Bench, spanning three diverse subsets. Experiments demonstrate that OnePred reduces per-turn token consumption by up to 22

\times

compared to full-history inputs while consistently exceeding all baselines in prediction quality, with larger gains on longer conversations. Our code is publicly available at https://github.com/ZBWpro/OnePred.

Jiangwang Chen, Bowen Zhang, Zixin Song +4

1Tsinghua University · 2Qwen Applications Business Group of Alibaba

Long-Term Conversational Memory Efficient Language Model Inference Language Model Post-Training

Ask PDF

Apr 26, 2026

DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute

Scientific Deep Research (DR) agents answer user queries by synthesizing research papers into multi-section reports. User feedback can improve their utility, but existing protocols only score the final report, making it hard to study and learn which intermediate actions DR agents should take to improve reports. We collect DRACULA, the first dataset with user feedback on intermediate actions for DR. Over five weeks, nineteen expert CS researchers ask queries to a DR system that proposes actions (e.g., "Add a section on datasets"). Our users select actions they prefer, then judge whether an output report applied their selections successfully, yielding 8,103 action preferences and 5,230 execution judgments. After confirming a DR agent can execute DRACULA's actions, we study the predictability of user-preferred actions via simulation-how well LLMs predict the actions users select-a step toward learning to generate useful actions. We discover: (1) LLM judges initially struggle to predict action selections, but improve most when using a user's full selection history, rather than self-reported or extrapolated user context signals; (2) Users' selections for the same query differ based on unstated goals, bottlenecking simulation and motivating affordances that let users steer reports; and (3) Our simulation results inform an online intervention that generates new actions based on the user's past interactions, which users pick most often in follow-up studies. Overall, while work extensively studies execution, DRACULA reveals a key challenge is deciding which actions to execute in the first place. We open-source DRACULA's study design, user feedback, and simulation tasks to spur future work on action feedback for long-horizon agents.

Nishant Balepur, Malachi Hamada, Varsha Kishore +9

University of Maryland · New York University · Allen Institute for Artificial Intelligence (Ai2)

Deep Research Agents LLM Agent Evaluation LLM Agents Human-AI Decision Making

Ask PDF

Abstract

Human-Computer Interaction

Explore similar work

NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video

OnePred: Next-Query Prediction via Recursive Intent Memory in Multi-Turn Conversations

DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute