cs.CLMar 6, 2026
SaveLearning Next Action Predictors from Human-Computer Interaction
Organizations: Stanford University · Hasso Plattner Institute · New York University
Abstract
Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.
Figures & tables
Figure 1: Long-Context Next Action Predictors (LongNAPs) draw from the entirety of a user’s multimodal context (e.g. screenshots)—retrieving over an unbounded history—to predict what they will do next. We train LongNAP end-to-end on data from 20 users over a month, spanning 1.9M screenshots or 1,800 hours of screen on-time. Predictions are rewarded based on LLM-judged similarity to a set of ground truth future actions.
Figure 2: LongNAP significantly outperforms all baselines by at least 39.4% relative to the strongest baseline . We evaluate with LLM-judge, which outputs similarity to ground truth future actions (0-1 score). Performance is averaged across 20 models trained on individual users.
Figure 3: NAPsack enables passive collection of human-computer interaction data . It ingests screenshots and input events, compresses them to retain only meaningful frames, and annotates with action descriptions.
| Ground-Truth | Predicted | Predicted | Predicted | |
|---|---|---|---|---|
| Score = 1.0 | Score = 0.72 | Score = 0.52 | Score = 0.15 | |
| 1 | Switched to “Australia vs Zimbabwe” tab, entered fullscreen | Switched to “Australia vs Zimbabwe” tab, entered fullscreen | Switched to “Australia vs Zimbabwe” tab, clicked player | Clicked on YouTube home page, browsed recommended videos |
| 2 | Clicked thumbnail in “Key Moments” section | Clicked on video player timeline to skip forward | Pressed “f” to enter fullscreen on cricket video | Clicked video titled “Best Cricket Catches of 2025” |
| 3 | Pressed “esc” to exit fullscreen | Pressed “esc” to exit fullscreen | Pressed “esc” to exit fullscreen | Pressed “esc” to exit fullscreen |
| 4 | Switched to W&B workspace, clicked “Runs” icon | Switched to W&B workspace, scrolled through dashboard | Opened new tab, typed “wandb.ai” into address bar | Typed “australia cricket schedule” into Google search bar |
| 5 | Switched to “Australia vs Zimbabwe,” entered fullscreen | Switched to “Australia vs Zimbabwe” tab, entered fullscreen | Scrolled through W&B charts for training metrics | Clicked ESPN Cricinfo link in search results |
Table 1: Qualitative examples of candidate trajectories at decreasing LLM judge similarity scores (left to right), all evaluated against the same ground-truth human annotated trajectory. For readability, we highlight in red actions with substantially wrong intent.
| Method | Judge Score ( ) | Human WR % ( ) | Size ( ) |
|---|---|---|---|
| naive | 0.48 0.03 | 19.2 6.7 | |
| + split | 0.57 0.03 | 45.4 8.7 | |
| + split + compress | 0.60 0.03 | 49.6 8.9 | |
| + split + compress + IO | 0.70 0.03 | 85.8 6.0 |
Table 2: NAPsack reduces the amount of data we have to save for effective captioning by 75% without compromising quality . Event-driven compression (where frames are saved only when a user interacts with their computer) yields improved efficiency; and I/O signals further improve performance. Performance is measured across both LLM-judge scores ([0, 1]) and human eval (win rate). 95% conf. intervals are computed through bootstrapping.
Figure 4: Predictions from LongNAP are generated in a two-phase process. In the first phase, LongNAP Reasons to Retrieve : conditioned on what the user sees right now (e.g. a set of paper reviews), LongNAP generates a reasoning trace and uses it to retrieve past traces. Using retrieved traces, LongNAP Reasons to Predict : generating a final reasoning trace, adding it back to memory, and then predicting the next steps a user might take. The predicted trajectory is compared against a ground truth (with an LLM-as-a-judge similarity score), and then LongNAP is optimized with GRPO ( Shao et al., 2024b ) .
| User | Gemini | Qwen-2.5-VL-7B | ||||
|---|---|---|---|---|---|---|
| Zero-shot | RAG | Zero-shot | RAG | SFT | LongNAP | |
| 0.28 | 0.29 | 0.21 | 0.23 | 0.23 | +0.12 0.41 | |
| 0.21 | 0.22 | 0.15 | 0.17 | 0.18 | +0.12 0.34 | |
| 0.23 | 0.23 | 0.19 | 0.18 | 0.17 | +0.10 0.33 | |
| 0.24 | 0.23 | 0.14 | 0.16 | 0.17 | +0.09 0.32 | |
| 0.23 | 0.24 | 0.18 | 0.16 | 0.16 | +0.10 0.34 | |
Table 3: Training on individual users substantially outperforms baseline methods, improving average performance by 39.4% ( +0.11 ) relative to the strongest baseline (Gemini Few-shot RAG). We report the similarity score (0-1) to ground truth future actions, as determined by our LLM judge ( §3.2 ). denotes a LongNAP instance trained on a single user from our dataset of 20 users, and represents the mean performance across all 20 individually trained models.
Figure 5: Some users are more predictable than others. When LongNAP is trained on a single user (in our generalizing over time experiments, §5.1 ), LLM-as-a-judge evals vary substantially from one user’s LongNAP to another user’s. In the above figure, we re-evaluate across checkpoints from training epochs for each user, highlighting variance.
| Model | Method | Win Rate (%) |
| Qwen 2.5 VL | LongNAP | 79.0 6.6 |
| SFT | ||
| Few-shot RAG | ||
| Zero-shot | ||
| Gemini | Few-shot RAG | |
| Zero-shot |
Table 4: Human evaluation shows significantly higher win rates for LongNAP. We show pairwise human evaluation win rates across each method (combined across 2 annotators, 300 total comparisons, 95% bootstrapped CIs).
| Model | Method | ||||||
|---|---|---|---|---|---|---|---|
| Gemini | Zero-shot | 0.22 | 0.27 | 0.22 | 0.20 | 0.20 | 0.21 |
| Few-shot RAG | 0.23 | 0.26 | 0.23 | 0.22 | 0.21 | 0.22 | |
| Qwen VL | Zero-shot | 0.16 | 0.22 | 0.13 | 0.14 | 0.16 | 0.14 |
| Few-shot RAG | 0.17 | 0.23 | 0.15 | 0.18 | 0.14 | 0.18 | |
| SFT | 0.17 | 0.22 | 0.14 | 0.16 | 0.15 | 0.17 | |
| LongNAP | 0.26 | 0.31 | 0.28 | 0.22 | 0.21 | 0.26 |
Table 5: When trained jointly on many users, LongNAP exhibits modest generalization to unseen users, improving average performance by 13.0% relative to the strongest baseline (again, Gemini Few-Shot RAG). We report similarity to ground truth future actions, as determined by our LLM judge ( §3.2 ). To test generalization to unseen users, we train randomly on 10 users; validate on 5; and then test on 5 (reported above). denotes the mean performance across the evaluated users, and represents performance on user .
| Model | ||||||
|---|---|---|---|---|---|---|
| LongNAP | ||||||
| Ablations | ||||||
| Remove reasoning | (-0.07) | (-0.09) | (-0.06) | (-0.06) | (-0.09) | (-0.06) |
| Remove retriever | (-0.06) | (-0.07) | (-0.07) | (-0.01) | (-0.08) | (+0.01) |
| Shuffle dataset | (-0.04) | (-0.05) | (-0.04) | (+0.00) | (-0.05) | (-0.00) |
Table 6: Both reasoning and retrieval are critical to LongNAP’s performance. We pick a random subset of users to perform ablations on. Removing the reasoning component leads to the largest drop in average performance ( absolute), while removing the retriever reduces performance by on average. Shuffling the dataset also degrades results ( ), suggesting that preserving temporal structure is important. denotes mean performance across the evaluated users, and denotes performance on user . We perform ablations on a random subset of 5 users.
Figure 6: At a given time (top), LongNAP is likely to retrieve over a substantial part of its past context (bottom) to predict what a user will do next. The visualization above (for a random user) shows what context from the past is retrieved for a query at the current time.
Figure 7: LongNAP learns diverse reasoning strategies when trained on different users. Different colors correspond to different users. We embed (using the all-MiniLM-L6-v2 model; Wang et al. (2020) ; Reimers and Gurevych (2019) ) and visualize traces from the reasoning to predict phase across a subset of users with tSNE. For some users (e.g. , blue, almost exclusively takes online surveys), reasoning strategies are homogenous. For others, LongNAP generates a library of reasoning patterns for specific contexts ( ; in orange).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Event type | GAP threshold (s) | MAX duration (s) |
|---|---|---|
| Click | 0.2 | 0.3 |
| Move | 0.5 | 4.0 |
| Scroll | 0.5 | 3.0 |
| Key | 0.5 | 6.0 |
Table 7: Event burst thresholds used by NAPsack.
Figure 8: Reasoning traces grow shorter across model training. Traces for the retrieve phase grow far shorter than queries for the predict phase (avg. 10.11 retrieval tokens v.s. 85.34 prediction). Qualitatively, we find that reasoning for the retrieve phase resembles queries to a retriever; while reasoning for the prediction phase resembles higher order descriptions of a user’s behavior.
| Hyperparameter | LongNAP | SFT |
| Effective Batch Size | 16 | 16 |
| Learning Rate | 3e-5 | 1e-4 |
| Group Size | 4 | - |
| Epochs | 10 | 10 |
| Actions in history | 16 | 16 |
| Actions to predict | 8 | 8 |
Table 8: Hyperparameter settings. We the best checkpoint based on our validation performance, early stopping if appropriate. We only sweep learning rate on baselines (with SFT) due to budget constraints. We train on 8 B200 GPUs.
Figure 9: Pass@k scores for LongNAP. To count as a “pass,” we selected an LLM-judge threshold of 0.5; trajectories that get this score are often well aligned with the actual intent of the user, but miss minor details or skip a few actions (see §5.2 for an example). At this threshold, LongNAP achieves 17.1% at pass@1 across users, rising to 36.3% at pass@20 .
Figure 10: Empirical calibration of LongNAP. Confidence percentiles are computed per-user from intra-cluster variance of 20 sampled trajectories. Higher confidence prompts yield substantially higher pass@1 accuracy (25.9% vs. 10.3%).
| Score | Ground Truth | Predicted Actions |
|---|---|---|
| 0.05 | 1. Typed “icc world cup” into search bar 2. Clicked “STANDINGS” tab in ICC T20 World Cup widget 3. Scrolled through standings table 4. Clicked “SL” (Sri Lanka) in standings 5. Switched to W&B dashboard tab 6. Scrolled through W&B charts 7. Clicked “train/entropy” chart in W&B 8. Clicked chat with “Omar Shaikh” in Slack | 1. Typed “x” into address bar, pressed Enter 2. Clicked Twitter logo in top left 3. Typed “arch” into Twitter search bar 4. Clicked search result for “Arch Labor” 5. Clicked “Home” button in Twitter sidebar 6. Scrolled through Twitter feed 7. Clicked search button on Twitter 8. Typed “forms” into Twitter search bar |
| 0.12 | 1. Clicked “rouge-score” on PyPI, copied install command 2. Ran uv pip install rouge-score in terminal 3. Scrolled through “rouge-score” PyPI page 4. Ran accelerate launch training command 5. Scrolled through “rouge-score” PyPI page 6. Switched between W&B, Google Doc, and terminal 7. Typed training command, cancelled with Ctrl+C 8. Ran cd sim-v2 and git pull | 1. Scrolled through search results for “install rouge score” 2. Clicked on a search result for “install rouge score” 3. Scrolled down “crag” repository on GitHub 4. Clicked “hub/parouza” link in GitHub repo 5. Clicked “eval” directory in file explorer 6. Pressed Enter to run evaluate_qm9.py 7. Switched to Google Chrome window 8. Copied pip install command from GitHub |
| 0.20 | 1. Clicked on a purple property pin on the map 2. Clicked on the “2.8k” property pin 4. Clicked on the “3.3k” property pin 6. Clicked on the “5.3k” property pin 8. Clicked on the “$2.8k” property pin | 1. Scrolled through search results and map on Zillow 2. Clicked on a property marker on the Zillow map 3. Clicked “Down” arrow to expand price filter 4. Clicked “OK” to apply price filter settings 5. Scrolled through filtered search results 6. Selected a property listing card from results 7. Clicked “Show on map” on the listing page 8. Scrolled through rental listings on Zillow map |
| 0.35 | 1. Scrolled through W&B dashboard 2. Clicked “kanishkgandhi” project link in W&B 3. Clicked on a run in W&B sidebar 4. Clicked “kanishkgandhi’s workspace” tab 5. Scrolled through Claude conversation 6. Clicked “New Tab” button in Chrome 7. Switched to W&B tab 8. Scrolled through W&B for “context-distillation-goyal” | 1. Scrolled through W&B charts on “NAP Time” dashboard 2. Closed W&B dashboard tab in Chrome 3. Switched to Claude.ai tab 4. Closed current tab in Chrome 5. Switched to Claude tab 6. Inspected text in Claude chat 7. Re-fetched view selection content from Claude 8. Re-fetched view selection content from Claude |
| 0.42 | 1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked thumbnail in “Key Moments” section 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, clicked “Runs” icon 5. Switched to “Australia vs Zimbabwe,” entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, scrolled through charts 8. Scrolled through W&B dashboard | 1. Switched to “Australia vs Zimbabwe” tab 2. Clicked full screen button on cricket video 3. Clicked History channel logo at bottom of player bar 4. Pressed “escape” in Chrome 5. Clicked on the video player 6. Clicked History channel logo at bottom of player bar 7. Clicked History channel logo, pressed “space” 8. Pressed “space” in Chrome |
Table 9: Qualitative examples of predicted vs. ground-truth actions at different accuracy scores assigned by the Gemini judge (Part 1: low scores). The first five rows are from model predictions.
| Score | Ground Truth | Predicted Actions |
|---|---|---|
| 0.52 | 1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked thumbnail in “Key Moments” section 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, clicked “Runs” icon 5. Switched to “Australia vs Zimbabwe,” entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, scrolled through charts 8. Scrolled through W&B dashboard | 1. Switched to “Australia vs Zimbabwe” tab, clicked player 2. Pressed “f” to enter fullscreen on cricket video 3. Pressed “esc” to exit fullscreen 4. Opened new tab, typed “wandb.ai” into address bar 5. Scrolled through W&B charts for training metrics 6. Clicked “Table” view in W&B dashboard 7. Switched to YouTube, clicked recommended video 8. Scrolled through comments section on YouTube |
| 0.72 | 1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked thumbnail in “Key Moments” section 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, clicked “Runs” icon 5. Switched to “Australia vs Zimbabwe,” entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, scrolled through charts 8. Scrolled through W&B dashboard | 1. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 2. Clicked on video player timeline to skip forward 3. Pressed “esc” to exit fullscreen 4. Switched to W&B workspace, scrolled through dashboard 5. Switched to “Australia vs Zimbabwe” tab, entered fullscreen 6. Pressed “esc” to exit fullscreen 7. Switched to W&B tab, clicked on a chart 8. Clicked “Table” view in W&B workspace |
| 0.78 | 1. Clicked on a purple property pin on the map 2. Clicked on the “2.8k” property pin 4. Clicked on the “3.3k” property pin 6. Clicked on the “5.3k” property pin 8. Clicked on the “$2.8k” property pin | 1. Clicked on a purple property pin on the map 2. Clicked on the “2.8k” property pin 4. Clicked on the “3.3k” property pin 6. Clicked on the “4.2k” property pin 8. Clicked on the “$3.1k” property pin |
| 0.82 | 1. Typed “icc world cup” into search bar 2. Clicked “STANDINGS” tab in ICC T20 World Cup widget 3. Scrolled through standings table 4. Clicked “SL” (Sri Lanka) in standings 5. Switched to W&B dashboard tab 6. Scrolled through W&B charts 7. Clicked “train/entropy” chart in W&B 8. Clicked chat with “Omar Shaikh” in Slack | 1. Typed “icc world cup” into search bar 2. Clicked “STANDINGS” tab in ICC T20 World Cup widget 3. Scrolled through standings table 4. Clicked “IND” (India) in standings 5. Switched to W&B dashboard tab 6. Scrolled through W&B charts 7. Clicked “train/loss” chart in W&B 8. Clicked chat with “Omar Shaikh” in Slack |
| 0.98 | 1. Clicked “rouge-score” on PyPI, copied install command 2. Ran uv pip install rouge-score in terminal 3. Scrolled through “rouge-score” PyPI page 4. Ran accelerate launch training command 5. Scrolled through “rouge-score” PyPI page 6. Switched between W&B, Google Doc, and terminal 7. Typed training command, cancelled with Ctrl+C 8. Ran cd sim-v2 and git pull | 1. Clicked “rouge-score” on PyPI, copied install command 2. Ran uv pip install rouge-score in terminal 3. Scrolled through “rouge-score” PyPI page 4. Ran accelerate launch with dataset arguments 5. Scrolled through “rouge-score” PyPI page 6. Switched between W&B, Google Doc, and terminal 7. Typed CUDA training command, cancelled with Ctrl+C 8. Ran cd sim-v2 and git pull |
Table 10: Qualitative examples continued (Part 2: high scores). These examples are manually constructed to illustrate what the upper score range represents.