Runtime traces are becoming a central substrate for understanding agentic systems, yet interpretation has focused largely on what the agent did. Workplace agents face the complementary problem: interpreting the human activity that surrounds them. Hours of low-level events carry rich evidence about a user's state but are too granular to reason over directly, and flattening them into one stream or compressing them into a single embedding both treat "summarize the user's behavior" as if it had one correct answer. We argue instead that behavioral interpretation is resolution-dependent: the same trace should admit multiple addressable interpretations at different temporal resolutions. We construct a multi-resolution vocabulary of semantically normalized operators, recurring motifs, coherent episodes, and day-level rhythms, each preserving the structure salient at its own horizon. Applied to 667 million human-attributed events from a large commercial productivity suite (50,000 users, 100 organizations), it yields 120 operator types, thousands of motifs, 25 episode types, and five day-rhythm archetypes. We validate it on real telemetry: re-running the entire pipeline on a disjoint 2,000-user sample recovers the same taxonomy (structural stability), and on held-out users the full representation forecasts a user's next episode more accurately than a flat-operator baseline, a 17% relative macro-F1 gain (predictive validity), so the abstractions preserve future-relevant information rather than merely describe it. A controlled resolution ablation then shows that no single level is optimal across questions: different agent-facing questions about the same trace are best answered at different resolutions. Behavioral trace interpretation for agents should therefore be multi-resolution and query-conditioned: an agent should access the temporal grain a question needs, not one universal summary.
Figures & tables
Figure 1: The same workday admits four simultaneous interpretations at different temporal resolutions, read here on one shared clock. The whole day is a single rhythm ; it is a sequence of episodes ; each episode is a stream of operators ; and short recurring sub-sequences of operators are motifs . Because every track shares the same time axis, containment is visible directly: an episode is a span of operators, a motif a shorter recurring span. The left column states each level’s invariance: the time horizon it covers, what it preserves, and what it abstracts away, which is what lets the levels be addressed separately rather than fused into one summary. The highlighted motif is a word-processor “Save-As Formatting Pass” that recurs twice in the afternoon; both occurrences normalize to the same operator sequence, Format → FileModified → FileSaveAs (3 of 120 canonical operators). The four levels are a conceptual multi-resolution vocabulary; the learning pipeline need not implement a strictly sequential generative hierarchy.
Level
Instances / coverage
Vocabulary
Typical scale
Raw events (human)
667M events
∼ 14K app–action pairs
sub-second to minutes
Canonical operators
667M mapped events
120 types
seconds
Motifs
20.6M occurrences
3,597 motifs
seconds to minutes
Episodes
16.7M instances
25 types
minutes to hours
Day rhythms
1.82M user-days
5 archetypes
hours to days
Table 1: Scale of the corpus and learned representation. All counts refer to the human-attributed event stream.
Figure 2: Next-episode forecasting on held-out users (2,975,438 transitions, 10,000 users), one fixed linear probe per condition. Left: both metrics rise as representation scales are added cumulatively (macro-F1 primary, top-3 accuracy secondary); adding episodes lifts macro-F1 by 11% (the pre-registered primary; absolute +0.023, 95% CI [0.022, 0.023]). Right: the same conditions split by what happens next. Adding episode context improves new-mode prediction ( +20% relative) but degrades same-type repeats ( −27% ), where the current type already predicts the next. The average gain is real forward-looking signal, and added scale is not uniformly useful, which is the predictive-side motivation for scale-selective retrieval.
Figure 3: Which interpretation answers each question? Macro-F1 for seven capabilities across eight context shapes, on the instrumented fixtures (640 questions, three runs); each column exposes a different interpretation of the same underlying trace. Rows are grouped into the three families (Perception, Inference, Action). In each row a filled dot marks the best observable context and an open dot its runner-up, so the distance between them shows how sharply the winning grain leads. The best cell moves across four different columns (motif, episode, rhythm, and the stacked hierarchy), so no single interpretation wins everywhere. The two columns to the right of the rule, oracle and all , carry declared metadata that telemetry cannot observe.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Shape
Content on top of the common baseline
baseline
persona role, time of day, the artifact under the cursor, today’s calendar, recent collaborators
+raw
the last 60 minutes of raw events (timestamp, app, action, operation)
+motif
a step-level trace: per step, its goal/purpose/app and a run-compressed action sequence
+episode
the current and recent episode labels, durations, and boundaries
+rhythm
the day-so-far summary: cumulative active time, episode-type mixture, meeting and AI-assistant share
+hierarchy
baseline + motif + episode + rhythm together (the bottom-up pipeline’s output)
Appendix
Table 2: The eight context shapes. The six observable shapes add layers a deployed agent could plausibly compute; oracle and all add declared metadata as reference points.
Table 3: The seven capabilities, their families, and sub-class sample counts (before the ×8 shapes ×3 runs).
Probe (Fig. 3 row)
Question, verbatim
Answer classes (ground truth)
Perception
motif recog.
“Looking at the user’s recent activity (the last hour or so), which best describes the kind of work they have been doing?”
drafting, revising, investigating, wrap-up
episode recog.
“What is the user’s current activity primarily aimed at?”
create, communicate, consume
day recog.
“Looking at the user’s full day so far (when they started, what kinds of work, how much overall activity), which best describes the day’s overall rhythm?”
full workday, deep work, light day
Inference
short-horizon proj.
“Based on the user’s recent activity and the patterns of their day so far, what is the user most likely to do in the next 30 minutes?”
same-task continues, task transition
Appendix
Table 4: The seven probe questions, verbatim, with the answer classes that define ground truth. Short labels match the rows of Figure 3 ; per-class sample counts are in Table 3 . Each question is posed as a single multiple-choice item (with an added “none of the above” escape option), and every context shape answers the identical question.
Figure 4: Perception, per sub-class. Rows are grouped by capability (motif, episode, and day recognition); a filled dot marks each row’s best observable context and an open dot its runner-up. Day recognition is uniformly best at the rhythm layer, while motif recognition spreads its best cell across baseline, raw, motif, and rhythm.
Figure 5: Inference, per sub-class. Both projection sub-classes and both anomaly sub-classes peak at the episode grain; for anomaly detection the observable episode cell already matches the all ceiling, so declared oracle metadata adds nothing.
Figure 6: Action, per sub-class. Interruption timing (ideal, mild, high, prohibitive) is a broad hierarchy lead that never dominates a single row; help selection splits its best cell between motif (drafting, scheduling) and the hierarchy (analysis, entry, polish).
Stratum
Group
Episode increment
Transitions
Overall
all
+0.023
2,975,438
Transition kind
state change
+0.035
2,570,957
Transition kind
same-label revisit
−0.098
404,481
Future gap
0–10 s
+0.032
363,134
Future gap
≥ 300 s
+0.020
2,612,304
Always-on
excluded
+0.022
2,967,600
Appendix
Table 5: Episode-increment (S-C3 − S-C2) macro-F1 differences by stratum, held-out test set. The increment is also stable across the five user-persona strata ( +0.017 to +0.025 ), omitted here for space. Descriptive; not part of the pre-registered primary comparison.
Figure 7: Independent held-out replication. Left: log-gap quantiles for the 50K run overlap those of a disjoint 2K run (Jensen–Shannon divergence 0.0040). Right: per-application episode clusters matched by Hungarian assignment; 92% (23/25) of 25 types are recovered above the 0.7 cosine threshold.
Figure 8: The five day-rhythm archetypes over 1.82M user-days, ordered by prevalence: Core Collaboration (48%), Quick-Check (31%), Meeting-Block (13%), AI-Assisted Document (5%), and Extended Collaboration Span (3%). Each card shows the rhythm’s share of user-days, a short profile, and the episode types that compose it, and doubles as the color key. Bottom: the ten personas as dialects , each a longitudinal mixture over the same five rhythms. All names are generated from behavioral profiles after clustering and are never inputs to the objective.