Rhythms of Work: Multi-Scale Interpretation of Human Behavioral Traces for Workplace Agents

Lin Ai; Scott Counts

cs.CLSep 3, 2026

Rhythms of Work: Multi-Scale Interpretation of Human Behavioral Traces for Workplace Agents

Authors: Lin Ai, Scott Counts

Organizations: Microsoft

Abstract

Runtime traces are becoming a central substrate for understanding agentic systems, yet interpretation has focused largely on what the agent did. Workplace agents face the complementary problem: interpreting the human activity that surrounds them. Hours of low-level events carry rich evidence about a user's state but are too granular to reason over directly, and flattening them into one stream or compressing them into a single embedding both treat "summarize the user's behavior" as if it had one correct answer. We argue instead that behavioral interpretation is resolution-dependent: the same trace should admit multiple addressable interpretations at different temporal resolutions. We construct a multi-resolution vocabulary of semantically normalized operators, recurring motifs, coherent episodes, and day-level rhythms, each preserving the structure salient at its own horizon. Applied to 667 million human-attributed events from a large commercial productivity suite (50,000 users, 100 organizations), it yields 120 operator types, thousands of motifs, 25 episode types, and five day-rhythm archetypes. We validate it on real telemetry: re-running the entire pipeline on a disjoint 2,000-user sample recovers the same taxonomy (structural stability), and on held-out users the full representation forecasts a user's next episode more accurately than a flat-operator baseline, a 17% relative macro-F1 gain (predictive validity), so the abstractions preserve future-relevant information rather than merely describe it. A controlled resolution ablation then shows that no single level is optimal across questions: different agent-facing questions about the same trace are best answered at different resolutions. Behavioral trace interpretation for agents should therefore be multi-resolution and query-conditioned: an agent should access the temporal grain a question needs, not one universal summary.

Figures & tables

Figure 1: The same workday admits four simultaneous interpretations at different temporal resolutions, read here on one shared clock. The whole day is a single rhythm ; it is a sequence of episodes ; each episode is a stream of operators ; and short recurring sub-sequences of operators are motifs . Because every track shares the same time axis, containment is visible directly: an episode is a span of operators, a motif a shorter recurring span. The left column states each level’s invariance: the time horizon it covers, what it preserves, and what it abstracts away, which is what lets the levels be addressed separately rather than fused into one summary. The highlighted motif is a word-processor “Save-As Formatting Pass” that recurs twice in the afternoon; both occurrences normalize to the same operator sequence, Format

\rightarrow

FileModified

\rightarrow

FileSaveAs (3 of 120 canonical operators). The four levels are a conceptual multi-resolution vocabulary; the learning pipeline need not implement a strictly sequential generative hierarchy.

Level	Instances / coverage	Vocabulary	Typical scale
Raw events (human)	667M events	$\sim$ 14K app–action pairs	sub-second to minutes
Canonical operators	667M mapped events	120 types	seconds
Motifs	20.6M occurrences	3,597 motifs	seconds to minutes
Episodes	16.7M instances	25 types	minutes to hours
Day rhythms	1.82M user-days	5 archetypes	hours to days

Table 1: Scale of the corpus and learned representation. All counts refer to the human-attributed event stream.

Figure 2: Next-episode forecasting on held-out users (2,975,438 transitions, 10,000 users), one fixed linear probe per condition. Left: both metrics rise as representation scales are added cumulatively (macro-F1 primary, top-3 accuracy secondary); adding episodes lifts macro-F1 by 11% (the pre-registered primary; absolute +0.023, 95% CI [0.022, 0.023]). Right: the same conditions split by what happens next. Adding episode context improves new-mode prediction (

+20\%

relative) but degrades same-type repeats (

-27\%

), where the current type already predicts the next. The average gain is real forward-looking signal, and added scale is not uniformly useful, which is the predictive-side motivation for scale-selective retrieval.

Figure 3: Which interpretation answers each question? Macro-F1 for seven capabilities across eight context shapes, on the instrumented fixtures (640 questions, three runs); each column exposes a different interpretation of the same underlying trace. Rows are grouped into the three families (Perception, Inference, Action). In each row a filled dot marks the best observable context and an open dot its runner-up, so the distance between them shows how sharply the winning grain leads. The best cell moves across four different columns (motif, episode, rhythm, and the stacked hierarchy), so no single interpretation wins everywhere. The two columns to the right of the rule, oracle and all , carry declared metadata that telemetry cannot observe.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Shape	Content on top of the common baseline
baseline	persona role, time of day, the artifact under the cursor, today’s calendar, recent collaborators
+raw	the last 60 minutes of raw events (timestamp, app, action, operation)
+motif	a step-level trace: per step, its goal/purpose/app and a run-compressed action sequence
+episode	the current and recent episode labels, durations, and boundaries
+rhythm	the day-so-far summary: cumulative active time, episode-type mixture, meeting and AI-assistant share
+hierarchy	baseline + motif + episode + rhythm together (the bottom-up pipeline’s output)

Appendix

Table 2: The eight context shapes. The six observable shapes add layers a deployed agent could plausibly compute; oracle and all add declared metadata as reference points.

Family	Capability	Sub-classes (sample counts)	$n$
Perception	Motif-scale recognition	drafting 33, revising 33, investigating 33, wrap-up 33	132
Perception	Episode-scale recognition	create 35, communicate 35, consume 35	105
Perception	Day-scale recognition	full workday 33, deep work 33, light day 26	92
Inference	Behavior projection	same-task 25, transition 25	50
Inference	Anomaly detection	normal 25, fragmented 25	50
Action	Interruption timing	ideal 25, mild 25, high 25, prohibitive 22	97

Appendix

Table 3: The seven capabilities, their families, and sub-class sample counts (before the

\times 8

shapes

\times 3

runs).

Perception
Probe (Fig. 3 row)	Question, verbatim	Answer classes (ground truth)
motif recog.	“Looking at the user’s recent activity (the last hour or so), which best describes the kind of work they have been doing?”	drafting, revising, investigating, wrap-up
episode recog.	“What is the user’s current activity primarily aimed at?”	create, communicate, consume
day recog.	“Looking at the user’s full day so far (when they started, what kinds of work, how much overall activity), which best describes the day’s overall rhythm?”	full workday, deep work, light day
Inference
short-horizon proj.	“Based on the user’s recent activity and the patterns of their day so far, what is the user most likely to do in the next 30 minutes?”	same-task continues, task transition

Appendix

Table 4: The seven probe questions, verbatim, with the answer classes that define ground truth. Short labels match the rows of Figure 3 ; per-class sample counts are in Table 3 . Each question is posed as a single multiple-choice item (with an added “none of the above” escape option), and every context shape answers the identical question.

Figure 4: Perception, per sub-class. Rows are grouped by capability (motif, episode, and day recognition); a filled dot marks each row’s best observable context and an open dot its runner-up. Day recognition is uniformly best at the rhythm layer, while motif recognition spreads its best cell across baseline, raw, motif, and rhythm.

Figure 5: Inference, per sub-class. Both projection sub-classes and both anomaly sub-classes peak at the episode grain; for anomaly detection the observable episode cell already matches the all ceiling, so declared oracle metadata adds nothing.

Figure 6: Action, per sub-class. Interruption timing (ideal, mild, high, prohibitive) is a broad hierarchy lead that never dominates a single row; help selection splits its best cell between motif (drafting, scheduling) and the hierarchy (analysis, entry, polish).

Stratum	Group	Episode increment	Transitions
Overall	all	$+0.023$	2,975,438
Transition kind	state change	$+0.035$	2,570,957
Transition kind	same-label revisit	$-0.098$	404,481
Future gap	0–10 s	$+0.032$	363,134
Future gap	$\geq$ 300 s	$+0.020$	2,612,304
Always-on	excluded	$+0.022$	2,967,600

Appendix

Table 5: Episode-increment (S-C3

-

S-C2) macro-F1 differences by stratum, held-out test set. The increment is also stable across the five user-persona strata (

+0.017

+0.025

), omitted here for space. Descriptive; not part of the pre-registered primary comparison.

Figure 7: Independent held-out replication. Left: log-gap quantiles for the 50K run overlap those of a disjoint 2K run (Jensen–Shannon divergence 0.0040). Right: per-application episode clusters matched by Hungarian assignment; 92% (23/25) of 25 types are recovered above the

0.7

cosine threshold.

Figure 8: The five day-rhythm archetypes over 1.82M user-days, ordered by prevalence: Core Collaboration (48%), Quick-Check (31%), Meeting-Block (13%), AI-Assisted Document (5%), and Extended Collaboration Span (3%). Each card shows the rhythm’s share of user-days, a short profile, and the episode types that compose it, and doubles as the color key. Bottom: the ten personas as dialects , each a longitudinal mixture over the same five rhythms. All names are generated from behavioral profiles after clustering and are never inputs to the objective.

Human Activity Recognition

Human activity recognition classifies actions and behaviors from wearable sensors, video, radio signals, and other measurements.

See all 228 papers

Aug 31, 2026

Using Grounded Theory for Agent Behavior Analysis at Scale

Understanding agent behavior requires methods that scale to thousands of trajectories and surface new patterns in long, often unfamiliar tasks where pre-built classifiers fall short. We propose to bring grounded theory into agent trajectory analysis: a six-decade-old qualitative method from the social sciences, with a principled saturation criterion and an auditable trail from data to theory. We propose AutoTraceGT (Automated Trace analysis through Grounded Theory), the first multi-agent pipeline that automates grounded theory on agent trajectories. It iteratively performs open, axial, and theoretical coding until saturation, producing a behavioral taxonomy tailored to each task. Across six trajectory corpora, AutoTraceGT produces codebooks that recover 73-91 percent of the failure modes in human-annotated taxonomies and surface additional patterns that those taxonomies miss. The emergent theoretical narrative aligns with prior expert accounts. Used as a deductive feature space, the codebook outperforms zero-shot and few-shot LLM baselines on downstream failure prediction. These results suggest Grounded Theory offers a scalable analytic tool for ML researchers and agent developers studying what agents actually do.

Zhuoran Lu, Yangyang Yu, Zhuoyan Li +5

1Purdue University · 2Stevens Institute of Technology · 3Cornell University +2

Agent Failure Analysis Agent Trajectory Analysis

Ask PDF

May 13, 2026

How to Interpret Agent Behavior

Autonomous agents such as Claude Code and Codex now operate for hours or even days. Understanding their runtime behavior has become critical for downstream tasks such as diagnosing inefficiencies, fixing bugs, and ensuring better oversight. A primary way to gain this understanding is analyzing the reasoning trajectories and execution traces these agents generate. Yet such data remains in unstructured natural-language form, making it difficult for humans to interpret at scale. We introduce ACTONOMY (a combination of Action and Taxonomy), a taxonomy for describing and analyzing agent behavior at runtime. ACTONOMY has two components: (1) the taxonomy itself, developed through Grounded Theory and structured as a three-level hierarchy of 10 actions, 46 subactions, and 120 leaf categories; and (2) an open repository that hosts the living taxonomy, provides an automated analysis pipeline that applies it to agent trajectories analysis, and defines an extension protocol for customization and growth. Our experiments show that ACTONOMY can compare behavioral profiles across agents and characterize a single agent's behavior across diverse trajectories, surfacing patterns indicative of failure modes. By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently, enabling better oversight and control.

Jie Gao, Kaiser Sun, Jen-tse Huang +8

1Johns Hopkins University · 2California Institute of Technology · 3Northeastern University +1

Hierarchical Classification AI Agent Monitoring LLM Agents Agent Trajectory Analysis

Ask PDF

Aug 20, 2026

Inducing Task Models from Computer-Use Traces

Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and real-world work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, and produce step-level summaries rather than structured task models. We introduce Task Model Induction (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.

Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen +1

Stanford University · Carnegie Mellon University

Task Decomposition Process Mining Computer-Use Agents

Ask PDF

Abstract

Human Activity Recognition

Explore similar work

Using Grounded Theory for Agent Behavior Analysis at Scale

How to Interpret Agent Behavior

Inducing Task Models from Computer-Use Traces