Task handover requires communicating enough current state for a successor to resume work, yet the relevant information is often divided between system records and human observations. System records can be precise and timestamped but only partially observe the task, while human reports capture intent and task knowledge that no log contains but are vulnerable to omission and memory error. We present a provenance-aware pipeline that converts task telemetry and human-authored reports into a shared typed task-state representation, aligns and reconciles their facts, detects conflicts, and generates structured handover reports. We evaluate the approach on 13 paired task states collected in a controlled spatial multitask environment, using task-grounded metrics that estimate the state-reconstruction cost a report would spare a hypothetical recipient and the misinformation burden it would impose. Reconciling both sources preserved greater estimated task-state utility than either the user report or telemetry alone. Relative to a direct end-to-end LLM given the same inputs, structured reconciliation maintained comparable estimated utility while incurring substantially less misinformation, and task-aware rendering retained utility more efficiently (per token) than exhaustive rendering. An exploratory content analysis further shows that human reports contain substantial strategic knowledge that lies outside state-focused metrics. These results support provenance-aware state reconciliation as a design pattern for safer AI-assisted handover.
Figures & tables
Figure 1. Provenance-aware state reconciliation pipeline. From left to right: (1) Raw input: source evidence (timestamped telemetry events and sentences from the human-authored report); (2) Structured graphs: source-specific typed facts extracted from each source; (3) Reconciled graph: alignment and conflict detection, classifying each fact as corroborated, telemetry-only, human-only, or conflicting; (4) Narrative view: temporal reconciliation, which prunes superseded facts to yield a current-state estimate with provenance retained, and conversion to top-down entity-centric view; and (5) Natural language report: constrained report realization. The pipeline’s contribution is not conversion of text into a graph per se, but preservation of where each fact came from, explicit detection of disagreement, computation of current (rather than historical) state, and constrained generation over the result. A five-column pipeline diagram showing telemetry events and human report text being converted to typed facts, aligned and checked for conflicts, reconciled into a current task state with provenance labels, and rendered as a structured handover report.
Condition
Human report
Telemetry
Structured repr.
Reconciliation
Renderer
User report (baseline)
x
-
-
-
Human (verbatim)
Task-aware (TA)
x
x
x
x
Task-aware LLM prompt
Full-realization (FR)
x
x
x
x
Exhaustive LLM prompt
End-to-end (end2end)
x
x
-
-
Task-aware LLM prompt
Telemetry-only (TA -user)
-
x
x
x
Task-aware LLM prompt
Table 1. Experimental conditions, decomposed by source input, intermediate representation, explicit state reconciliation, and renderer. All generated conditions are paired transformations of the same 13 participant task states.
Figure 2. Walk-through of the data collection, showing the task interface and map. Participants controlled the player character with keyboard controls detailed in a short tutorial. Left: the interface includes a display of the current room, the player character (blue), and non-player characters within, along with the room name, their score (number of tasks complete), and their time progress in the task. Right: tasks required participants to navigate a map consisting of several rooms, interacting with non-player characters (indicated in purple) and item resources (indicated in pink). Participants did not have access to the map during the task. The environment creates concurrent, spatially distributed task threads and a controlled information asymmetry between system telemetry and human observation (Table 2 ). The task was also interrupted with a distractor task designed to further challenge participants’ working memory of their task threads and progress. After 10 minutes of in-game time had elapsed, participants were asked to write a handover report detailing their task state and progress to a hypothetical successor. A figure showing the flow of the user data collection. There are 5 components with accompanying illusations: Onboarding and tutorial (5 minutes), Simulated hospital floor task, part 1 (5 minutes), Distractor task (2 minutes), Simulated hospital floor task, part 2 (5 minutes), and Reporting (5 minutes). A screenshot of the task user interface and an illustration of the map used for the game task are included.
Information type
Telemetry can observe
Human can observe/report
Room transitions
Yes
Yes
Item pickup/delivery
Yes
Yes
Dialogue content (requests, hints)
No
Yes
Outstanding need inferred from dialogue
Partial
Yes
User intent and strategy
No
Yes
Exact event timestamps
Yes
Usually no
Table 2. Observability matrix for the controlled environment: which information types are available to system telemetry versus to the outgoing human. The engineered asymmetry makes the two sources complementary by design.
Credit Type ( κ )
Condition
Csaved
FULL
Perfect match or correct omission
Cmax
PARTIAL
Correct but vague or under-specified information
P⋅Cmax
NONE
No correct relevant information provided
0
Table 3. Cost breakdown by classification of information extracted from handover reports.
Condition
Combined cost (search-cost units)
Omission share (%)
Misinformation share (%)
Baseline (user-only)
1150.51 [22.50, 2023.93]
64.6 [34.7, 100.0]
35.4 [0.0, 65.3]
Task-aware
370.21 [150.00, 965.94]
63.0 [17.6, 100.0]
37.0 [0.0, 82.4]
Telemetry-only
274.70 [150.00, 551.50]
100.0 [34.7, 100.0]
0.0 [0.0, 65.3]
Full-realization
361.20 [191.50, 789.24]
44.6 [18.2, 100.0]
55.4 [0.0, 81.8]
End-to-end
773.20 [370.21, 1345.51]
32.5 [14.2, 50.4]
67.5 [49.6, 85.8]
Table 4. Descriptive cost metrics by condition. Combined cost is computed at α=3 and reported for descriptive purposes only; estimated cost saved and misinformation cost are reported separately as primary inferential metrics (see 6 ). Values are median [range] across n=13 participants. Units are task-derived search-cost units, not literal movement steps.
Figure 3. Estimated information access cost saved (positive axis) and misinformation cost incurred (negative axis) by report condition. TA = task-aware (ours); end2end = end-to-end (LLM-only ablation); TA -user = telemetry-only ablation; n.s. = not significant; * = p<.05 ; ** = p<.01 ; *** = p<.001 . Task-aware reports saved significantly more estimated cost than the user-only baseline and the telemetry-only ablation, and incurred significantly less misinformation than the user-only baseline and the end-to-end ablation. Estimated cost saved did not differ significantly between the task-aware, full-realization, and end-to-end conditions, and the telemetry-only ablation incurred the least misinformation of all conditions. A bar chart with one bar group per report condition. Bars above the horizontal axis show estimated cost saved and bars below the axis show misinformation cost incurred, with significance brackets over selected condition pairs.
Condition
Combined Cost
Omission Share (%)
Misinformation Share (%)
Task-aware (GPT 4.1-mini)
370.21 [150.00, 965.94]
63.0 [17.6, 100.0]
37.0 [0.0, 82.4]
Task-aware (GPT 5.6-terra)
233.20 [101.50, 485.90]
100.0 [36.8, 100.0]
0.0 [0.0, 63.2]
End-to-end (GPT 5.6-terra)
203.20 [116.50, 444.40]
100.0 [36.5, 100.0]
0.0 [0.0, 63.5]
Table 5. Comparison with frontier model, combined cost and omission/misinformation shares by condition
Figure 4. Breakdown of content types of user reports across all participants, by token count. Human-authored reports contained both task-state content and strategic-knowledge content. State was the largest category on average, while the proportion of strategic content varied substantially across participants. The value of this strategic knowledge is more difficult to evaluate objectively, and it is not captured by our state-focused pipeline or metrics. A stacked bar chart showing, for each participant's report, the proportion of tokens categorized as state transfer, knowledge transfer, ambiguous, or meta content. The mix varies widely across participants.