cs.CLOct 27, 2026
SaveBenchmarking Task Takeover Across Agent Handoffs
Organizations: Hokkaido University Sapporo, Japan · The University of Tokyo Tokyo, Japan
Abstract
As agents collaborate on longer tasks, responsibility may pass between models before the work is complete. The receiving model must build on prior decisions, observations, and actions while addressing the remaining requirements. Evaluating this transition requires connecting the information preserved at handoff to how the receiver continues the work, a relationship that task-success and memory-retention scores alone do not capture. We introduce the Handoff State Continuity Benchmark (HSC-Bench), comprising 1,039 tasks that evaluate two complementary aspects of task takeover: recovering relevant repository evidence and continuing from established conversational, tool, and file-work state. Comparisons across six receiving models show that structured handoff state improves repository evidence recovery for every model and continuation performance for most models. The strongest receivers differ between the two tracks, showing that evidence-recovery rankings alone do not characterize performance when continuing from inherited state. By combining these comparisons with checks of subsequent responses and actions, HSC-Bench evaluates both access to prior evidence and adherence to established commitments and completed work. This provides a basis for selecting receiving models and assessing handoff representations according to the work that must be carried forward.
Figures & tables
Figure 1. Task takeover from a shared history. A recent transcript may omit earlier decisions and completed work, leaving the receiver uncertain about the next step. Structured state preserves this information explicitly: in the illustrated example, the receiver identifies the required input, follows the established constraint, and recognizes that an output has already been created. The comparison holds the receiving model and visible token count fixed. The motivating illustration shows a shared task history and two handoff messages: a recent-history suffix and explicitly organized task state, including a completed file creation that should not be repeated.
Figure 2. Overview of HSC-Bench. Shared source histories and fixed handoff points support comparisons across receivers and paired comparisons of context conditions within each receiver. Recent transcripts and structured state are evaluated on both tracks for all six receivers. GPT-4.1 constructs narrative summaries and structured state from observable history, with compact contexts matched in visible token count. The continuation track additionally evaluates full history for all six receivers, and narrative summaries and a target-state reference for GPT-5.4. The target-state reference supplies privileged task targets. The two tracks assess repository evidence recovery and state-dependent continuation separately. The framework diagram links a fixed source execution to alternative receiver-visible messages and two evaluation tracks. The caption specifies the scope and additional information access of reference conditions.
| Task type | Tasks | Criteria |
|---|---|---|
| Conversational state | 205 | 17 |
| Tool state | 386 | 10 |
| File-work progress | 148 | 8 |
| Total | 739 | — |
Table 1. Controlled tasks and per-task scoring criteria.
| Repository | State-Dependent | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Line-F1 | Mean score | Pass (%) | State by task type (Mean score) | ||||||||
| Receiver | Recent | State | Recent | State | Full | Recent | State | Full | Conv. | Tool | File |
| GPT-5.4 | 0.075 | 0.229 | 89.73 | 92.76 | 94.51 | 44.7 | 52.1 | 57.9 | 87.63 | 96.81 | 89.27 |
| Claude Sonnet 5 | 0.121 | 0.263 | 87.19 | 91.05 | 92.77 | 41.1 | 50.2 | 52.1 | 86.74 | 96.61 | 82.52 |
| DeepSeek V3.2 | 0.033 | 0.189 | 85.29 | 89.54 | 92.65 | 35.0 | 47.2 | 55.3 | 85.45 | 95.18 | 80.49 |
| Qwen 3.5 Plus | 0.089 | 0.250 | 86.71 | 89.84 | 92.37 | 36.7 | 47.2 | 51.6 | 85.80 | 95.98 | 79.39 |
Table 2. Receiver performance on the Repository Evidence Recovery and State-Dependent Continuation tracks, abbreviated as Repository and State-Dependent in the table. Repository scores are line-F1 on 300 tasks; continuation results report mean scores on a 0–100 scale and all-criteria pass rates in percent on 739 tasks. Recent, State, and Full denote recent transcripts, structured state, and full history, respectively. The task-type breakdown reports structured-state mean scores for conversational, tool, and file-work state. Bold indicates the highest value in each column.
Figure 3. Effects of handoff context within the State-Dependent Continuation Track. Panels show conversational, tool, and file-work state tasks. Points represent paired mean score differences, computed as structured state minus recent transcript; positive values favor structured state. Error bars show 95% bootstrap confidence intervals from resampling tasks or task families. Sample sizes appear below each panel. Model abbreviations correspond to Table 2 .
Figure 4. Evidence available at takeover. Path retention is the fraction of previously observed, annotated target paths retained in the received context. Bars average this fraction over tasks. The 259 eligible tasks comprise 217 earlier-only and 42 still-recent tasks; retention is undefined for the remaining 41. Earlier-only transcript retention is zero by definition. Path retention is 12.6 versus 93.0 percent overall, zero versus 92.7 percent for earlier-only evidence, and 77.8 versus 94.4 percent for still-recent evidence, for recent transcripts and structured state.
Figure 5. Evidence recovery and subsequent action in task takeover. Read left to right: prior progress and the remaining request, information received, and actual behavior. (a) GPT-5.4 locates the earlier keyboard handler when its location is retained. (b) Both contexts record the completed address update, yet receivers submit the report, ask for the request again, or repeat the update. Input excerpts are quoted; behavior descriptions are condensed from recorded outputs. Gray values give complete-prediction line-F1 in (a) and mean scores (0–100) over ten requirements in (b). Appendix D details case selection and action sequences. Two cases read from prior progress through received information to receiver behavior. GPT-5.4 misses an earlier code location under recent transcripts and recovers it under structured state. After a completed address update, Claude asks for the request under recent transcripts and reports under structured state; Qwen reports under recent transcripts and repeats the update under structured state.
| Context condition | Mean score | Completion (%) |
|---|---|---|
| Recent transcript | 89.73 | 44.7 |
| Narrative summary | 87.34 | 42.8 |
| Structured state | 92.76 | 52.1 |
| Full history | 94.51 | 57.9 |
| Target state | 98.30 | 85.3 |
Table 3. GPT-5.4 across context conditions on all 739 controlled tasks. Mean scores use a 0–100 scale; completion requires all task criteria over the full record. Target state supplies privileged target information.
Figure 6. Shared and receiver-specific improvements. Each bin counts tasks on which structured state scores higher than the recent transcript for the given number of receivers. Heights are task percentages; labels give repository and controlled task counts under the main scoring rules. Two histograms show the number of benefiting receivers, from zero to six, on 300 repository tasks and 739 controlled tasks.
| Receiver | Benchmark | + status match |
|---|---|---|
| GPT-5.4 | +7.44 [3.11, 12.59] | +6.09 [1.79, 11.19] |
| Claude | +9.07 [5.26, 13.32] | -9.88 [-18.49, 0.43] |
| DeepSeek | +12.18 [8.31, 15.60] | -7.58 [-14.50, 0.81] |
| Qwen | +10.55 [5.26, 15.41] | -6.09 [-10.98, -0.15] |
| Gemini | +12.18 [5.71, 17.96] | -3.92 [-7.92, 0.42] |
| Kimi | +9.07 [5.14, 12.76] | -5.28 [-10.28, 0.74] |
Table 4. Completion differences on 739 tasks (structured state minus recent transcript, percentage points). Benchmark uses the criteria in Section 4 ; + status match adds order-status matching in the tool-observation field. Both columns give task-family 95% intervals.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Conversational state (205) | |||
|---|---|---|---|
| Requirement | Recent | State | |
| C1 | Primary choice | 83.6 | 93.1 |
| C2 | Backup choice | 85.3 | 94.8 |
| C3 | Status label | 78.2 | 81.1 |
| C4 | Review focus | 80.1 | 88.8 |
| C5 | Audience | 79.5 | 86.8 |
Table 5. Criterion satisfaction rates (%), averaged equally across six receivers. Recent and State denote recent transcripts and structured state. Each row contributes one binary score; passing conditions are defined in Appendix B .
| Line-F1 difference [95% CI] | |||||
|---|---|---|---|---|---|
| Receiver | File-F1 | All tasks | Earlier only (217) | Still recent (42) | Not observed (41) |
| GPT-5.4 | 0.5858 | +0.154 [0.107, 0.186] | +0.182 [0.133, 0.216] | +0.108 [0.016, 0.195] | +0.049 [-0.011, 0.101] |
| Claude | 0.5911 | +0.142 [0.095, 0.172] | +0.168 [0.118, 0.200] | +0.094 [-0.008, 0.198] | +0.052 [-0.050, 0.135] |
| DeepSeek | 0.5485 | +0.156 [0.112, 0.186] | +0.178 [0.127, 0.212] | +0.147 [0.097, 0.198] | +0.048 [-0.001, 0.090] |
| Qwen | 0.5988 | +0.160 [0.102, 0.201] | +0.189 [0.127, 0.233] | +0.089 [-0.015, 0.172] | +0.081 [0.010, 0.151] |
| Gemini | 0.5724 | +0.114 [0.079, 0.145] | +0.132 [0.103, 0.166] | +0.071 [-0.070, 0.182] | +0.064 [-0.017, 0.135] |
Table 6. Repository recovery overall and by visibility of annotated paths. File-F1 uses structured state; differences are structured state minus recent transcript, with repository-level 95% bootstrap intervals. The three visibility groups partition all 300 tasks.
Figure 7. Evidence recovery for task takeover (300 tasks per receiver). Counts below panels give gains, ties, and losses in line-F1 with structured state relative to recent transcripts. Point area counts tasks sharing a score pair.
| Task | State before continuation | Remaining request | Observed continuation |
|---|---|---|---|
| Conversational state | Eight choices established, including launch readiness and no invented dates. Later updates: teal/APAC orange/EMEA; total . | Report the current brief state. | Retains all eight choices; reports orange, EMEA, and 14; omits obsolete values. 17/17 criteria. |
| Tool state | Address update already succeeded for order BX-591204. Order facts and a do-not-repeat instruction remain relevant. | Report the final order state. | Reports order facts and the completed update; marks it as non-repeatable; makes no further update call. 10/10 criteria. |
| File-work state | slugify : two tests pass, one fails. Hyphen-replacement loop rejected; separator normalization proposed. No patch or successful rerun; test file protected. | Report current progress. | Retains test observations, rejected attempt, hypothesis, and edit boundary; does not claim verified success. 8/8 criteria. |
Table 7. Three recorded continuation examples, abridged from task histories and GPT-5.4 responses under structured state. The last column summarizes the observed continuation and the number of satisfied criteria.
| Subset | Tasks | Difference [task CI] | Family CI |
|---|---|---|---|
| All | 739 | +5.42 [4.44, 6.41] | [2.46, 9.20] |
| Non-file | 591 | +5.84 [4.74, 6.94] | [2.20, 10.68] |
| Intact non-file | 429 | +3.04 [2.07, 4.03] | [1.35, 6.39] |
| Padding | 97 | +4.12 [2.02, 6.41] | [1.22, 8.89] |
Table 8. GPT-5.4 mean-score differences (State minus Summary), with task-level and task-family 95% bootstrap intervals. Intact summaries are untruncated; non-file tasks equalize source access.
More in this topic
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,217 papers