Organizations: East China Normal University, Shanghai, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Institute of Deep Perception Technology, JITRI, Wuxi, China
People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the faithfulness of reports, and it leaves open whether the requester can understand and use the reply. We call this property reply usability and measure it with a reader model that takes the requester's position and tests whether the reply can be interpreted from what the requester has seen, is in the requester's language and answers the request. Its agreement with the majority of five annotators lies within the range of the individual annotators. Across eight models from seven families, 54% of 480 replies fail, and the best model still fails on 30%. Failed replies mostly point to material the requester has not seen or use labels that the agent coined during its work. Stating explicit requirements for the reply reduces the failure rate from 0.52 to 0.43 together with errors and omissions, and a secondary model that rewrites the reply reduces it to 0.36, which human annotators confirm in a blind comparison. Reply usability can thus be scored alongside task success and improved with one appended message or one call to a secondary model.
Figures & tables
Figure 1: A reply of Claude Opus 5 on long-document question answering and its reading by the reader model. The requester does not see the agent’s work of 41 tool calls. The reader model finds no answer to its questions about the three marked names.
Line of work
Judged against
Property
Agent benchmarks
environment state
task success
Self-reports of agents
the trajectory
faithfulness, completeness
Hand-offs between agents
the next agent’s task
downstream success
Decontextualization
the empty context
interpretability of a sentence
This work
what the requester has seen
usability of the reply
Table 1: What related lines of work evaluate.
Figure 2: (a) Cohen’s κ with the majority label of the gold set, which for an annotator is the majority of the other four. (b) Replies failed by the majority, by the number of gaps the reader finds.
Project
Type
Language
Detecting mislabeled examples in crowd-labeled datasets
research
Chinese
Long-document question answering with a small model and retrieval
research
English
Price change for a subscription product
business
Chinese
Redesign of the onboarding flow of a software product
business
English
Table 2: The four projects of the collaborations.
Figure 3: (a) Failure rate of each model with 95% intervals, divided into replies with an information gap and other failures. (b) Failure rate by tool-calling turns before the reply. (c) First occurrence of the coined labels in the agent’s context, where red marks the agent’s files, project memory and tool-call arguments.
Kind of failure
What the requester lacks
Example span and the reader’s question
Failed replies (%)
Pointer to unseen material
a file section, an index entry or the output of a run that the requester has not seen
R03 . Which paper or reference does R03 refer to?
46.1
Coined label
the meaning of a name, number or short form that the agent introduced during the work
crn_hierbm25_L . What does “crn_hierbm25_L” refer to?
42.2
Unexplained term
the meaning of an abbreviation or term that is established in the field
CUPED adjustment. What does CUPED stand for in the context of variance reduction in A/B testing?
23.6
Missing background
a premise that the reply depends on and does not state
confirm whether the difference between 80.2 and 76.0 is stable. At what sample size […] was the comparison of 80.2 and 76.0 run?
1.6
Request not answered
the result or answer that the latest message asked for
28.3
Wrong language
a reply in the requester’s language
6.6
Table 3: Kinds of failure among the 258 failed replies, where a reply can have several kinds. The upper four are the categories of information gaps, shown with a span of a reply and the reader’s unanswered question.
Condition
Written in
Added instruction
Failure rate
Reduction
Errors + omissions
Default
working context
none
0.52
1.41
Notice
working context
notice
0.50
0.027 [ − 0.052, 0.112]
1.34
Requirements
working context
requirements
0.43
0.098 [0.030, 0.168]
1.07
Separate pass
new conversation
none
0.47
0.051 [ − 0.004, 0.109]
1.37
Fact list
new conversation
none
0.49
0.031 [ − 0.020, 0.081]
1.45
Separate pass with requirements
new conversation
requirements
0.42
0.102 [0.023, 0.181]
1.16
Table 4: The six writing conditions on 256 reply points, with 95% intervals for the reduction from the default.
Figure 4: (a) Failure rate by default, under the requirements and after the rewrite. (b) Default replies and rewrites at 83 reply points, judged by annotators and by the reader. (c) Share of reply points with a usable reply among k samples.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Reader’s gaps
Replies
Failed by the majority
Annotators who fail
0
108
26%
1.4
1
75
71%
3.5
2 or more
117
94%
4.6
Annotators who fail
Replies
Failed by the reader
0
57
19%
1
28
39%
Appendix
Table 5: Human judgments by the number of gaps the reader finds, and the reader’s failure rate by the number of annotators who fail the reply.
Rater
κ
Accuracy
Annotator 1
0.68
–
Annotator 2
0.57
–
Annotator 3
0.64
–
Annotator 4
0.59
–
Annotator 5
0.63
–
Reader
0.60
82%
Appendix
Table 6: Agreement with the majority label of the gold set (Cohen’s κ ). For each annotator the majority is taken over the other four.
Model
Tool-calling turns
Reply length
Compactions
Claude Opus 5
5
2,645
6.25
DeepSeek V4.1 Flash
4
2,136
3.75
GLM-5.3
2
1,710
0.25
DeepSeek V4 Pro
1
2,169
0.25
Kimi K2.6
1
2,394
0.25
GPT-5.6 Sol
1
2,552
0
Appendix
Table 7: Median number of tool-calling turns before a reply, median reply length in characters, and mean number of context compactions per collaboration.
Tool-calling turns before the reply
Model
Weights
Failure rate
Information gap
0
1–2
3 or more
Claude Opus 5
closed
0.82 [0.75, 0.90]
0.78
0.67
0.56
0.89
DeepSeek V4.1 Flash
open
0.65 [0.53, 0.77]
0.63
0.71
0.50
0.71
MiniMax M3
open
0.53 [0.37, 0.70]
0.37
0.43
0.58
0.62
GLM-5.3
open
0.52 [0.43, 0.60]
0.42
0.43
0.42
0.68
DeepSeek V4 Pro
open
0.50 [0.42, 0.62]
0.43
0.56
0.28
0.76
Appendix
Table 8: Failure rate of the 60 replies of each model with 95% intervals, share of replies with at least one information gap, and failure rate by the number of tool-calling turns before the reply. The three groups of turns contain 120, 174 and 186 replies.
At the reply points
Project
Type
Failure rate
Default
Requirements
Rewrite
Mislabeled examples
research
0.66
0.73
0.53
0.38
Long-document QA
research
0.50
0.41
0.34
0.26
Subscription pricing
business
0.41
0.44
0.35
0.39
Onboarding redesign
business
0.58
0.50
0.47
0.38
Appendix
Table 9: Failure rate by project over the 120 replies of each project, and at the reply points by default, under the requirements and after the rewrite.
Model
Default
Notice
Requirements
Separate pass
Fact list
Separate pass with requirements
Claude Opus 5
0.72
0.84
0.53
0.72
0.75
0.50
DeepSeek V4.1 Flash
0.66
0.44
0.53
0.69
0.59
0.50
DeepSeek V4 Pro
0.47
0.44
0.34
0.28
0.38
0.34
GLM-5.3
0.59
0.59
0.47
0.50
0.66
0.41
GPT-5.6 Sol
0.25
0.16
0.22
0.25
0.25
0.25
Kimi K2.6
0.56
0.41
0.47
0.50
0.44
0.44
Appendix
Table 10: Failure rate by model under the six writing conditions, on 32 reply points per model.
Family
n
Default
Rewrite
Claude
32
0.72
0.47
DeepSeek
64
0.56
0.38
GLM
32
0.59
0.59
GPT
32
0.25
0.13
Kimi
32
0.56
0.38
MiniMax
32
0.53
0.28
Appendix
Table 11: Failure rate of the default replies and their rewrites by family.
Family
Failed replies
Pointer to unseen material
Coined label
Unexplained term
Missing background
Request not answered
Wrong language
Claude
49
61.2
65.3
12.2
0.0
18.4
8.2
DeepSeek
69
44.9
49.3
33.3
0.0
27.5
0.0
GLM
31
48.4
45.2
16.1
0.0
35.5
0.0
GPT
18
16.7
0.0
44.4
11.1
38.9
0.0
Kimi
29
51.7
37.9
24.1
3.4
20.7
6.9
MiniMax
32
34.4
31.2
12.5
0.0
43.8
15.6
Appendix
Table 12: Share of failed replies (%) with each kind of failure, by family.