RECEIPT: Measuring and Repairing the Return Assumption of LLM Agents in Web Service Operations
Organizations: Yanshan University · Tencent · Shenzhen Intelligent Strong Technology Co., Ltd. · Aalto University
Abstract
LLM agents operate Web infrastructure from the command line and rely on a return assumption, namely that each tool call returns within a bounded time and with its requested action finished. Many commands, however, ask a question before they act, and the agent's harness either closes the program's input or keeps it open as a terminal at which nobody types. With closed input, the program can return status 0 without the requested action. With open input, it waits until a time limit, and the agent pays in time and tokens. To the best of our knowledge, this paper is the first to measure and repair the return assumption. We collect 91,469 production tool calls and 89,047 public agent trajectories, test how 8 widely used harnesses launch commands, and build Litmus, a catalog of 158 checked command-line invocations, and Tripwire, a benchmark of 25 task pairs that differ only in whether a question guards the requested action. We then propose RECEIPT, which keeps the input open, reads what each process waits for from system calls, and returns this state to an unmodified agent through a model-checked protocol. On Litmus, closed input causes 11.5% of tested invocations to return status 0 without performing the requested action, and 7 of 8 harnesses close input when no terminal is attached. An unanswered question at a plain terminal multiplies a task's token use by 2.9, and the input connection moves an LLM from 7th to 3rd place among 8. Across 5 LLMs, RECEIPT reduces false claims of success by 73.5% and completes all 25 guarded tasks on a second harness. An agent's reliability therefore depends on what its harness lets it observe as well as on its LLM. Agent builders should report what each tool call did, not only how it ended, and benchmark maintainers should fix and report the input connection before they rank LLMs.
Figures & tables
| Harness | Input under a terminal | At the limit |
|---|---|---|
| Claude Code [ 3 ] | closed | background, 120 s |
| Codex CLI [ 29 ] | closed | tool returns, 10 s |
| Gemini CLI [ 17 ] | own terminal | SIGTERM, 300 s idle |
| OpenCode [ 30 ] | closed | SIGTERM, 120 s |
| mini-SWE-agent [ 42 ] | user’s terminal | SIGKILL, 30 s |
| Goose [ 7 ] | closed, user’s tty | none |
| Kind | Programs | Share |
|---|---|---|
| Version control and pipelines | git , gh , pipeline APIs | 22.9% |
| Remote hosts | ssh , scp , rsync | 13.8% |
| HTTP endpoints | curl , wget | 6.1% |
| Services and servers | systemctl , nginx | 4.6% |
| Containers | docker , kubectl | 3.8% |
| Package registries | apt , npm , pip | 3.8% |
| Terminal | Submissions | Observed calls | Calls not returned | Trials affected |
|---|---|---|---|---|
| Multiplexer [ 27 ] | 12 | 205,500 | 6.0% | 56.1% |
| Pseudo-terminal [ 29 ] | 3 | 51,924 | 6.6% | 40.2% |
| Shell session [ 6 , 12 ] | 5 | 60,074 | 2.3% | 21.8% |
| Pipes [ 3 , 17 ] | 19 | 138,587 | 0.7% | 8.5% |
| Condition | Successes | False successes | Median run time | Million tokens |
|---|---|---|---|---|
| Closed input | 153 | 34 | 53 s | 4.4 |
| Terminal | 158 | 28 | 105 s | 10.4 |
| Receipt w/o report | 162 | 20 | 66 s | 7.8 |
| Receipt | 176 | 9 | 62 s | 7.9 |
| Condition | Control successes | Trap successes | False successes |
|---|---|---|---|
| Closed input | 45 | 33 | 6 |
| Terminal | 45 | 40 | 2 |
| Terminal + text hint | 44 | 41 | 2 |
| Published baselines | |||
| Outcome monitor [ 31 ] | 41 | 30 | 6 |
| DeepSeek inspector [ 9 ] | 40 | 35 | 3 |
| LLM | Receipt w/o report | Receipt |
|---|---|---|
| Qwen2.5-Coder-7B [ 18 ] | 21 / 1 | 23 / 1 |
| Qwen2.5-Coder-14B [ 18 ] | 31 / 4 | 31 / 2 |
| Qwen2.5-Coder-32B [ 18 ] | 39 / 0 | 39 / 0 |
| Qwen2.5-72B [ 38 ] | 35 / 8 | 40 / 4 |
| Llama-3.1-70B [ 25 ] | 35 / 7 | 41 / 1 |
| All 5 LLMs | 161 / 20 | 174 / 8 |
| First 20 pairs | Last 5 pairs | ||||
|---|---|---|---|---|---|
| Condition | Trap successes | False successes | Trap successes | False successes | Control successes |
| Native | 14 | 2 | 3 | 1 | 25 |
| Receipt w/o report | 18 | 1 | 2 | 0 | 24 |
| Receipt | 20 | 0 | 5 | 0 | 25 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Task and program | Control trap | EOF outcome | Verifier target |
|---|---|---|---|
| Cleanup, written script | Assume yes request confirmation | Skip | Build cache absent |
| Amend, git | Editor true vim | Hang | Clean tree, both files in HEAD, same message and commit count |
| Configure, written script | Positional profile --profile or menu | Proceed | Release profile selected |
| Decrypt, written script | Passphrase file /dev/tty | Fail | Expected secrets file produced |
| Deploy, written script | Positional target --target or prompt | Proceed | Staging version updated, production version preserved |
| Prune, find | -exec -ok | Skip | No temporary files, 3 CSV files remain |
| Closed | Pipe | Terminal | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Category | Hang | Fail | Skip | Unasked | Unaffected | Hang | Fail | Skip | Unasked | Unaffected | Hang | Fail | Skip | Unasked | Unaffected |
| Confirmation | 2 | 29 | 10 | 5 | 1 | 34 | 8 | 0 | 4 | 1 | 30 | 0 | 0 | 0 | 1 |
| Credential | 4 | 26 | 0 | 2 | 0 | 20 | 13 | 0 | 0 | 0 | 28 | 0 | 0 | 0 | 0 |
| Interpreter | 1 | 2 | 1 | 0 | 17 | 19 | 2 | 0 | 0 | 0 | 14 | 0 | 0 | 0 | 0 |
| Editor | 5 | 9 | 2 | 1 | 0 | 7 | 8 | 1 | 1 | 0 | 15 | 0 | 0 | 0 | 0 |
| Pager | 0 | 0 | 0 | 0 | 13 | 1 | 0 | 0 | 0 | 12 | 10 | 0 | 0 | 0 | 1 |
| Launch | True outcome | Agent’s answer |
|---|---|---|
| foreground | killed at 10-14 s of 100 s | “still running” |
| Start-Job | did not start | “unknown” |
| Start-Process | outlives the session, completes | “unknown” |
| cmd start /b | outlives the session, completes | “still running” |
| bash nohup & | outlives the session, completes | “still running” |
| Launcher | Launch | Outcome | Notice |
|---|---|---|---|
| main | harness background | completes | yes |
| main | foreground past limit | background, completes | yes |
| main | shell background | completes | no |
| sub-agent | harness background | killed at 2-6 s of 150 s | no |
| sub-agent | nohup or & | completes | no |
| LLM | Closed | Terminal | w/o report | Receipt |
|---|---|---|---|---|
| Qwen2.5-Coder-7B [ 18 ] | 26 / 5 / 43 | 25 / 7 / 78 | 21 / 1 / 112 | 26 / 0 / 55 |
| Qwen2.5-Coder-14B [ 18 ] | 29 / 7 / 62 | 31 / 4 / 211 | 31 / 4 / 80 | 32 / 2 / 68 |
| Qwen2.5-Coder-32B [ 18 ] | 33 / 6 / 109 | 40 / 2 / 111 | 40 / 0 / 57 | 40 / 1 / 51 |
| Qwen2.5-72B [ 38 ] | 35 / 10 / 47 | 33 / 6 / 102 | 35 / 8 / 63 | 38 / 6 / 66 |
| Llama-3.1-70B [ 25 ] | 30 / 6 / 52 | 29 / 9 / 99 | 35 / 7 / 66 | 40 / 0 / 62 |
| All 5 LLMs | 153 / 34 / 53 | 158 / 28 / 105 | 162 / 20 / 66 | 176 / 9 / 62 |
| LLM | Closed | Terminal | w/o report | Receipt |
|---|---|---|---|---|
| Qwen2.5-72B [ 38 ] | 1 | 2 | 3 | 3 |
| Mistral-Small-24B [ 28 ] | 3 | 5 | 2 | 4 |
| Qwen2.5-Coder-32B [ 18 ] | 2 | 1 | 1 | 1 |
| Llama-3.1-70B [ 25 ] | 4 | 5 | 3 | 1 |
| Qwen2.5-Coder-14B [ 18 ] | 5 | 4 | 6 | 5 |
| Qwen2.5-Coder-7B [ 18 ] | 6 | 7 | 8 | 7 |
| Outcome with input closed | Pipe | Terminal | |
| Skip | 14 | 13 | 14 |
| Unasked | 10 | 4 | 5 |
| Hang | 7 | 7 | 7 |
| Fail | 68 | 51 | 63 |
| Unaffected | 20 | 13 | 12 |
| No outcome with input closed | 1 | 1 | 1 |
| Forbidden order | Rule | Test |
|---|---|---|
| Stale input after an interruption reaches the shell | Sender and observer share a single lock, and the sender defers input published after the agent’s last observation | input_guard , 3 cases |
| A scheduled input is interrupted before delivery | The observer pauses while the batch still schedules input | queued_input , 2 cases |
| A receipt reports a wait of a program that exited by itself | The observer rechecks the reader’s system call under the lock before it signals | exit_race , 8 cases |
| A scheduled keystroke answers a prompt the agent has not seen | The sender holds the keystroke, records it with the prompt, and the program keeps waiting | hold_on_prompt , 4 cases |
| A receipt published after the observation defers the agent’s next batch | The observation of a batch waits for a pending candidate wait, at most 1.2 s | await_receipt , 4 cases |
| A keystroke whose form does not fit the reader’s class is delivered | The sender holds it once with the reason and delivers it when sent again | state_receipt , 6 cases |
| Compared with | Fewer | More | Tied | Change in pp |
|---|---|---|---|---|
| Closed input | 8 | 1 | 6 | [ , ] |
| Terminal | 8 | 1 | 6 | [ , ] |
| Receipt w/o report | 7 | 2 | 6 | [ , ] |
| Pipe | Terminal | |||
|---|---|---|---|---|
| Predictor | AUROC | AP | AUROC | AP |
| Rule set | 0.603 | 0.170 | 0.572 | 0.199 |
| Logistic regression | 0.737 | 0.126 | 0.704 | 0.222 |
| Tuned LLM | 0.761 | 0.194 | 0.666 | 0.240 |
| Zero-shot 32B | 0.900 | 0.477 | 0.717 | 0.395 |
Tool-Using Language Agents
Tool-using language agents combine language models with APIs, code, and other tools to answer questions and carry out multi-step tasks.
See all 64 papers