Action Verification through Conjecture and Refutation for Language Model Agents
Organizations: Pohang University of Science and Technology · National AI Research Lab
Abstract
Language model agents increasingly act on users' behalf by manipulating external environments through tools, often from high-level instructions that leave critical task- and environment-specific context unstated. To prevent resulting misaligned actions, recent work verifies proposed actions before execution, largely by establishing whether the conditions supporting them hold. We argue that this confirmation-oriented view leaves a fundamental blind spot: a wrong action often passes every check that asks whether it is right. This motivates a different question for verification: rather than asking what evidence supports an action, can the agent find evidence that refutes it? We pursue this direction with BlackSwan, a refutation-oriented, expect-suspect-test procedure that predicts the intended effect of each state-mutating action, formulates specific and empirically testable hypotheses for how it could be wrong, and tests them using existing observations or additional non-mutating actions, executing the action only if none survives testing. On challenging AppWorld tasks, BlackSwan improves success rate over the strongest baseline by 14.9 and 25.2%p with and without the backbone model's thinking mode, respectively. Even without thinking or in-context demonstrations, it outperforms stronger baselines equipped with them. The gains also transfer beyond AppWorld, with BlackSwan improving WebShop performance by 12.7%p. More broadly, our results suggest that language model agents have an underused capacity to recognize and self-correct misaligned actions, which can be systematically elicited through active conjecture and refutation before execution.
Figures & tables
| Suspicions | |||
| Orientation | Explicit | Implicit | |
| Refutation | 81.6 | 72.7 | |
| Confirmation | 76.0 | 72.6 | |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Limit |
|---|---|
| Word–endpoint table | once per app, at most 8 entries |
| Expect | one expectation, at most 500 characters |
| Suspect | at most 5 suspicions, each at most 1,000 characters |
| Test | at most 50 read-only probes per verification; grounds at most 400 characters |
| Redrafts of a blocked cell | at most 3 per step |
| Decoding | temperature 0, at most 8,192 generated tokens per call |
| Kind | The falsifier asserts that | Derived from |
|---|---|---|
| Wrong object | the records acted on are not the ones the task names | the action |
| Incomplete set | the task names more records than the action covers | the action |
| Wrong value | an argument or endpoint is not what the task means | the action |
| Hidden state | existing state changes what the call does | the action |
| The task’s words | the part of the task this action carries out asks for something the action does not do | the task and the word–endpoint table |
| Left open by the paper | Our choice |
|---|---|
| Which calls are sensitive | Every endpoint whose HTTP method is not GET (POST, PATCH, DELETE), plus supervisor.complete_task. The set is computed from the code before execution, by a syntax-tree walk over apis.<app>.<endpoint>( calls. |
| Grounding-agent model | The same model and decoding as the agent. |
| Grounding-agent memory | One chat per task, kept across verifications. Each verification after the first shows only the steps taken since the previous one. |
| Where observations run | In the agent’s own namespace, so the grounding agent can read the agent’s variables. Write endpoints raise instead of executing, and the observation code runs inside a function so that it cannot rebind the agent’s variables. |
| Observation budget | Five observation rounds per verification. A reply with neither code nor a decision is answered with a one-line reminder of the output format and costs a round. When the budget is exhausted the decision is forced to APPROVE. |
| Rejections per task | Three. After the third, a REJECT is ignored and the code executes. |
| test-normal | test-challenge | |||||
|---|---|---|---|---|---|---|
| Thinking | Source | Demos | TGC | SGC | TGC | SGC |
| on | reported ( Jiang et al., 2026 ) | – | 81.5 | 67.9 | 69.8 | 53.2 |
| on | reproduced (3 runs) | 3 | 77.2 1.4 | 62.5 4.7 | 71.9 1.0 | 59.0 1.9 |
| on | reproduced (3 runs) | 1 | 79.4 1.8 | 65.5 5.7 | 63.5 1.7 | 48.4 1.5 |
| off | reproduced (3 runs) | 3 | 76.4 3.9 | 62.5 3.1 | 69.3 1.8 | 52.3 1.5 |
| off | reproduced (3 runs) | 1 | 76.2 1.2 | 59.5 2.7 | 55.0 0.4 | 44.8 2.3 |
| Test (normal) | Test (challenge) | ||||||
| Method | TGC | Tokens | Cost | TGC | Tokens | Cost | |
| Qwen3.5-27B | |||||||
| ✓ | ReAct | 72.4 1.9 | 2.7 (1.0 ) | 0.032 (1.0 ) | 58.4 0.8 | 3.6 (1.0 ) | 0.047 (1.0 ) |
| Accord | 79.4 1.8 | 12.9 (4.7 ) | 0.092 (2.9 ) | 63.5 1.7 | 10.5 (2.9 ) | 0.130 (2.7 ) | |
| BlackSwan | 88.3 3.0 | 17.7 (6.5 ) | 0.235 (7.4 ) | 84.7 1.3 | 22.9 (6.4 ) | 0.447 (9.4 ) | |
| ✗ | ReAct | 70.0 2.5 | 2.4 (0.9 ) | 0.028 (0.9 ) | 48.0 0.8 | 2.2 (0.6 ) | 0.044 (0.9 ) |
| Accord (approves with one alarm still on) | BlackSwan (refutes and revises) |
|---|---|
| Task: I have alarms for all my work meetings. Some of my coworkers have canceled their meeting with me via email. Please disable the corresponding alarms from my phone. (GT: set enabled=False on alarms 744, 745, and 743. Three coworkers canceled; Stephen’s email is titled “Skip this time?” and reads “Cannot make it.”) | |
| Shared prefix: both agents search Gmail for "cancel meeting", find the two “Cancel Meeting?” threads (Stacy alarm 744, Kristin alarm 745), and propose phone.delete_alarm(alarm_id=744). | |
| Round 1 Grounding agent: reads the update_alarm docs, then: DECISION: REJECT. OBSERVE: The task instruction says to “disable” the alarms, not delete them. The update_alarm API has an enabled parameter that can be set to false to disable an alarm without deleting it. […] | Round 1 Suspect: (1) The task asks to ‘disable’ the alarms, but the code calls delete_alarm which permanently removes them. (2) The code only deletes the alarm for Stacy (ID 744). […] alarm labeled ‘1-1 with Kristin’ (ID 745) also needs to be disabled, but this code cell does not address it. Test: reads the update_alarm docs both confirmed . |
| Both revise to update_alarm(alarm_id=744, enabled=False) and update_alarm(alarm_id=745, enabled=False). | |
| Round 2 — the two updates run; the agent proposes supervisor.complete_task(). Grounding agent: “Let me verify that no other cancellation emails were missed” probes show_inbox_threads(query="cancel") Thread 47823: “Cancel Meeting?” from Stacy Maldonado — handled. Thread 47825: “Cancel Meeting?” from Kristin White — handled. Thread 47831: “Cancel Reunion” from Leslie Ball — this is about a reunion, not a work meeting. Other threads are Amazon notifications, dinner party RSVPs, […] none are meeting cancellations. DECISION: APPROVE. OBSERVE: […] Confirmed that only two work meeting cancellations were found (Stacy Maldonado and Kristin White), both of which have been handled […] The task is complete. (alarm 743 “1-1 with Stephen” stays enabled) | Round 2 — the agent proposes the two update_alarm calls. Expect: Disables alarms 744 (‘1-1 with Stacy’) and 745 (‘1-1 with Kristin’) by setting enabled=False, matching the two meeting cancellations found in emails […] Suspect: […] The real risk is if there are other cancellation emails not captured by the query ‘cancel meeting’ (e.g., ‘reschedule’, ‘postpone’, ‘cannot make it’) that correspond to alarms. If other coworkers canceled via different phrasing, their alarms would remain enabled […] ref: phone.update_alarm.alarm_id. [Two further suspicions, on the label–sender mapping and on the meaning of ‘disable’, are dismissed in the test and omitted here.] Test: probes show_inbox_threads(query="reschedule OR postpone OR cannot make OR conflict") thread 47821 “Skip this time?”; then show_thread(47821) sender Stephen Mccoy, “Cannot make it.”, and alarm 743 “1-1 with Stephen” is enabled. Verdict: confirmed — Email thread 47821 from Stephen Mccoy says ‘Cannot make it’ (cancellation), and alarm 743 ‘1-1 with Stephen’ exists but was not included in the proposed code. Reframe: disables 744, 745, and 743. (all three alarms disabled) |
| Accord (sees the wrench, approves “no”) | BlackSwan (asks what would make “no” wrong) |
|---|---|
| Task: Have I returned any crescent wrench on amazon in this or last month? Say yes or no. (GT: yes — “Craftsman 8-Inch Adjustable Wrench”, product_type “crescent wrench”, returned on 2023-04-28.) | |
| Shared prefix: both agents fetch all 33 returns, keep those from April/May 2023 whose product_name contains “crescent wrench” (none), and propose supervisor.complete_task(answer="no"). | |
| Grounding agent: “Let me observe to verify the agent’s logic by checking if there are any returns with ‘crescent’ or ‘wrench’ in the product name that might have been missed.” probes all_returns Output: Total wrench returns in April/May 2023: 1 — Craftsman 8-Inch Adjustable Wrench (initiated: 2023-04-28). Total crescent returns in April/May 2023: 0. DECISION: APPROVE. OBSERVE: Verified by re-analyzing all 33 returns that (1) the date range April/May 2023 is correct […], (2) there is 1 wrench return (“Craftsman 8-Inch Adjustable Wrench”) but 0 crescent wrench returns in the target months, (3) the agent’s case-insensitive search for “crescent wrench” correctly found no matches. The answer “no” is accurate. (answer: no; GT: yes) | Expect: The code will mark the current task as complete with the answer ‘no’, indicating that the user has not returned any crescent wrench on Amazon in the current or previous month, based on the analysis of all returned items. Suspect: The code searched for the exact phrase ‘crescent wrench’ in the product name. However, the returned item ‘Craftsman 8-Inch Adjustable Wrench’ is a type of crescent wrench. […] The code did not account for synonyms or broader category matching. ref: task. Test: probes amazon.search_products(query="crescent wrench") Output: product_id 2139, product_type “crescent wrench”, name “Craftsman 8-Inch Adjustable Wrench” […] Verdict: confirmed — Search results show product_id 2139 has product_type ‘crescent wrench’ but name ‘Craftsman 8-Inch Adjustable Wrench’. The return list contains this product_id. Reframe: re-checks the returns by product_id found, 2023-04-28 complete_task(answer="yes"). (answer: yes) |
LLM Agent Verification
LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.
See all 166 papers