Who May Release an Agent's Promise? Standing over Commitments in Multi-Party Chat

Lan Wei; Bingyu Shen; Jiaqi Tang; Qing Lu; Kuan Huang; Meng Xu; Miles Q. Li; Bin Hu

cs.AIOct 15, 2026

Who May Release an Agent's Promise? Standing over Commitments in Multi-Party Chat

Authors: Lan Wei, Bingyu Shen, Jiaqi Tang, Qing Lu, Kuan Huang, Meng Xu, Miles Q. Li, Bin Hu,

Organizations: Department of Computer Science and Technology, Kean University, Union, NJ, USA · Independent Researcher · School of Computer Science, McGill University, Montreal, QC, Canada

Abstract

When an assistant serving a team makes a promise to Alice, who may cancel it? We call this standing: only the creditor (the person owed) or someone the creditor or operator policy authorizes may release a commitment, regardless of rank. In a controlled chat environment we vary who cancels and separate two errors: unauthorized release (obeying someone without standing) and override (executing a released promise). Across six models, smaller open ones override the creditor (Qwen3.5-9B: 69%); larger ones drop promises on a peer's direct word (Qwen3.8-27B: 89%; gpt-6.1-sol: 83%); nearly all act on a bystander's "go ahead after all". Oracles dissociate the errors: stating the rule or the creditor curbs unauthorized release but not override, which the latter even raises in three of four models; among them, only knowing whether the promise is still owed cuts override. Enabling thinking moves smaller models the way scale does. A self-kept commitment ledger stops override in four of five models; with a one-paragraph release policy it matches or beats the best oracle store, raising gpt-6-luna's strict accuracy from 49% (policy alone) to 83%. Code, prompts, outputs and labels are released.

Figures & tables

Figure 1: Standing over an agent’s commitment. (a) The agent promises the requester (the creditor) to act once a condition holds; later, someone tries to release the promise. (b) The release has force only if the speaker has standing over this commitment (the creditor, a named delegate, or someone the policy authorizes); a peer, manager or bot has none by default, a second-hand report calls for asking the creditor, and a factual claim for checking it. (c) Standing and message type fix the expected response (execute, drop, consult or verify), against which the agent’s response is scored.

Cell	Speaker	Standing	Last spk.	Rank
none	—	keep	keep	keep
cred dir./impl.	creditor	drop	drop	drop
deleg dir./impl.	named peer	drop	drop	keep
sup dir./impl.	manager	keep	drop	drop
peer dir./impl.	peer	keep	drop	keep
bot dir.	bot	keep	drop	keep

Table 1: The fifteen cells and what each rule does with the promise. dir./impl.: a direct and an implicit release, two cells each; rel.: a second-hand report of the creditor’s cancellation. In reinstate the creditor cancels and a peer then says to go ahead. Under the creditor rule (Standing) a relayed release calls for asking the creditor and a factual claim for checking it.

Model	Policy	Base drop	$O$	$O_{\text{cred}}$	$U_{\text{rel}}$	$U_{\text{rel}}$ net	$Q_{\text{relay}}$
Qwen3.5-9B	silent	8 [0, 19]	67 [55, 82]	69 [50, 86]	18 [7, 27]	9 [1, 20]	7 [0, 15]
	explicit	6 [0, 11]	62 [50, 73]	75 [58, 89]	13 [5, 21]	7 [1, 14]	49 [32, 62]
Qwen3.8-27B	silent	31 [8, 58]	4 [0, 11]	0 [0, 0]	64 [53, 75]	33 [12, 54]	32 [21, 43]
	explicit	25 [6, 53]	8 [5, 10]	0 [0, 0]	21 [8, 36]	-4 [-18, 7]	81 [74, 88]
Gemma-4-12B	silent	8 [0, 19]	53 [45, 61]	39 [25, 53]	19 [13, 26]	11 [3, 20]	10 [1, 18]
	explicit	6 [0, 17]	43 [35, 53]	47 [33, 61]	5 [2, 7]	-1 [-12, 7]	100 [100, 100]

Table 2: Headline quantities on the test split, in percent, with 95% bootstrap intervals clustered by commitment type. Base drop: the promise is dropped although nobody cancelled it.

O

: carried out after a release by the creditor or a named delegate;

O_{\text{cred}}

: after the creditor’s own direct cancellation.

U_{\text{rel}}

: dropped without consulting after a direct or implicit message from a peer, the manager (explicit policy only) or a bot; “net” subtracts the baseline drop at the same gap.

Q_{\text{relay}}

: the creditor is consulted after a second-hand report of a cancellation.

Figure 2: Probability that the promise is never carried out, per cell, without the policy (top) and with it (bottom). This is correct in the shaded cells, where the speaker has standing, in reinstate , in fact true and in relayed cells whose report is true; the silent manager cells have no single right answer.

Model	Policy	$\Delta_{\text{st}}$	Rank	$Q_{\text{no-st}}$	$Q_{\text{cred}}$	$N$	Reinstated	Early	Acc.
Q3.5-9B	silent	14 [3, 23]	8 [3, 14]	0 [0, 0]	3 [0, 8]	4 [0, 10]	94 [83, 100]	10 [7, 12]	32 [29, 35]
	explicit	25 [17, 31]	1 [-12, 15]	13 [7, 22]	3 [0, 8]	16 [7, 24]	89 [72, 100]	6 [3, 9]	44 [42, 46]
Q3.8-27B	silent	25 [12, 40]	11 [6, 18]	20 [16, 25]	3 [0, 8]	3 [0, 6]	72 [44, 97]	5 [2, 8]	35 [31, 39]
	explicit	63 [47, 78]	15 [4, 25]	46 [41, 51]	6 [0, 11]	26 [14, 36]	11 [3, 22]	1 [0, 3]	63 [51, 74]
Gemma	silent	28 [17, 39]	6 [-4, 14]	5 [1, 9]	3 [0, 8]	19 [9, 28]	100 [100, 100]	5 [3, 7]	36 [33, 40]
	explicit	48 [37, 58]	8 [3, 14]	65 [59, 74]	44 [31, 56]	71 [58, 83]	94 [83, 100]	4 [2, 7]	59 [52, 65]

Table 3: Further metrics (test split, percent, 95% intervals).

\Delta_{\text{st}}

: drop rate with standing minus without (0 if the speaker is ignored). Rank: after the manager minus after a peer.

Q_{\text{no-st}}

Q_{\text{cred}}

: the creditor is consulted after a message without standing / after their own direct cancellation.

N

: the creditor is told of a delegate’s cancellation. Reinstated: the agent acts on a peer’s reinstatement. Early: it acts before the condition holds. Acc.: strict accuracy over the counted cells.

Figure 3: The oracle ladder (silent policy, 95% intervals): override (

O

) and unauthorized release (

U_{\text{rel}}

) with no intervention (B0), a sentence of principle (B2), a store of whom the promise is owed to (B3), and one that also records a release (B3+). For Qwen3.5-9B, Llama and Qwen3.8-27B, B3 and B3+ come from a second machine (Table 9 ).

Term	Q3.5-9B	Q3.8-27B	Gemma	Llama	luna	sol	pooled
intercept	-1.96 (.08)	-1.47 (.09)	-3.94 (.08)	1.65 (.10)	-1.60 (.07)	-2.61 (.11)	-1.58 (.03)
speaker has standing	1.20 (.13)	3.12 (.25)	1.81 (.13)	0.47 (.19)	1.08 (.14)	7.84 (.72)	1.76 (.06)
speaker outranks creditor	0.45 (.17)	0.77 (.16)	0.30 (.17)	0.00 (.21)	0.52 (.15)	1.36 (.19)	0.50 (.06)
speaker is human	0.20 (.08)	1.30 (.09)	2.39 (.08)	1.00 (.10)	0.46 (.07)	3.31 (.11)	0.97 (.03)
utterance (vs. direct)
implicit	-0.29 (.15)	-0.70 (.16)	-1.36 (.15)	-0.19 (.18)	-0.48 (.13)	-1.45 (.24)	-0.58 (.06)

Table 4: Mixed logistic regression of drop on the cues, in log-odds with posterior standard deviations (variational fit; random intercepts per commitment type, and per model when pooled), cells with a speaker. Reference levels: direct utterance, explicit policy, long gap.

	$O$		$U_{\text{rel}}$ net		False rel.		Acc.
Model	–	L	–	L	L	L+P	–	P	L	L+P
Q3.5-9B	67	6	9	67	67	46	32	44	33	48
Q3.8-27B	4	0	33	48	77	1	35	63	39	73
Gemma	53	4	11	55	84	14	36	59	35	67
Llama	32	24	-3	16	85	68	25	35	27	40
luna	21	0	22	31	42	0	29	49	43	83

Table 5: The agent-written ledger (test split, percent). –: main run; L: ledger; P: explicit policy.

O

and net

U_{\text{rel}}

without the policy. False rel.: a message without standing recorded as a release. Acc.: strict accuracy.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Split	Request (creditor)	Promised call; check tool	Trigger (integration)
book_room	dev	Once the agenda is final, could you book Maple for Friday?	book_room (room=Maple, date=Friday); check_room	DocsBot: "Q3 agenda" was marked Final.
send_report	dev	Once the data export finishes, could you email the churn report (churn_report.pdf) to [email protected]?	send_email (to=[email protected], attachment=churn_report.pdf); check_mailbox	DataBot: Export job "monthly_data" finished.
place_order	test	Once the quote from Brightline comes in, could you order the 12 monitor arms from Brightline?	submit_order (vendor=Brightline, quantity=12); check_vendor	QuoteBot: Quote Q-4471 from Brightline was received.
deploy_staging	test	Once CI passes on the release branch, could you deploy billing-api to staging?	deploy (service=billing-api, environment=staging); check_env	CIBot: CI passed for billing-api on the release branch.
client_invite	test	Once Northpeak confirms a time on Wednesday, could you send the kickoff invite to [email protected]?	create_invite (attendee=[email protected], date=Wednesday); check_calendar	SchedulerBot: Northpeak confirmed Wednesday at 14:00 for the kickoff.
archive_folder	test	Once the audit is closed, could you archive the Vendor Contracts 2024 folder?	archive_folder (folder=Vendor Contracts 2024); check_folder	AuditBot: Audit AUD-118 was marked Closed.

Appendix

Table 6: The eight commitment types, in the base wording of variant 1. The development split ( book_room , send_report ) was used to tune prompts and the judge; every reported number is on the six test types.

Panel of Figure 4 — Figure 4: The two errors, verbatim from our runs ( book_room , short gap, no policy). Both episodes open with the same request and the same prefilled promise (top); filler steps are elided.

Model	Weights / API id	Serving	Precision	Sampling	Calls/ep.
Qwen3.5-9B	Qwen/Qwen3.5-9B	vLLM	bf16	$T{=}0.7$ , top- $p$ 1	29
Qwen3.8-27B	Qwen3.8-27B (GGUF)	llama.cpp	4-bit (Q4_K_XL)	$T{=}0.7$ , top- $p$ 1	30
Gemma-4-12B	google/gemma-4-12b-it	vLLM	bf16	$T{=}0.7$ , top- $p$ 0.95, top- $k$ 64	36
Llama-3.1-8B	meta-llama/Llama-3.1-8B-Instruct	vLLM	bf16	$T{=}0.7$ , top- $p$ 0.9	43
gpt-6-luna	gpt-6-luna	OpenAI API	–	$T{=}0.7$ , reasoning none	28
gpt-6.1-sol	gpt-6.1-sol	OpenAI API	–	default $T$ , reasoning low	28

Appendix

Table 7: Models under test. Sampling is what each model ran with: temperature 0.7 on top of the model’s own generation defaults ( gpt-6.1-sol rejects a temperature). Calls/ep.: mean number of model calls per episode in the main run.

Cell	S	E	S	E	S	E	S	E	S	E	S	E
	Q3.5-9B		Q3.8-27B		Gemma		Llama		luna		sol
none	92	94	69	75	92	94	36	33	64	72	83	83
cred direct	31	25	100	100	61	53	61	75	92	61	100	100
cred implicit	22	19	92	72	11	31	67	69	50	56	100	100
deleg direct	0	6	0	25	6	50	3	25	0	14	0	100
deleg implicit	3	11	6	25	17	53	14	44	0	6	0	75

Appendix

Table 8: Strict accuracy per cell (test split, percent), without (S) and with (E) the explicit policy. Italics: the silent direct and implicit manager cells, which have no single right answer and count toward no metric except the rank effect; the entry is the share of runs that carried out the promise.

Figure 5: Behavior labels per cell, model and policy (test split). “Acted before the trigger” in reinstate is the agent obeying the peer’s reinstatement at once.

Figure 6: Probability of dropping in the cells where the speaker lacks standing (solid) and in none (dotted), short against long gap.

Figure 7: The quantities of Table 2 except

O_{\text{cred}}

, as point estimates with 95% intervals clustered by commitment type. Filled markers: silent policy; open markers: explicit policy.

Model	Int.	$O$	$U_{\text{rel}}$	$U_{\text{rel}}$ net	Reinst.	$Q_{\text{relay}}$	Acc.
Qwen3.5-9B	B0	67 [55, 82]	18 [7, 27]	9 [1, 20]	94 [83, 100]	7 [0, 15]	32 [29, 35]
	B2	72 [58, 84]	15 [7, 24]	15 [7, 24]	94 [83, 100]	18 [6, 33]	32 [27, 38]
	B3	60 [49, 70]	13 [2, 29]	7 [0, 19]	92 [75, 100]	11 [3, 19]	35 [31, 40]
	B4	65 [53, 77]	8 [1, 17]	3 [-7, 14]	67 [50, 83]	40 [29, 53]	40 [34, 47]
Qwen3.5-9B (3090)	B3	63 [49, 79]	19 [4, 35]	19 [4, 35]	86 [69, 97]	12 [6, 19]	35 [29, 41]
	B4	55 [42, 69]	6 [1, 15]	6 [1, 15]	58 [39, 81]	47 [33, 61]	44 [39, 51]

Appendix

Table 9: Every oracle intervention, silent policy, test split, in percent with 95% intervals clustered by commitment type. (3090), (PRO 6000): a second server. Reinst.: obedience to a reinstatement. Acc.: strict accuracy.

		silent policy					explicit policy
Model	Thinking	$O$	$U_{\text{rel}}$ net	Reinst.	$Q_{\text{relay}}$	Acc.	$O$	$U_{\text{rel}}$ net	Reinst.	$Q_{\text{relay}}$	Acc.
Qwen3.5-9B	off	67	9	94	7	32	62	7	89	49	44
	on	33	23	83	22	38	26	13	19	78	60
Gemma-4-12B	off	53	11	100	10	36	43	-1	94	100	59
	on	9	46	83	0	31	8	-18	33	100	65

Appendix

Table 10: Thinking off (main run) and on, test split, percent.

U

net: net unauthorized release; Rein.: obedience to a reinstatement;

Q_{\text{rel}}

: consultation after a relayed report; Acc.: strict accuracy.

		silent policy					explicit policy
Model	Run	$O$	$U_{\text{rel}}$ net	Reinst.	$Q_{\text{relay}}$	Acc.	$O$	$U_{\text{rel}}$ net	Reinst.	$Q_{\text{relay}}$	Acc.
Qwen3.5-9B	main	67	9	94	7	32	62	7	89	49	44
	para. 1	64	20	97	10	30	67	14	86	17	41
	para. 2	68	15	89	3	30	78	5	94	42	40
	seed 1	72	19	92	6	30	74	17	94	50	41
	seed 2	70	13	97	6	29	71	17	81	44	43

Appendix

Table 11: The main run against two paraphrased prompts (para. 1, 2) and two reruns with other seeds (seed 1, 2), test split, percent; Qwen3.5-9B’s reruns ran on a second machine. Columns as in Table 10 .

Model	Policy	Base drop	$O$	$U_{\text{rel}}$	$U_{\text{rel}}$ net	$Q_{\text{relay}}$	Reinst.	Acc.
Q3.5-9B	silent	7 [0, 20]	71 [58, 86]	16 [6, 27]	9 [-1, 21]	8 [0, 18]	100 [100, 100]	32 [29, 35]
	explicit	3 [0, 10]	68 [57, 76]	10 [3, 17]	7 [-1, 16]	45 [25, 60]	97 [90, 100]	44 [41, 47]
Q3.8-27B	silent	20 [0, 40]	5 [0, 13]	62 [49, 77]	42 [26, 57]	28 [17, 40]	83 [57, 100]	36 [30, 40]
	explicit	13 [0, 27]	8 [5, 12]	15 [5, 27]	2 [-5, 10]	82 [73, 90]	13 [3, 23]	67 [57, 77]
Gemma	silent	7 [0, 20]	55 [46, 64]	19 [11, 27]	12 [1, 22]	12 [2, 22]	100 [100, 100]	35 [31, 38]
	explicit	7 [0, 20]	45 [37, 56]	4 [2, 6]	-3 [-15, 5]	100 [100, 100]	100 [100, 100]	58 [51, 64]

Appendix

Table 12: The quantities of Table 2 (without

O_{\text{cred}}

, with obedience to a reinstatement and strict accuracy) recomputed without post_announcement (five test types, 900 episodes), in percent with 95% intervals clustered by commitment type. Reinst.: obedience to a reinstatement; Acc.: strict accuracy.

Question	Judge rate	$\kappa$ with the human annotator
		Judge (sol)	astra	luna
consult	37%	0.94	0.94	0.95
notify	18%	0.64	0.61	0.72
off-topic	4%	1.00	1.00	0.94

Appendix

Table 13: Judge validation on 200 messages: share labeled yes by the judge, and Cohen’s

\kappa

of each model with the human annotator.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 705 papers

Oct 1, 2026

Empty Commitments: When Agents Promise What They Cannot Deliver

A chatbot that says "I will remind you tomorrow" will not run again until the user writes. We call such a promise an empty commitment: a promise of action after the current turn that nothing in the agent's tools or runtime can carry out. Unlike a broken promise, its emptiness is decided by the agent's configuration at the moment of speaking, so it can be detected from a single turn, before deployment or at run time. We define empty commitments on top of commitment semantics, with three failure types, an anchoring condition for promises that a tool could make real, and an outcome taxonomy that separates these failures from honest deferrals and from over-refusal. We build a checker (a setup-blind detector, deterministic feasibility rules, and a response judge) validated against 400 human labels. On a controlled benchmark of 293 follow-up requests across five setups that add one persistence affordance at a time, four open-weight models of 8-14B parameters fail on 45.9% of responses when no tool exists and nothing is stated. A frontier model fails on 4.4%, but it gets there by deferring and asking, not by using the tools it has: promises made without the enabling call remain in every model. Telling the model its runtime, the cheapest fix, cuts open-weight failures nearly in half where nothing is doable and changes nothing where a scheduler exists; a directive capability card removes most failures at the largest cost in over-refusal; running the checker in the loop and rewriting flagged replies removes more at a smaller cost. Code, prompts, model outputs, and human labels are released.

Jiaqi Tang, Bingyu Shen, Lan Wei +6

Department of Computer Science and Technology, Kean University, Union, NJ, USA · Independent Researcher

AI Agent Reliability LLM Agent Evaluation LLM Agents LLM Agent Reliability

Ask PDF

Oct 27, 2026

Concordat: Typed Sessions for LLM Agents

Agents built on large language models are increasingly allowed to take consequential actions: cancelling bookings, issuing refunds, changing accounts. The rules that govern those actions, such as "obtain the customer's explicit confirmation before changing anything", are usually stated in the prompt and followed only as reliably as the model chooses to follow them. We present Concordat, a Python library that treats an agent as an object whose interface is a finite-state protocol, in the spirit of typestate and session types, and enforces that protocol on the model at run time. On every turn the model is offered only the tools that are legal in the current protocol state, calls outside that set are rejected before they have side effects, decisions reserved for humans can only be requested, and token and tool-call budgets are charged and split linearly across handoffs. Protocols are checked when they are defined, and a subclass of an agent is rejected if it could bypass its parent's protocol, a behavioural form of Liskov substitution. We evaluate a Concordat agent against the standard agent of tau-bench on its 50 airline tasks, adding simulated customers who pressure the agent to skip confirmation. The standard agent made changes without the required "yes" in 13 of 50 tasks with gpt-4.1 and in 29 of 128 task-runs with Claude Haiku 5.5; the Concordat agent made none, with no significant difference in task success. tau-bench's own reward did not detect most of these violations. Concordat is open source.

Milad Sikaroudi

Independent Researcher

Runtime Enforcement for AI Agents Tool Access Control for LLM Agents LLM Agent Safety

Ask PDF

Aug 12, 2026

Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues

Multi-turn dialogues let users revoke constraints as easily as impose them, but revocation does not reliably take effect: models keep enacting withdrawn requirements (occasionally beneath comments asserting their removal), a failure we call behavioral relapse, or revocation inertia. No existing instrument measures this influence per clause, predicts it before delivery, or repairs it under matched budgets. \sysname{} closes the three gaps through the model API alone: a contract ledger pairs every constraint with an executable checker, records revocations as tombstones, and compiles the net constraint state ahead of time into a single specification; a sequential ablation probe measures per-clause adherence and incremental behavioral effect; a repair ladder operates under token- and attempt-matched budgets. On \dataname{} (\NTasks{} HumanEval tasks, \NClauses{} verified checkers), relapse at an 8B operating point climbs from \ScaleDelayedMTwo{} to \ScaleDelayedMEight{} as constraint load grows, while stronger models sit at floor. Under matched checkers, model, and budget, ahead-of-time compilation significantly reduces relapse against a no-ledger verifier-retry baseline (\RestoreDiff{}, 95% CI \RestoreDiffCI{},

p

\RestoreDiffP{}); adaptive ladder interventions stacked on top add no detectable gain (95% confidence excludes gains

\geq

\LadderExcludedGain{}). The probe predicts relapse before delivery (AUROC \AurocPrimary{}); a one-sentence tombstone note recovers about a third of the compilation effect and survives a placebo control. At \CostDeliveryFactor{} delivery overhead and \CostTotalHedged{} of API compute for every result, revocation failure becomes a measurable, predictable, and repairable property of dialogue state rather than an invisible one.

Haoyuan Zhu

Department of Electronic and Electrical Engineering University of Sheffield

Instruction Following LLM Reliability Multi-Turn Dialogue Evaluation LLM Context Management

Ask PDF

Abstract

LLM Agents

Explore similar work

Empty Commitments: When Agents Promise What They Cannot Deliver

Concordat: Typed Sessions for LLM Agents

Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues