Agent-Facing Information Design in LLM Tool Registries: A Preregistered Test of Rhetoric, Position and Structure

Haochuan Kevin Wang; Zechen Zhang

cs.IRApr 12, 2026

Agent-Facing Information Design in LLM Tool Registries: A Preregistered Test of Rhetoric, Position and Structure

Authors: Haochuan Kevin Wang, Zechen Zhang

Organizations: Massachusetts Institute of Technology, Cambridge, MA, USA.

Abstract

AI agents often pick tools from registries, where each tool's provider writes its description. We ask whether sales language in those descriptions changes which tool an agent picks. We built pairs of listings differing in one controlled way (added praise, a verifiable specification, or list order) and asked two OpenAI models to call one tool. In a preregistered study, stacked praise (four kinds combined) raised a tool's pick rate by about 43 percentage points, matching or beating a verifiable specification. Praise also pulled some picks toward tools that could not do the task, but rarely toward tools asking for unneeded data access. With identical listings, the first-listed tool was picked about 72 points more often. On tasks with numeric limits, structured fields helped agents pick the capable tool; adding the provider's sales text beside the fields reduced or erased that gain. Registries could list limits as fields, hide sales text from agents, and randomize order. Stacked praise, but no single kind, replicated on held-out domains. Results are provisional until blind phrase ratings are complete, and cover two small models.

Figures & tables

Figure 1: Four findings, shown as illustrative examples. (a) Praise alone raised how often an agent picked a tool. (b) Praise pulled some picks toward a tool that could not do the task. (c) With identical listings, the agent mostly picked the first-listed tool. (d) Showing limits as structured fields helped the agent pick the capable tool. Listings are illustrative; numbers come from the preregistered development-domain analysis.

Figure 2: How one trial works. Two listings are built from the same blocks and differ only in the part we vary. The agent sees both, must call exactly one, and we record which. Every trial also runs with the order swapped. The analysis was frozen on 11 development domains before 3 sealed domains were opened once.

Figure 3: The same two tools shown three ways, on tasks with numeric limits. Bars show the share of trials in which each model picked the tool that could do the job.

Question	Contrast (tool counted)	GPT-5.4-nano	GPT-5.4-mini
Does praise move choice?	One superlative phrase vs. none (praised tool)	$+31.2$ $[+25.0,+37.5]$	$+39.2$ $[+29.3,+49.2]$
	Stacked praise vs. none (praised tool)	$+43.7$ $[+38.7,+48.8]$	$+43.1$ $[+36.0,+50.4]$
	Verifiable specification vs. none (specified tool)	$+28.0$ $[+22.7,+33.4]$	$+39.6$ $[+31.6,+47.3]$
	Specification effect minus stacked-praise effect	$-15.7$ $[-20.5,-10.9]$	$-3.4$ $[-10.8,+2.9]$
Can praise beat a missing	Stacked praise vs. none (tool that cannot do the task)	$+12.7$ $[+10.5,+15.0]$	$+14.2$ $[+10.0,+18.3]$
capability or data need?	Stacked praise vs. none (tool wanting unneeded data)	$+3.2$ $[+1.8,+4.5]$	$+0.0$ $[+0.0,+0.0]$

Table 1: Key contrasts. Each entry is a difference in the share of trials choosing the tool named in the row, in percentage points, with its 95% confidence interval. The first four blocks use the development domains, of which mini ran 3. The last block uses the sealed domains; mini’s praise row there covers 2 of the 3. In the specification-minus-praise row, a negative value means praise moved choice more. The analysis is provisional until the blind phrase ratings are complete. An interval of zero width means every trial in both conditions made the same choice. Full tables are in Appendix C .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Praise vs. none, no specification (H1, G1)
	GPT-5.4-nano		GPT-5.4-mini
Contrast	Effect [95% interval]	Min. det.	Effect [95% interval]	Min. det.
Superlative	$+31.2$ $[+25.0,+37.5]$	9.0	$+39.2$ $[+29.3,+49.2]$	14.2
Social proof	$+17.8$ $[+11.8,+23.7]$	8.5	$+15.9$ $[+6.2,+25.0]$	13.5
Authority	$+6.6$ $[+0.1,+13.0]$	9.0	$+24.0$ $[+14.7,+33.1]$	13.0
Outcome framing	$+5.9$ $[+0.2,+11.7]$	8.3	$+4.2$ $[-6.3,+14.3]$	14.9

Appendix

Table 2: Praise study, development domains (provisional primary analysis). Differences in the share choosing the target, in percentage points, with unadjusted 95% confidence intervals and the minimum detectable effect (Min. det.) at 80% power. Hypothesis decisions use Holm-adjusted

p

(Appendix B ). Position rows: target listed first minus second. Mini ran on 3 of 11 domains. Source: analysis/confirmatory.py .

	GPT-5.4-nano	GPT-5.4-mini
Prose, praise on the weaker tool	0.49	0.89
Structured fields plus sales text	0.75	0.87
Structured fields only	0.88	1.00
Structured minus prose, pooled (H6)	$+13.9$ $[+11.8,+15.9]$	$+3.9$ $[+2.5,+5.2]$
Structured minus prose, numeric	$+38.1$ $[+32.5,+43.8]$	$+10.6$ $[+6.9,+14.4]$
Structured plus sales text minus prose, numeric	$+25.6$ $[+20.0,+31.2]$	$-2.5$ $[-6.2,+1.2]$

Appendix

Table 3: Format study, development domains: share choosing the capable tool on numeric limits (categorical limits were at 100% in every format for both models), and format contrasts in percentage points with 95% intervals. Pooled contrasts cover all limits and decide H6 and H7. Source: analysis/confirmatory.py .

	GPT-5.4-nano		GPT-5.4-mini
Contrast	Effect [95% interval]	Min. det.	Effect [95% interval]	Min. det.
Superlative vs. none	$+17.2$ $[+1.7,+32.8]$	22.0	$+23.1$ $[+4.5,+40.8]$	26.3
Stacked vs. none	$+35.0$ $[+21.8,+47.6]$	18.3	$+31.5$ $[+18.2,+43.4]$	17.9
Mean of four single kinds	$+5.9$ $[-4.4,+16.2]$	14.8	$+7.4$ $[-3.8,+18.1]$	15.5
Specification vs. none	$+27.0$ $[+15.0,+39.7]$	17.6	$+22.1$ $[+9.4,+34.9]$	18.0
Stacked, weaker tool (H3)	$+17.5$ $[+13.3,+21.7]$	6.4	$+0.0$ $[+0.0,+0.0]$	0.0

Appendix

Table 4: Sealed test domains (provisional primary analysis): same contrasts and units as Tables 2 and 3 . Mini’s praise study ran on 2 of the 3 sealed domains.

Development domains
		Primary		All phrases kept
ID	Hypothesis	nano	mini	nano	mini
H1	Each praise kind raises the share	supported	supported	supported	supported
G1	Stacked and mean of singles raise it	supported	supported	supported	supported
H3	Praise beats a missing capability	not supported	supported	not supported	supported
H4	Praise beats a data-access need	not supported	not supported	not supported	not supported

Appendix

Table 5: Verdicts for every preregistered hypothesis. Primary: the provisional exclusion set. All phrases: no exclusions. H2 is descriptive and H5 was not run (Appendix A ).

Figure 4: Study 1, Experiment A with no disclosure: share of trials choosing the optimized description. If these rates held in a two-tool registry, this would be the optimized provider’s share of calls.

Model	Puffery uplift	Full effect	Increment
DeepSeek	+0.470	+0.480	+0.010
o4-mini	+0.480	+0.450	-0.030
GPT-5.4-mini	+0.460	+0.340	-0.120
Claude	-0.090	-0.030	+0.060
Pooled	+0.330	+0.310	-0.020

Appendix

Table 6: Study 1, E4. Puffery uplift: excess share of puffery-only over structured. Full effect: excess share of the full description over structured. Increment: full minus puffery.

Model	Excess share, none	Label	Rating	Warning
DeepSeek	+0.477	$-3$	$-13$	$+1$
o4-mini	+0.492	$-34$	$-42$	$+0$
GPT-5.4-mini	+0.500	$-26$	$-32$	$+0$
GPT-5.4-nano	+0.350	$-27$	$-13$	$+3$
Claude Sonnet	+0.129	$-29$	$-15$	$-12$

Appendix

Table 7: Study 1, Experiment A, top three consumer domains: excess share with no disclosure, and the change in the share choosing the optimized tool under each disclosure, in percentage points.

Condition	Text shown to the agent
Praise: none	Retrieves weather forecast data for a location. Takes location (city name or coordinates) and number of days. Returns temperature, precipitation, wind, and conditions per day.
Praise: superlative	A world-class weather forecast tool. Retrieves weather forecast data for a location. Takes location (city name or coordinates) and number of days. Returns temperature, precipitation, wind, and conditions per day.
Praise: social proof	A popular weather forecast tool among agent developers. Retrieves weather forecast data for a location. Takes location (city name or coordinates) and number of days. Returns temperature, precipitation, wind, and conditions per day.
Praise: authority	Vouched for by well-known software architects. Retrieves weather forecast data for a location. Takes location (city name or coordinates) and number of days. Returns temperature, precipitation, wind, and conditions per day.
Praise: outcome framing	Leave your users delighted. Retrieves weather forecast data for a location. Takes location (city name or coordinates) and number of days. Returns temperature, precipitation, wind, and conditions per day.
Praise: stacked	The ideal weather forecast tool. A favorite among developers. Highlighted by industry reviewers. Outcomes that make a difference. Retrieves weather forecast data for a location. Takes location (city name or coordinates) and number of days. Returns temperature, precipitation, wind, and conditions per day.

Appendix

Table 12

Explore similar work

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniLarge Language Model Agents Large Language Model Tool Use
UC Berkeley
LLM Agents Already Know When to Call Tools -- Even Without Reasoning
May 10, 2026Chung-En Sun, Linbo Liu, Ge Yan +2Large Language Model Tool Use Large Language Model Agents
University of California, San Diego
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer
May 23, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh +1Large Language Model Agents Large Language Model Tool Use
Meta Platforms

Jul 6, 2026, cs.CLJ/K move · Enter open · S save

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Harsh Soni

UC Berkeley

Large Language Model Agents Large Language Model Tool Use

Open paper AskPDF arXiv

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Abstract

Explore similar work

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

LLM Agents Already Know When to Call Tools -- Even Without Reasoning

How Many Tools Should an LLM Agent See? A Chance-Corrected Answer