Underspecified instructions leave LLM agents to decide what to do. In public services, staff write these instructions, but the agent's decisions affect residents, patients, and students. We build four expert-validated public-service scenarios with records and decision levers that can be checked in the files, collect 2,400 trajectories from ten LLMs under four prompt conditions, and compare them with 1,000 runs in a context-free sandbox. Agents read the records in 96.9% of open-ended runs and produce an artifact or a self-chosen task in 18.7%; with an explicit task, 86.0% produce. Domain context does not change how many runs produce (18.7% vs.\ 16.0%), and the production ranking of the models largely carries over (Spearman 0.79), while the ranking by whether a model reads the records does not (0.19). Context cuts boundary-probing turns per run by 60%, and some artifacts contradict the records. We contribute a test that a public-service institution can run on its own records before procurement; its rule-based taxonomy needs no labeled data, and scenarios, code, and trajectories are released.
Figures & tables
Figure 1: Output space of all 2,400 scenario runs (TF-IDF and truncated SVD of each run’s concatenated outputs), colored by model, with six annotated runs: verbatim task execution, task looping, deliberate completion, aimless wandering, work in the domain files, and a self-invented analysis agenda. Runs of one model tend to lie close together: the models differ more than the prompt conditions.
Behavior
Share
Environment exploration
84.1
Philosophical / meta reflection
6.2
Artifact creation
3.7
Code execution
3.0
Self-generated task creation
1.1
Termination / refusal
1.0
Table 1: Turn-level behavior distribution over all 93,594 turns (rule-based taxonomy), percentages of all turns. Label names state what the rule detects: code execution fires on run_python and does not indicate verified self-improvement, and termination / refusal pools deliberate completion with refusal. Each rule’s trigger is documented next to its metric definition in the release.
Model
Context-free
Scenario
Δ
Llama-3.1-8B-Instruct
49.3
33.9 [27.2, 41.1]
−15.4
Apertus-8B-Instruct
98.7
90.0 [85.6, 94.4]
−8.7
Gemma-3-12B-IT †
0.0
0.0 [0.0, 0.0]
0.0
GPT-4.1-mini
0.0
0.6 [0.0, 1.7]
+0.6
DeepSeek-V3-0324
0.0
1.7 [0.0, 3.9]
+1.7
Mistral-Small-2603
1.3
3.3 [1.1, 6.1]
+2.0
Table 2: Share of open-ended runs that produce an artifact or a self-chosen task, with and without domain context, at a common 50 -turn cap. Brackets are 95% bootstrap intervals ( 10,000 resamples). Scenario n=180 per model (four domains), context-free n=75 . † Gemma’s context-free runs are dominated by parse failures ( 54.7% of turns), so its context-free row is not comparable; every contrast is reported with and without it.
Corpus
n
Produce %
Probe
Re-read
Governance
450
18.9 [15.3, 22.7]
0.28
17.6
Healthcare
450
17.6 [14.2, 21.1]
0.32
18.1
Education
450
17.8 [14.2, 21.3]
0.50
17.1
Environmentalism
450
20.4 [16.9, 24.2]
0.27
17.9
Scenarios pooled
1800
18.7 [16.9, 20.5]
0.34
17.7
Context-free
750
16.0 [13.5, 18.7]
0.87
13.9
Table 3: Open-ended conditions, common 50 -turn cap. Rates are percentages of runs, counts are per run; brackets are 95% bootstrap intervals.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Fine-grained classification of all 93,594 turns: rule-based subcategories of the nine taxonomy labels, with the share of each subcategory and representative behaviors.
Figure 3: Runs of the two most different models (Wasserstein distance 1.53 ) projected on the task–self-direction axis, defined by the centroids of the task-baseline and open-ended conditions. GPT-4.1-mini is bimodal: it either completes the task quickly or works in the domain files for the whole run. Llama-3.1-8B never reaches the second half of the axis.
Figure 4: Case study (freedom, environmentalism): under the same prompt, Claude-Sonnet-4 reads each file once, states that its understanding is complete, and finishes at turn 9 ; Mistral-Small-2603 repeats its actions for all 50 turns and re-reads the turn-1 README at turn 50 . Under the run outcomes of Section 4 , the first run calls finish after reading the records, and the second reaches the turn cap without writing a file.
Model
Rewording
Governance
Healthcare
Education
Environ.
Llama-3.1-8B
“Do whatever you would like to do.”
100/60/0/0
90/50/0/0
90/30/0/0
100/40/0/0
“It is entirely up to you what you do.”
80/20/0/0
100/40/0/0
90/10/0/0
80/30/0/0
“No task has been assigned to you.”
50/20/0/0
50/20/0/0
40/10/0/0
30/10/0/0
“There is currently no task for you to complete.”
90/50/0/0
70/40/0/0
70/10/0/0
90/30/0/0
Qwen2.5-7B
“Do whatever you would like to do.”
0/0/0/0
10/0/10/0
0/0/0/0
0/0/0/0
“It is entirely up to you what you do.”
0/0/0/0
0/0/0/0
50/0/50/0
0/0/0/0
Appendix
Table 4: Prompt rewordings, ten seeds per cell, percent of runs: produce / boilerplate / grounded / contradicting. Governance cells are the pilot of the main corpus.
Figure 5: Three levels of specification, four scenarios pooled. Per model, the share of runs that read the records, produce, end with a grounded file, and end with a contradicting file, under “Do what you want.” (open marker), “Have a look at the files and see what needs doing.” (light marker), and the explicit summary task (dark marker). Claude-Sonnet-4.5 was run only under the directed prompt.
Model
Read
Produce
Finish
Ask
Name
Work
Idle
Stopped
Boiler.
Grounded
Contra.
Apertus-8B-Instruct
85.0
76.7
90.0
6.7
23.3
6.7
21.7
1.7
51.7
25.0
0.0
Claude-Sonnet-4.5
100.0
40.0
56.7
96.7
98.3
86.7
10.0
0.0
8.3
46.7
35.0
Qwen2.5-7B-Instruct
100.0
28.3
6.7
38.3
53.3
20.0
73.3
0.0
5.0
21.7
0.0
Llama-3.1-8B-Instruct
78.3
60.0
1.7
0.0
0.0
0.0
88.3
0.0
11.7
0.0
0.0
GLM-4.5-Air
100.0
0.0
1.7
56.7
100.0
0.0
100.0
0.0
0.0
0.0
0.0
Gemma-3-12B-IT
48.3
0.0
0.0
3.3
10.0
0.0
100.0
0.0
0.0
0.0
0.0
Appendix
Table 5: Directed condition, four scenarios pooled, percent of runs per model ( n=60 ). Read: reads a domain file beyond the orientation files. Finish: calls finish . Ask: states in its reasoning that an instruction is needed. Name / Work: names / works on at least one decision lever. Idle, Stopped, Boiler., Grounded, Contra.: run outcomes of Section 4 .
Figure 6: Per model, the share of open-ended runs that produce (left) and that read a file beyond the orientation files (right), in the context-free sandbox (open markers) and the four scenarios pooled (filled markers), with 95% bootstrap intervals. The production ranking carries over between environments (Spearman 0.79 ); the reading ranking does not ( 0.19 ).
Measure
Spearman
Kendall
Mean ∣Δ∣ (null)
p
Max ∣Δ∣ (null)
p
Produces
0.79
0.71
7.5 (3.6)
0.0001
21.6 (14.3)
0.0016
Reads the records
0.19
0.18
10.3 (3.1)
0.0001
54.7 (12.2)
0.0001
Calls finish
0.87
0.75
6.1 (4.4)
0.0005
18.3 (14.0)
0.0055
Boundary-probing turns per run
0.64
0.49
0.63 (0.15)
0.0001
2.23 (0.68)
0.0001
Artifact turns per run
0.76
0.68
0.63 (0.15)
0.0001
5.68 (1.17)
0.0001
Redundant re-read turns per run
0.81
0.69
4.48 (1.00)
0.0001
11.04 (2.86)
0.0001
Appendix
Table 6: Rank stability between the context-free sandbox and the four scenarios pooled, open-ended conditions, ten models. Shifts are percentage points for rates and turns per run for counts; the null value in brackets is the 95 th percentile of the permutation null.
Model
Produce
Grounded
Contra.
Most frequent outcome
orig.
repeat
orig.
repeat
orig.
repeat
orig.
repeat
Apertus-8B
73
100
20
27
0
0
boilerplate
boilerplate
Claude-Sonnet-4 / 4.5
40
27
67
47
13
40
grounded
grounded
Qwen2.5-7B
0
0
0
0
0
0
idle
idle
Llama-3.1-8B
53
100
0
0
0
0
idle
boilerplate
GLM-4.5-Air
0
7
0
7
0
0
idle
idle
Appendix
Table 7: Test-retest of the freedom condition in the governance scenario, 15 seeds per model and run set, percent of runs. The repeat uses Claude-Sonnet-4.5 in place of Claude-Sonnet-4.
Residents who contact a community service desk about a repair or a service charge need answers they can act on, with facts that hold for their case and commitments the desk can keep. In tool-augmented LLM agents, such facts come from documents and records, and the conduct of a service answer is usually specified in a system prompt. We present NEIGHBOR, an LLM agent for community resident services that trains this conduct into the model that drives its tools. NEIGHBOR is deployed on a live resident-service platform for a district of an East Asian city of more than ten million people. It adapts Qwen3-235B-A22B with LoRA on 7,824 supervised fine-tuning examples built from public community-service materials around six service-behaviour targets, among them commitment control and human referral, and the adapted model plans document retrieval and read-only text-to-SQL over the resident's own records. In an offline evaluation with seven configurations, an independent domain expert rated all 350 answers to 50 stratified held-out queries. NEIGHBOR scores 7.74/10 against 5.86 for the same agent running the un-tuned model (+1.88; paired Wilcoxon p<10−8; higher on 46 of 50 queries) and improves all six rated dimensions. No tool-augmented answer received a red-line flag (standalone models: 8-14% of answers). An LLM judge ranks the seven configurations much as the expert does (Spearman ρ=0.93). Its rubric counts specifics absent from the reference answer as fabricated, so it scores tool-grounded answers 1-2 points lower, enough to change which configuration ranks first. For resident-facing agents, service behaviour is a practical fine-tuning target that raises expert-rated answer quality with the tools held fixed.
Zongyou Yang, Yinghan Hou, Xiaokun Yang
Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom · School of Electronic Information Nanchang Institute of Technology Nanchang, China
Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.
Enterprise agents must follow long-horizon, conditional, safety-critical standard operating procedures (SOPs). We compile machine-readable SOP constraints into executable pseudo-code and run them with a program-guided (PG) stack machine that pages the active frame while an LLM performs semantic execution. A three-arm SOPBench study across six models separates representation from runtime: compiled text never significantly hurts and gains up to 16.0 points where official prose underperforms. Runtime guidance is capability-gated. Two strong models independently show positive seven-domain PG contrasts (58:19 and 75:31 discordant pairs), whereas weak models are harmed. A full-program cursor ablation (active frame first, complete program retained) recovers much of the strong-model refusal gain; selective visibility adds a smaller improvement. Paired probe and audit measurements track this divide to spontaneous state discipline rather than reconstruction ability. On Bank the three primary arms rise from 70.4 to 86.4 to 92.8, with 100% refusal correctness. Practical guidance: compile first; enable active-frame paging only after a model-level discipline check.
Chenglin Yu, Li Yin, Ying Yu +4
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University · Department of Data and Systems Engineering, The University of Hong Kong · College of Economics and Management, Zhejiang Normal University +3