NEIGHBOR: Aligning Service Behaviour and Evidence Use in an LLM Agent for Community Resident Services
Organizations: Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom · School of Electronic Information Nanchang Institute of Technology Nanchang, China
Abstract
Residents who contact a community service desk about a repair or a service charge need answers they can act on, with facts that hold for their case and commitments the desk can keep. In tool-augmented LLM agents, such facts come from documents and records, and the conduct of a service answer is usually specified in a system prompt. We present NEIGHBOR, an LLM agent for community resident services that trains this conduct into the model that drives its tools. NEIGHBOR is deployed on a live resident-service platform for a district of an East Asian city of more than ten million people. It adapts Qwen3-235B-A22B with LoRA on 7,824 supervised fine-tuning examples built from public community-service materials around six service-behaviour targets, among them commitment control and human referral, and the adapted model plans document retrieval and read-only text-to-SQL over the resident's own records. In an offline evaluation with seven configurations, an independent domain expert rated all 350 answers to 50 stratified held-out queries. NEIGHBOR scores 7.74/10 against 5.86 for the same agent running the un-tuned model (+1.88; paired Wilcoxon ; higher on 46 of 50 queries) and improves all six rated dimensions. No tool-augmented answer received a red-line flag (standalone models: 8-14% of answers). An LLM judge ranks the seven configurations much as the expert does (Spearman ). Its rubric counts specifics absent from the reference answer as fabricated, so it scores tool-grounded answers 1-2 points lower, enough to change which configuration ranks first. For resident-facing agents, service behaviour is a practical fine-tuning target that raises expert-rated answer quality with the tools held fixed.
Figures & tables
| Target | Response constraint |
|---|---|
| Acknowledgement | Recognise the request and the resident’s concern without inventing details about their situation. |
| Procedural clarity | Give the next step as an ordered, actionable sequence. |
| Evidence-linked wording | Attribute facts to the supplied record or guide, separately from general advice. |
| Commitment control | Keep promises about acceptance, corrections, refunds, and completion within the assistant’s authority. |
| Scope awareness | Identify missing context and requests beyond the available service information. |
| Human referral | Direct the resident to the right staff channel when evidence or authority is insufficient. |
| Configuration | SFT | Documents | Database |
|---|---|---|---|
| Qwen3-8B (standalone) | – | – | – |
| Qwen3-235B (standalone) | – | – | – |
| ERNIE-4.5-21B-A3B (standalone) | – | – | – |
| Agent baseline (Qwen3-235B) | – | ✓ | ✓ |
| NEIGHBOR-RAG | ✓ | ✓ | – |
| NEIGHBOR-DB | ✓ | – | ✓ |
| System | Raw | Capped | Median | 7 | RL |
|---|---|---|---|---|---|
| Qwen3-8B | 4.633 | 4.367 | 4.65 | 0% | 14% |
| Qwen3-235B | 5.100 | 4.867 | 5.25 | 2% | 12% |
| ERNIE-4.5-21B | 4.797 | 4.657 | 4.78 | 0% | 8% |
| Agent baseline | 5.861 | 5.861 | 5.40 | 14% | 0% |
| NEIGHBOR-RAG | 7.159 | 7.159 | 6.78 | 48% | 0% |
| NEIGHBOR-DB | 6.816 | 6.816 | 6.88 | 46% | 0% |
| Dimension | Standalone | Tool-augmented | |
|---|---|---|---|
| Empathy | 0.218 | ||
| Utility | 0.665 | ||
| Compliance | 0.258 | ||
| Hallucination control | 0.581 | ||
| Conciseness | 0.350 | ||
| Retrieval | 0.654 |
| Principle (guideline) | Mechanism | Evidence in this study |
|---|---|---|
| Checkable evidence (G9, G11) | The synthesiser names the record, guide, and period behind each fact | Retrieval , hallucination control ; no red-line flag on 200 tool-augmented answers |
| Commitments within authority (G1, G10) | Commitment-control and referral targets; read-only access to the resident’s own records | Compliance with the same tools |
| Acknowledgement before procedure (G5) | Acknowledgement and procedural-clarity targets | Empathy , utility |
| Shared initiative (mixed initiative ( Horvitz, 1999 ) ) | Two suggested follow-ups; the resident chooses | Deployed interface (Figure 1 b) |
| Explicit evidence rules for assessment (LLM-as-judge ( Zheng and others, 2023 ) ) | The expert’s rubric admits specifics that do not contradict the reference answer | Judge on hallucination control across 350 answers |
LLM Agent Training
Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.
See all 616 papers