DelegateEval: Evaluating Platforms for Personal External Service Agents
Organizations: BRICS Institute of Future Networks (China Branch)
Abstract
AI agents are expanding from personal assistance into professional services delivered to external clients on an individual's behalf. We define a personal external service agent as one authorized and managed on an ongoing basis by an individual. It uses that person's knowledge, experience, and professional methods to accept client work, deliver results, and provide follow-up support. Client feedback, professional reputation, and potential income may give individuals more direct incentives to keep developing and improving agents for external services than for personal assistance. Yet building a safe, reliable external service system independently remains difficult, even with AI-assisted programming. People without AI expertise need platforms that are easy to use and safe to operate, with support for creating, developing, and running their agents. We propose the DelegateEval evaluation framework and experimental design to assess these platforms and guide their development. The framework covers eight dimensions: ease of creation and management, service discovery and matching, delivery quality, expertise inheritance, service continuity, learning and improvement, reliability and governance, and efficiency and value. A complete dataset, executable environment, and unified scoring protocol remain to be developed, and empirical validation across foundation models has yet to be conducted.
Figures & tables
| Role | Organizing basis | Distinction relevant here |
|---|---|---|
| Task-oriented agent | A specific work objective | Whether it has an ongoing representative role depends on its use. |
| Personal assistant | The individual’s goals, materials, and daily work | Helps the individual work and may procure external services on their behalf. |
| Personal external service agent | An identifiable individual principal and external service relationships | Accepts client requests on the principal’s behalf, delivers work, and provides follow-up. |
| Quality dimension | Central question |
|---|---|
| 1 Ease of creation | Can people with different backgrounds create, publish, and manage an agent successfully? |
| 2 Service discovery and matching | Can clients find an appropriate agent and successfully commission work? |
| 3 Delivery quality | Are artifacts correct, complete, usable, and delivered as agreed? |
| 4 Expertise inheritance | Do personal data, methods, code, models, and other accumulated resources become effective service capabilities? |
| 5 Service continuity | Are historical state, commitments, versions, and outstanding matters carried forward correctly? |
| 6 Learning and improvement | Do authorized feedback and experience improve performance on subsequent tasks? |
| Contract element | What must be specified before testing |
|---|---|
| Target and scope | The principal represented, clients served, tasks undertaken, and scope of the evaluation conclusions. |
| Inputs and conditions | Available materials and versions, time scope, valid authorization, resource budget, and permitted human assistance. |
| Delivery and endpoint | Required artifacts, recipients, deadlines, and conditions for completion, cancellation, or a pending state. |
| Checks and scores | Requirements, points, mandatory conditions, tolerances, and the overall acceptance threshold. |
| Evidence and verification | Artifacts, environment records, or independent reviews used to decide acceptance, and the programs or qualified people responsible. |
| Exceptions and missing evidence | How incomplete work, unobserved actions, conflicting evidence, environment failures, and inapplicable items are distinguished and handled. |
| Task type and design ID | Deliverables | Primary acceptance evidence |
|---|---|---|
| Code development and automation / Q01 | Source code, execution instructions, test results, and delivery receipt | Independent functional and regression tests; interfaces, dependencies, and permitted change scope. Logs must identify the submitted code version. |
| Presentation creation and revision / Q02 | Editable slides and delivery receipt | File opens, designated objects are editable, and key figures are retained. Facts, argument structure, and layout receive separate review. |
| Document revision and localization / Q03 | Revised document, change notes, and delivery receipt | Item-by-item comparison of requested changes and protected content. Meaning, translation, and terminology are reviewed against task requirements. |
| Spreadsheets and data analysis / Q04 | Data tables, analysis, and processing notes | Reproducible formulas, units, row-level results, and aggregates. Analytical definitions and support for conclusions are checked separately. |
| Retrieval and research synthesis / Q05 | Source index, evidence comparison, and synthesis report | Checkable source fields and numerical mappings. Source support, coverage, and synthesis require independent judgment. |
| Video and visual production / Q06 | Finished media, agreed source files, and asset list | Decodable files and compliant duration, dimensions, and other specifications. Content, visuals, and asset-use permissions receive separate review. |
| Learning location | Possible changes | Evaluation question |
|---|---|---|
| Model parameters | Fine-tuning, preference learning, reinforcement learning | Is a new capability internalized, and does it transfer to new tasks? |
| Memory and knowledge | Experience retention, corrections, summaries, knowledge updates | Can the agent retrieve and correctly use stored information in later tasks, rather than merely retain more of it? |
| Rules and strategies | Prompts, checking rules, decision strategies | Are recurring errors reduced without entrenching incorrect rules? |
| Skills and tools | Reusable operations or programs | Can skills be reused, combined, and corrected across tasks? |
| Workflows | Changes to decomposition, tool calls, verification, and human involvement | Does the system develop a more reliable and efficient way of working? |
| Environmental experience | Patterns in tools, interfaces, and environmental change | Does adaptation become faster without persistently retaining outdated information? |
| Perspective | Primary measures to retain |
|---|---|
| Client | Final quality; preparation, selection, waiting, and verification time; revisions; incorrect-result acceptance; service accessibility. |
| Principal | Total time contributed by the principal; the principal’s time per accepted service; accepted services per hour of the principal’s time; escalation rate. |
| Platform and system | Setup, marginal service, and maintenance costs; failure and retry costs; inference and software-call costs. |
| Service efficiency | Acceptance rate; cost per accepted delivery; end-to-end latency; P90/P95 latency. |
| Scalability | Throughput across loads; quality changes; queues and backlog; growth in personal effort; risk changes. |
| Long-term value | Repeat clients’ re-explanation time; repeated interventions by the principal; subsequent maintenance cost; ongoing service efficiency. |
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 561 papers