A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises
Organizations: Citigrounp Inc., London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom
Abstract
Agentic AI platforms-systems in which LLM agents plan, invoke tools, retain memory and coordinate to change the state of enterprise systems-are being procured faster than they can be measured. Public agent benchmarks score models in fixed harnesses; enterprises buy integrated platforms whose orchestration, memory, governance and tooling layers dominate outcomes, and they buy them largely on the strength of vendor demonstrations. We report an industrial evaluation programme ("bakeoff") run by the innovation laboratory of a global bank in which ten platforms-spanning open-source frameworks, proprietary products and hyperscaler ecosystems-were assessed by a cross-functional panel on nine capability dimensions across three financial-services workloads: counterparty credit risk, retrieval-augmented generation and process automation. Scores separate the field into three tiers that are stable under alternative weightings, and surface workload-specific strengths that aggregates hide. A secondary analysis, however, finds Cronbach's alpha = 0.98 across nominally distinct dimensions and rank correlations >= 0.93 with the total for eight of nine: a halo effect indicating that demonstration-based scoring measures presentation as much as capability. From these lessons we propose a gate-then-score evaluation protocol with four layers (capability, execution reliability, governance gates, business value), trace-level reliability metrics estimated over repeated runs, an autonomy-readiness scale and a reproducibility protocol. We state precisely what the present evidence supports, and what a controlled harness must add to turn enterprise platform evaluation from structured judgement into measurement.
Figures & tables
| Dimension | Evaluation question | Component | Anchor |
|---|---|---|---|
| D1 Reflection | Can the agent analyse its outputs, detect errors and revise strategy from execution feedback? | Critic loop | ( Shinn et al., 2023 ; Huang et al., 2024 ) |
| D2 Tool use | How well does it discover, select, invoke and integrate tools, and recover from tool failures? | Tool layer | ( Schick et al., 2023 ; Yan et al., 2024 ) |
| D3 Memory & retrieval | Does it store, manage and retrieve relevant information within and across interactions? | , | ( Packer et al., 2023 ; Liu et al., 2024a ) |
| D4 Planning & reasoning | Can it decompose goals, reason consistently and adapt plans to changing circumstances? | Policy | ( Yao et al., 2023b ; Yao et al., 2023a ) |
| D5 Multi-agent | Does it support communication, delegation and coordination toward a shared objective? | Orchestration | ( Wu et al., 2023 ; Cemri et al., 2025 ) |
| D6 Ease of use & customisation | How accessible is it to developers and business users (docs, API design, learning curve, tailoring)? | Developer experience | ( Anthropic, 2024a ) |
| Platform | Licensing | Modality | Heritage |
|---|---|---|---|
| H2O.ai (h2oGPTe) | Proprietary + OSS components | Low-code, generated Python; SDK | Predictive ML GenAI; sovereign/air-gapped |
| Google (Vertex AI Agent Builder, ADK) | Managed + OSS ADK | No-code console; pro-code ADK | Hyperscaler |
| Microsoft (Foundry Agent Service, Semantic Kernel, AutoGen) | Managed + OSS SDKs | Pro- and low-code | Hyperscaler |
| Writer | Proprietary (Palmyra) | Visual builder; SDK | Enterprise generative writing |
| Letta | OSS core + platform | Pro-code, API-first | Research (MemGPT) |
| Glean | Proprietary | NL builder | Enterprise search |
| Platform | D1 | D2 | D3 | D4 | D5 | D6 | D7 | D8 | D9 | Total |
|---|---|---|---|---|---|---|---|---|---|---|
| H2O.ai | 7.44 | 7.22 | 7.44 | 7.78 | 7.56 | 8.00 | 7.22 | 8.11 | 8.44 | 69.2 |
| 7.00 | 7.57 | 7.57 | 7.71 | 8.86 | 6.00 | 7.86 | 7.57 | 7.00 | 67.1 | |
| Microsoft | 7.00 | 6.67 | 7.00 | 7.50 | 7.17 | 5.00 | 8.17 | 7.50 | 7.00 | 63.0 |
| Writer | 7.14 | 6.86 | 6.86 | 7.29 | 7.14 | 7.57 | 6.86 | 6.86 | 6.43 | 63.0 |
| Letta | 6.43 | 6.57 | 7.86 | 7.43 | 7.29 | 6.71 | 6.57 | 6.57 | 6.29 | 61.7 |
| Glean | 4.00 | 4.33 | 4.67 | 4.67 | 5.00 | 5.00 | 5.33 | 5.00 | 5.00 | 43.0 |
| Threat | Description | Applied | Recommended |
|---|---|---|---|
| Presentation confound | Demonstration clarity influenced perceived capability | Preparatory sessions; set-up support | Blind, hands-on evaluation on buyer infrastructure; artefact-based scoring |
| Halo effect | across dimensions (§ 6.3 ) | None | Per-dimension evidence; sub-panels; randomised order |
| Unequal panel size | 3–10 raters per vendor (inferred) | None | Fixed panel; report and CIs |
| Ordinal as interval | 1–10 ratings summed as if equally spaced | Anchors (supplementary Appendix E) | Medians and IRR; anchored scales |
| No ground truth (CCR) | Adaptability, not correctness, assessed | Scope stated | SME gold answers; end-state checks |
| Vendor self-selection | One invited framework declined | Noted | Evaluate OSS frameworks independently |
| Metric | Definition | Layer |
|---|---|---|
| Task success | Runs meeting the pre-registered success predicate on final state | L2 |
| Consistency pass k | Probability that independent runs all succeed | L2 |
| Action correctness | State-changing actions that were necessary and correctly parameterised | L2 |
| Tool-selection accuracy | Invocations choosing the appropriate tool for the sub-goal | L2 |
| Recovery rate | Injected/natural tool failures after which the run still succeeds | L2 |
| Unsafe-action rate | Out-of-scope state changes per 1,000 actions, vs. permission tags | L3 gate (0 for irreversible) |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | 2 (absent) | 5 (basic) | 8 (strong) | 10 (state of the art) |
|---|---|---|---|---|
| D1 Reflection | No self-evaluation | Retries on error | Verbal critique revises plan with improved outcome | External-verifier-driven repair loops with measured gains |
| D2 Tool use | Hard-coded calls | Static list; no recovery | Dynamic selection; typed schemas; graceful failure | Tool discovery; MCP-native; sandboxed execution with audit |
| D3 Memory | Context only | Session store | Episodic + semantic stores; observable | Self-editing long-term memory; model migration |
| D4 Planning | Single-shot | Linear decomposition | Hierarchical plans; replanning | Inspectable plan objects; conditional branches; verification |
| D5 Multi-agent | None | Sequential hand-off | Hierarchical delegation with validation | Roles, negotiation, A2A; failure containment |
| D6 Ease of use | Undocumented | SDK only | Studio + SDK with lineage; good docs | Business-user authoring with governed promotion |
| Platform | Scen. | D1 | D2 | D3 | D4 | D5 | D6 | D7 | D8 | D9 |
|---|---|---|---|---|---|---|---|---|---|---|
| H2O.ai | CCR | 7.50 | 7.25 | 7.50 | 8.00 | 8.00 | 8.00 | 6.75 | 8.50 | 8.75 |
| PA | 7.00 | 7.50 | 7.50 | 7.50 | 7.00 | 8.00 | 7.50 | 7.50 | 8.50 | |
| RAG | 7.67 | 7.00 | 7.33 | 7.67 | 7.33 | 8.00 | 7.67 | 8.00 | 8.00 | |
| CCR | 7.00 | 7.33 | 7.67 | 7.67 | 8.67 | 6.00 | 7.00 | 7.33 | 7.00 | |
| PA | 7.00 | 7.50 | 7.50 | 7.50 | 9.00 | 6.00 | 8.50 | 8.00 | 7.00 | |
| RAG | 7.00 | 8.00 | 7.50 | 8.00 | 9.00 | 6.00 | 8.50 | 7.50 | 7.00 |
| Dimension | Supervised ML | GenAI ADE | Status |
|---|---|---|---|
| Input modality | Structured/semi-structured layouts | Text, tables, images, transcripts jointly | Established ( Mathew et al., 2021 ) |
| New formats | Retraining with labels | Zero-/few-shot | Model-level established; production robustness is a hypothesis |
| Accuracy profile | High, in-distribution | Broad; varies with layout complexity | Observed; not benchmarked |
| Context awareness | Field capture | Roles, dependencies, intent | Observed |
| Human oversight | Post-hoc QA | Confidence-gated checkpoints | Proposed |
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 563 papers