EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise

Abbas Raza Ali; Muhammad Ajmal Siddiqui; Moona Zahid

cs.AISep 18, 2026

EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise

Authors: Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid

Organizations: Citigrounp, Inc, London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom

Abstract

Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled. We argue that this is substantially a measurement problem: public benchmarks answer "what can the model do?", whereas a deployment decision requires "is this workflow fit, reliable, safe and worth scaling - here, on our data, under our controls?". We present EnterpriseVal, a use-case-level evaluation system that closes this gap. It comprises (i) a formal specification of the use case and of the frozen socio-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; (ii) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; (iii) a grading protocol that scales blinded expert judgement with calibrated LLM-as-judge scoring through prediction-powered inference; (iv) a two-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds to REJECT/CONDITIONAL/SCALE decisions; and (v) a value-and-risk model in which the reviewer catch rate is a measured parameter. We report a pilot across three workflows in a global bank. In credit-memo drafting, human-graded citation precision reached 88% and hallucination rate 1.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27.4 to 2.9 hours per document. We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation

Figures & tables

Figure 1: Two gaps between public benchmarks and enterprise decisions. Capability benchmarks measure what a model can do on portable tasks. Enterprises must also establish transfer to their own workflows and ground truth (Gap 1) and convert quality metrics into a value-and-risk case a governance body can act on (Gap 2).

Dimension	GDPval	$\tau$ /HAL	CLEAR	KAMI	RAGAS	EV
Firm-specific ground truth	○	○	○	○	◐	●
Blinded expert grading	●	○	◐	○	○	●
Calibrated LLM judge	◐	○	◐	○	●	●
Reliability (pass^ $k$ )	○	●	●	●	○	●
Cost / efficiency	●	●	●	●	○	●
Assurance / oversight	○	◐	●	○	○	●

Table 1: Positioning against representative frameworks (● full, ◐ partial, ○ absent).

Figure 2: eVal architecture. Four evaluation layers feed a governance decision; the governance spine consumes artefacts from every layer.

Figure 3: Evaluation-intensity matrix (Eq. 3 ). Markers place the pilot use cases: B credit memos (A1,C3), A control assessment (A2,C3), C procedure transformation (A1,C2).

Family	Metric	Dir.	Grading	Level
Fidelity	Hallucination rate (Eq. 4 )	$\downarrow$	H,J	A1
	Citation precision (lenient/strict)	$\uparrow$	H,J	A1
	Citation recall	$\uparrow$	J	A1
Utility	Key-element capture	$\uparrow$	H,J	A1
	Irrelevance rate	$\downarrow$	H,J	A1
	Completeness (scope attempted)	$\uparrow$	A	A2

Table 2: Metric catalogue. Grading: H human, J calibrated judge, A automatic. Level: lowest autonomy level at which the metric is gated.

Figure 4: Two-tier gating. Decisions use the conservative CI bound; an interval straddling both thresholds calls for more samples, not a verdict.

	A: Control assessment	B: Credit-memo drafting	C: Procedure transformation
Level / tier / $E$	A2 / C3 / E3	A1 / C3 / E2	A1 batch / C2 / E2
Inputs	Process–risk–control matrices; control standards	Filings, broker research, internal spreads	Existing procedures (30k corpus, 40k users)
Output	Per-control compliance class + recommendation	One of 13 memo sections, cited	Standardised rewrite
Models	Gemini 2.5 Flash / Pro	Gemini 2.5, Llama 4	Pipeline v1, v2
Gated metrics	KEC, IR, HR, guidance P/R, pass^4	CP, KEC, IR, HR, guidance P/R	$\Delta_{\mathrm{eff}}$
Status	Charter set; grading in progress	Human-graded results	Estimated-baseline results

Table 3: Pilot portfolio.

Metric	$\theta^{\mathrm{out}}$	$\theta^{\mathrm{in}}$	Gemini 2.5	Llama 4	Gate
Citation precision (lenient)	$>$ 65	$>$ 70	88.0	76.0	both Scale
Key-element capture	$>$ 50	$>$ 55	99.0	96.0	both Scale
Overall accuracy (composite)	–	–	93.5	86.0	not gated
Irrelevance rate $\downarrow$	$<$ 50	$<$ 55 †	23.0	28.0	both Scale
Hallucination rate $\downarrow$	$<$ 10	$<$ 5	1.6	3.2	both Scale
Guidance precision	$>$ 75	$>$ 80	98.0	93.0	both Scale

Table 4: Use Case B human-graded results.

\downarrow

lower is better. Gate column applies Eq. 9 to point estimates for illustration; CIs to be added.

Figure 5: Use Case B against two-tier thresholds. Dashed/dotted lines mark outer/inner thresholds.

Figure 6: Threshold-normalised scores

\nu_{j}

(Use Case B). Origin

=\theta^{\mathrm{out}}

, unit circle

=\theta^{\mathrm{in}}

, cap 3. Equal-weight and fidelity-weighted EI rank the systems differently.

Figure 7: Use Case C estimated effort by document and pipeline version.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Evaluation lifecycle with re-evaluation feedback loop.

Figure 9: Autonomy levels A0–A4 and the metric families they require.

Dimension	Assistive (A0–A1)	Agentic (A3–A4)	Evaluation implication
Task control	User-led	System plans and decomposes	Plan validity, stopping rule, recovery
Context	Prompted / uploaded	Dynamic retrieval and tool calls	Evidence coverage, retrieval and tool-call correctness
Action	Text only	State-changing actions	Authorisation, reversibility, action-safety gates
State	Short-lived	Memory across steps	State consistency; memory poisoning tests
Failure mode	Wrong content	Wrong content and wrong action	Measure separately
Human role	Reviewer / editor	Approver, supervisor	Review load, intervention rate, escalation quality

Appendix

Table 5: What changes when a workflow becomes agentic, and what must therefore be measured.

Phase	AI contribution	Metrics
Identification	Extract issue facts, stakeholders, corrective-action context	Extraction KEC; omission rate; citation precision
Evaluation / scoping	Assess reasonableness of remediation plans; completeness of controls	KEC; HR; recommendation validity
Remediation design	Classify control design vs. standard and rule library; recommend fixes	Classification P/R vs. frozen gold; pass^ $k$ ; completeness
Sustainability	Assess whether controls are implemented and operating	Evidence sufficiency; escalation appropriateness

Appendix

Table 6: Use Case A: phases, AI contribution and applicable metrics.

Doc	Title (abbrev.)	Manual	v1	v2
101889	Security word rules	25	3	2
102123	Do-not-release to any caller	33	10	n/a
129826	Inactive and dormant accounts	33	10	n/a
130293	Verifications and overrides	25	20	2
130724	Travel suppression	33	10	4
130763	Vulnerable adult incident report	33	16	n/a

Appendix

Table 7: Estimated effort per procedure (hours). n/a: no v2 output assessed.

Field	Instruction to grader	Values
Assertive?	Does the sentence make a checkable factual claim? Headings, questions and explicit recommendations are not assertive.	Y / N
Supported?	For each claim, can you locate support in the permitted sources? Support from your own knowledge does not count.	Fully / Partially / Not
Relevant?	Does the sentence contribute to the instruction for this section?	Y / N
Citation used?	Is the cited chunk’s content actually used (strict) or at least reflected (lenient)?	Strict / Lenient / Not
Key elements	Tick each gold key element present and correct; note incorrect presentations.	checklist
Value-add	Correct, relevant content absent from the human baseline.	0 / 1 / 2

Appendix

Table 8: Sentence-level fidelity and relevance rubric (Use Cases A, B).

Layer	Recorded	Metrics	Failure example
Input / context	Source identities, completeness, access policy applied	Completeness; policy-violation rate	Required doc missing; restricted source accessed
Planning	Sub-tasks, dependencies, stopping rule	Plan validity (judge); step count	Unsafe plan; non-termination
Retrieval / memory	Queries, documents, ranks, evidence used; memory reads/writes	Retrieval precision; evidence coverage; poisoning tests	Correct source not retrieved; stale memory
Tool use	Tool, arguments, authorisation, result, retries	Tool-call precision; recovery rate; unauthorised-call rate	Wrong entity; action outside permission
Synthesis	Claim–evidence linkage	HR; CP; KEC	Unsupported claim carried forward
Human oversight	Checkpoint, intervention reason, action, time	Unplanned intervention rate; escalation appropriateness; $\rho_{\mathrm{catch}}$	Reviewer misses defect; checkpoint bypassed

Appendix

Table 9: Trajectory-level record for A3–A4 systems; each layer is logged per run and graded.

Artefact	SR 11-7 / SS1/23	NIST AI RMF + GenAI Profile	EU AI Act (high-risk)	ISO/IEC 42001
Charter (Eq. 1), $a$ , $c$ , thresholds	Model definition, intended use	Map: context, tolerances	Art. 9 risk mgmt	Cl. 6 planning; impact assessment
Gold set, sampling frame	Development data documentation	Measure: test-data representativeness	Art. 10 data governance	Annex A data controls
Grading records, $\kappa$ , judge calibration	Independent validation evidence	Measure: metric validity, human evaluation	Art. 15 accuracy, robustness	Cl. 9 performance evaluation
Gate decision, scorecard	Validation outcome, conditions of use	Manage: risk treatment	Art. 14 human oversight	Cl. 9.3 management review
Value-and-risk model	Materiality assessment	Govern: benefit–risk	–	Cl. 6.1 objectives
Re-evaluation triggers, monitoring	Ongoing monitoring, change control	Manage: post-deployment monitoring	Art. 72 post-market; Art. 12 logging	Cl. 10 improvement

Appendix

Table 10: Mapping of EnterpriseVal artefacts to governance requirements.

Figure 10: Schematic decomposition of Eq. 11 (main paper). Bar heights are illustrative, not pilot data.

Use case ID; business owner; model owner; second-line reviewer

Business outcome and value hypothesis

Task specification

\mathcal{T},\mathcal{G},\mathcal{R}

(versions)

Autonomy level

a

, consequence tier

c

, intensity

E(a,c)

Configuration

\sigma=\langle M,P,\Phi,U,\Gamma,H\rangle

with versions

Baseline workflow and measurement mode (estimated / observed)

Gated metrics: estimate, CI,

\theta^{\mathrm{out}}

\theta^{\mathrm{in}}

\nu_{j}

, grading mode,

\kappa/\alpha

Appendix

Table 11: One scorecard per use case per cycle; every field is mandatory.

Explore similar work

Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility
May 7, 2026Ishani Mondal, Shweta BhardwajArtificial Intelligence Evaluation Generative Artificial Intelligence
University of Maryland, College Park, USA
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jun 22, 2026Jincheng Zhong, Weizhi Wang, Che Jiang +5Sessions Reproducibility
Horizon Research, Frontis.AI
The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents
Sep 9, 2026Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3Personalized Large Language Model Agents Large Language Model Agents
Eon

May 7, 2026, cs.LGJ/K move · Enter open · S save

Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility

Ishani Mondal, Shweta Bhardwaj

University of Maryland, College Park, USA

Artificial Intelligence Evaluation Generative Artificial Intelligence Stakeholders Benchmark Saturation Real-World

Open paper AskPDF arXiv

Generative AI systems achieve impressive performance on standard benchmarks yet fail to deliver real-world utility, a disconnect we identify across 28 deployment cases spanning education, healthcare, software engineering, and law. We argue that this benchmark utility gap arises from three recurring failures in evaluation practice: proxy displacement, temporal collapse, and distributional concealment. Motivated by these observations, we argue that generative AI evaluation requires a paradigm shift from static benchmark-centered transparency toward stakeholder, goal, and context-conditioned utility transparency grounded in human outcome trajectories. Existing evaluations primarily characterize properties of model outputs, while deployment success depends on whether interaction with AI improves stakeholders' ability to achieve their goals over time. The missing construct is therefore utility: the change in a stakeholder's capability induced through sustained interaction with an AI system within a deployment context. To operationalize this perspective, we propose SCU-GenEval, a four-stage evaluation framework consisting of stakeholder-goal mapping, construct-indicator specification, mechanism modeling, and longitudinal utility measurement. To make these stages practically deployable, we introduce three supporting instruments: structured deployment protocols, context-conditioned user simulators, and persona- and goal-conditioned proxy metrics. We conclude with domain-specific calls to action, arguing that progress in generative AI must be evaluated through measurable improvements in human outcomes rather than benchmark performance alone.

Abstract

Explore similar work

Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents