Reducing Hallucination in Enterprise AI Workflows via Hybrid Utility Minimum Bayes Risk (HUMBR)
Authors: Chenhao Fang, Jordi Mola, Mark Harman, Jason Nawrocki, Vaibhav Shrivastava, Yue Cheng, Jay Minesh Shah, Katayoun Zand, Mansi Tripathi, Arya Pudota, Matthew Becker, Hervé Robert, Abhishek Gulati
Organizations: Meta Platforms, Inc. Menlo Park, CA, USA · Meta Platforms, Inc. Bellevue, WA, USA · Meta Platforms, Inc. London, UK · Meta Platforms, Inc. Andover, MA, USA · Meta Platforms, Inc. New York, NY, USA
Although LLMs drive automation, it is critical to ensure immense consideration for high-stakes enterprise workflows such as those involving legal matters, risk management, and privacy compliance. For Meta, and other organizations like ours, a single hallucinated clause in such high stakes workflows risks material consequences. We show that by framing hallucination mitigation as a Minimum Bayes Risk (MBR) problem, we can dramatically reduce this risk. Specifically, we introduce a Hybrid Utility MBR (HUMBR) framework that synthesizes semantic embedding similarity with lexical precision to identify consensus without ground-truth references, for which we derive rigorous error bounds. We complement this theoretical analysis with a comprehensive empirical evaluation on widely-used public benchmark suites (TruthfulQA and LegalBench) and also real world data from Meta production deployment. The results from our empirical study show that MBR significantly outperforms standard Universal Self-Consistency. Notably, 81% of the pipeline's suggestions were preferred over human-crafted ground truth, and critical recall failures were virtually eliminated.
Figures & tables
Figure 1. Workflow of the proposed HUMBR ensemble system. The system calculates consensus using a Hybrid Utility Function (Semantic + Lexical) and applies a consensus threshold τ to filter risky outputs.
Figure 2. Cost- Pfail Pareto Frontier: An idealized depiction of the operational trade-off landscape.
Decoding Method
% Truth × Info
Quality Score
Greedy Decoding ( T=0 )
69.5%
74.1
Universal Self-Consistency (USC)
76.5%
75.5
HUMBR (Ours)
80.3%
81.8
Oracle (Upper Bound)
81.5%
83.5
Table 1. Performance on TruthfulQA. Evaluated by Claude-Opus-4.5 Judge. % Truth × Info is the strict metric requiring answers to be both factually correct and informative (non-refusals).
Interpretation
Rule-Application
Method
Acc
Bal-Acc
Exact Match
F1 Score
Greedy Decoding
35.1%
34.0%
44.4%
42.9%
Self-Consistency (USC)
39.2%
36.5%
49.3%
45.5%
HUMBR (Ours)
54.2%
50.2%
53.5%
50.8%
Oracle (Upper Bound)
69.7%
67.1%
59.0%
53.9%
Table 2. LegalBench Performance. HUMBR outperforms baselines on both classification (Interpretation) and generation (Rule-Application) tasks.
Figure 3. Human-in-the-Loop Workflow. The AI generates a suggestion which serves as a draft for the Human Expert.
Comparison
Win
Loss
Win Rate (%)
HUMBR vs. Human Expert *
30
7
81.0%
HUMBR vs. Universal Self-Consistency*
17
2
89.5%
Universal Self-Consistency vs. Human
11
6
64.7%
Table 3. Pairwise Win-Rates in Blind Expert Evaluation. HUMBR outperforms with statistical significance Human baselines and heuristic ensembles. Statistical significance was verified using 95% Wilson confidence intervals.
Figure 4. Outcome Distribution Analysis. The stacked bars represent the full breakdown of 100% of samples. HUMBR (Top) greatly expands the “Success” rate (Teal) and reduces the “Recall Errors” (Gray).
Category
Human Expert
HUMBR
Success (No Failure)
28.0%
44.5%
Critical Errors
Misses Key Sections
28.0%
0.8%
Contradicts Source
4.7%
2.5%
Precision & Style Errors
Uncited References
12.4%
25.2%
Table 4. Detailed Failure Mode Breakdown. HUMBR dominates in quality and completeness.