Integrating the Analytic Hierarchy Process with Large Language Models for Transparent Multi-Criteria Decision-Making

Han Zhiguang; Farah Benamara; Pascale Zaraté

cs.AISep 15, 2026

Integrating the Analytic Hierarchy Process with Large Language Models for Transparent Multi-Criteria Decision-Making

Authors: Han Zhiguang, Farah Benamara, Pascale Zaraté

Organizations: IRIT-MELODI, UT3, IPAL · IRIT, Université de Toulouse, Toulouse, France · IPAL-CNRS-NUS-A*STAR, Singapore · IRIT, UT Capitole, IRIT-ADRIA ·

Abstract

LLMs are increasingly employed in a wide range of decision-making tasks. However, the opacity of their internal reasoning makes it difficult to validate or interpret their outputs, and the need for interpretability becomes especially critical in high-stakes settings. This study examines the decision-making capabilities of LLMs through the Analytic Hierarchy Process (AHP), a classical and widely used multicriteria decision-making framework. We construct a new annotated benchmark based on AHP and propose the first end-to-end approach that enables LLMs to perform the complete AHP workflow. Experiments in real-world decision problems in the legal and higher-education ranking domains show that our method significantly improves alignment with expert judgments.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge
Apr 4, 2026Yulong He, Ivan Smirnov, Dmitry Fedrushkov +2LLM Evaluation LLM-as-a-Judge
St. Petersburg State University, University emb., 7/9, St. Petersburg, Russia · ITMO University, Kronverkskiy av., 49, St. Petersburg, Russia
IDEA: An Interpretable and Editable Decision-Making Framework for LLMs via Verbal-to-Numeric Calibration
Apr 14, 2026Yanji He, Yuxin Jiang, Yiwen Wu +3LLM Decision-Making Language Model Calibration
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd · The Hong Kong University of Science and Technology
Learning Transferable Latent User Preferences for Human-Aligned Decision Making
May 12, 2026Alina Hyk, Sandhya SaisubramanianLLM Alignment LLM Decision-Making
Oregon State University

Apr 4, 2026J/K move · Enter open · S save

Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge

Yulong He, Ivan Smirnov, Dmitry Fedrushkov, Sergey Kovalchuk+1

St. Petersburg State University, University emb., 7/9, St. Petersburg, Russia · ITMO University, Kronverkskiy av., 49, St. Petersburg, Russia

LLM Evaluation LLM-as-a-Judge Multi-Criteria Decision Analysis

Open paper AskPDF arXiv

Effective evaluation of large language models (LLMs) remains a critical bottleneck, as conventional direct scoring often yields inconsistent and opaque judgments. In this work, we adapt the Analytic Hierarchy Process (AHP) to LLM-based evaluation and, more importantly, propose a confidence-aware Fuzzy AHP (FAHP) extension that models epistemic uncertainty via triangular fuzzy numbers modulated by LLM-generated confidence scores. Systematically validated on JudgeBench, our structured approach decomposes assessments into explicit criteria and incorporates uncertainty-aware aggregation, producing more calibrated judgments. Extensive experiments demonstrate that both crisp and fuzzy AHP consistently outperform direct scoring across model scales and dataset splits, with FAHP showing superior stability in uncertain comparison scenarios. Building on these insights, we propose DualJudge, a hybrid framework inspired by Dual-Process Theory that adaptively fuses holistic direct scores with structured AHP outputs via consistency-aware weighting. DualJudge achieves state-of-the-art performance, underscoring the complementary strengths of intuitive and deliberative evaluation paradigms. These results establish uncertainty-aware structured reasoning as a principled pathway toward more reliable LLM assessment. Code is available at https://github.com/hreyulog/AHP_llm_judge.

Abstract

LLM Decision-Making

Explore similar work

Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge

IDEA: An Interpretable and Editable Decision-Making Framework for LLMs via Verbal-to-Numeric Calibration

Learning Transferable Latent User Preferences for Human-Aligned Decision Making