LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
Apr 9, 2024Yu Zhou, Xingyu Wu, Jibin Wu +2LLM Evaluation Causal Discovery
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
LLMRec: Benchmarking Large Language Models on Recommendation Task
Aug 23, 2023Junling Liu, Chao Liu, Peilin Zhou +8Large Language Model-Based Recommendation Recommender Systems
Alibaba Group · Ant Group · The Hong Kong University of Science and Technology (Guangzhou) +5

Apr 9, 2024J/K move · Enter open · S save

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng+1

Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China

LLM Evaluation Causal Discovery Causal Structure Learning Causal Reasoning in Language Models Causal Reasoning

Open paper AskPDF arXiv

The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual reasoning, as causality reveals the underlying data distribution. However, the lack of a comprehensive benchmark currently limits the evaluation of LLMs' causal learning capabilities. To fill this gap, this paper develops CausalBN-Bench based on data from the causal research community, enabling comparative evaluations of LLMs against traditional causal learning algorithms. To provide a comprehensive investigation, we offer three tasks of varying difficulties, including correlation, causal skeleton, and causality identification. Evaluations of 19 leading LLMs reveal that, while closed-source LLMs show potential for simple causal relationships, they significantly lag behind traditional algorithms on larger-scale networks (

>50

nodes). Specifically, LLMs struggle with collider structures but excel at chain structures, especially at long-chain causality analogous to Chain-of-Thought techniques. This supports the current prompt approaches while suggesting directions to enhance LLMs' causal reasoning capability. Furthermore, CausalBN-Bench incorporates background knowledge and training data into prompts to thoroughly unlock LLMs' text-comprehension ability during evaluation, whose findings indicate that, LLM understand causality through semantic associations with distinct entities, rather than directly from contextual information or numerical distributions.

Latest papers

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

LLMRec: Benchmarking Large Language Models on Recommendation Task