LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model

Ergan Shang; Weijing Tang; Yinqiu He

cs.CLAug 23, 2026

LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model

Authors: Ergan Shang, Weijing Tang, Yinqiu He

Organizations: Department of Statistics and Data Science, Carnegie Mellon University · Department of Statistics, University of Madison

Abstract

Evaluation of large language models (LLMs) increasingly requires predicting how a model will perform on new questions or tasks before collecting large amounts of new annotations. This problem is challenging because question difficulty, scenario, and underlying capability demands can vary substantially. Simple retrospective averages may confound model ability with item characteristics. In this paper, we study a model-based evaluation framework that combines multidimensional item response theory model with question contexts to predict LLM performance on unseen questions. The framework represents LLMs through latent capability profiles while using question content to inform item characteristics, allowing information to transfer beyond previously observed items. Empirically, we find that for within-scenario evaluation, incorporating question embeddings improves prediction relative to model-free baselines, and that multidimensional latent structure provides a richer description of capability variation than unidimensional alternatives. At the same time, our results reveal an important limitation that the generalizability does not necessarily translate into reliable prediction under cross-scenario shift. These findings suggest that context-aware psychometric modeling is a promising direction for efficient and interpretable LLM evaluation, while also highlighting cross-scenario generalization as a central open challenge.

Figures & tables

Figure 1: Heatmap of the 90th percentile of question-wise AUC distribution predicted by the C-MIRT method across scenarios.

Figure 2: Heatmap of the 90th percentile of LLM-wise AUC distribution predicted by the C-MIRT method across scenarios.

Figure 3: Heatmap of mean differences in the 90th percentile of question-wise AUC between C-MIRT and competing methods across scenarios.

Figure 4: Heatmap of mean differences in the 90th percentile of LLM-wise AUC between C-MIRT and competing methods across scenarios.

Explore similar work

Apr 14, 2026

Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities

Current evaluations of large language models aggregate performance across diverse tasks into single scores. This obscures fine-grained ability variation, limiting targeted model improvement and ability-guided selection for specific tasks. Motivated by this gap, we propose a cognitive diagnostic framework that estimates model abilities across multiple fine-grained dimensions. For mathematics, we construct a 35-dimensional ability taxonomy grounded in cognitive theory and domain knowledge. The framework employs multidimensional Item Response Theory with an item-ability association matrix to estimate fine-grained ability levels, which in turn enable prediction of performance on unseen items (questions of benchmark). Evaluated on 41 models, our approach demonstrates strong criterion validity, consistent ability estimates across benchmarks, and accurate prediction of unseen items with AUC ranging from 0.80 to 0.89 within benchmarks and from 0.77 to 0.86 across benchmarks, substantially exceeding trivial baselines. The framework generalizes across scientific domains, producing consistent diagnostic performance in physics (27 dimensions), chemistry (58 dimensions), and computer science (12 dimensions). This work establishes a principled framework for fine-grained assessment of abilities, with potential applications in targeted training, ability-guided model selection, and ability-aware benchmark design.

Xu Zhang, Xudong Gong, Jiacheng Qin +5

College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan, China · State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Humanities and Social Sciences, School of Public Administration, Beihang University, Beijing, China +1

LLM Evaluation

Ask PDF

Sep 8, 2026

Evaluation of Contextual Understanding in Large Language Models

Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information--a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA) benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.

Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe +4

Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, Coral Gables, Florida, USA.

LLM Evaluation LLM Grounding LLM Reasoning with Graphs Knowledge Graphs

Ask PDF

Jun 14, 2026

Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation

Multilingual benchmarks are central to evaluating large language models (LLMs) across languages, but they suffer from three issues: exhaustive evaluation scales linearly with the number of languages, automatic translation introduces errors that are easily missed at scale, and some items conflate general and culture-specific knowledge. We address all three with a unified statistical framework, Multilingual-IRT, which extends Item Response Theory with per-language difficulty deviations, split discriminability separating content from language effects, and per-language ability residuals. Fitting Multilingual-IRT on 25 LLMs across 29 languages of MMLU-Pro-X, we show that its fitted parameters support three practical applications: predicting unobserved (item, LLM, language) instances with 11-16% lower binary cross-entropy than the strongest accuracy-based baseline, surfacing candidate translation errors distributed across all 28 non-English languages, whereas accuracy-based baselines concentrate detections in a few languages, and recovering culture-specific items that accuracy-based baselines miss.

Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1

Google Research · The Hebrew University of Jerusalem · PhaseV Trials

Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation Item Response Theory

Ask PDF