Using Vision + Language Models to Predict Item Difficulty

Samin Khan

cs.AIMar 4, 2026

Using Vision + Language Models to Predict Item Difficulty

Authors: Samin Khan

Organizations: Stanford Graduate School of Education

Abstract

This project investigates the capabilities of large language models (LLMs) to determine the difficulty of data visualization literacy test items. We explore whether features derived from item text (question and answer options), the visualization image, or a combination of both can predict item difficulty (proportion of correct responses) for U.S. adults. We use GPT-4.1-nano to analyze items and generate predictions based on these distinct feature sets. The multimodal approach, using both visual and text features, yields the lowest mean absolute error (MAE) (0.224), outperforming the unimodal vision-only (0.282) and text-only (0.338) approaches. The best-performing multimodal model was applied to a held-out test set for external evaluation and achieved a mean squared error of 0.10805, demonstrating the potential of LLMs for psychometric analysis and automated item development.

Figures & tables

Figure 1. MAE for each predictive model on the validation subset. Error bars represent the standard error of the mean.

Figure 2. Distribution of predicted easiness scores (proportion correct) for the three models on the validation subset.

Item Difficulty Estimation

See all 17 papers

Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling
Aug 5, 2026Han Chen, Ming Li, Hong Jiao +1Vision-Language Models Educational Assessment
Mohamed bin Zayed University of Artificial Intelligence · University of Maryland
Estimating Item Difficulty with Large Language Models as Experts
May 18, 2026Diana Kolesnikova, Kirill Fedyanin, Abe D. Hofman +2LLM Evaluation LLM-as-a-Judge
Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands · Smart Business Technologies, Belgrade, Serbia · Department of Psychological Methods, University of Amsterdam, Amsterdam, Netherlands +2
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
Jun 17, 2026Han Chen, Ming Li, Chenguang Wang +4LLM Evaluation Educational Assessment
MBZUAI · University of Maryland · Virginia Tech

Aug 5, 2026J/K move · Enter open · S save

Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling

Han Chen, Ming Li, Hong Jiao, Tianyi Zhou

Mohamed bin Zayed University of Artificial Intelligence · University of Maryland

Vision-Language Models Educational Assessment Item Difficulty Estimation

Open paper AskPDF arXiv

Predicting item difficulty from content can provide an initial estimate for newly developed questions before sufficient student responses are available. Existing approaches typically represent the question stem and answer choices as text. When mathematics items contain visual components, a common pipeline first textualizes that evidence and then applies a text predictor. We ask: how should visual evidence be represented for item difficulty prediction? We compare question text alone, visual textualization, which expresses visual evidence in language, and image-native modeling, which retains the original image. Using Eedi items with difficulty calibrated from student responses, we train large language models (LLMs) and vision-language models (VLMs) directly for difficulty regression. Both visual interfaces achieve the lowest point estimates, although the leading systems cannot be reliably ordered. Open-VLM textualization yields lower RMSE point estimates for all evaluated LLMs, while broader adaptation does so for all image-native VLMs. Test-time interventions show dependence on the paired full-item image, but do not isolate the additional visual component. The two visual interfaces also make partially complementary item-level errors and differ substantially in computational workflow. Thus, textualization should not be treated as the only practical interface: image-native modeling is a competitive alternative whose effectiveness depends on how the VLM is adapted.

Abstract

Item Difficulty Estimation

Explore similar work

Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling

Estimating Item Difficulty with Large Language Models as Experts

LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment