cs.AINov 10, 2026

Benchmarking LLM Next-Step Recommendations with Knowledge Space Theory

Authors: Peter Steiner, Jan Hochweber

Organizations: Institute of Educational Psychology, St.Gallen University of Teacher Education (PHSG), St.Gallen, Switzerland

Abstract

Selecting an LLM for next-step recommendation requires benchmarks that reflect the intended educational task and domain. We present a benchmarking method that uses Knowledge Space Theory (KST) as a formal evaluation reference. Given a task catalogue and an existing prerequisite relation, the method derives admissible knowledge states and the complete set of immediate next steps for each state. Recommendations generated by a model under a specified prompt are compared with these reference sets. A sample evaluation study uses 2,400 responses from six models across four mathematics domains, four prompt conditions, and two catalogue orders. Exact-set accuracy was 61.3% with descriptions only, 62.8% when models were asked to reconstruct prerequisites, 96.3% with supplied prerequisite relations, and 98.2% with the complete knowledge space. Performance varied across models and domains, and recommendation sets agreed across the two presentations in 85.25% of pairs. The benchmark supports comparisons of model-prompt combinations against a context-specific reference. Its scores describe structural recommendation accuracy under the evaluated conditions; they do not establish learning benefit or the necessity of using an LLM for selection.

Figures & tables

Explore similar work

CardsList
  1. LLMRec: Benchmarking Large Language Models on Recommendation Task

    Aug 23, 2023Junling Liu, Chao Liu, Peilin Zhou +8Large Language Model-Based RecommendationRecommender Systems

  2. RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation

    Sep 1, 2026Zhongru Chen, Yuan Wu, Yi ChangLLM EvaluationLarge Language Model-Based Recommendation

  3. Knowledge Index of Noah's Ark

    Jun 3, 2026Sheng Jin, Minghao Liu, Yunze Xiao +24LLM EvaluationHuman-in-the-Loop Annotation