Difficulty Measure

Difficulty measure in machine learning and game development focuses on quantifying the inherent challenge posed by individual data points or tasks, aiming to improve model training efficiency and game design. Current research explores diverse approaches, including leveraging large language models as game testers, developing standardized benchmark datasets with difficulty annotations (using methods like Item Response Theory), and creating objective difficulty metrics based on factors like gradient variance or sample generalization error. These advancements have implications for optimizing model training through curriculum learning and sample weighting, as well as for creating more engaging and adaptable games through procedural difficulty adjustment.

Papers

October 1, 2024

LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
Chang Xiao, Brenda Z. Yang
Large Language Model Autonomous Agent LLM Agent Strategy Game Game Testing Expertise Level Difficulty Measure

September 27, 2024

Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Anima Anandkumar, Furong Huang
Language Model Strong Generalization MT Bench LLM Performance Difficulty Measure

August 27, 2024

CL4KGE: A Curriculum Learning Method for Knowledge Graph Embedding
Yang Liu, Chuan Zhou, Peng Zhang, Yanan Cao, Yongchao Liu, Zhao Li, Hongyang Chen
Knowledge Graph Curriculum Learning Borda Counting Difficulty Measure

April 9, 2024

A Lightweight Measure of Classification Difficulty from Application Dataset Characteristics
Bryan Bo Cao, Abhinav Sharma, Lawrence O'Gorman, Michael Coss, Shubham Jain
Neural Network Model Efficient Model Class Similarity Characterization Data Difficulty Measure Classification Difficulty

July 7, 2023

Procedurally generating rules to adapt difficulty for narrative puzzle games
Thomas Volden, Djordje Grbic, Paolo Burelli
Simple RULE Difficulty Level Game Narrative Puzzle Game Difficulty Measure Educational Game

February 2, 2023

Human not in the loop: objective sample difficulty measures for Curriculum Learning
Zhengbo Zhou, Jun Luo, Dooman Arefan, Gene Kitamura, Shandong Wu
Human Generated Curriculum Learning Difficulty Measure Irreducible Curriculum

January 12, 2023

Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure
Xiaoling Zhou, Ou Wu, Weiyao Zhu, Ziyang Liang
Deep Learning Difficulty Measure Difficulty Based Weighting

May 16, 2022

Exploring the Learning Difficulty of Data Theory and Measure
Weiyao Zhu, Ou Wu, Fengguang Su, Yingjun Deng
Machine Learning Bias Variance Difficulty Measure Difficulty Based Weighting

Difficulty Measure

Papers

LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents

Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization

CL4KGE: A Curriculum Learning Method for Knowledge Graph Embedding

A Lightweight Measure of Classification Difficulty from Application Dataset Characteristics

Procedurally generating rules to adapt difficulty for narrative puzzle games

Human not in the loop: objective sample difficulty measures for Curriculum Learning

Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure

Exploring the Learning Difficulty of Data Theory and Measure