Psychometric Property

Psychometric properties, the reliability and validity of measurements, are being rigorously investigated in the context of large language models (LLMs). Current research focuses on adapting existing psychological questionnaires and scales for LLMs, employing techniques like natural language inference and Item Response Theory (IRT) to analyze LLM responses and compare them to human data, often using latent variable modeling. This work aims to understand the nature of LLM "personalities" and biases, assess their capabilities, and improve their trustworthiness and safety by leveraging psychometric principles. The findings have implications for both the development of more robust and reliable AI systems and the advancement of psychological measurement itself.