The Ultimate Tutorial for AI-driven Scale Development in Generative Psychometrics: Releasing AIGENIE from its Bottle

Lara Russell-Lasalandra; Hudson Golino; Luis Eduardo Garrido; Alexander P. Christensen

cs.AIMar 30, 2026

The Ultimate Tutorial for AI-driven Scale Development in Generative Psychometrics: Releasing AIGENIE from its Bottle

Authors: Lara Russell-Lasalandra, Hudson Golino, Luis Eduardo Garrido, Alexander P. Christensen

Organizations: Department of Psychology, University of Virginia, Charlottesville, VA 22903, USA · Department of Psychology, Pontificia Universidad Madre y Maestra, Dominican Republic · Department of Psychology, Vanderbilt University, USA

Abstract

Psychological scale development has traditionally required extensive expert involvement, iterative revision, and large-scale pilot testing before psychometric evaluation can begin. The \texttt{AIGENIE} R package implements the AI-GENIE framework (Automatic Item Generation and Validation with Network-Integrated Evaluation), which integrates large language model (LLM) text generation with network psychometric methods to automate the early stages of this process. The package generates candidate item pools using LLMs, transforms them into high-dimensional embeddings, and applies a multi-step reduction pipeline --- Exploratory Graph Analysis (EGA), Unique Variable Analysis (UVA), and bootstrap EGA --- to produce structurally validated item pools entirely \textit{in silico}. This tutorial introduces the package across eight parts: installation and setup, text generation, embeddings, item generation, the full AI-GENIE pipeline, the GENIE pipeline for researcher-supplied items, advanced prompt engineering, and fully local operation. Two running examples illustrate the package's use: the Big Five personality model (a well-established construct) and AI Anxiety (an emerging construct). The package supports multiple LLM providers (OpenAI, Anthropic, Groq, HuggingFace, and local models), offers a fully offline mode with no external API calls, and provides the \texttt{GENIE()} function for researchers who wish to apply the psychometric reduction pipeline to existing item pools regardless of their origin. The \texttt{AIGENIE} package is freely available on CRAN at https://CRAN.R-project.org/package=AIGENIE.

Figures & tables

Figure 1: AI-GENIE Reduction Pipeline. This figure illustrates the entire AI-GENIE pipeline. Step 0, which precedes item reduction, is obtaining the initial item pool, either by generating items with an LLM or by writing them manually. Step 1 obtains the item embeddings using an encoder LLM. Steps 2 through 6 apply network psychometric techniques to reduce the item pool algorithmically.

Provider	Model Types	Cost	Website
OpenAI	text and embedding	“Pay as you go” billing. Prepaid credits are also available.	https://platform.openai.com
Anthropic	text only	Prepaid credits only.	https://console.anthropic.com
Groq	text only	Free base plan with opportunity to upgrade to better rate limits.	https://console.groq.com
HuggingFace	text and embedding	Typically free with costs incurring only in special circumstances.	https://huggingface.co
Jina AI	embedding only	Free for the first 10 million tokens.	https://jina.ai/embeddings

Table 1: API providers compatible with the AIGENIE package

Provider	Notes	Example	Website
OpenAI	Offers state-of-the-art embedding models that have been extensively validated in the AI-GENIE simulations	text-embedding-3-small (the package default)	https://platform.openai.com
HuggingFace	An extremely wide ecosystem of open-source embedding models. Most usage is free.	BAAI/bge-base-en-v1.5	https://huggingface.co
Jina AI	Offers competitive open-source models. The first 10 million tokens are free.	jina-embeddings-v4	https://jina.ai/embeddings

Table 2: Embedding model providers compatible with the AIGENIE package

Figure 2: How AIGENIE Constructs a Prompt in Built-In Mode. The user supplies descriptive components such as domain , scale.title , audience , item.type.definitions , item.examples , and prompt.notes . The package assembles these into a complete prompt and sends it to the LLM, so that the researcher specifies what the items should cover without writing the prompt itself.

Figure 3: Conscientiousness Items: EGA Network Before and After Reduction. The estimated network before item reduction (left) compared with the network after reduction (right). Node color indicates the community assigned by EGA.

Figure 4: Conscientiousness Items: Item Stability Before and After Reduction. Item stability before reduction (left) versus after reduction (right). Stability refers to how consistently each item is assigned to the same community across bootstrap samples, with values closer to 1 indicating more reliable assignment. Stability improves after reduction, and NMI increases by roughly 4 percentage points.

Figure 5: Learning Anxiety Items: EGA Network Before and After Reduction. The estimated network before reduction (left) versus after reduction (right). NMI improved by 9.48 percentage points, reaching a final value of 86.91%.

Figure 6: Job Replacement Items: EGA Network Before and After Reduction. The estimated network before reduction (left) versus after reduction (right). NMI improved by 9.39 percentage points, reaching a final value of 100%.

Figure 7: Sociotechnical Blindness Items: EGA Network Before and After Reduction. The estimated network before reduction (left) versus after reduction (right). NMI improved by 6.88 percentage points, reaching a final value of 90.93%.

Figure 8: AI Configuration Items: EGA Network Before and After Reduction. The estimated network before reduction (left) versus after reduction (right). NMI improved by 9.37 percentage points, reaching a final value of 100%.

Figure 9: Bundled GPT-5.4 Example: Pooled EGA Network Before and After Reduction. The pooled item network before (left) and after (right) type-level reduction for the bundled example. Node color indicates the community assigned by EGA; the six communities correspond to the three attributes within each of the two item types. NMI rises from 98.64 to 100 once the five redundant and unstable items are removed.

Explore similar work

GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing
May 30, 2026Ming Wang, Shuang Wu, Bixuan Wang +7Psychometric Properties Psychology
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · Mental Health Education Center, Northeastern University, Shenyang 110819, China · School of Psychology, Northeast Normal University, Changchun 130024, China +3
Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models
Dec 10, 2024Chang-Jin Li, Jiyuan Zhang, Yun Tang +1Personality
Faculty of Psychology, Beijing Normal University, Beijing, 100875, China · School of Psychology, Central China Normal University, Wuhan, Hubei, 430079, China · Key Laboratory of Adolescent Cyberpsychology and Behavior, Ministry of Education, Wuhan, Hubei, 430079, China +1
Automated Generation of Complexity-Validated Decision Scenarios Using Large Language Models
Aug 9, 2026Abdalla Doleh, Toni Somers, Ratna Babu ChinnamPsychometric Properties Cognitive Science
Department of Industrial & Systems Engineering, Wayne State University, Detroit, MI, 48202, USA · Mike Ilitch School of Business, Wayne State University, Detroit, MI, 48202, USA

May 30, 2026, cs.SIJ/K move · Enter open · S save

GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing

Ming Wang, Shuang Wu, Bixuan Wang, Lu Lin+6

School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · Mental Health Education Center, Northeastern University, Shenyang 110819, China · School of Psychology, Northeast Normal University, Changchun 130024, China+3

Psychometric Properties Psychology Survey Test Generation

Open paper AskPDF arXiv

Self-report questionnaires remain the prevailing tool for probing the psychological states of persona-conditioned agents (PC-Agents). However, classical instruments inherit two well-known threats: contamination from training corpora and directional bias driven by social-desirability or contextual framing. To overcome these methodological bottlenecks, we ask whether projective paradigms can be adapted into a robust psychometric tool. We introduce \textbf{GenPT} (Generative Projective Testing), which reformulates TAT, Rorschach, and SCT with newly generated stimuli and organizes assessment as a three-stage pipeline to derive standardized psychological indicators and target states. Evaluating PC-Agents induced via CharacterRAG and AnnaAgent profiles, we benchmark GenPT's reliability and validity against classical questionnaires. The results indicate that questionnaires exhibit systematic directional shifts under social-desirability framing, most strongly on suicide ideation. In contrast, GenPT's collected behavioral patterns stay near the symmetric baseline. Furthermore, under a longitudinal counselling context, GenPT-based depression assessment shifts by roughly an order of magnitude more than the questionnaire counterpart when Qwen3 serves as the backbone. Overall, GenPT complements self-report methods in scenarios where contamination resistance, bias asymmetry, and context sensitivity matter. Code and stimuli can be found at https://github.com/sci-m-wang/GenPT.

Abstract

Explore similar work

GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing

Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models

Automated Generation of Complexity-Validated Decision Scenarios Using Large Language Models