A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments
Authors: Jing Huang, Jihong Zhang, Hua-Hua Chang
Organizations: College of Education, Purdue University, West Lafayette, USA · Educational Statistics & Research Methods, University of Arkansas, Fayetteville, USA
The rapid expansion of large-scale assessments and the growing adoption of automatic item generation have intensified concerns about incidental content redundancy, where construct-irrelevant elements such as wording or contextual framing become unintentionally repetitive across items. Traditional similarity metrics like BLEU or cosine similarity, often fail to capture the nuanced structural and semantic layers that drive perceived redundancy simultaneously. This study proposes a dual-dimensional framework for Automated Item Similarity Analysis (AISA) powered by Large Language Models (LLMs), operationalizing similarity through Structured Decomposition and Semantic Relatedness. Psychometric validation indicates that LLM-derived metrics align more closely with indicators of construct-irrelevant local dependence and yield more coherent item parameter groupings than traditional text-based measures. The framework is further evaluated through its application in Computerized Adaptive Testing (CAT). Simulations reveal that incorporating LLM-based similarity constraints into item selection improves estimation stability and reduces bias with minimal efficiency trade-offs, outperforming constraints based on conventional metrics. These findings highlight the potential of LLM-powered AISA to support scalable bank curation, content-aware test assembly, and experience-sensitive adaptive testing across diverse assessment contexts.
Figures & tables
Role Setup:
You are a researcher working on one study of content analysis of survey. You aim to evaluate the similarity of the item content.
Task:
Using the 36-items the Experiences in Close Relationships (ECR) scale shown in <item samples>. Calculate pairwise Structured decomposition similarity between items based on <rules>.
Components:
1. Item format (10 points) : similarity in layout or phrasing patterns (e.g., question type, use of options). 2. Sentence structure (10 points) : similarity in grammatical constructions or word order. 3. Lexical overlap and paraphrasing (10 points) : use of shared vocabulary, stems, or synonymous expressions.
Rules:
• Take the provided <BLEU similarity> matrix as an extra information into consideration, and do not recalculate the BLEU. • The final Structured decomposition similarity relies on your own judgment on the three <Components> in this prompt. • The final similarity is number ranging from 0 to 30, which adds up all scores of components together. For example, if two items have 4 points, 3 points, 3 points for three components. The final similarity score of structured decomposition is 10 points. Lower value indicating lower similarity of structured decomposition and semantic relatedness or vice versa.
Output:
Return a JSON array of objects, each containing: ‘item_i’, ‘item_j’, ‘score on Overlapping themes or contexts’, ‘score on General lexical semantics’, ‘score on Shared connotations or emotional tone’, ‘final_similarity’, ‘explanation’ (a brief but important explanation of how the scores were determined).
Table 1: Prompt Template for Structured Decomposition Similarity Calculation
1. I prefer not to show a partner how I feel deep down.
2. I worry about being abandoned.
3. I am very comfortable being close to romantic partners.
4. I worry a lot about my relationships.
5. Just when my partner starts to get close to me, I find myself pulling away.
6. I worry that romantic partners won’t care about me as much as I care about them.
7. I get uncomfortable when a romantic partner wants to be very close.