CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems

Wanyong Feng; Alexander Scarlatos; Ruochen Sun; Andrew Lan

cs.CLFeb 23, 2026

CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems

Authors: Wanyong Feng, Alexander Scarlatos, Ruochen Sun, Andrew Lan

Organizations: University of Massachusetts Amherst · Independent Researcher

Abstract

Existing Computerized Adaptive Testing (CAT) frameworks typically select questions based on the predicted likelihood that the student will answer correctly. This design ignores information contained in students' open-ended responses, especially in domains such as programming education, where code structures and bugs contain rich information on student knowledge. In this work, we propose Code GENerative CAT (CodeGENCAT), a generative CAT framework that selects questions using predicted student code responses. First, we develop a Generative Item Response Theory (GIRT) model that generates code responses conditioned on estimated student knowledge, trained with supervised fine-tuning followed by direct preference optimization for knowledge-response alignment. Second, we introduce three question-selection algorithms that measure uncertainty, coding style diversity, and information from predicted student code responses. Experiments on two real-world programming education datasets show that CodeGENCAT outperforms all CAT baselines, achieving an AUC improvement of up to 4.32% over the strongest baseline in the early stages of adaptive testing.

Figures & tables

Figure 1: Overview of one CodeGENCAT cycle with Diversity as the question selection algorithm.

Framework	Step $t$
	CodeWorkout							ProgFeed
	$1$	$3$	$5$	$7$	$10$	$15$	$20$	$1$	$3$	$5$	$7$	$10$	$15$	$20$
AUC $\uparrow$
CodeGENCAT (Uncertainty)	0.6737	$0.7258^{*}$	0.7415	0.7599	0.7693	0.7838	0.7873	0.6375	0.6725	0.6875	0.6930	0.7020	0.7063	0.7101
CodeGENCAT (Diversity)	0.7073	$\textbf{0.7352}^{*}$	$\textbf{0.7529}^{*}$	$\textbf{0.7632}^{*}$	0.7687	0.7801	0.7855	0.6437	0.6319	0.6834	0.6870	0.6901	0.7064	0.7161
CodeGENCAT (Information)	0.6854	0.7264	0.7492	0.7616	0.7718	0.7842	0.7901	0.6730	0.6845	0.6910	0.7080	0.7056	0.7204	0.7191

Table 1: 5-fold averaged knowledge estimate performance (AUC and Accuracy) on both datasets across varying test lengths (

t

). CodeGENCAT consistently outperforms baselines, particularly in the early stages (

t=1

7

), with Diversity, and late stages (

t=10

20

) with Information on the CodeWorkout dataset. CodeGENCAT consistently outperforms baselines for most time steps with Information on the ProgFeed dataset. * denotes statistically significant improvement over the best baseline using a paired sample t-test (p < 0.05). For AUC, the best baselines are 1PL IRT and NCAT, and for accuracy, the best baselines are NCAT and LACAT, for CodeWorkout and ProgFeed respectively. Best results are in bold and second best are underlined .

Method	Exp.% (med) $\downarrow$	Over.% (mean) $\downarrow$
CodeGENCAT (Uncertainty)	12.80	25.33
CodeGENCAT (Diversity)	11.60	36.21
CodeGENCAT (Information)	15.60	35.41
1PL IRT	10.60	28.20
LACAT	6.40	25.92
NCAT	41.40	58.92

Table 2: 5-fold averaged median Exposure rate (Exp.) and mean overlap rate (Over.) at step

t=5

on CodeWorkout dataset. Best results in bold and second best are underlined .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

AUC $\uparrow$
Method	Step $t$
	CodeWorkout							ProgFeed
	$1$	$3$	$5$	$7$	$10$	$15$	$20$	$1$	$3$	$5$	$7$	$10$	$15$	$20$
1B Uncertainty	0.6737	0.7258	0.7415	0.7599	0.7693	0.7838	0.7873	0.6375	0.6725	0.6875	0.6930	0.7020	0.7063	0.7101
3B Uncertainty	0.6612	0.7121	0.7531	0.7626	0.7746	0.7845	0.7893	0.6091	0.6582	0.6789	0.6852	0.7081	0.7175	0.7200
1B Diversity	0.7073	0.7352	0.7529	0.7632	0.7687	0.7801	0.7855	0.6437	0.6319	0.6834	0.6870	0.6901	0.7064	0.7161

Appendix

Table 3: 5-fold averaged CodeGENCAT results under different GIRT backbone sizes (1B vs. 3B) across question selection algorithms. Bold indicates the better result between 1B and 3B for the same question selection algorithm at each step.

AUC $\uparrow$
Method	CodeWorkout
Method	$1$	$3$	$5$	$7$	$10$	$15$	$20$
DPO Uncertainty	0.6612	0.7121	0.7531	0.7626	0.7746	0.7845	0.7893
SFT Uncertainty	0.6768	0.7146	0.7389	0.7632	0.7749	0.7870	0.7893
DPO Diversity	0.7235	0.7355	0.7547	0.7673	0.7777	0.7833	0.7910
SFT Diversity	0.7079	0.7277	0.7444	0.7563	0.7691	0.7821	0.7872

Appendix

Table 4: 5-fold averaged CodeGENCAT results under different GIRT training stages (SFT vs. DPO) across question selection algorithms. Bold indicates the better result between SFT and DPO for the same question selection algorithm at each step.

AUC $\uparrow$
Method	CodeWorkout
Method	$1$	$3$	$5$	$7$	$10$	$15$	$20$
Sample 3 Uncertainty	0.6918	0.7329	0.7522	0.7616	0.7734	0.7772	0.7834
Sample 10 Uncertainty	0.6612	0.7121	0.7531	0.7626	0.7746	0.7845	0.7893
Sample 3 Diversity	0.6927	0.7424	0.7645	0.7703	0.7746	0.7856	0.7876
Sample 10 Diversity	0.7235	0.7355	0.7547	0.7673	0.7777	0.7833	0.7910

Appendix

Table 5: 5-fold averaged CodeGENCAT results under different sampling numbers (3 vs. 10) across question selection algorithms. Bold indicates the better result between _3 and _10 for the same question selection algorithm at each step.

Model	Method	Step $t$
		CodeBLEU $\uparrow$
		CodeWorkout							ProgFeed
		$1$	$3$	$5$	$7$	$10$	$15$	$20$	$1$	$3$	$5$	$7$	$10$	$15$	$20$
3B	Uncertainty	0.5598	0.5807	0.5845	0.5827	0.5902	0.5899	0.5915	0.4852	0.4865	0.4855	0.4855	0.4850	0.4850	0.4846
	Diversity	0.5638	0.5797	0.5845	0.5887	0.5898	0.5905	0.5901	0.4855	0.4861	0.4852	0.4855	0.4874	0.4847	0.4848
	Information	0.5669	0.5775	0.5855	0.5901	0.5906	0.5917	0.5947	0.4860	0.4861	0.4853	0.4863	0.4859	0.4858	0.4860

Appendix

Table 6: 5-fold averaged Code similarity results (CodeBLEU) on CodeWorkout and ProgFeed across varying test lengths (

t

) under the 1B and 3B GIRT models and baselines. Best results are in bold .

Model	Method	Step $t$
		Pearson Correlation $\uparrow$
		CodeWorkout							ProgFeed
		$1$	$3$	$5$	$7$	$10$	$15$	$20$	$1$	$3$	$5$	$7$	$10$	$15$	$20$
3B	Uncertainty	0.1909	0.2518	0.2635	0.2640	0.2758	0.2862	0.2821	0.1353	0.1454	0.1502	0.1231	0.1246	0.1301	0.1307
	Diversity	0.2117	0.2721	0.2747	0.2769	0.2839	0.2871	0.2815	0.1348	0.1493	0.1597	0.1359	0.1296	0.1318	0.1390
	Information	0.1799	0.2714	0.2831	0.3095	0.2996	0.3029	0.3206	0.1439	0.1621	0.1266	0.1403	0.1297	0.1549	0.1334

Appendix

Table 7: 5-fold averaged Pearson correlation coefficient results on CodeWorkout and ProgFeed across varying test lengths (

t

) under the 1B and 3B GIRT models and baselines. Best results are in bold .

Student KC mastery	Code
0	public int noTeenSum(int a, int b, int c) { a = fixTeen(a); b = fixTeen(b); c = fixTeen(c); return a + b + c; } public int fixTeen(int n) { if (n >= 13 && n <= 19) { if (n == 15) { n = 0; } else { n = 0; } } else { n = n; } return n; }
0.5	public int noTeenSum(int a, int b, int c) { a = fixTeen(a); b = fixTeen(b); c = fixTeen(c); return a + b + c; } public int fixTeen(int n) { if (n >= 13 && n <= 19) { if (n == 15 \|\| n == 16) { return n; } else { return 0; } } else { return n; } }
1	public int noTeenSum(int a, int b, int c) { return (fixTeen(a) + fixTeen(b) + fixTeen(c)); } public int fixTeen(int n) { if(n >= 13 && n <= 19) { if(n == 15 \|\| n == 16) { return n; } else { return 0; } } else { return n; } }

Appendix

Table 8: Generated code for problem noTeenSum using the DPO-refined model across varying KC mastery levels.

Student KC mastery	Code
0	public int noTeenSum(int a, int b, int c) { a = fixTeen(a); b = fixTeen(b); c = fixTeen(c); return a + b + c; } public int fixTeen(int n) { if (n >= 13 && n <= 19) { if (n == 15) { return 15; } else if (n == 16) { return 16; } else { return 0; } } else { return n; } }
0.5	public int noTeenSum(int a, int b, int c) { a = fixTeen(a); b = fixTeen(b); c = fixTeen(c); return a + b + c; } public int fixTeen(int n) { if (n >= 13 && n <= 19) { if (n == 15 \|\| n == 16) { return n; } else { return 0; } } else { return n; } }
1	public int noTeenSum(int a, int b, int c) { a = fixTeen(a); b = fixTeen(b); c = fixTeen(c); return a + b + c; } public int fixTeen(int n) { if (n >= 13 && n <= 19) { if (n == 15 \|\| n == 16) { return n; } else { return 0; } } else { return n; } }

Appendix

Table 9: Generated code for problem noTeenSum using the SFT baseline model across varying KC mastery levels.

System: You are a helpful assistant. You generate plausible student-written Java code given a coding problem and the student knowledge of knowledge components (KCs) that are related to the problem. Correctness is not required — the code should reflect the student partial understanding, and may contain inefficiencies, bugs, or non-idiomatic style. User: Given a Java coding problem and the student knowledge on several knowledge components (KCs) that are related to the problem, your job is to generate the code that the student would write to solve the problem. Follow these rules carefully: 1. Output only one complete method or function in Java. If a student is unlikely to have knowledge of KCs, the generated code may be incomplete. 2. Do not include any explanations, comments, or formatting symbols (such as backticks). 3. Correctness is not required. The goal is to produce plausible student code influenced by the provided knowledges. Inefficiency, minor bugs, and convoluted logic are expected if the student is unlikely to have knowledge of KCs. This rule is the most important!!!! The Java coding problem is:<question> The KC 1 is: <kc1>. The student has knowledge of <kc1>: <|placeholder|>. The KC 2 is: <kc2>. The student has knowledge of <kc2>: <|placeholder|>.

\dots

The KC

K

is: <kcK>. The student has knowledge of <kcK>: <|placeholder|>. The code that the student writes to solve this problem based on their knowledge of these KCs is:

Appendix

Table 10: Prompt for the GIRT model.

System: You are a teacher scoring student written code to a java coding question. User: Score the student written code to the java coding question by only outputting a single integer between 0 and 1 inclusive. Question: <question> Student written code: <code> Score:

Appendix

Table 11: Prompt for the scoring model.

Explore similar work

AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models
Jun 20, 2026Weiyuan Zhou, Haiping Ma, Xiaoshan Yu +3Multi-Agent LLM Systems Educational Assessment
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Institute of Physical Science and Information Technology, Anhui University, China · School of Artificial Intelligence, Anhui University, China · School of Computer Science and Technology, Dalian University of Technology, China +1
ACE-TA: An Agentic Teaching Assistant for Grounded Q&A, Quiz Generation, and Code Tutoring
Feb 20, 2026Himanshu Tripathi, Charlottee Crowell, Kaley Newlin +3Intelligent Tutoring Systems Generative AI in Education
University of Alabama, Tuscaloosa, Alabama, USA · Brown University, Providence, Rhode Island, USA · Meharry Medical College, Nashville, Tennessee, USA +1
CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation
Apr 5, 2026Xiaojing Duan, Frederick Nwanganga, Chaoli WangGenerative AI in Education Human-in-the-Loop AI
University of Notre Dame, Notre Dame IN 46556, USA

Jun 20, 2026J/K move · Enter open · S save

AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models

Weiyuan Zhou, Haiping Ma, Xiaoshan Yu, Changqian Wang+2

State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Institute of Physical Science and Information Technology, Anhui University, China · School of Artificial Intelligence, Anhui University, China · School of Computer Science and Technology, Dalian University of Technology, China+1

Multi-Agent LLM Systems Educational Assessment AI Agent Benchmarks

Open paper AskPDF arXiv

Computerized Adaptive Testing (CAT), as a key technology for personalized education, aims to accurately assess examinee proficiency by retrieving exercises dynamically matching current ability estimates. However, existing CAT research is constrained by limitations of static offline data and isolated component optimization. Restricted by partial labels in offline logs, researchers degrade the dynamic assessment process into static sequence prediction. Current research focuses on isolated perspectives, e.g., selection or diagnosis, neglecting the overall CAT interaction process. To address this, we propose AgentCAT, a Large Language Model-based multi-agent simulation system, to construct a high-fidelity benchmarking environment for dynamic testing. This framework comprises three modules: (1) The examinee agent with memory retrieval and Chain-of-Thought reasoning simulates responses based on cognitive profiles; (2) The selection agent uses coarse-to-fine bucketing and knowledge graph exploration to balance local difficulty and global coverage; (3) The supervisor uses dual-auditing and robust update to ensure convergence and validity. To validate the framework, we evaluated on two real-world datasets across three dimensions: macro-level ability convergence, micro-level interaction logic, and data sparsity resilience. Results show AgentCAT achieves effective ability estimation, and its selection strategy balances difficulty adaptation and instructional coherence, aligning with human pedagogical intuition.

Abstract

Explore similar work

AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models

ACE-TA: An Agentic Teaching Assistant for Grounded Q&A, Quiz Generation, and Code Tutoring

CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation