Large language models (LLMs) play an increasing role in programming education, challenging traditional assessment methods as they can generate fully functional code instantly. This disrupts the reliability of student submissions as indicators of learning and effort. To address this issue, code provenance is essential for tracing the origins of code and distinguishing student-authored, AI-generated, and externally sourced components. We extend the Min-K% probability approach by incorporating synthetic non-member code generation and apply it to Python code for membership inference and provenance tracking. Evaluations cover free LLMs such as GPT-2, DistilGPT-2, and DialoGPT, as well as state-of-the-art commercial models including GPT-5, Gemini 2.5, Mistral, DeepSeek, and LLaMA 3.3. Before evaluating our approach on higher-education student assignments, we validate its feasibility using Stack Overflow data. The results show that free models enable more sensitive code memorization detection than commercial models. By systematically comparing these LLMs, we assess trade-offs between detection accuracy and privacy protection, taking an initial step toward effective assessment of AI-assisted programming education through combined provenance detection and process visibility.
Figures & tables
Figure 1: Workflow for detecting LLM-generated code in students’ code using the Min-K% membership inference attack.
Programming Concept
Frequency
Tuples, Lists, Aliasing, Mutability, Cloning
∼ 1.5M
String manipulation, Guess and Check, Approximations, Bisection
∼ 1.26M
Decomposition, Abstractions, Functions
∼ 993K
Branching and iterations
∼ 963K
Recursion, Dictionaries
∼ 806K
Testing, Debugging, Exceptions, Assertions
∼ 596K
Table 1: Distribution of key Python programming concepts and the number of programming questions related to each concept in the ProCQA dataset.
Model
AUC
TPR@5%FPR
Precision
F1
DialoGPT-medium
0.577
9.24
0.564
0.543
DialoGPT-large
0.585
10.28
0.608
0.471
GPT-2
0.664
18.19
0.651
0.631
DistilGPT-2
0.713
17.24
0.680
0.674
Gemini-2.5-Flash
0.654
6.16
0.567
0.666
Mistral-Small-24B-Instruct
0.535
7.29
0.509
0.639
Table 2: Performance metrics of general-purpose LLMs on membership inference evaluation. The top portion presents free LLMs, while the bottom portion lists paid/commercial LLMs. The highest scores within each group are highlighted in bold .
Figure 2: We show AUC performance across various LLMs plotted against their model size (number of parameters, in billions (B)). The models are grouped into free and paid categories to illustrate differences in membership inference detection effectiveness relative to model scale.
Group
AUC
TPR@5%FPR
Precision
F1
Group 1
0.584
6.25
0.723
0.780
Group 2
0.594
7.03
0.895
0.757
Group 3
0.640
8.24
0.688
0.700
Group 4
0.644
7.86
0.542
0.644
Group 5
0.629
7.10
0.233
0.368
Table 3: Performance of the enhanced Min-K% method on student group submissions.