Large language models (LLMs) play an increasing role in programming education, challenging traditional assessment methods as they can generate fully functional code instantly. This disrupts the reliability of student submissions as indicators of learning and effort. To address this issue, code provenance is essential for tracing the origins of code and distinguishing student-authored, AI-generated, and externally sourced components. We extend the Min-K% probability approach by incorporating synthetic non-member code generation and apply it to Python code for membership inference and provenance tracking. Evaluations cover free LLMs such as GPT-2, DistilGPT-2, and DialoGPT, as well as state-of-the-art commercial models including GPT-5, Gemini 2.5, Mistral, DeepSeek, and LLaMA 3.3. Before evaluating our approach on higher-education student assignments, we validate its feasibility using Stack Overflow data. The results show that free models enable more sensitive code memorization detection than commercial models. By systematically comparing these LLMs, we assess trade-offs between detection accuracy and privacy protection, taking an initial step toward effective assessment of AI-assisted programming education through combined provenance detection and process visibility.
Figures & tables
Figure 1: Workflow for detecting LLM-generated code in students’ code using the Min-K% membership inference attack.
Programming Concept
Frequency
Tuples, Lists, Aliasing, Mutability, Cloning
∼ 1.5M
String manipulation, Guess and Check, Approximations, Bisection
∼ 1.26M
Decomposition, Abstractions, Functions
∼ 993K
Branching and iterations
∼ 963K
Recursion, Dictionaries
∼ 806K
Testing, Debugging, Exceptions, Assertions
∼ 596K
Table 1: Distribution of key Python programming concepts and the number of programming questions related to each concept in the ProCQA dataset.
Model
AUC
TPR@5%FPR
Precision
F1
DialoGPT-medium
0.577
9.24
0.564
0.543
DialoGPT-large
0.585
10.28
0.608
0.471
GPT-2
0.664
18.19
0.651
0.631
DistilGPT-2
0.713
17.24
0.680
0.674
Gemini-2.5-Flash
0.654
6.16
0.567
0.666
Mistral-Small-24B-Instruct
0.535
7.29
0.509
0.639
Table 2: Performance metrics of general-purpose LLMs on membership inference evaluation. The top portion presents free LLMs, while the bottom portion lists paid/commercial LLMs. The highest scores within each group are highlighted in bold .
Figure 2: We show AUC performance across various LLMs plotted against their model size (number of parameters, in billions (B)). The models are grouped into free and paid categories to illustrate differences in membership inference detection effectiveness relative to model scale.
Group
AUC
TPR@5%FPR
Precision
F1
Group 1
0.584
6.25
0.723
0.780
Group 2
0.594
7.03
0.895
0.757
Group 3
0.640
8.24
0.688
0.700
Group 4
0.644
7.86
0.542
0.644
Group 5
0.629
7.10
0.233
0.368
Table 3: Performance of the enhanced Min-K% method on student group submissions.
The rapid proliferation of large language models (LLMs) in the context of education has introduced significant challenges in enforcement of academic integrity, especially in programming courses. We present Argus, an automated detection system for LLM-assisted student work in undergraduate C programming assignments. Argus integrates behavioral and stylistic indicators to create a holistic picture of the student's progress through an assignment and surfaces anomalies that point to potential misuse of LLM assistance. We quantify and analyze data over six years of Spring semester offerings in a large-enrollment CS2 course at Purdue University using Argus, finding that 45% of enrolled students exhibited patterns consistent with LLM-assisted code development in Spring 2026. To contextualize these results, we analyze the relationship between flagged LLM use and student performance on written, in-person proctored examinations, and find a significant negative correlation. We also explore the problem of mitigating false positives, recognizing that erroneous accusations of academic integrity carry significant consequences for students and instructors alike, particularly in the context of large enrollment courses. We argue that any automated detection system must be accompanied by a structured process for human review. We discuss the consequences for future course design, changing academic policy as these tools become more ubiquitous, and the pedagogical implications of LLM-based tools in computer science education.
The rapid emergence of Large Language Models (LLMs) presents both opportunities and challenges for programming education. While students increasingly use generative AI tools, direct access often hinders the learning process by providing complete solutions rather than pedagogical hints. Concurrently, educators face significant workload and scalability challenges when providing timely, personalized feedback. This study investigates the capabilities of LLMs to safely and effectively assist educators in answering student questions within a CS1 programming course. To achieve this, we established a rigorous, reproducible evaluation process by curating a benchmark dataset of 170 authentic student questions from a learning management system, paired with ground-truth responses authored by subject matter experts. Because traditional text-matching metrics are insufficient for evaluating open-ended educational responses, we developed and validated a custom LLM-as-a-Judge metric optimized for assessing pedagogical accuracy. Our findings demonstrate that models, such as Gemini 3 flash, can surpass the quality baseline of typical educator responses, achieving high alignment with expert pedagogical standards. To mitigate persistent risks like hallucination and ensure alignment with course-specific context, we advocate for a "teacher-in-the-loop" implementation. Finally, we abstract our methodology into a task-agnostic evaluation framework, advocating for a shift in the development of educational LLM tools from ad-hoc, post-deployment testing to a quantifiable, pre-deployment validation process.
The rapid proliferation of Large Language Models (LLMs) in software development has made distinguishing AI-generated code from human-written code a critical challenge with implications for academic integrity, code quality assurance, and software security. We present LLMSniffer, a detection framework that fine-tunes GraphCodeBERT using a two-stage supervised contrastive learning pipeline augmented with comment removal preprocessing and an MLP classifier. Evaluated on two benchmark datasets - GPTSniffer and Whodunit - LLMSniffer achieves substantial improvements over prior baselines: accuracy increases from 70% to 78% on GPTSniffer (F1: 68% to 78%) and from 91% to 94.65% on Whodunit (F1: 91% to 94.64%). t-SNE visualizations confirm that contrastive fine-tuning yields well-separated, compact embeddings. We release our model checkpoints, datasets, codes and a live interactive demo to facilitate further research.
Mahir Labib Dihan, Abir Muhtasim
Department of Computer Science and Engineering Bangladesh University of Engineering and Technology (BUET) Dhaka, Bangladesh