Beyond Simple Input-Output Assessment Tasks: Leveraging Automated Programming Assessment for Non-Trivial Courses

Active learning is widely recognized as an effective approach for improving learning outcomes in introductory programming courses. However, insufficient instructional support often limits students' access to timely, personalized feedback, which is crucial for mastering foundational programming concepts. Although recent advances in AI, particularly large language models, offer scalable opportunities for feedback, concerns about explainability and reliability remain. In this paper, we present an AI-driven classroom assistant that leverages an explainable AI model to analyze student code, map logical errors to instructor-identified misconceptions, and deliver instructor-authored feedback, thereby grounding reliability in instructor-defined pedagogical knowledge. To evaluate the effectiveness of our framework, we conducted an expert evaluation to examine its alignment with instructor-verified feedback and deployed the system in a classroom setting to assess students' perceptions of its usability. Results indicate that the assistant can provide accurate, instructor-verified feedback to students while fostering a positive experience.

AI-native course assessments in senior computer science courses and related fields should grade students by \emph{AI-resilient skill}: the ability to achieve outcomes beyond a strong AI baseline. Such assessments should allow students to use AI freely, while reducing the extent to which greater private AI budget or more intensive AI use, by itself, becomes a grading advantage. This paper proposes a minimal formal framework for this goal. The framework specifies a real task, an executable evaluator, a declared AI-native Pareto frontier, and a grading rule based on Pareto surplus. The central claim is simple: Pareto surplus provides a measurable, protocol-relative certificate that a submitted artifact achieves a tradeoff not already supplied by the declared AI baseline, and grading by this surplus is AI-resilient with respect to that baseline. Interpreting surplus as evidence of student skill requires the surrounding assessment protocol--for example, design reports, ablations, prompt traces, oral checks, or reproducibility explanations--but the grading certificate itself is behavioral and executable. The framework is then extended to practical complications, including self-improving AI loops, budget neutrality, server-mediated feedback, and prompt-based red teaming. As a concrete instantiation, we describe an AI-resilient approximate-membership assignment centered on Bloom filters for COMP 480/580 at Rice University, designed to test whether students can improve beyond AI-generated implementations.

The rapid adoption of generative AI has created new opportunities for teaching, learning, and quality assurance. Existing applications, however, remain largely student-facing, with comparatively limited attention to institution-level needs. This paper presents a course intelligence platform deployed across more than 100 universities and serving over 10,000 instructors in China. By linking competency requirements, knowledge structures, teaching activities, and assessment evidence, it establishes a shared foundation for knowledge organization, instructional design, learning assessment, and quality evaluation. The course evaluation module is examined as a representative institution-facing application of the platform, which integrates national evaluation standards, structured educational evidence, customized prompting strategies, and domain-adapted LLMs to generate quantitative scores and qualitative feedback. A case study involving 100 authentic university courses is conducted to evaluate its alignment with expert judgments and the interpretability of its outputs. Statistical analyses show substantial agreement between AI-generated assessments and expert ratings, while qualitative results highlight the credibility of the feedback. The findings further suggest that AI-assisted evaluation requires not only capable models but also structured domain knowledge and transparent criteria. In this context, human ratings should be treated as an informative reference rather than an error-free gold standard, and the objective is to achieve consistent, interpretable, and defensible judgments instead of merely replicating expert scores.