cs.AIAug 18, 2026

Measuring the Partial-Credit Gap: A Strict Benchmark on Vietnam's 2025 Convex Marking Scheme

Authors: Nguyen Quoc Hung, Nguyen Dang Minh, Le Nhu Quynh, Tran Khanh Linh, Nguyen Kieu Linh

Organizations: Posts and Telecommunications Institute of Technology, Hanoi, Vietnam

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When evaluating language models on human exams, benchmarks typically score each response as right or wrong and report the overall accuracy. This approach assumes that partial knowledge is worth proportional credit, an assumption that fails when an examination uses a non-additive grading scheme. The 2025 reform of Vietnam's National High School Graduation Examination demonstrates the cost of this substitution. In Part II of the exam, candidates evaluate four true/false statements per question. The grading is convex: the number of correct statements earns 0, 0.10, 0.25, 0.50, or 1.00 points. Identifying three statements correctly pays 0.50 points, not the 0.75 points that standard accuracy metrics would award. Because Part II accounts for 4.00 of the exam's 10.00 points, reporting accuracy inflates the score by rewarding partial knowledge that the state explicitly penalizes. We introduce THPT-Ladder, a benchmark of 632 items from 21 official exams across 11 subjects, graded exactly as the ministry grades its students. The ministry publishes the marks of over a million candidates, allowing us to place models directly into the human cohort. Across eight models, the official rubric pays 0.020 to 0.159 points less per Part II question than proportional credit. This shortfall changes a model's apparent competence. For Qwen3.5-27B on the 2025 History exam, a 0.042-point shortfall drops its standing from the 90th to the 77th percentile among 481,293 candidates. A model's accuracy does not predict this penalty. At Claude Sonnet 5's accuracy level, different distributions of errors yield scores varying from 0.869 to 0.932 points per question. Official marks depend on how correct statements are grouped, meaning standard benchmarks report a competence the institution would not certify.

Figures & tables

Figure 1: Marks of the 282,519 candidates who sat Economics & Law in 2026, with all eight models and a fixed answer string placed on the same scale. Each marker is drawn at its mark, with height proportional to the number of candidates who obtained that mark. On this exam the best fixed string is SDSS , which earns 1.75 without reading any question. Scoring under the official rules locates a model within the population the examination was written to assess, an outcome that standard accuracy cannot achieve.
Figure 2: One Part II question from the corpus, as the ministry prints and marks it. The Vietnamese original is on the left, and our English translation is on the right. All four statements are true ( Đ = đúng ). This single question demonstrates the convex grading ladder: judging three out of four statements correctly pays 0.50 points, not the 0.75 points that proportional credit would award. The right panel shows how this rule leaves all eight models short by 0.020 to 0.159 points per Part II question across the corpus. Four models are drawn, showing that their ranking by shortfall differs from their ranking by statement accuracy.
 structureitems 
SubjectIIIIIIrnd20252026Fig.
Biology18462.350282828
Chemistry18462.35028289
Economics & Law24402.72528280
English40002.50040400
Geography18462.35028288
Table 1: Official structure (Decision 764) and corpus coverage.
Figure 3: The marking rule and where the answers actually fall. The state pays the step function, while proportional credit would pay the straight diagonal line. The gap is widest when three out of four statements are correct, paying 0.50 points instead of the implied 0.75 points. Notably, 17.6% of the Part II questions our models answered landed exactly on this rung. The shortfall in Eq. 2 represents the vertical distance between the two lines, weighted by how often each count occurs. Because the mass of answers sits where the gap is widest, the penalty is a real effect rather than a theoretical edge case.
Figure 4: How often each Part II statement is true across the twenty subject-years where we hold the keys. In Mathematics, position (a) is always true in all 96 questions every year. However, in Geography 2026, position (a) is true in only 29% of questions and position (d) in 14%. This shows that the answer keys are not statement-balanced and that the imbalance changes direction depending on the subject. This imbalance allows a fixed answer string to beat random chance without reading a single question.
Figure 5: Each open-weight model’s rank among the candidates who sat the same exam, for the eighteen subject-years where the corpus holds a whole 10.00-mark exam; the closed models’ standings are given in the text. Rows run from the exam the cohort found hardest to the one it found easiest, with the cohort’s own mean mark at the right. Percentiles are lower bounds: a mark on the ministry’s 0.25 grid ties with every candidate who earned it. If difficulty transferred from candidates to models the markers would drift rightwards down the figure; they do not, which indicates that a model’s standing is set by something other than what the exam cost a human.
Figure 6: The same eighteen exams, with the cohort’s mean mark against each open-weight model’s on one 0–10 scale; the closed models’ correlations are given in the text. The diagonal marks equal marks; each dashed horizontal is one model’s own mean. A cloud running parallel to the diagonal would mean difficulty transfers from candidates to models, and a cloud along a horizontal that it does not. Every cloud here is flat: a model’s mark is therefore near-independent of what the exam cost its candidates. No cohort averaged above 7.7, which is why the upper right is empty. Labelled: the cohort’s hardest exam and its easiest.
Model% avail.III stmtII pts/qshf/qIII
Qwen3.5-27B93.80.9730.9260.8840.0420.917
Qwen3.5-9B90.70.9650.8960.8300.0660.850
InternVL3.5-8B69.30.8380.7560.5970.1590.183
Claude Opus 597.00.9880.9730.9490.0240.950
GPT-5.597.00.9880.9670.9480.0200.950
Claude Sonnet 593.60.9770.9350.8810.0540.917
Table 2: Eight models on the 632 scored items: three open-weight, then five closed.
More in this topic

Educational Assessment

Educational assessment measures what learners know and can do, using tests, rubrics, and learning evidence to guide decisions.

See all 151 papers

Explore similar work

CardsList
Jul 27, 2026

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.

Zongyou Yang, Yinghan Hou

Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom

LLM EvaluationLLM Answer VerificationLanguage Model Generation EvaluationLLM Reliability
AskPDF
Sep 24, 2026

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

One long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce; LLM graders are a tempting alternative. To show its pitfalls we grade a practical Computer Vision exam (570570 dual-graded students) under 171171 configurations spanning closed and open-weights models; the best reaches mean absolute error 1.64/351.64/35, below the 2.61/352.61/35 two human graders achieve against each other. The catch is the prompt: a short ''strict grader'' preamble drives 1414 of 1717 open-weights models out of the graded band (MAE≥8\text{MAE} \ge 8), three stopping grading altogether. The damage traces to the preamble's two credit-withholding sentences, not to tone or model scale; one of them, ''never give partial credit'', alone makes two of three probed models stop grading. The closed flagships of three vendors shift calibration under it but stay in the band. In 162162 further configurations on a second, independent Machine Learning exam from another course (1,0381{,}038 dual-graded students), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over-mark: the vulnerability replicates, but its direction is exam-specific. Light LoRA fine-tuning repairs it: one adapter on the two exams' pooled ∼3,900\sim 3{,}900 graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes (≤0.32\le 0.32 MAE). We release the anonymised dataset, full ablation grid, and grading, fine-tuning and analysis pipelines.

Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2

KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · Visual Artificial Intelligence Laboratory Oxford Brookes University · KAUST Academy & CEMSE, KAUST +1

Prompt SensitivityLLM-as-a-JudgeLLM AuditingLanguage Model CalibrationAutomated Grading
AskPDF
Aug 30, 2026

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy), but in educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward. We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy: distractor elimination, abstention, confidence calibration, and self-correction. On LLM benchmarks, these schemes: 1) shift rankings of 31 LLMs beyond rephrased number right prompts; 2) better predict the LLMs users prefer in LLM Arena; and 3) reveal distinct model capabilities, like that GPT-5 rarely abstains and readily self-corrects, while weaker open-weight models often abstain and hesitate to eliminate choices. Given the benefits of alternative scoring schemes, we discuss ways to extend them to tasks beyond MCQA.

Nishant Balepur, Paiheng Xu, Wei Ai +3

University of Maryland · New York University · Nanyang Technological University

LLM EvaluationMultiple-Choice Question AnsweringAutomated GradingLLM Abstention
AskPDF