cs.CVFeb 19, 2026

Sketch2Feedback: Grammar-in-the-Loop Framework for Rubric-Aligned Feedback on Student STEM Diagrams

Authors: Aayam Bansal

Organizations: IEEE

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Providing timely, rubric-aligned feedback on student-drawn diagrams is a persistent challenge in STEM education. While large multimodal models (LMMs) can jointly parse images and generate explanations, their tendency to hallucinate undermines trust in classroom deployments. We present Sketch2Feedback, a grammar-in-the-loop framework that decomposes the problem into four stages -- hybrid perception, symbolic graph construction, constraint checking, and constrained VLM feedback -- so that the language model verbalizes only violations verified by an upstream rule engine. We evaluate on two synthetic micro-benchmarks, FBD-10 (free-body diagrams) and Circuit-10 (circuit schematics), each with 500 images spanning standard and hard noise augmentation tiers, comparing our pipeline against end-to-end LMMs (LLaVA-1.5-7B, Qwen2-VL-7B), a vision-only detector, a YOLOv8-nano learned detector, and an ensemble oracle. On n=100 test samples per benchmark with 95% bootstrap CIs, results are mixed and instructive: Qwen2-VL-7B achieves the highest micro-F1 on both FBDs (0.570) and circuits (0.528), but with extreme hallucination rates (0.78, 0.98). An ensemble oracle that selects the best prediction per sample reaches F1=0.556 with hallucination 0.320 on FBDs, demonstrating exploitable complementarity between grammar and end-to-end approaches. Confidence thresholding at tau=0.7 reduces circuit hallucination from 0.970 to 0.880 with no F1 loss. Hard noise augmentation reveals domain-dependent robustness: FBD detection is resilient while circuit detection degrades sharply. An LLM-as-judge evaluation confirms that the grammar pipeline produces more actionable circuit feedback (4.85/5) than the end-to-end LMM (3.11/5). We release all code, datasets, and evaluation scripts.

Figures & tables

Figure 1: Sketch2Feedback pipeline overview. Stage 1 : Hybrid CV perception detects primitives (arrows, wires, components, junctions) via CLAHE preprocessing, adaptive thresholding, contour analysis, and HoughLinesP. Stage 2 : Detected primitives form a typed symbolic graph G=(V,E)G=(V,E) with spatial proximity edges. Stage 3 : Domain-specific constraints are checked against the scenario key, producing a list of verified violations. Stage 4 : Only verified violations are passed to a compact VLM (Qwen2-VL-2B), which generates rubric-aligned feedback. The VLM cannot fabricate errors the constraint checker did not detect.
DataModelMi-F1PRCorr.Act.Hall. ↓\downarrow
FBDGrammar.263 [.13,.35].385.2003.363.65.375
E2E.471 [.36,.62].571.4003.914.35.375
Vis..077 [.18,.45]1.00.0402.202.05.000
Circ.Grammar.329.522.2402.955.00.925
E2E.038.333.0203.814.03.750
Vis..000.000.0001.702.20.200
Table 1: Main results on FBD-10 and Circuit-10 test sets ( n=40n{=}40 each). Best values per metric in bold . 95% bootstrap CIs in brackets. Grammar+VLM uses Qwen2-VL-2B; E2E-LMM uses LLaVA-1.5-7B.
DataError TypeGram.E2EVis.
FBDwrong direction.500.000.000
missing force.000.745.182
extra force.000.154.000
Circ.missing ground.558.091.000
missing component.000.000.000
wrong polarity.000.000.000
Table 2: Per-error-type F1 scores. Zeros indicate the model never correctly detected that error type.
Figure 2: Model complementarity across error types. The grammar pipeline excels at structural constraint violations (wrong direction, missing ground), while the E2E-LMM detects omission-type errors (missing force). Neither model detects missing components or wrong polarity, indicating a shared perception bottleneck.
More in this topic

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 227 papers

Explore similar work

CardsList
Feb 12, 2026

Beyond Pixels: A Vector-to-Graph Framework for Reliable Schematic Auditing

Multimodal Large Language Models (MLLMs) have shown remarkable progress in visual understanding, yet they suffer from a critical limitation: structural blindness. Even state-of-the-art models fail to capture topology and symbolic logic in engineering schematics, as their pixel-driven paradigm discards the explicit vector-defined relations needed for reasoning. To overcome this, we propose a Vector-to-Graph (V2G) pipeline that converts CAD diagrams into property graphs where nodes represent components and edges encode connectivity, making structural dependencies explicit and machine-auditable. On a diagnostic benchmark of electrical compliance checks, V2G yields large accuracy gains across all error categories, while leading MLLMs remain near chance level. These results highlight the systemic inadequacy of pixel-based methods and demonstrate that structure-aware representations provide a reliable path toward practical deployment of multimodal AI in engineering domains. To facilitate further research, we release our benchmark and implementation at https://github.com/gm-embodied/V2G-Audit.

Chengwei Ma, Zhen Tian, Zhou Zhou +5

Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China · Guangdong Power Grid Co., Ltd., Yangjiang Power Supply Bureau, Yangjiang, China · Shanghai University, Shanghai, China +1

Multimodal Large Language ModelsMultimodal Model Evaluation
AskPDF
Apr 23, 2026

SketchVLM: Vision language models can annotate images to explain thoughts and guide users

When answering questions about images, humans naturally point, label, and draw to explain their reasoning. In contrast, modern vision-language models (VLMs) such as Gemini-3-Pro and GPT-5 only respond with text, which can be difficult for users to verify. We present SketchVLM, a training-free, model-agnostic framework that enables VLMs to produce non-destructive, editable SVG overlays on the input image to visually explain their answers. Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, and object counting) and drawing (part labeling, connecting-the-dots, and drawing shapes around objects), SketchVLM improves visual reasoning task accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x relative to image-editing and fine-tuned sketching baselines, while also producing annotations that are more faithful to the model's stated answer. We find that single-turn generation already achieves strong accuracy and annotation quality, and multi-turn generation opens up further opportunities for human-AI collaboration. An interactive demo and code are at https://sketchvlm.github.io/.

Brandon Collins, Logan Bolton, Hung Huy Nguyen +3

Auburn University · Adobe Research

Vision-Language ModelsLLM-Assisted AnnotationVLM ReasoningVisually Grounded ReasoningMultimodal Understanding
AskPDF
Aug 9, 2026

Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities

The generation of mathematically precise diagrams from tex- tual prompts has emerged as a critical yet underexplored capability of Large Language Models (LLMs). This has been of interest to researchers in the areas of curriculum preparation, automated ranking of problem sets, and scientific publishing. For LLMs to achieve this, it requires per- fect coordination between Spatial Reasoning, Mathematical Reasoning, and Rendering systems. While existing benchmarks such as MathVision, MathVista are built for Math Reasoning or DiagramGenBenchmark, Mer- maidSeqBench on general purpose diagram generation, no prior work provides a standardized set of prompt, image pairs that can be used to evaluate the LLMs specifically on math diagram generation. This includes fields that span both both text-to-code and text-to-image paradigms. We introduce Math-Vision Diagrams, the first benchmark specifically designed to evaluate LLMs on mathematical diagram generation, and the first to assess text-to-code and text-to-image generation paradigms together in a single unified setting, agnostic of the underlying coding lan- guage or model type. Building on the Math-Vision benchmark, we select a subset of 2920 images out of 3040 from high-quality competition problems with essential visual context. A novel pipeline combining an ensemble of LLMs with Subject Matter Expert (SME) curation is presented, together with a suite of evaluation metrics. Testing several leading models against this benchmark, we demonstrate that LLMs struggle with math diagram generation. All code, data, curation pipeline, and evaluation scripts will be fully open-sourced.

Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy +2

Pandita AI Inc., 2540 N First St, San Jose CA 95131, USA

LLM EvaluationT2I Generation Evaluation
AskPDF