cs.AINov 17, 2026

A Visual Information Extraction System Using Multimodal Large Language Models for Scanned Cognitive and Behavioral Assessments

Authors: Enshuo Hsu, Linh Nguyen, Sunyang Fu, Heath Goodrum, Sandra Stansberry, Kirk Roberts, Hongfang Liu, Yen-Chi L. Le

Organizations: McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA · Enterprise Development and Integration, University of Texas MD Anderson Cancer Center, Houston, Texas, USA · UT Physicians Center for Population Health Management & Quality, The University of Texas Health Science Center at Houston, Houston, Texas, USA · Department of Quantitative and Systems Health Sciences, Dell Medical School, The University of Texas at Austin, Austin, TX, USA.

Abstract

Objective. Paper-based assessments are common in patient care, but handwriting and scanning artifacts make these forms difficult to process at scale. This study proposes a modular, multimodal LLM-based system with human oversight for extracting fields and scores. We conducted a qualitative analysis of the use of LLMs for hand-drawing analysis. Materials and Methods. The system processes cognitive assessments (i.e., MMSE, MoCA, Mini-Cog) and the NICHQ Vanderbilt Assessment using two modules: 1) a form classifier that assigns a scanned page to a form type, followed by a corresponding 2) score extractor that extracts fields and scores into a schema. A human-in-the-loop design utilizing dual-model validation engages human annotators when two LLMs disagree, limiting manual review to challenging pages only. Results. End-to-end accuracy for cognitive assessments was 88%, with most form types exceeding 90%. For the NICHQ Vanderbilt, overall accuracy reached 97.5%, with all form types above 96%. The human-in-the-loop design further improved cognitive assessment accuracy to 97.5% while reducing manual work by 63.8%. In the Mini-Cog Clock Drawing task, qualitative analysis showed that 62.3% of all extraction errors stemmed from ambiguous drawing, in which cases the LLM provided reasonable readings. Discussion. Compared to current manual data extraction, our LLM-based system reduces processing time by 10 to 20 times and costs by over 12 times. The modular design allows seamless expansion to additional form types. Conclusion. Our system provides an accurate, scalable, and cost-efficient solution for structuring data trapped in handwritten paper forms for clinical decision support.

Explore similar work

CardsList
  1. Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring

    May 11, 2026Jiaqing Zhang, Sandeep Elluri, Bhanu Cherukuvada +7LLM AuditingMultimodal Large Language Models

  2. From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

    Apr 14, 2026Nicholas Pather, Joshua Fouché, Sitwala Mundia +5Document Information ExtractionDocument Image Understanding