An Offline Smartphone Check for Drug Interactions and Past Reactions in Handwritten Prescriptions: A Technical Feasibility Study in Rural Bangladeshi Outpatient Care
Organizations: Department of Computer Science and Engineering, United International University, Dhaka, Bangladesh
Abstract
Rural Bangladeshi outpatient clinics keep no patient records, prescriptions are handwritten, and physicians have under a minute per patient, so they easily miss a known adverse drug reaction (ADR) or drug-drug interaction (DDI). We built and evaluated an offline Android application that photographs a handwritten prescription, detects and reads each medicine line, resolves brand names to active ingredients, and retrieves, but never predicts, documented interactions and the patient's recorded reactions. This quantitative technical-feasibility study, with qualitative error analysis, involved no patients or clinicians. Reference data combine two Bangladeshi drug registries (27,880 brands) with 190,421 DDI pairs. A YOLOv8n line detector reached [email protected] = 0.687 on held-out prescriptions but recalled only 60.6% of lines at the deployed cutoff. On-device ML Kit read 11.8% of 780 benchmark handwritten words exactly. We trained a CRNN-CTC reader on public handwriting data, decoded it against the brand lexicon, and fine-tuned it on 660 author-labelled prescription crops. On benchmark words, the reader accepted the correct ingredient for 41.5% and a wrong one for 4.2%, versus 21.0% and 5.3% for ML Kit at its preset cutoff (McNemar p<10^-20). On scorable lines from held-out real prescriptions, the reader accepted the correct ingredient for 36.0% [26.8, 46.3] and a wrong one for 2.2% [0.6, 7.8]. In 62 simulated runs on a mid-range phone, median time per patient was 26-31 s and 59 of 62 runs finished within 60 s, but 62% of medicine lines still needed a manual check. A cross-visit medication history added after the evaluation passed seven automated tests and one two-visit functional test but was not evaluated. The pipeline is feasible within the clinic workflow; reading handwriting, not retrieval, is the bottleneck. Nothing here demonstrates a clinical benefit.
Figures & tables
| Approach | ADR/DDI | LMIC fit | No-records infra. | Rural BD | Main limitation for our setting |
|---|---|---|---|---|---|
| EHR allergy/interaction alerts (high-income) [ 5 , 6 ] | ✓ | ✗ | ✗ | ✗ | Requires full EHR/CPOE; severe alert fatigue |
| LMIC point-of-care CDS / mHealth [ 9 , 10 , 11 ] | ✗ | ✓ | ✓ | ✗ | Targets other conditions; not medication safety |
| Bangladeshi prescription-error studies [ 7 , 8 ] | ✓ | ✓ | Measure the problem; no deployed system | ||
| Handwritten-prescription reading / OCR [ 19 , 20 ] | ✗ | ✓ | ✓ | Reading only; no ingredient resolution, history or check | |
| This work | ✓ | ✓ | ✓ | ✓ | Simulated use only; reading accuracy is the bottleneck |
| Planned axis | Status in this paper |
|---|---|
| Workflow fit (s/patient) | Executed as simulated use by the author: 62 logged runs, two sessions (Sec. VIII-D ). |
| Component accuracy | Executed : detection, reading, resolution, with CIs (Secs. VIII-A – VIII-C ). |
| Retrospective detection of known site cases | Not executed : no site records were used. |
| Identity re-linking via phone number | Not executed : dummy numbers only. |
| Usability / acceptability (e.g. SUS [ 21 ] , interviews) | Not executed : no clinicians took part. |
| Reaction-note reading (history capture) | Implemented, not tested : no reaction note was marked in any logged run. |
| # | Source | Content | Role |
|---|---|---|---|
| 1 | BdMedex [ 22 ] + DGDA registered products [ 23 ] | 21,714 + 41,201 brand rows | Reference: brand-to-ingredient lookup. Never trained on. |
| 2 | DrugBank-derived DDI table [ 24 , 25 , 26 ] | 191,541 drug pairs with text | Reference: interaction knowledge base. Never trained on. |
| 3 | Handwritten prescriptions with medicine-line boxes [ 27 ] | 200 photos, 1,086 boxes (YOLO format) | Detector: train 140 / val 30 / test 30 photos (seed 42). The author also added text labels to the boxes (Sec. IV-B ). |
| 4 | Doctor’s Handwritten Prescription BD [ 20 ] | 4,680 word images, 78 names (+generic) | OCR benchmark only : test split, 780 images. Never trained or tuned on. |
| 5 | RxHandBD [ 19 ] | 5,578 word images, 1,509 distinct texts | We train the handwriting reader on 4,463 images and use the test split (1,115) to select checkpoints and the first cutoff. No image appears in both this dataset and Dataset 4, but 33 of Dataset 4’s 78 names appear in Dataset 5’s training labels. |
| # | Change | Trigger | Effect |
|---|---|---|---|
| 1 | Ingredient canonicalisation (salt, synonyms) | Only 39.3% of dictionary ingredients existed in the DDI table; same drug stored under two spellings | Coverage 60.6%; one spelling per ingredient, so reactions match across brands |
| 2 | Full-dictionary fuzzy match | Scaffold matched only the first 50 brands sharing a 3-letter prefix; OCR gets the first 3 letters right 23% of the time | +1.7 pp ingredient-correct (EasyOCR proxy) |
| 3 | ML Kit first real test | Author’s first photographs: “Napa Extend” read as “e. Nopa Euknd” | 3/3 lines unreadable; correct brand ranked 19th and 39th by fuzzy match |
| 4 | DGDA merge, vet filter, typo snapping | Brands on real prescriptions missing from BdMedex | 13,611 27,880 brands |
| 5 | CRNN-conv reader + lexicon CTC | Need a handwriting model; free text unrecoverable | D4: IC 30.0% / WA 4.6% @0.99 (cutoff read off D4, see #8) |
| 6 | CRNN-LSTM reader | LSTM export to TFLite initially failed; conv fallback overwrote one LSTM run | D4 greedy exact 15.1% 21.3% |
| [email protected] / [email protected]:0.95 | 0.687 / 0.314 |
|---|---|
| Precision / recall (at max-F1 confidence) | 0.845 / 0.564 |
| Box precision / recall at cutoff 0.15 (IoU 0.5) | 71.4% / 60.6% |
| Exact box count per photo | 16.7% [7.3, 33.6] |
| Mean absolute count error | 1.70 boxes |
| Reader | Exact | Cutoff | IC (%) | WA (%) | Top-1 IC (%) | ms/line |
|---|---|---|---|---|---|---|
| EasyOCR + fuzzy (desktop proxy) | — | 0.82 | 17.9 [15.4, 20.8] | 4.5 [3.2, 6.2] | 28.1 | — |
| ML Kit + fuzzy (original) | 11.8% | 0.82 | 21.0 [18.3, 24.0] | 5.3 [3.9, 7.1] | 30.0 | 128 |
| CRNN-conv + lexicon | 15.1% | 0.99 † | 30.0 [26.9, 33.3] | 4.6 [3.4, 6.3] | 54.4 | 160 |
| CRNN-LSTM + lexicon (brand posterior) | 21.3% | 0.9999 | 22.9 [20.1, 26.0] | 2.1 [1.3, 3.3] | 62.3 | 207 |
| CRNN-LSTM + lexicon (ingredient-set) | 21.3% | 0.9999 | 26.8 [23.8, 30.0] | 2.1 [1.3, 3.3] | 62.3 | 198 |
| Fine-tuned LSTM + lexicon (ingredient-set), deployed | 24.5% | 0.995 | 41.5 [38.1, 45.0] | 4.2 [3.0, 5.9] | 63.7 | 200 |
| RxHand-only | Fine-tuned | |
| CER (121 medicine lines) | 0.63 | 0.40 |
| Brand part read exactly | 4.1 [1.8, 9.3] | 16.5 [11.0, 24.2] |
| Top-1 IC (no cutoff) | 33.7 [24.7, 44.0] | 58.4 [48.0, 68.1] |
| IC @0.98 (5% cap on val) | 18.0 [11.4, 27.2] | 44.9 [35.0, 55.3] |
| WA @0.98 | 5.6 [2.4, 12.5] | 7.9 [3.9, 15.4] |
| IC @0.995 (2% cap, deployed) | 14.6 [8.7, 23.4] | 36.0 [26.8, 46.3] |
| S1 ( ) | S2 ( ) | |
| Phone entry | 6.2 [5.6, 7.2] | 6.2 [5.6, 7.4] |
| Photo / pick | 8.2 [6.7, 8.7] | 8.7 [8.1, 9.4] |
| Detection | 1.1 [1.1, 1.1] | 1.8 [1.8, 1.8] |
| Confirm boxes | 7.6 [6.5, 8.3] | 11.1 [9.6, 13.6] |
| Reading (all lines) | 0.9 [0.8, 1.1] | 1.1 [0.9, 1.3] |
| Total per patient | 26.4 [24.0, 33.3] | 30.7 [28.3, 33.7] |
| Visit 1 | Visit 2 | |
|---|---|---|
| History before check | none stored | visit 1 (in window) |
| Read automatically | Pantosil pantoprazole | Voltalin D diclofenac |
| Manual / “Is this X?” | 2 manual | 2 manual; “NAB (diclofenac)?” No |
| Result | not cleared (unreadable) | flagged : diclofenac (today) + pantoprazole (visit 1) |
| Stored for later visits | Pantosil | Voltalin D (NAB not stored) |
| Total time | 200.6 s | 52.9 s |
Healthcare
Healthcare AI develops and evaluates models and systems for clinical care, medical research, public health, and healthcare operations.
See all 617 papers