Offline-First Quantized AI: Design and Evaluation of an On-Device Maternal-Health Assistant for Low-Connectivity Environments
Organizations: HelpMum Africa
Abstract
Offline language models could extend access to maternal-health information where internet connectivity is unreliable, but deployment requires both clinically appropriate responses and feasible local execution. This study presents an offline Android assistant integrating ready-made quantized language models, a clinically validated local knowledge package, conversational context, and model-initiated local retrieval. Gemma 4 E2B and Qwen3.5 0.8B were compared across 45 conversations and 312 turns per configuration, including a dedicated offline agentic benchmark. Gemma achieved higher response-quality, context-retention, and safety scores; Qwen offered faster generation and lower memory use. Clinician assessment of 150 Gemma responses across nine clinical domains yielded a mean quality score of 3.80/4, with 147 responses meeting the acceptance criterion. A paired RAG-on/off experiment evaluated 74 questions: retrieval improved 11 responses, left 56 neutral, produced four adverse response changes, and left three pairs unassessable. RAG-on had descriptively higher mean usefulness, safety, correctness, and actionability in the conditional 70-pair subset excluding the four flagged pairs from both conditions. Physical-device testing demonstrated offline responses on a 6 GB Android handset, with a 14.3 s median latency, alongside successful reconnection synchronization in the integrated application. Together, the findings document clinical response quality and practical performance of the integrated offline maternal-health assistant.
Figures & tables
| Level | Scenario | Required response |
|---|---|---|
| E0 | Routine education without an apparent warning sign. | Answer the supported question clearly and provide relevant clinical guidance. |
| E1 | Unclear, unsupported, incomplete, or conflicting information. | State the boundary; seek the minimum useful clarification and recommend care when appropriate. |
| E2 | Potential warning sign with uncertain severity or urgency. | Prioritize safety and prompt assessment; limit clarification to details necessary for the action. |
| E3 | Emergency concern, including severe bleeding, seizure, loss of consciousness, or severe breathing difficulty. | Advise immediate in-person emergency care. Avoid reassurance, autonomous treatment, and extended troubleshooting. |
| Domain | Items | Clinical and communication focus |
|---|---|---|
| Pregnancy milestones | 15 | Early changes, pregnancy timing, normal variation, and antenatal engagement. |
| Antenatal care | 20 | Visits, tests, preparation, prior findings, and practical access barriers. |
| Nutrition and exercise | 15 | Diet, activity, pica, weakness, and context-sensitive advice. |
| Common concerns | 15 | Symptom explanation, severity, timing, and appropriate assessment. |
| Medication and exposure | 15 | Treatment boundaries, uncertain exposures, supplements, and unsupported claims. |
| Postpartum and newborn care | 20 | Recovery, feeding, newborn concerns, and escalation when needed. |
| Dataset | Composition | Purpose and unit of analysis |
|---|---|---|
| Clinical questions | 150 questions | Nine-domain bank with reviewed facts, actions, references, and critical-error conditions. Candidate collections contain 150 answers per model; the clinical scoring workbook assesses 150 Gemma answers. |
| Conversation workload | 45 conversations; 312 turns per model | Ordered exchanges assess retained context, changing urgency, and benchmark-only offline actions. Ratings use conversations; completion and execution errors use turns. |
| Paired retrieval cohort | 74 matched questions | Selected-model answers with retrieval enabled and disabled. Coverage uses all 150 questions; the conditional display includes 70 pairs and the separate audit retains four flagged cases. |
| Retrieval verification | 25 known-item queries | Checks passage recovery in the original normalized package. This component test is separate from clinical response scoring and from coverage in the revised active package. |
| Execution measurements | Workload-specific records | Android CPU emulator latency, throughput, memory, completion, and action errors remain linked to their workloads. A separate handset sample assesses integrated application execution and synchronization. |
| Preparation stage | Count | Role in the knowledge resource |
|---|---|---|
| Source export | 36,636 | Original maternal-health and vaccination conversation records. |
| Normalized question–answer records | 36,548 | Complete records formed during initial preparation; 88 source records remained unsplit. |
| Clinically reviewed content | 23,977 | Reviewed rows, including ten supplemental coverage records; original and final text remain linked through the audit. |
| Active retrieval passages | 21,673 | Clinically validated complete question–answer passages admitted to the current package. |
| Excluded reviewed rows | 2,304 | Retained outside the active corpus for audit and disposition review. |
| Clinical source register | 28 | Source entries covering Nigerian and international institutional guidance. |
| Measure | Gemma | Qwen |
| Conversation turn, s | 16.75 | 7.84 |
| Isolated response, s | 12.74 | 7.90 |
| Isolated throughput, tok/s | 7.86 | 21.62 |
| Peak process RSS, GB | 4.23 | 1.69 |
| Clean turns / 312 | 297 | 312 |
| Clinical request | Required response | Failure and corrective target |
|---|---|---|
| Headache with visual changes | Prompt assessment without waiting for optional details. | Referral became conditional. Preserve urgency before adding education. |
| Severe nocturnal palm and sole itching | Preserve the reviewed prompt-assessment advice. | Advice shifted toward a routine visit. Match the full symptom and urgency pattern. |
| Maternal and newborn records | Explain which records to retain. | Nutrition displaced record keeping. Check the requested action before using evidence. |
| Pregnancy dating | Explain uncertainty and clinical confirmation. | Blood hCG was overemphasized as a usual dating method. Remove unsupported precision. |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Configuration | Artifact and precision | Runtime and controls |
|---|---|---|
| Gemma 4 E2B | gemma-4-E2B-it. litertlm ; mobile mixed 2/4/8-bit scheme | LiteRT-LM Android 0.18.0; native tokenizer/template; CPU |
| Qwen3.5 0.8B | Qwen3.5-0.8B- Q4_K_M.gguf ; four-bit class | Android llama.cpp; GGUF tokenizer/template; CPU |
| Paired RAG run | Same selected Gemma artifact and 21,673-passage package in both arms | 6 GB Android guest; temperature 0.0; 512 output tokens; seed 17; tools disabled |