Offline language models could extend access to maternal-health information where internet connectivity is unreliable, but deployment requires both clinically appropriate responses and feasible local execution. This study presents an offline Android assistant integrating ready-made quantized language models, a clinically validated local knowledge package, conversational context, and model-initiated local retrieval. Gemma 4 E2B and Qwen3.5 0.8B were compared across 45 conversations and 312 turns per configuration, including a dedicated offline agentic benchmark. Gemma achieved higher response-quality, context-retention, and safety scores; Qwen offered faster generation and lower memory use. Clinician assessment of 150 Gemma responses across nine clinical domains yielded a mean quality score of 3.80/4, with 147 responses meeting the acceptance criterion. A paired RAG-on/off experiment evaluated 74 questions: retrieval improved 11 responses, left 56 neutral, produced four adverse response changes, and left three pairs unassessable. RAG-on had descriptively higher mean usefulness, safety, correctness, and actionability in the conditional 70-pair subset excluding the four flagged pairs from both conditions. Physical-device testing demonstrated offline responses on a 6 GB Android handset, with a 14.3 s median latency, alongside successful reconnection synchronization in the integrated application. Together, the findings document clinical response quality and practical performance of the integrated offline maternal-health assistant.
Figures & tables
Level
Scenario
Required response
E0
Routine education without an apparent warning sign.
Answer the supported question clearly and provide relevant clinical guidance.
E1
Unclear, unsupported, incomplete, or conflicting information.
State the boundary; seek the minimum useful clarification and recommend care when appropriate.
E2
Potential warning sign with uncertain severity or urgency.
Prioritize safety and prompt assessment; limit clarification to details necessary for the action.
E3
Emergency concern, including severe bleeding, seizure, loss of consciousness, or severe breathing difficulty.
Early changes, pregnancy timing, normal variation, and antenatal engagement.
Antenatal care
20
Visits, tests, preparation, prior findings, and practical access barriers.
Nutrition and exercise
15
Diet, activity, pica, weakness, and context-sensitive advice.
Common concerns
15
Symptom explanation, severity, timing, and appropriate assessment.
Medication and exposure
15
Treatment boundaries, uncertain exposures, supplements, and unsupported claims.
Postpartum and newborn care
20
Recovery, feeding, newborn concerns, and escalation when needed.
TABLE II: Clinical question bank domain profile
Dataset
Composition
Purpose and unit of analysis
Clinical questions
150 questions
Nine-domain bank with reviewed facts, actions, references, and critical-error conditions. Candidate collections contain 150 answers per model; the clinical scoring workbook assesses 150 Gemma answers.
Conversation workload
45 conversations; 312 turns per model
Ordered exchanges assess retained context, changing urgency, and benchmark-only offline actions. Ratings use conversations; completion and execution errors use turns.
Paired retrieval cohort
74 matched questions
Selected-model answers with retrieval enabled and disabled. Coverage uses all 150 questions; the conditional display includes 70 pairs and the separate audit retains four flagged cases.
Retrieval verification
25 known-item queries
Checks passage recovery in the original normalized package. This component test is separate from clinical response scoring and from coverage in the revised active package.
Execution measurements
Workload-specific records
Android CPU emulator latency, throughput, memory, completion, and action errors remain linked to their workloads. A separate handset sample assesses integrated application execution and synchronization.
TABLE III: Evaluation datasets and their analytical roles
Preparation stage
Count
Role in the knowledge resource
Source export
36,636
Original maternal-health and vaccination conversation records.
Normalized question–answer records
36,548
Complete records formed during initial preparation; 88 source records remained unsplit.
Clinically reviewed content
23,977
Reviewed rows, including ten supplemental coverage records; original and final text remain linked through the audit.
Active retrieval passages
21,673
Clinically validated complete question–answer passages admitted to the current package.
Excluded reviewed rows
2,304
Retained outside the active corpus for audit and disposition review.
Clinical source register
28
Source entries covering Nigerian and international institutional guidance.
TABLE IV: Local knowledge package composition and clinical provenance
Fig. 1: Offline architecture with model-initiated retrieval. The generator answers directly or requests the allowlisted local tool. Deterministically admitted evidence returns through a bounded tool loop; conversation history, clinical response policy, and separately stored citation metadata retain distinct roles.
Fig. 2: Clinical single-response evaluation. Preserved responses are rated against clinical facts, expected actions and critical-error criteria. Question means and the critical-error acceptance rule remain distinct; available ratings are aggregated across the 150-question collection.
Fig. 3: Multi-turn benchmark assessment separates LLM judgments from scripted action scoring. The action interface is used only for evaluation. Complete transcripts and runtime evidence are identity-masked; quality, context and safety use 0–4, action behavior uses -5 to +5, and critical errors remain separate.
Fig. 4: Normalized rubric means for 45 conversations per model. Response quality, context retention, and safety use 0–4 scales. Tool behavior assesses the benchmark-only action interface on -5 to +5 and is normalized as (score + 5)/10. Percentages are rescaled ratings, not accuracy or deployment action-success rates.
Fig. 5: Quality averages five normalized benchmark behavior measures, including action-tool performance; resources average memory headroom, relative speed, and clean benchmark-turn completion. The shaded region and dashed segment are project decision illustrations rather than fitted or clinically validated thresholds. The component definitions are given in Appendix B.
Measure
Gemma
Qwen
Conversation turn, s
16.75
7.84
Isolated response, s
12.74
7.90
Isolated throughput, tok/s
7.86
21.62
Peak process RSS, GB
4.23
1.69
Clean turns / 312
297
312
TABLE V: Android CPU emulator results. Candidate artifacts and native runtimes differ. Clean-turn counts include benchmark action checks.
Fig. 6: Mean clinician ratings for 150 Gemma responses on five 0–4 dimensions. Error bars show the question-level bootstrap 95% intervals in the analysis. Item means use available reviewer ratings.
Fig. 7: Domain-level breakdown of correctness, completeness, safety, relevance, and clarity. Colors summarize mean ratings on the common 0–4 scale; question counts differ across domains as reported in Table II .
Fig. 8: Conditional comparison of the same 70 paired questions after excluding four audit-flagged IDs from both conditions. Scores assess usefulness, safety, correctness, and actionability; outcome bars retain the project retrieval-effect categories. The four excluded cases are reported in Table VI and the complete 74-pair estimates in Section IV-E . The subset is selected by observed outcomes.
Clinical request
Required response
Failure and corrective target
Headache with visual changes
Prompt assessment without waiting for optional details.
Referral became conditional. Preserve urgency before adding education.
Severe nocturnal palm and sole itching
Preserve the reviewed prompt-assessment advice.
Advice shifted toward a routine visit. Match the full symptom and urgency pattern.
Maternal and newborn records
Explain which records to retain.
Nutrition displaced record keeping. Check the requested action before using evidence.
Pregnancy dating
Explain uncertainty and clinical confirmation.
Blood hCG was overemphasized as a usual dating method. Remove unsupported precision.
TABLE VI: Audit-flagged retrieval failures and corrective targets
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Configuration
Artifact and precision
Runtime and controls
Gemma 4 E2B
gemma-4-E2B-it. litertlm ; mobile mixed 2/4/8-bit scheme
LiteRT-LM Android 0.18.0; native tokenizer/template; CPU
Qwen3.5 0.8B
Qwen3.5-0.8B- Q4_K_M.gguf ; four-bit class
Android llama.cpp; GGUF tokenizer/template; CPU
Paired RAG run
Same selected Gemma artifact and 21,673-passage package in both arms
Emergencies can occur where there is no phone signal and no ambulance nearby. A small language model on a phone can be helpful in such situations, but its advice needs to be safe and accurate. We built HandsOnAid, a 4-bit Gemma 2B model fine-tuned with QLoRA on FirstAidQA dataset, a corpus of 5,550 first-aid question-answer pairs. We compared seven configurations, including the base, 4-bit and 8-bit fine-tuned models, and versions with a length floor, safety gate, or retrieval. We used a 41-pair question-answer bank with medically reviewed references that followed ANZCOR guidelines, and a strict 0-5 score rubric, with twelve safety overrides. Four frontier LLM judges (GPT-5.6 Sol, Claude Opus 5, DeepSeek V4 Pro, GLM-5.3) blindly scored every answer, with 45 hidden controls used to check judge reliability. Fine-tuning improved the mean score by +1.08 over the base model to 2.34. The best configuration scored 2.51 out of 5 which was the fine-tuned 4-bit model with retrieval. A medical professional also conducted a blind evaluation of 40 answers across the base, the 4-bit fine-tuned, and the retrieval configurations, and the scores were similar to the LLM judges' average scores. A keyword audit found limited coverage of several details of ANZCOR guidelines in the training corpus, that are required by the rubric, which suggests that missing information in the training data could be a cause of the low scores. On a smartphone device (Pixel 7 Pro) with six CPU threads and no network connection, the 4-bit fine-tuned model produced its first displayable text in 6.86 seconds and a complete answer in 14.88 seconds on average over ten questions.
Rezwan Islam Salvi, Saiyma Sittul Muna, Mushfiqur Rahman Mushfique
Islamic University of Technology Dhaka, Bangladesh
The Offline AI Modules workstream enables practical, low-power, and community-accessible deployment of voice-first AI systems that operate fully offline. Designed for African language communities where speech is the dominant mode of interaction and internet connectivity is unreliable or absent, the workstream delivers three reinforcing components: a modular voice-first offline architecture, a low-cost hardware reference bill of materials, and a reproducible quantization and a reproducible quantization and benchmarking pipeline for instruction-tuned language models in the 2-5B parameter class. This paper presents the first end-to-end benchmark evaluation of the stack across two hardware tiers: an NVIDIA Jetson Orin NX (TierB) and a Raspberry Pi5 (TierA). Three instruction-tuned models are evaluated across four quantization formats, assessed for deployment metrics (decode throughput, chat latency, memory, power) and multilingual quality (topic classification accuracy on MasakhaNEWS across English, Hausa, Igbo, Nigerian Pidgin, and Yoruba; per-language perplexity drift). Speech recognition is evaluated using Ethio-ASR on Amharic and Oromo across both tiers. The principal finding is that Q4_K_M quantization represents the best size-to-quality trade-off for deployment on both tiers: gemma-4-E2B-it achieves 28.8t/s decode throughput and 89.2% topic classification accuracy at Q4_K_M on TierB, while all three models run within the 16GB memory budget on TierA.
Maternal and newborn mortality remain among the highest in sub-Saharan Africa, where midwifery care is often delivered by nurses who lack midwifery training to international standards, and consulting authoritative guidance at the point of care is hard: the guidelines are long and connectivity is intermittent. We present MAM-AI, a medical question-answering assistant for nurse-midwives in Zanzibar that runs entirely on a commodity Android device: a question is embedded (EmbeddingGemma, 300M) and matched against a curated corpus of 87 guideline documents (63,650 passages), then answered with citations by a 4B int4 generator (Gemma 4 E4B), fully offline, with no query leaving the device. We evaluate the exact deployed configuration with a layered methodology -- retriever, generator under oracle context, end-to-end, and latency -- scored by LLM judges validated against physician rubrics. The evaluation relocates the hard problem. On-device retrieval is essentially solved: the 300M embedder ranks third of seven retrievers and rivals cloud systems, so the passages the system needs are usually found. The small generator is what remains in doubt: adding retrieved context does not improve its answers, and at 4B it cannot be both helpful and safe at once -- of two same-size candidates, the more helpful one commits genuine dangerous errors, so we deploy the other, which is about twice as faithful to its sources (as faithful as a frontier model), and recover its helpfulness with a redesigned prompt that cuts deflection from 33% to 3%. Corpus quality is decisive for the same reason: where the corpus holds the right passage the answer is specific and actionable, and where it does not it goes vague. MAM-AI is a thoroughly evaluated, open-source research prototype, not a fielded product; the system, knowledge base, benchmarks, and evaluation harness are released.
Yi Ren
École Polytechnique Fédérale de Lausanne, Switzerland