cs.CRMay 8, 2025

LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities

Authors: Kalyan Nakka, Jimmy Dani, Ausmit Mondal, Nitesh Saxena

Organizations: SPIES Research Lab, Dept. of CSE, Texas A&M University

Abstract

The growing adoption of Large Language Models (LLMs) has influenced the development of Small Language Models (SLMs) for on-device deployment across smartphones and edge devices, offering enhanced privacy, reduced latency, server-free functionality, and improved user experience. However, due to on-device resource constraints, SLMs undergo size optimization through compression techniques like quantization, which inadvertently introduce fairness, ethical and privacy risks. Critically, quantized SLMs may respond to harmful queries directly, without requiring adversarial manipulation, raising significant safety and trust concerns. To address this, we propose LiteLMGuard, an on-device guardrail that provides real-time, prompt-level defense for quantized SLMs. Additionally, our guardrail is designed to be model-agnostic such that it can be seamlessly integrated with any SLM, operating independently of underlying architectures. Our LiteLMGuard formalizes deep learning (DL)-based prompt filtering by leveraging semantic understanding to classify prompt answerability for SLMs. Built on our curated Answerable-or-Not dataset, LiteLMGuard employs ELECTRA as the candidate model with 97.75% answerability classification accuracy. The on-device deployment of LiteLMGuard enabled real-time offline filtering with over 85% defense-rate against harmful prompts (including jailbreak attacks), 94% filtering accuracy and ~135 ms average latency. These results demonstrate LiteLMGuard as a lightweight robust defense mechanism for effectively and efficiently securing on-device SLMs against Open Knowledge Attacks.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GUARD-SLM: Token Activation-Based Defense Against Jailbreak Attacks for Small Language Models

    Mar 28, 2026Md Jueal Mia, Joaquin Molto, Yanzhao Wu +1Jailbreak DefenseJailbreak Detection

  2. Distilling Safe LLM Systems via Soft Prompts for On Device Settings

    Jun 8, 2026Motasem Alfarra, Cristina Pinneri, Dana Kianfar +2On-Device Language Model InferenceLLM Safety Alignment

  3. MOMAT: Mixture of Multiple Atlases for Low-Power Jailbreak Defense of Quantized LLMs

    Oct 1, 2026Boyang Li, Bingyu Shen, Weihao Hong +6Retrieval-Augmented GenerationLLM Safety