Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs

Md. Rafiul Biswas; George Mikros; Wajdi Zaghouani

cs.CLNov 5, 2026

Compression Through a Context Lens: Evaluating Adaptation, Quantization, and Distillation in Arabic Legal LLMs

Authors: Md. Rafiul Biswas, George Mikros, Wajdi Zaghouani

Organizations: Hamad Bin Khalifa University, Doha, Qatar · Northwestern University in Qatar, Doha, Qatar

Abstract

Model compression can facilitate the deployment of large language models in resource-constrained legal settings, but its effects on domain-specific performance may depend on the evaluation protocol. We investigate this issue using ArabLegalEval, a Saudi Arabic legal benchmark, under both with-context and no-context evaluation. The former provides supporting legal passages, while the latter assesses performance without external context. Removing the context produces accuracy gaps of up to 27 percentage points for the base model. We examine three stages: domain adaptation, quantization, and knowledge distillation across FANAR-9B, JAIS-8B, and Qwen2.5-7B. Four-bit quantization reduces model memory by 64-71%, with changes in no-context MCQ accuracy ranging from -3.2 to +1.6 percentage points. Distillation produces more variable results, with accuracy generally declining as student size decreases. A Qwen2.5-7B student distilled from FANAR achieves 76.6% no-context accuracy, compared with 77.2% for its teacher. Our findings suggest that no-context evaluation provides a useful complement to context-supported testing when assessing the effects of adaptation and compression on Arabic legal language models.

Figures & tables

Figure 1: Overview of the proposed methodology.

		MCQ Accuracy (%)			QA ROUGE-L
Model	Variant	With-Ctx	No-Ctx	Gap ( $\Delta$ )	With-Ctx	No-Ctx
FANAR-9B	Base	97.8	70.8	$+27.0$	0.0083	0.0058
	CPT	42.6	74.8	$-32.2$	0.1700	0.1450
	SFT	98.8	77.2	$+21.6$	0.1700	0.0490
	CPT $\to$ SFT	98.8	79.2	$+19.6$	0.1683	0.0637
JAIS-8B	Base	79.2	71.6	$+7.6$	0.0194	0.0179

Table 1: Stage 1 results for three base models and four adaptation variants. MCQ accuracy is reported on the fixed 500-item set and QA ROUGE-L on 200 items;

\Delta

denotes With-Ctx minus No-Ctx. Bold marks the best variant per model and protocol (ties both bolded). Raw counts and ROUGE-1 are given in Appendix Table 4

				MCQ Accuracy (%)
Model	Method	Mem. (GB)	Compr. (%)	With-Ctx	No-Ctx	$\Delta$ No-Ctx
FANAR-9B	BF16 (Baseline)	16.36	—	98.8	79.2	—
	INT8	8.61	47.4	99.2	79.4	$+0.2$
	INT4 NF4	4.73	71.1	99.0	79.4	$+0.2$
	INT4 FP4	4.73	71.1	98.2	79.2	$0.0$
JAIS-8B	BF16 (Baseline)	15.07	—	98.6	75.8	—

Table 2: Stage 2 quantization results on the fixed 500-item MCQ evaluation set. BF16 baselines are the CPT

\to

SFT models from Table 1 . Compr. denotes the percentage reduction in model memory relative to BF16, and

\Delta

No-Ctx indicates the change in no-context accuracy in percentage points.

				MCQ Accuracy (%)
Teacher	Model	Params	BF16 Mem. (GB)	With-Ctx	No-Ctx
FANAR-9B	Teacher (SFT)	8.7B	16.36	98.8	77.2
	Qwen2.5-7B	7.62B	$\approx$ 14.19	97.8	76.6
	Qwen2.5-3B	3.09B	$\approx$ 5.76	87.4	73.6
	Qwen2.5-1.5B	1.54B	$\approx$ 2.87	81.4	59.4
	Qwen2.5-0.5B	0.49B	$\approx$ 0.92	53.2	52.8

Table 3: Stage 3 distillation results on the fixed 500-item MCQ evaluation set. Teachers are the Stage 1 SFT checkpoints (Table 1 ). Qwen and JAIS use same-family students, whereas FANAR is distilled into Qwen2.5 students. Student BF16 memory values marked

\approx

are weights-only estimates; teacher memory values are measured.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: With-context and no-context MCQ prompts. The prompts are identical except for the instruction and the inclusion of the {context} context in the with-context setting.

Model	Setting	Experiment	MCQ Accuracy (%)	MCQ Correct	QA ROUGE-1	QA ROUGE-L
FANAR	With-Ctx	Exp A (Base)	97.8	489/500	0.0083	0.0083
		Exp B (CPT)	42.6	213/500	0.1700	0.1700
		Exp C (SFT)	98.8	494/500	0.1700	0.1700
		Exp D (CPT $\to$ SFT)	98.8	494/500	0.1683	0.1683
	No-Ctx	Exp A (Base)	70.8	354/500	0.0058	0.0058
		Exp B (CPT)	74.8	374/500	0.1450	0.1450

Appendix

Table 4: Stage 1 results for FANAR, JAIS, and Qwen under with-context and no-context evaluation. MCQ results are reported on 500 items; bold denotes the best-performing variant within each model and setting.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 294 papers

Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model
Sep 21, 2026MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan +4Retrieval-Augmented Generation Legal NLP
Computer Science and Engineering BRAC University Dhaka, Bangladesh
Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark
Aug 31, 2026Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa +3Multilingual Language Model Evaluation LLM Evaluation
North South University · BRAC University
Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval
Feb 18, 2026Subrit DikshitSmall Language Models Legal IR
Department of Computer Science and Engineering, Indian Institute of Information Technology, Pune, India · Director of AI & Data Science, Compunnel Inc

Sep 21, 2026J/K move · Enter open · S save

Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model

MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan, Nadifa Zaman+3

Computer Science and Engineering BRAC University Dhaka, Bangladesh

Retrieval-Augmented Generation Legal NLP On-Device Language Model Inference Language Model Distillation Legal QA

Open paper AskPDF arXiv

Legal information in Bangladesh is inaccessible to most citizens. Statutory text is English-only, trained lawyers are concentrated in urban centres, and cloud-dependent AI fails where mobile connectivity is unreliable, a setting in which hallucinated legal text causes direct harm. The system addresses statutory interpretation only; queries that require judicial precedent or case-law reasoning fall outside its scope. We target the statutory access gap by compressing a 9-billion-parameter Gemma-2 teacher into a 2-billion-parameter student through two-phase progressive knowledge distillation. Phase 1 performs supervised fine-tuning on 9,429 quality-gated legal question-answer pairs (65% acceptance from 14,514 generated queries); Phase 2 minimises sparse Kullback-Leibler divergence against the teacher's top-50 per-token logits at temperature tau = 4.0, implemented via QLoRA (4-bit NF4, rank-32 LoRA adapters). Prior legal language models target general legal English; this system specialises in Bangladeshi statutory law. Every response is grounded through hybrid retrieval combining dense semantic search (60%) and BM25 (40%) across 36,029 statutory passages from the Bangladesh Constitution and national legislation. On a 50-query English benchmark, the distilled model reaches ROUGE-L 0.4715 and BERTScore F1 0.5679, a 103% ROUGE-L and 143% BERTScore gain over the retrieval-augmented undistilled baseline (ROUGE-L 0.2323, BERTScore 0.2340). The adapter quantises to 1.6 GB (GGUF Q4_K_M) and runs at 4-8 tokens per second on a Pixel 6 with no network access. Cross-lingual evaluation on 50 Bangla queries yields ROUGE-L 0.4083 and BERTScore 0.8133, showing effective retrieval from Bangla input against an English-only corpus. In a single-evaluator pilot, a practising lawyer rated 50 responses at a weighted mean of 4.16/5 (90% rated 4 or 5), supporting utility beyond text-overlap metrics.

Abstract

LLM Compression

Explore similar work

Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model

Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark

Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval