Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language

Remigiusz Kinas; Paweł Kiszczak; Sergio P. Perez; Krzysztof Ociepa; Łukasz Flis; Krzysztof Wróbel; Adrian Gwoździej

cs.CLMar 12, 2026

Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language

Authors: Remigiusz Kinas, Paweł Kiszczak, Sergio P. Perez, Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej

Organizations: Bielik.AI, Ingenix.ai · Bielik.AI, Vstorm · NVIDIA · Bielik.AI, Azurro.pl ·

Abstract

This report details the creation of Bielik-Minitron-7B, a compressed 7.35B parameter version of the Bielik-11B-v3.0 model, specifically optimized for European languages. By leveraging a two-stage compression methodology inspired by the NVIDIA Minitron approach, we combined structured hybrid pruning and knowledge distillation to reduce the model's parameter count by 33.4%, from 11.04B to 7.35B. We utilized the NVIDIA Model Optimizer for structural pruning and the NVIDIA NeMo Framework for logit-based distillation for quality recovery. Following distillation, the model underwent a rigorous alignment pipeline consisting of Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO-P), and Reinforcement Learning (GRPO). Our final model successfully recovered approximately 90% of the baseline model's performance while providing up to 50% inference speedup. This approach demonstrates an efficient pathway to create language models for less-represented languages, preserving the original model quality while reducing inference deployment costs.

Figures & tables

Method	Granularity	Importance Metric	Retraining	One-Shot	Hardware-Friendly
Magnitude [ 8 ]	Unstructured	$\|w\|$	Required	No	No
SparseGPT [ 9 ]	Unstructured/Semi	Hessian-based	No	Yes	Partial (2:4)
Wanda [ 10 ]	Unstructured/Semi	$\|w\|\cdot\\|x\\|$	No	Yes	Partial (2:4)
ShortGPT [ 11 ]	Structured (Depth)	Block Influence	Optional	Yes	Yes
LLM-Pruner [ 12 ]	Structured	Taylor / $\ell_{1}$ / $\ell_{2}$	Light (LoRA)	Yes	Yes
Sheared LLaMA [ 14 ]	Structured	Learned $\ell_{0}$ masks	Integrated	No	Yes

Table 1: Comparison of LLM pruning methods.

Experiment ID	Hidden size	Intermediate size	Layers	Total Parameters [B]
Original model	4096	14336	50	11.04
EXP_001	3072	—	—	8.28
EXP_002	—	9216	—	7.90
EXP_003	—	—	36	7.99
EXP_004	—	12288	40	7.85
EXP_005	—	10240	—	8.52

Table 2: Model pruning scenarios.

Experiment	Parameters	Reduction	Structural Modifications	Status
Original (11B-v3)	11.04B	—	Baseline Configuration	Baseline
EXP_006	8.07B	26.9%	Layers: 50 $\rightarrow$ 44, Intermediate: 14336 $\rightarrow$ 11264	Sub-optimal
EXP_009	7.26B	34.2%	Intermediate: 14336 $\rightarrow$ 8192	Unstable
EXP_010	7.35B	33.4%	Layers: 40, Intermediate: 11264	Selected

Table 3: Systematic Search: Comparison of Key Pruning Candidates.

Figure 1: Training convergence profiles of distillation loss. The legend indicates the three architectural variants evaluated during the search phase - EXP_006, EXP_009 and EXP_010.

Task Category	Recovery %	Evaluation Metric
Overall Aggregate Score	90.1%	Avg. Benchmark Accuracy
Open PL LLM Leaderboard	94.7%	Average Score
Polish EQ-Bench	90.0%	Average Score
CPTUB	90.6%	Average Score
Polish Medical Leaderboard	88.3%	Accuracy
INCLUDE-base-44	88.6%	Regional Knowledge

Table 4: Performance Recovery Analysis: Student (7.35B) vs. Teacher (11.04B).

Model	Parameters (B)	Average
Mistral-Large-Instruct-2411	123.0	69.84
Meta-Llama-3.1-405B-Instruct-FP8	405.0	69.44
Mistral-Large-Instruct-2407	123.0	69.11
Qwen2.5-72B-Instruct	72.7	67.92
QwQ-32B-Preview	32.8	67.01
Llama-3.3-70B-Instruct	70.6	66.40

Table 5: Open PL LLM Leaderboard results for instruction-tuned models (5-shot evaluation)

Model	Parameters (B)	Score
Mistral-Large-Instruct-2407 †	123.0	78.07
Mistral-Large-Instruct-2411 †	123.0	77.29
Meta-Llama-3.1-405B-Instruct-FP8	405.0	77.23
gpt-4o-2024-08-06	Unknown	75.15
gpt-4-turbo-2024-04-09	Unknown	74.59
Bielik-11B-v2.6-Instruct	11.2	73.8

Table 6: Polish EQ-Bench results for various models.

Model	Params (B)	Overall	Implicatures	Senti-	Language	Phrase-	Tricky
		Average	Average	ment	Understanding	ology	Questions
gemini-2.0-flash-001	Unknown	4.29	4.39	4.52	4.32	4.34	3.99
DeepSeek-R1	685.0	4.14	4.14	4.49	4.35	3.60	4.12
gemini-2.0-flash-lite-001	Unknown	4.09	4.17	4.23	4.05	4.24	3.85
DeepSeek-V3-0324	685.0	4.03	4.03	4.36	4.20	3.54	4.02
Mistral-Large-Instruct-2411 †	123.0	4.00	4.10	4.33	3.98	3.99	3.72

Table 7: Complex Polish Text Understanding Benchmark (CPTUB) results across different evaluation categories

Model	Parameters (B)	Average (%)
Meta-Llama-3.1-405B-Instruct-FP8	405.0	69.20
Mistral-Large-Instruct-2407 †	123.0	64.28
Qwen2.5-72B-Instruct	72.7	63.89
Meta-Llama-3.1-70B-Instruct	70.6	61.75
Qwen2-72B-Instruct	72.7	61.35
Meta-Llama-3-70B-Instruct	70.6	57.51

Table 8: Polish Medical Leaderboard results (5-shot setting) showing model performance on Polish Board Certification Examinations.

Model	Params (B)	AVG	Polish
Bielik-11B-v3-Instruct	11	64.8	69.0
Qwen2.5-14B-Instruct	14	61.7	58.9
Bielik-11B-v3.0	11	60.6	63.9
phi-4	15	58.8	49.6
Apertus-8B-Instruct-2509	8	57.9	49.6
Bielik-Minitron-7B-v3.0-Instruct	7	57.4	59.3

Table 9: INCLUDE-base-44 benchmark results showing average performance across European languages (20 language subset) and Polish-specific scores.

Model	Params (B)	Score
Qwen2.5-14B-Instruct	14	85.91
Bielik-11B-v3.0-Instruct	11	82.98
phi-4	15	81.71
Bielik-Minitron-7B-v3.0-Instruct	7	78.03
Qwen2.5-7B	7	74.60
Mistral-Nemo-Instruct-2407	12	74.14

Table 10: Belebele benchmark results showing model performance on reading comprehension across European languages (28 language subset).

Model	Params (B)	AVG	to Polish	from Polish
EuroLLM-9B-Instruct ∗	9	20.61	19.28	21.95
Bielik-11B-v3.0-Instruct	11	19.22	18.54	19.91
Bielik-11B-v3.0	11	17.85	17.60	18.11
phi-4 (15B)	15	15.58	14.55	16.61
Bielik-Minitron-7B-v3.0-Instruct	7	15.53	15.74	15.32
Qwen3-14B	14	15.37	14.18	16.56

Table 11: FLORES machine translation benchmark results showing translation performance across European languages (20 language pairs) measured by BLEU scores.

Category	Metric	Teacher (11B)	Student (7.35B)	$\Delta$	Recovery %
Common-sense reasoning	Test MCC	48.77	36.44	-12.33	74.71%
Common-sense reasoning	Test Accuracy	69.85	62.70	-7.15	89.75%
European values	Test European values	11.52	10.28	-1.24	89.24%
Knowledge	Test MCC	49.75	38.39	-11.35	77.18%
Knowledge	Test Accuracy	62.30	53.81	-8.49	86.37%
Linguistic acceptability	Test MCC	45.10	39.83	-5.27	88.32%

Table 12: Detailed Knowledge Recovery Analysis: Bielik-Minitron-7B-v3.0-Instruct vs. Bielik-11B-v3.0-Instruct parent model across EuroEval categories.

Model	Params (B)	Average (%)
Bielik-11B-v3.0-Instruct	11.2	65.93
Bielik-Minitron-7B-v3.0-Instruct	7.35	62.46
Bielik-Minitron-7B-v3.0-Instruct.Q6_K.gguf	7.35	62.29
Bielik-Minitron-7B-v3.0-Instruct.Q8_0.gguf	7.35	62.06
Bielik-Minitron-7B-v3.0-Instruct.Q4_K_M.gguf	7.35	61.89
Bielik-Minitron-7B-v3.0-Instruct FP8	7.35	61.53

Table 13: Open PL LLM benchmark results for Bielik-Minitron-7B-v3.0-Instruct across different quantization methods, compared to other models. Higher scores are better.

Stage	Open LLM EN	Open LLM PL
Pruning & Distillation	60.04	50.67
SFT	66.3	62.26
DPO-P	66.54	62.5
GRPO	66.6	62.46

Table 14: Bielik-Minitron-7B-v3.0 performance across training stages. Higher scores are better.

Model	Params	Non-Live	Non-Live	Non-Live	Live	Live	Live	Live Parallel
	(B)	Multiple	Parallel	Parallel	Simple	Multiple	Parallel	Parallel Multiple
		AST	AST	Multiple AST	AST	AST	AST	AST
Bielik-4.5B-v3.0-Instruct (FC)	4.6	92.50%	82.00%	86.00%	70.16%	68.66%	50.00%	54.17%
Bielik-Minitron-7B	7.3	94.50%	92.00%	85.00%	71.32%	71.51%	31.25%	66.67%
Bielik-11B-v2.6-Instruct	11.0	94.50%	87.50%	86.00%	75.97%	76.07%	37.50%	66.67%
Bielik-11B-v2.3-Instruct (Prompt)	11.0	93.50%	47.00%	50.00%	72.87%	69.71%	43.75%	54.17%

Table 15: Berkeley Function-Calling Leaderboard (BFCL) results. Comparison of family of Bielik models against the student, Bielik-Minitron, 11B teacher Bielik-11B-v3 and industry baselines.

Model Variant	Throughput	Median TTFT	Median TPOT
	(tok/s) $\uparrow$	(ms) $\downarrow$	(ms) $\downarrow$
Bielik-11B-v3 (bf16)	54.42	24.64	18.28
Bielik-minitron-7B-v3 (bf16)	81.41	27.29	12.32

Table 16: Hardware Performance Comparison: Throughput (Tokens per Second), Time to First Token (TTFT), and Time per Output Token (TPOT).

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 283 papers

Knowledge Distillation for Large Language Models
Mar 14, 2026Alejandro Paredes La Torre, Barbara Flores, Diego RodriguezLanguage Model Distillation LLM Compression
Duke University
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
Jul 27, 2026Darina Gold, Alexander Schwirjow, Viktor Haag +4On-Device Language Model Inference Training Data Curation
IIS Fraunhofer
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST

Mar 14, 2026J/K move · Enter open · S save

Knowledge Distillation for Large Language Models

Alejandro Paredes La Torre, Barbara Flores, Diego Rodriguez

Duke University

Language Model Distillation LLM Compression

Open paper AskPDF arXiv

We propose a resource-efficient framework for compressing large language models through knowledge distillation, combined with guided chain-of-thought reinforcement learning. Using Qwen 3B as the teacher and Qwen 0.5B as the student, we apply knowledge distillation across English Dolly-15k, Spanish Dolly-15k, and code BugNet and PyTorrent datasets, with hyperparameters tuned in the English setting to optimize student performance. Across tasks, the distilled student retains a substantial portion of the teacher's capability while remaining significantly smaller: 70% to 91% in English, up to 95% in Spanish, and up to 93.5% Rouge-L in code. For coding tasks, integrating chain-of-thought prompting with Group Relative Policy Optimization using CoT-annotated Codeforces data improves reasoning coherence and solution correctness compared to knowledge distillation alone. Post-training 4-bit weight quantization further reduces memory footprint and inference latency. These results show that knowledge distillation combined with chain-of-thought guided reinforcement learning can produce compact, efficient models suitable for deployment in resource-constrained settings.

Abstract

LLM Compression

Explore similar work

Knowledge Distillation for Large Language Models

From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training