H2LooP Spark Preview: Continual Pretraining of Large Language Models for Low-Level Embedded Systems Code

Amit Singh; Vedant Nipane; Pulkit Agrawal; Jatin Kishnani; Sairanjan Mishra

cs.LGMar 11, 2026

H2LooP Spark Preview: Continual Pretraining of Large Language Models for Low-Level Embedded Systems Code

Authors: Amit Singh, Vedant Nipane, Pulkit Agrawal, Jatin Kishnani, Sairanjan Mishra

Abstract

Large language models (LLMs) demonstrate strong code generation abilities in general-purpose programming languages but remain limited in specialized domains such as low-level embedded systems programming. This domain involves hardware register manipulation, vendor-specific SDKs, real-time operating system APIs, and hardware abstraction layers that are underrepresented in standard pretraining corpora. We introduce H2LooP Spark Preview, a continual pretraining (CPT) pipeline that adapts the OLMo-3-7B-a fully open language model to the embedded systems domain using BF16 LoRA with rank-stabilized scaling on 8 NVIDIA H100 GPUs. Our training corpus is constructed from repository-datasheet pairs covering 100B tokens of raw embedded systems data across 117 manufacturers, processed using the hierarchical datasheet-to-code mapping approach proposed in SpecMap (Nipane et al., 2026). The resulting curated dataset split contains 23.5B tokens across 13 embedded domains. Continual pretraining with high-rank LoRA (r=512) yields substantial gains, reducing in-domain perplexity by 70.4% and held-out repository perplexity by 66.1%. On generative code completion benchmarks spanning 13 embedded domains, our 7B model outperforms Claude Opus 4.6 and Qwen3-Coder-30B on 8 categories in token accuracy, showing that targeted continual pretraining enables smaller open-weight models to rival frontier systems on specialized technical tasks. We release the production training checkpoint on Huggingface as an open-source artifact.

Figures & tables

Property	Value
Raw data size	76.4 GB (818 repo-datasheet pairs)
Processed training data	$\sim$ 20 GB (289 Parquet files)
Estimated total samples	$\sim$ 11.5M
Estimated total tokens	$\sim$ 23.5B
Maximum sequence length	2,048 tokens
Average sample length	$\sim$ 1,450 tokens

Table 1: Processed Training Corpus Statistics

Repository	URL	Categories
torvalds/linux	github.com/torvalds/ linux	linux_kernel, amd_gpu_registers, device_tree, wireless_ble_wifi, usb_stack
Infineon/AURIX_ code_examples	github.com/Infineon/ AURIX_code_examples	infineon_aurix
STMicroelectronics/ STM32CubeF4	github.com/ STMicroelectronics/ STM32CubeF4	stm32_hal
nxp-mcuxpresso/ mcux-sdk	github.com/nxp- mcuxpresso/mcux-sdk	nxp_imx
ARM-software/ CMSIS_5	github.com/ARM- software/CMSIS_5	arm_cortex_asm
zephyrproject-rtos/ zephyr	github.com/ zephyrproject-rtos/ zephyr	zephyr_rtos

Table 2: Held-Out Evaluation Repositories

Parameter	Value	Rationale
LoRA rank ( $r$ )	512	Optimal rank-performance tradeoff (Section 5.2)
LoRA alpha ( $\alpha$ )	1,024	Standard 2 $\times$ rank scaling
LoRA dropout	0.05	Mild regularization
RSLoRA	Enabled	Stable scaling for high-rank adapters
Target modules	All (attn + MLP + embed)	Full targeting optimal (Section 5.2)

Table 3: LoRA Configuration

Parameter	Value
Per-device batch size	4
Gradient accumulation steps	8
Number of GPUs	8 $\times$ H100 80GB SXM
Effective batch size	256
Max sequence length	2,048 tokens (with packing)
Tokens per optimizer step	524,288

Table 4: Training Hyperparameters

Parameter Group	Learning Rate	Rationale
Attention + MLP LoRA adapters	1.5 $\times 10^{-5}$	Standard adaptation rate
embed_tokens + lm_head adapters	7.5 $\times 10^{-6}$ (0.5 $\times$ )	Conservative embedding updates

Table 5: Optimizer Parameter Groups

Parameter	Value
Base model	allenai/OLMo-3-1025-7B
LoRA alpha ( $\alpha$ )	$2\times r$ (rank-scaled)
LoRA dropout	0.05
RSLoRA	Enabled
Per-device batch size	16
Gradient accumulation	8 steps

Table 6: Fixed Hyperparameters Across All Sweep Configurations

Figure 2: Bayesian Hyperparameter Sweep: Training dynamics for 10 configurations. Each curve represents a unique (rank, target, LR) triple. Top-left: Training loss stratified by rank. Top-center: Learning rate schedules. Top-right: Post-clip gradient norms. Bottom: Global step and epoch progression. Labels encode rank ( $r$ ), target (A = attention-only, F = full), and learning rate.

Configuration	$r$	Target	$\eta$	Init Loss	Final Loss	Min Loss	Reduction
r512-F-3.5e-5	512	Full	$3.45\times 10^{-5}$	1.426	1.015	0.961	28.8%
r512-A-3.5e-5	512	Attn	$3.45\times 10^{-5}$	1.419	1.034	0.981	27.1%
r256-F-5e-5	256	Full	$5.00\times 10^{-5}$	1.418	1.068	1.009	24.7%
r256-A-5e-5	256	Attn	$5.00\times 10^{-5}$	1.420	1.078	1.024	24.1%
r256-F-3.5e-5	256	Full	$3.45\times 10^{-5}$	1.421	1.095	1.038	22.9%
r256-A-3.5e-5	256	Attn	$3.45\times 10^{-5}$	1.422	1.109	1.054	22.0%

Table 7: Bayesian Sweep Results

Axis	Levels	Mean Final Loss	$\Delta$ (absolute)
LoRA rank	$r=128$	1.175	0.150 (128 $\to$ 512)
	$r=256$	1.087
	$r=512$	1.025
Target modules	Attention-only	1.115	0.010
	Full (attn+MLP+embed)	1.105
Learning rate	$3.45\times 10^{-5}$	1.105	0.013

Table 8: Marginal Effect of Each Hyperparameter Axis on Final Loss

Configuration	$r$	Target	$\eta$	Peak Grad Norm	Mean Grad Norm
r256-F-5e-5	256	Full	$5.00\times 10^{-5}$	28.46	5.73
r128-F-3.5e-5	128	Full	$3.45\times 10^{-5}$	12.47	3.22
r256-F-3.5e-5	256	Full	$3.45\times 10^{-5}$	8.37	3.91
r512-F-3.5e-5	512	Full	$3.45\times 10^{-5}$	5.40	4.30
r128-F-5e-5	128	Full	$5.00\times 10^{-5}$	4.37	2.14
r512-A-3.5e-5	512	Attn	$3.45\times 10^{-5}$	3.11	2.22

Table 9: Gradient Norm Statistics Across Sweep Configurations

Hyperparameter	Sweep Finding	Production Choice
LoRA rank	$r=512$ best ( $\Delta=0.150$ vs $r=128$ )	$r=512$
Target modules	Full $>$ attn-only ( $\Delta=0.010$ )	Full (all 9 modules)
Learning rate	$3.45\times 10^{-5}$ more stable	$1.5\times 10^{-5}$ (further reduced)
Gradient clipping	Spikes up to 28.5 observed	max_grad_norm = 5.0
Batch size	128 (sweep)	256 (production, 8 $\times$ H100)
Training steps	150 (sweep)	45,000+ (production)

Table 10: Mapping from Sweep Findings to Production Configuration

Metric	Value
Total configurations	10
Steps per configuration	150
Mean runtime per configuration	1.56 hours
Total GPU-hours (A100)	15.6 hours
Fraction of production compute	$\sim$ 5.3%
Hyperparameter axes explored	3 (rank, target, LR)

Table 11: Computational Cost of the Bayesian Hyperparameter Sweep

Figure 3: Training Loss — Training cross-entropy loss as a function of optimizer steps for the hero run. The raw loss (light trace) and smoothed loss (bold trace, window=20) both exhibit monotonically decreasing behavior from 0.88 to 0.25 with no instabilities throughout the entire 294.7-hour run.

Step	Approx. Tokens	Loss	Token Accuracy	Entropy
10	5.0M	0.881	85.8%	–
280	147M	0.827	86.0%	–
1,000	0.5B	0.583	87.7%	0.446
2,000	1.0B	0.503	88.7%	–
5,000	2.6B	0.399	90.0%	–
10,000	5.2B	0.324	92.0%	–

Table 12: Training Progress (From W&B Logs)

Figure 4: Hero Run LR Schedule and Token Accuracy — Left: Cosine LR schedule with 10% warmup. Right: Mean token accuracy climbing monotonically from 86% to 93.5%.

Parameter	Value	Rationale
LoRA rank ( $r$ )	512	Optimal rank-performance tradeoff (Section 5 )
LoRA alpha ( $\alpha$ )	1,024	Standard 2x rank scaling
LoRA dropout	0.05	Mild regularization
RSLoRA	Enabled	Stable scaling for high-rank adapters
Target modules	All (attn + MLP + embed)	Full targeting optimal (Section 5 )

Table 13: LoRA Configuration

Figure 5: Throughput — Left: tokens per second, showing a sharp transition from $\sim$ 11,000 tok/s (steps 0-6,000) to $\sim$ 7,000 tok/s (steps 7,000+). Right: step time showing the corresponding increase from $\sim$ 45s to $\sim$ 70s per step.

Phase	Steps	Avg Tokens/sec	Avg Step Time	Regime
Phase 1 (early)	0–6,000	10,945	45.5s	Original launch
Phase 2 (resumed)	7,000–16,440	7,078	70.3s	Post-checkpoint resume
Overall weighted	0–16,440	8,644	60.3s

Table 14: Throughput by Training Phase (From W&B Logs)

Figure 6: Perplexity Convergence — Weighted perplexity on in-domain and held-out data across training. Both curves show rapid initial improvement followed by steady convergence.

Checkpoint	Tokens	In-Domain PPL	Held-Out PPL
Base (no training)	0	4.06	3.92
Step 1,000	0.5B	3.73 (-8.1%)	3.68 (-6.1%)
Step 4,000	2.0B	1.41 (-65.3%)	1.50 (-61.7%)
Step 7,000	3.5B	1.32 (-67.5%)	1.42 (-63.8%)
Step 9,000	4.5B	1.28 (-68.5%)	1.38 (-64.8%)
Step 13,000	6.5B	1.22 (-70.0%)	1.35 (-65.6%)

Table 15: Perplexity Convergence (Selected Checkpoints)

Figure 7: Category Perplexity — In-domain perplexity comparison across 13 categories. Improvement percentages shown above each category.

Category	Base PPL	Spark PPL	Improvement
Wireless/BLE/WiFi	7.79	1.48	81.0%
Linux Kernel	7.64	1.46	80.9%
NXP i.MX	6.00	1.22	79.7%
Device Tree	4.79	1.14	76.2%
USB Stack	4.80	1.29	73.1%
Zephyr RTOS	7.51	2.02	73.1%

Table 16: In-Domain Perplexity by Category (Checkpoint-15000)

Category	Base Loss	Spark Loss	Base Top-1	Spark Top-1	Spark Top-5
Infineon AURIX	0.965	0.007	0.819	0.998	1.000
Register Defines	0.908	0.049	0.821	0.988	0.996
ARM Cortex ASM	1.265	0.080	0.739	0.980	0.995
STM32 HAL	1.389	0.102	0.724	0.972	0.995
Linux Kernel	1.959	0.286	0.621	0.919	0.984
Wireless/BLE/WiFi	1.966	0.328	0.634	0.914	0.979

Table 17: In-Domain Completion (Checkpoint-15000, Selected Categories)

Figure 8: Token Accuracy Comparison — Generative token accuracy across 13 embedded domains for all four models.

Figure 9: BLEU-4 Comparison — BLEU-4 scores for the same comparison. Spark Preview achieves the highest BLEU-4 on 8 of 13 categories.

Category	Base OLMo	Spark	Claude Opus 4.6	Qwen3-30B	Winner
General	0.487	0.851	0.487	0.534	H2LooP Spark
Register Defines	0.374	0.596	0.477	0.508	H2LooP Spark
ARM Cortex ASM	0.184	0.558	0.274	0.293	H2LooP Spark
Infineon AURIX	0.299	0.543	0.265	0.313	H2LooP Spark
STM32 HAL	0.043	0.399	0.181	0.328	H2LooP Spark
Device Tree	0.132	0.352	0.257	0.197	H2LooP Spark

Table 18: Generative Token Accuracy Comparison

Category	Base Top-1	Spark Top-1	Improvement
Register Defines	0.837	0.991	+15.4pp
General	0.815	0.995	+18.0pp
Infineon AURIX	0.815	0.982	+16.7pp
STM32 HAL	0.696	0.979	+28.3pp
ARM Cortex ASM	0.754	0.971	+21.7pp
Linux Kernel	0.616	0.895	+27.9pp

Table 19: Held-Out Completion (Selected Categories)

Metric	Base	FT (ckpt-15k)	Delta
Compile Rate	44.9%	41.1%	-3.8%
Test Pass Rate	15.9%	13.1%	-2.8%

Table 20: FullStackBench Results: Overall Performance

Difficulty	Base Compile	FT Compile	Base Pass	FT Pass
Easy (8 samples)	62.5%	62.5%	50.0%	50.0%
Medium (37 samples)	45.9%	43.2%	18.9%	13.5%
Hard (62 samples)	41.9%	37.1%	9.7%	8.1%

Table 21: FullStackBench Results: Performance by Difficulty

Resource	Value
Wall-clock time	12.3 days
GPU-hours (8 $\times$ H100)	2,357
Tokens processed	8.6B
Average throughput	8,644 tok/sec

Table 22: Computational Resources (Hero Run)

Code Language Models

Code language models are AI models that generate, explain, edit, and reason about source code across programming tasks.

See all 113 papers

InCoder-32B: Code Foundation Model for Industrial Scenarios
Mar 17, 2026Jian Yang, Wei Zhang, Jiajun Wu +25Compiler Optimization Code Language Models
Beihang University · IQuest Research · Shanghai Jiao Tong University +6
Exploring different approaches to customize language models for domain-specific text-to-code generation
Mar 17, 2026Luís Freire, Fernanda A. Andaló, Nicki Skafte DetlefsenCode Generation LLM Code Generation
Technical University of Denmark · The LEGO Group
SLM Finetuning for Natural Language to Domain Specific Code Generation in Production
Apr 10, 2026Renjini R. Nair, Damian K. Kowalczyk, Marco Gaudesi +1Code Generation Fine-Tuning
Microsoft

Mar 17, 2026J/K move · Enter open · S save

InCoder-32B: Code Foundation Model for Industrial Scenarios

Jian Yang, Wei Zhang, Jiajun Wu, Junhang Cheng+24

Beihang University · IQuest Research · Shanghai Jiao Tong University+6

Compiler Optimization Code Language Models LLM Code Generation Code Generation GPU Kernel Optimization

Open paper AskPDF arXiv

Recent code large language models have achieved remarkable progress on general programming tasks. Nevertheless, their performance degrades significantly in industrial scenarios that require reasoning about hardware semantics, specialized language constructs, and strict resource constraints. To address these challenges, we introduce InCoder-32B (Industrial-Coder-32B), the first 32B-parameter code foundation model unifying code intelligence across chip design, GPU kernel optimization, embedded systems, compiler optimization, and 3D modeling. By adopting an efficient architecture, we train InCoder-32B from scratch with general code pre-training, curated industrial code annealing, mid-training that progressively extends context from 8K to 128K tokens with synthetic industrial reasoning data, and post-training with execution-grounded verification. We conduct extensive evaluation on 14 mainstream general code benchmarks and 9 industrial benchmarks spanning 4 specialized domains. Results show InCoder-32B achieves highly competitive performance on general tasks while establishing strong open-source baselines across industrial domains.

Abstract

Code Language Models

Explore similar work

InCoder-32B: Code Foundation Model for Industrial Scenarios

Exploring different approaches to customize language models for domain-specific text-to-code generation

SLM Finetuning for Natural Language to Domain Specific Code Generation in Production