Cognitively Layered Data Synthesis for Domain Adaptation of LLMs to Space Situational Awareness
Authors: Ding Linghu, Cheng Wang, Da Fan, Wei Shi, Kaifeng Yin, Xiaoliang Xue, Fan Yang, Haiyi Ren, Cong Zhang
Organizations: Qian Xuesen Laboratory of Space Technology, China Academy of Space Technology, Beijing, China · China Academy of Space Technology, Beijing, China · State Key Laboratory of Space Information System and Integrated Application, Beijing, China
Large language models (LLMs) demonstrate exceptional performance on general-purpose tasks. however, transferring them to complex engineering domains such as space situational awareness (SSA) remains challenging owing to insufficient structural alignment with mission chains, the absence of higher-order cognitive supervision, and poor correspondence between data quality criteria and engineering specifications. The core bottleneck is the construction of high-quality supervised fine-tuning (SFT) datasets. To this end, we propose BD-FDG (Bloom's Taxonomy-based Domain-specific Fine-tuning Data Generation), a framework that addresses incomplete knowledge coverage, shallow cognitive depth, and limited quality controllability through three mechanisms: structured knowledge organization, cognitively layered question modeling, and automated quality control. The framework uses a knowledge tree to ensure structured corpus coverage, designs a question generation scheme spanning nine categories and six cognitive levels from Remember to Create to produce samples with a continuous difficulty gradient, and applies a multidimensional scoring pipeline to enforce domain rigor and consistency. Using BD-FDG, we construct SSA-SFT, a domain dataset of approximately 230K samples, and fine-tune Qwen3-8B to obtain SSA-LLM-8B. Experiments show that SSA-LLM-8B achieves relative BLEU-1 improvements of 144% (no-think) and 176% (think) on the domain test set and a win rate of 82.21% over the baseline in arena comparisons, while largely preserving general benchmark performance (MMLU-Pro, MATH-500). These results validate SFT data construction driven by cognitive layering as an effective paradigm for complex engineering domains and provide a transferable framework for domain-specific LLM adaptation.
Figures & tables
Figure 1: Architecture of the BD-FDG framework. The domain knowledge base construction stage (left) parses mission chain literature into structured chunks and builds a hybrid dense–sparse retrieval index for multisource context assembly. The question generation stage (middle) applies nine Bloom’s Taxonomy-guided prompt templates across six cognitive levels to produce questions with graduated difficulty from retrieved context. The SFT data synthesis stage (right) distills reasoning traces and answers from teacher models, scores each sample along four quality dimensions, and filters by a composite threshold to yield the final SSA-SFT dataset.
No.
Type Name
Cognitive Level
Assessment Objective
Q1
Concept Discrimination
Remember/Understand
Understanding and distinguishing core concepts
Q2
Principle Explanation
Understand
In-depth understanding and explanation of technical principles
Q3
Formula Derivation
Understand/Apply
Mathematical modeling and theoretical analysis
Q4
Parameter Calculation
Apply
Design and calculation of key parameters
Q5
Algorithm Implementation
Apply/Analyze
Algorithm design and optimization
Q6
Performance Analysis
Analyze
System performance analysis and comparison
Table 1: Question type system guided by Bloom’s Taxonomy (Q1–Q9) for the space situational awareness domain.
Figure 2: Overview of the SSA-SFT and SSA-Test datasets. Left: three-tier mission chain knowledge system. Right: distribution of question types and sample counts.
Dataset
Task Type
no-think
think
Qwen3-8B
SSA-LLM-8B
Qwen3-8B
SSA-LLM-8B
SSA-Test
Knowledge
21.33
52.08
20.75
57.23
AIME24
Math
33.33
23.33
66.67
66.67
AIME25
Math
16.67
23.33
60.00
56.67
C-Eval
Exam
79.35
78.01
83.28
79.20
GPQA Diamond
Knowledge
51.52
44.44
60.10
60.10
Table 2: Test results of SSA-LLM-8B and Qwen3-8B on various test sets. All values are in %. Bold indicates the better or tied best value under the same mode.
Inference Mode
Win Rate (%)
Lower CI (%)
Upper CI (%)
no-think
82.21
81.50
82.91
think
73.54
72.84
74.44
Table 3: Arena Battle results on the SSA-Test dataset (SSA-LLM-8B vs. Qwen3-8B).
Metric
no-think
think
Qwen3-8B
SSA-LLM-8B
Qwen3-8B
SSA-LLM-8B
BLEU-1
21.33
52.08
20.75
57.23
BLEU-2
12.40
35.26
12.81
39.53
BLEU-3
7.80
25.70
8.54
29.18
BLEU-4
4.97
19.26
5.77
22.16
ROUGE-1-F
42.31
53.03
45.30
54.62
Table 4: Fine-grained metrics on the SSA-Test dataset. All values are in %. Bold indicates the better or tied best value under the same mode.
Figure 3: Two-dimensional ablation results. The central heatmap shows average quality scores for each (α,K) configuration; marginal bar plots depict the main effects of Top- K (top) and α (right). The optimal configuration (α=0.50,K=5) attains a score of 7.55.
Adapting Large Language Models (LLMs) to specialized domains requires high-quality instruction tuning datasets, which are expensive to create through human annotation. Existing data synthesis methods focus on general-purpose tasks and fail to capture domain-specific terminology and reasoning patterns. To address this, we introduce DS2-Instruct, a zero-shot framework that generates domain-specific instruction datasets without human supervision. Our approach first generates task-informed keywords to ensure comprehensive domain coverage. It then creates diverse instructions by pairing these keywords with different cognitive levels from Bloom's Taxonomy. Finally, it uses self-consistency validation to ensure data quality. We apply this framework to generate datasets across seven challenging domains, such as mathematics, finance, and logical reasoning. Comprehensive evaluation demonstrates that models fine-tuned on our generated data achieve substantial improvements over existing data generation methods.
High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. Data synthesis is a viable alternative and succeeds on closed tasks, yet the humanities and social sciences (HSS) are overlooked, and their open-ended nature makes synthesis challenging. Moving beyond prior capability-centric, fragmented attempts, we adopt a subject-centric paradigm, define the first HSS domain system covering 14 mainstream fields, and introduce HSS-Synth, the first data synthesis pipeline for HSS. HSS-Synth comprises: (1) constructing seed documents from web corpora via multi-step filtering and text refinement evaluated by a judge; (2) specifying "requirements + persona" to backtranslate seed documents into diverse yet faithful instructions with a strict Q&A alignment check; and (3) breaking LLM response limits via teacher-forced Answering that feeds seed documents during response generation to anchor semantics, reduce hallucinations, and preserve tone and integrity. HSS-Synth yields 237k high-quality, diverse instruction-tuning samples that outperform 14 leading baselines on 16 benchmarks. The fine-tuned Qwen3-8B-Base sets a new SOTA and approaches the official Qwen3-8B, improving both human preference and knowledge capabilities without performance seesaws. Extensive experiments demonstrate HSS-Synth's robustness and transferability. Our code is publicly available at https://github.com/pengr/HSS-Synth.
Ru Peng, Tianyu Zhao, Xijun Gu +9
Zhejiang University · Inclusion AI, Ant Group · Ant Group +2
Defense is a knowledge-intensive domain that requires precise understanding of specialized terminology, doctrinal concepts, operational procedures, and evolving military events. Although recent work has explored language technologies for military applications, existing efforts remain fragmented: they are often task-specific, rely on limited adaptation pipelines, or lack comprehensive defense-domain evaluation. In this paper, we present SALUTE, an end-to-end framework for benchmarking and adapting LLMs for the defense domain. SALUTE integrates Salute-Corpus, a curated corpus from open-access U.S. military doctrine and government documents; Salute-Conv, a grounded instruction dataset from doctrinal sources and decade-long defense news; Salute-Pref, a defense-aware preference dataset; and Salute-Bench, a rigorously filtered benchmark for evaluating defense-domain understanding and reasoning over doctrine and defense news. Based on these resources, we train Salute-LLM through multi-stage post-training with continual pretraining, supervised fine-tuning, and preference alignment. Extensive experiments show that Salute-LLM achieves strong defense-domain performance while retaining competitive general capabilities, demonstrating the effectiveness of SALUTE as an end-to-end framework for defense-domain LLM adaptation.
Hyeongcheol Park, Sumin In, Suyeon Myeong +5
Korea University, Republic of Korea · Sungkyunkwan University, Republic of Korea · Hanwha Systems, Republic of Korea