cs.AISep 11, 2026
SaveZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search
Organizations: Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence
Abstract
In this work, we present ZGCM-1, a fully open 7B dense foundation model trained from scratch with extreme data, system, and algorithmic efficiency. ZGCM-1 is founded on a core premise: compact models cannot passively memorize the open web, but can overcome parametric capacity limits by coupling deliberate internal thinking with active external tool use. To support this paradigm across a 256K context, we develop an end-to-end, high-efficiency open training recipe: Architecture & System Co-design: interleaved gated sliding-window and full attention, and a stable FP8 Muon optimizer; Progressive Curriculum & MDP Mid-Training: context scaling across 16K, 64K, and 256K, and the reformulation of interaction traces into Markov Decision Processes. Furthermore, we establish an AI-native R&D workflow where agent swarms autonomously manage cluster operations, data curation, and rapid diagnostic evaluation. Extensive evaluations show that ZGCM-1-7B is competitive across 7B model family on general benchmarks. On several challenging mathematical reasoning and agentic search suites, it remains competitive with frontier models orders of magnitude larger, such as Qwen3-235B-A22B and GLM-5.1. We also show that our pre-training design offers a ~4.2x efficiency improvement in 16K pre-training time-to-loss. Across the full development lifecycle, we distill eight actionable empirical findings-spanning architectural scaling, SFT quality pruning, long-context generalization, and agentic co-training dynamics. To facilitate community research, we open-source model weights from the pre-training, mid-training, and post-training stages, intermediate checkpoints, training code, per-stage data and data recipes, and W&B logs.
Figures & tables
Figure 1: Per-benchmark ranks for ZGCM-1-7B and models at comparable scale across 14 reasoning benchmarks. Full results appear in Section 5 .
Figure 2: Overview of ZGCM-1. Left: benchmark performance across reasoning and agentic tasks compared with 7B-scale and frontier models. Right: four key technical highlights: (1) FP8 pre-training with Muon and TWEO achieving time-to-loss speedup, (2) hybrid sliding-window/global attention, (3) MDP mid-training that reformulates interaction traces into state-action supervision with context scaling to 256K, and (4) AI-native R&D where each researcher directs agent swarms across the full development lifecycle. BFS denotes Binary Function Search. Qwen3-8B-Distill denotes DeepSeek-R1-0528-Qwen3-8B. Full results and evaluation protocols appear in Section 5 .
Figure 3: Hybrid attention architecture of ZGCM-1. The right panel shows the backbone formed by gated sliding-window and global attention layers; the left panel details the gated sliding-window attention module and its corresponding attention masks. The last global layer is layer 29 (0-indexed), followed by two trailing SWA layers.
Figure 4: Architecture experiments. (a) Quality-throughput trade-off among non-parameter-matched 7B attention configurations trained for 10B tokens (lower loss and higher throughput are better). (b) Throughput speedup of SWA schedules over full attention as context length grows from 4K to 256K. (c) KV cache memory footprint of three iso-parameter architectures (batch = 1, bf16): full attention (32 global layers), linear attention (GDN 3:1, 7 global + 21 linear, L=28), and our hybrid SWA 5:1 (5 global + 27 SWA).
Figure 5: General Pre-Training is organized into two data stages, followed by three Mid-Training stages with progressively longer context lengths.
Figure 6: Data mixture composition shown as donut charts for the two General Pre-Training stages and the three Mid-Training context stages. Wedge areas represent each category’s share of the corresponding stage’s total tokens.
Figure 7: Mid-Training data processing pipeline.
Figure 8: AI-driven self-iterating data governance pipeline. Each source family is routed and sampled independently, then processed through source-specific rule filtering, strong-model audit, failure mining, human spot checks, iterative script revision, and held-out validation before accepted examples enter the validated reasoning-data pool.
Figure 9: Token-level cross-entropy (CE) loss (blue, left axis) and learning-rate schedule (terracotta, right axis) for the staged Mid-Training run. The solid blue curve is a 200-point moving average, while the light-blue curve shows raw loss values. Both metrics share the cumulative-token axis; vertical dotted lines mark transitions between the 16K, 64K, and 256K context stages.
Figure 10: Supervised fine-tuning data curation pipeline. General data comprises general instruction and reasoning examples, while agentic data forms a separate branch. All sources undergo schema formatting, filtering and verification, mixture selection, benchmark decontamination, and preparation as packed training sequences with assistant-only loss.
| Component | Row share | Primary role |
|---|---|---|
| General data | 96.46% | Instruction following, knowledge, mathematics, science, code, dialogue, and reasoning |
| Agentic data | 3.54% | Deep research, software engineering, and terminal interaction through structured tools |
| Total | 100.00% | All supervised fine-tuning data |
Table 1: Composition of our supervised fine-tuning corpus.
Figure 11: Training dynamics of the two length-specific SFT variants. The 64K run (left) and the 256K run (right) exhibit similar loss-reduction patterns: a rapid initial decrease followed by stepwise convergence as the learning rate follows a cosine decay schedule. Curves are shown from optimizer step 50 to focus on steady-state training dynamics. Solid blue curves show a 160-step moving average of token-level cross-entropy (CE) loss, light-blue curves show raw loss values, and salmon curves show the learning rate. Both panels use the same CE-loss scale. The released model uses the 256K SFT variant.
| Benchmark | ZGCM-1 [1.5pt]7B | DeepSeek-R1 [1.5pt]0528-Qwen3 [1.5pt]8B | MiniCPM [1.5pt]4.1-8B | Qwen3 [1.5pt]8B | Olmo-3-7B [1.5pt]Think | MiMo-7B [1.5pt]RL | Open [1.5pt]Thinker [1.5pt]3-7B |
| Reasoning & General | |||||||
| MATH-500 | 97.13 | 96.32 | 95.60 | 96.20 | 95.10 | 96.20 | 88.40 |
| AIME 2024 | 80.62 | 83.33 | 83.33 | 80.00 | 71.60 | 66.67 | 43.33 |
| AIME 2025 | 73.33 | 75.21 | 73.33 | 63.33 | 64.60 | 53.33 | 43.33 |
| AIME 2026 | 75.00 | 69.17 | 71.67 | 66.67 | 66.16 | 56.67 | 40.00 |
| HMMT 2025 | 70.42 | 61.50 | 52.50 | 43.33 | 43.89 | 40.00 | 23.33 |
Table 2: Thinking-mode benchmark results (%) for ZGCM-1-7B and comparable-scale models. Dark blue cells in bold indicate the best result; light blue cells indicate the second-best result.
| Model / System | WebWalkerQA | BrowseComp | GAIA text-only |
|---|---|---|---|
| Our Model | |||
| ZGCM-1-7B | 63.09 | 19.43 | 42.52 |
| Open-source Research Agent | |||
| WebDancer-32B | 38.40 | 2.50 | 40.70 |
| Open-weight Models with Tools | |||
| Qwen3-235B-A22B | 59.60 | 2.30 | 45.60 |
Table 3: Web-environment deep-research results (%). Dark blue cells in bold indicate the best result; light blue cells indicate the second-best result. Dashes denote unavailable reported results.
Figure 12: Binary Function Search workflow and Ghidra command interface. The agent iterates between candidate exploration and evidence refinement, then submits an exact function-entry address for hidden-oracle validation.
| Model / System | Valid submissions | Correct | Accuracy (%) |
|---|---|---|---|
| Larger frontier models | |||
| DeepSeek-V4 Flash | 50/50 | 38/50 | 76.00 |
| Qwen3.5-397B-A17B | 50/50 | 38/50 | 76.00 |
| GLM-5.1 | 50/50 | 33/50 | 66.00 |
| ZGCM-1-7B | 50/50 | 31/50 | 62.00 |
| Kimi-K2 | 50/50 | 31/50 | 62.00 |
Table 4: Binary Function Search results on 50 tasks. Valid submissions counts answers in the required format; Correct counts exact oracle matches; Accuracy is Correct/50. The dark blue cell in bold and the light blue cell indicate the best and second-best accuracy.
Figure 13: AI-native R&D workflow. Each researcher directs agents across seven core stages of model development: data engineering, model architecture design, learning algorithm design, experimentation & monitoring, infrastructure engineering, evaluation, and deployment engineering. Completed agent work produces reusable experience (validated scripts, workflows, checklists, debugging history), and researcher discussions produce human context (conversations, meeting notes, decisions, plans); both feed into a shared agent harness that equips all agents with accumulated context, skills, tools, memory, orchestration, and verification capabilities.
| Level | Name | Defining criterion |
|---|---|---|
| L1 | Basic Assistance | Humans direct and execute the workflow; AI assists with individual steps. |
| L2 | Partial Automation | AI executes predefined workflows; humans specify procedures and handle exceptions. |
| L3 | Conditional Autonomy | AI adapts and manages routine workflows; consequential decisions require human approval. |
| L4 | High Autonomy | AI independently plans, executes, and iterates within human-defined objectives and constraints. |
| L5 | Full Autonomy | AI also identifies research objectives and coordinates sustained improvement across R&D stages without routine human direction. |
Table 5: Levels of AI autonomy in R&D workflows. The levels describe autonomy rather than productivity gains or output quality. L5 provides a reference for full autonomy, rather than an assertion of demonstrated capability.
Figure 14: AI4AI autonomy across R&D tasks, based on core contributor assessments. Bars show mean ratings, error bars indicate sample standard deviation across nine contributors, and points represent individual ratings. Vertical offsets separate coincident points and have no quantitative meaning. Levels L1–L5 are encoded as 1–5; zero is a plotting baseline, not a rating category. Error bars describe inter-contributor variation, not confidence intervals. The right column gives the autonomy level the team assigned to each task, with the number of filled markers indicating the level. Assignments are consensus judgments informed by the ratings.
| Stage | Context | TP/PP/CP/DP | MBS/GBS | Recompute | LR | RoPE base | Notes |
|---|---|---|---|---|---|---|---|
| General pre-training | 16K | 2/1/1/96 | 1/768 | None | 5M | 60% BF16-eq. MFU ( 585 TFLOP/s/GPU) | |
| Staged (step 1) | 64K | 2/1/4/24 | 1/192 | Selective | 5M | 10B tokens | |
| Direct 256K | 256K | 2/2/4/12 | 1/48 | Full | 10M | 30B tokens; 203 TFLOP/s/GPU | |
| Staged (step 2) | 256K | 2/2/4/12 | 1/48 | Full | 10M | 20B tokens; 204 TFLOP/s/GPU |
Table 6: Distributed-training configurations on H100 GPUs. TP = tensor parallelism, PP = pipeline parallelism, CP = context parallelism, DP = data parallelism, MBS = micro-batch size, GBS = global batch size.
Figure 15: Capability trends during General Pre-Training. Each point is obtained by directly evaluating the corresponding base checkpoint without supervised fine-tuning or other post-training adaptation. The x-axis reports cumulative tokens processed, truncated at 4.19T. The dotted line at 1.0T marks the end of the curriculum-ordered stage. Light lines show individual evaluations, and dark lines show five-point moving averages.
| Benchmark | Capability | Early | Around 1.0T | Around 2.5T | Final |
|---|---|---|---|---|---|
| (0.04T–0.07T) | (0.91T–1.16T) | (2.29T–2.79T) | (3.67T–4.18T) | ||
| MMLU | Knowledge | 26.37 | 48.32 | 56.94 | 59.52 |
| OpenBookQA | QA | 24.80 | 60.80 | 72.40 | 78.80 |
| CSQA | QA | 19.02 | 57.70 | 64.26 | 67.87 |
| HumanEval | Code | 13.66 | 28.29 | 36.46 | 43.54 |
| GSM8K | Math | 11.52 | 38.64 | 57.12 | 59.09 |
Table 7: Five-checkpoint window averages during General Pre-Training. Higher is better.
Figure 16: Capability trajectories across General Pre-Training and Mid-Training milestones after rapid fine-tuning on the same SFT corpus. The General Pre-Training probe and the 15%/30%/70%/77% Mid-Training probes use two SFT epochs; the final (100%) Mid-Training probe uses three. The curves show the downstream capability exposed by short SFT rather than direct base-model performance.
| Benchmark | Random | Curriculum |
|---|---|---|
| Coding | 1.99 | 0.81 |
| Math | 0.97 | 0.94 |
| MMLU | 1.03 | 1.12 |
| ARC | 1.05 | 1.09 |
| HellaSwag | 1.05 | 1.08 |
Table 8: Bits per byte (BPB) after 10B training tokens. Lower is better.
| Stage | Sampled tokens | Length composition |
|---|---|---|
| 16K | 180B | 180B at 16K |
| 64K | 240B | 180.89B at 16K + 59.11B at 16K–64K |
| 256K | 180.51B | 127.81B at 16K + 21.72B at 16K–64K + 30.98B above 64K |
Table 9: Mid-training context mixtures. Each longer-context stage retains shorter-context data.
| Data processing | Samples | IFEval | MATH-500 | BBH | MMLU | HumanEval+ | MBPP+ | Mean (6) |
|---|---|---|---|---|---|---|---|---|
| Minimal processing | 2.08M | 72.64 | 74.12 | 58.49 | 72.63 | 73.78 | 55.03 | 67.78 |
| Broader filtering | 1.833M | 71.20 | 77.63 | 65.08 | 70.60 | 65.00 | 58.36 | 67.98 |
| Quality-focused filtering | 1.145M | 71.94 | 75.43 | 68.59 | 71.50 | 67.56 | 57.94 | 68.83 |
Table 10: Ablation of SFT data filtering. All variants use the same base checkpoint, rapid SFT procedure, and evaluation settings. The final column is the mean over the six shared benchmarks.
| Tool | Arguments | Return |
|---|---|---|
| search | query : nonempty string array; topn : positive integer | Ranked results containing titles, URLs, and snippets for each query. |
| visit | url : nonempty URL array; goal : natural-language information goal | Goal-directed information extracted from the requested pages. |
Table 11: Tool schema for Deep Research trajectories.
| Benchmark | Before mixed SFT | After mixed SFT | Change |
|---|---|---|---|
| AIME 2025 | 10.00 | 43.33 | +33.33 |
| AIME 2024 | 6.67 | 33.33 | +26.66 |
| MBPP+ | 55.03 | 75.93 | +20.90 |
| BBH | 58.49 | 69.48 | +10.99 |
| LiveCodeBench v3 | 28.00 | 34.31 | +6.31 |
| IFEval | 72.64 | 78.00 | +5.36 |
Table 12: No-think performance before and after mixed think/no-think SFT. Both columns use matching benchmark versions and metrics; rows are sorted by score change.
| Category | Caps. | Probes | Primary evaluation scope |
|---|---|---|---|
| Agent | 8 | 120 | Environment interaction, long-horizon execution, recovery, memory, and coordinated action. |
| Arithmetic | 13 | 135 | Numerical operations, comparison, conversion, sorting, and exact calculation. |
| Causal | 5 | 75 | Mechanisms, interventions, counterfactuals, and latent-state inference. |
| Code | 26 | 389 | Program understanding, generation, debugging, optimization, testing, and artifact editing. |
| Dialogue | 6 | 90 | Multi-turn coherence, clarification, coreference, intent, persona, and subtext. |
| Instruction | 13 | 195 | Compliance with format, length, lexical, structural, stylistic, and multi-constraint requirements. |
Table 13: Evaluated ACE taxonomy, category scope, and probe allocation.
| Level | Example probe | Expected tool arguments |
|---|---|---|
| L1 | set_timer(minutes: int) . “Start a timer for 45 minutes.” | minutes=45 |
| L2 | ship_package(weight_grams: int, speed: standard/overnight) . “Send a 2.5 kg parcel by overnight delivery.” | weight_grams=2500, speed=overnight |
| L3 | submit_order(items, shipping) . “Order 3 of SKU A100 and 2 of SKU B250, and ship them to 12 Oak Lane in Canada.” | items=[(A100,3),(B250,2)] ; shipping=(12 Oak Lane, Canada) |
Table 14: Example L1–L3 probes for argument extraction.
Figure 17: Category-level ACE performance across models. Rows denote models and columns denote the 18 evaluated categories; each cell reports the corresponding pass rate (%).
| Core Contributor | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Task | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| Data cleaning | L3 | L2 | L3 | L3 | L2 | L3 | L3 | L2 | L2 |
| Data acquisition | L4 | L4 | L4 | L4 | L1 | L2 | L4 | L2 | L4 |
| Synthetic data generation | L4 | L3 | L3 | L3 | L3 | L3 | L3 | L3 | L3 |
| Model architecture design | L3 | L2 | L2 | L2 | L1 | L2 | L2 | L1 | L2 |
| Learning algorithm design | L3 | L2 | L2 | L3 | L2 | L1 | L2 | L1 | L2 |
Table 15: Individual assessments of AI autonomy across R&D tasks. Columns 1–9 denote Core Contributors 1–9, with identifiers held constant across tasks. Ratings follow the task-specific criteria in Appendix 13.2 .
| Benchmark | ZGCM-1-7B | Olmo 3 7B Instruct | Qwen3-8B (non-thinking mode) | Qwen2.5-7B |
|---|---|---|---|---|
| Reasoning & General | ||||
| MATH-500 | 89.40 | 87.30 | 82.30 | 71.00 |
| AIME 2024 | 33.33 | 44.30 | 26.20 | 11.30 |
| AIME 2025 | 43.33 | 32.50 | 21.70 | 6.30 |
| BBH | 69.48 | 71.20 | 73.70 | 68.80 |
| Code | ||||
Table 16: Selected no-think benchmark results (%) for ZGCM-1-7B and selected 7B–8B models. Dark blue cells in bold indicate the best result; light blue cells indicate the second-best result.