Supercharging Olmo-core for Efficient and Scalable MoE Training
Organizations: Allen Institute for AI · University of Washington
Abstract
Mixture-of-Experts (MoE) models add parameters without proportionally increasing per-token compute. However, training cost also depends on bytes moved, memory held, and kernels launched, and many of those costs follow total parameter size rather than active compute. On infrastructure built for dense models, this mismatch is costly: in early baselines, throughput fell as stored expert parameter size grew, even though the useful work per token stayed nearly fixed. To close this gap, we present a redesigned training stack in Olmo-core, built for efficient, scalable MoE training. We replace the Fully Sharded Data Parallel (FSDP) of earlier Olmo-core versions with an architecture built on Distributed Data Parallel (DDP), and add Expert Parallelism (EP), Pipeline Parallelism (PP), a distributed optimizer, and activation recompute. We propose NVSHMEM-based rowwise EP, which writes each token directly into its expert's buffer, avoiding dispatch and combine permutation overhead. We make EP fully synchronization-free by keeping routing metadata on the GPU and using device-scheduled grouped GEMMs. Closely integrated MXFP8 cuts compute, activation-memory, and transport costs, and topology-agnostic checkpoints let the parallel layout change between runs. On NVL8 B300 nodes, the stack supports training configurations from 12.9 billion to 1.2 trillion total parameters on up to 512 GPUs, reaching 858 useful-model TFLOP/s/GPU at 1.2 trillion with MXFP8 and per-layer recompute; an experimental capacity test with the optional DeepEP v2 backend reaches 2.38 trillion. We also explain each design choice and share findings for other training systems; for example, we identify a failure pattern we call Token Gerrymandering, in which the MoE router can learn to hack the load-balancing loss, and find that overlapping communication with computation can slow the overlapped kernels by more than it hides.
Figures & tables
| If you want to | Read |
| See the headline throughput and exactly what produced it | The panel at the end of Section 2 (p. 2 ), then Section 14 (p. 14 ) and Appendix B |
| Understand why FSDP was replaced for MoE | Sections 1.3 , 3 , and 4 (pp. 1.3 – 4.5 ) |
| Build or debug an MoE trainer | Sections 5 , 8 , 9 , and 12 (from p. 5 ), with Appendix A as the interface reference |
| Fit a larger model into memory | Sections 6 , 13 , and 14 , plus the offload budget in Section 15 (p. 15 ) |
| Adopt low-precision training | Section 11 (p. 11 ), then Sections 19 and 15 |
| Diagnose routing or load-balancing behavior | Section 10 (p. 10 ), with the rank-capacity policy in Section 10.3 |
| Model and primary source | Year | Routed experts | Parameters total/active | ||
| Mixtral 8x7B ( Jiang et al., 2024a ; Mistral AI Team, 2023 ) | 2023 | 8/2 | 47B/13B | ||
| DBRX ( Databricks AI Research Team, 2024 ) | 2024 | 16/4 | 132B/36B | ||
| Snowflake Arctic ( Snowflake AI Research, 2024 ) | 2024 | 128/2 | 480B/17B | ||
| OLMoE-1B-7B ( Muennighoff et al., 2024 ) | 2024 | 64/8 | 6.9B/1.3B | ||
| DeepSeek-V3 ( DeepSeek-AI et al., 2024 ) | 2024 | 256/8 | 671B/37B | ||
| Qwen3-235B-A22B ( Yang et al., 2025 ) | 2025 | 128/8 | 235B/22B |
| Method | What it divides | What one rank keeps during computation | When ranks communicate |
| Data-parallel and optimizer-sharding methods | |||
| DDP | Input examples across DP ranks | One complete copy of the rank-local model partition | Accumulated gradients synchronize before each optimizer update |
| Distributed optimizer | FP32 main weights and optimizer states within a replica group | Compute weights and the local optimizer shard | Gradient intake, local shard update, and compute-weight reconstruction once per optimizer step |
| FSDP / HSDP | Persistent training state—parameters, gradients, and optimizer data—across a shard group; HSDP repeats shard groups | Full weights for the current wrapped unit; other units remain sharded | Gather weights before each forward and backward use; free them after each use and reduce-scatter gradients in backward |
| Model and activation partitioning | |||
| EP | Routed experts | The assigned expert shard; dense and shared weights follow the other axes | Dispatch and combine in each routed layer and microbatch; synchronize expert copies once per optimizer step |
| No model parallelism: every DP/DDP rank holds the whole model | |||
| Training work | When it happens | Approximate cost | What changes for MoE |
| Forward and backward model compute | Every microbatch | for parameter-linear terms | This is the useful work. It follows active, not total, parameters. |
| Gradient storage and accumulation | Kept through the accumulation window; initialize once; add after each backward | Storage and zeroing ; additions recur for each microbatch | Buffer capacity follows total parameters. A rank’s experts share one stacked gradient tensor, so each accumulation can touch the whole tensor even when only some experts received tokens. |
| Gradient synchronization | Final microbatch, or every microbatch | Payload ; the exact factor depends on and the collective algorithm | Skipping it on the earlier microbatches (PyTorch’s no_sync() ) leaves one synchronization per optimizer step, but the payload still covers the full gradient replica, including experts that received no tokens. |
| Optimizer update | Once per optimizer step | Arithmetic and memory traffic | An inactive expert still has a weight and optimizer states; in this unsharded baseline every DP rank carries them. |
| Updated-parameter distribution | After the optimizer step | Local only in unsharded DDP | Every rank updates its complete replica, so there is no parameter collective. Parameter movement appears only after choosing a sharded optimizer. |
| Memory class | What it includes | Why it matters |
| Persistent training state | Compute weights; gradient-accumulation buffers; FP32 main weights; optimizer states and other long-lived optimizer buffers | Must remain available across microbatches or update boundaries. DDP, the distributed optimizer, EP, and PP determine which portions are replicated or sharded. |
| Execution working memory | Activations; materialization, routing, and communication buffers; kernel workspaces; allocator fragmentation; execution caches | Its contents and peak demand depend on microbatch shape, recomputation, routing, schedule, kernels, and runtime behavior. A preallocated fixed-address buffer can still belong to this class. |
| Phase | Block all-to-all EP | Rowwise EP | ||
| Operation | ms | ms | Operation | |
| Dispatch | Local permute | 0.351 | 2.298 | Route-map build + row dispatch |
| All-to-all | 1.960 | |||
| Global permute | 0.532 | |||
| Compute | Grouped GEMM | 3.630 | ||
| Combine | Global unpermute | 0.712 | 2.423 | Row gather + weighted reduce |
| MoE operation | Transfer | Row movement |
| Dispatch forward | PUT | The token-source rank pushes to the capacity row assigned to the selected expert. |
| Combine forward | GET | The token-source rank pulls the selected expert outputs and performs the router-weighted reduction locally. |
| Combine backward | PUT | The token-source rank pushes each router-weighted output gradient to the corresponding expert-output gradient row. |
| Dispatch backward | GET | The token-source rank pulls the route gradients and sums them into the gradient of the original token row. |
| Concurrent pair | GEMM time (ms) | GEMM slowdown |
| Grouped GEMM alone | 4.645–4.679 | — |
| GEMM + rowwise dispatch | 5.736–5.874 | 22.9–26.0% |
| GEMM + rowwise combine | 5.724–5.907 | 23.0–27.0% |
| Pair schedule (128-block launch) | Makespan (ms) | |
| Dispatch then GEMM, sequential | 7.768 | |
| Dispatch and GEMM, overlapped | 5.833 |
| Assumed activation copy: 10 GB | Assumed persistent training state: 12 GB | Budget: 80 GB | ||
| Schedule | Rank | Peak activation copies | Illustrative peak memory (GB) | Fits assumed budget? |
| Interleaved-1F1B | 0 | 11 | 122 | |
| 1 | 9 | 102 | ||
| 2 | 7 | 82 | ||
| 3 | 5 | 62 | ||
| 1F1B-V | 0 | 6 | 72 | |
| Axis | Traffic and clock | Placement consequence |
| DDP / optimizer | Parameter-sized gradient and updated-parameter buckets, once per optimizer step | Bandwidth and rail balance matter; accumulation microbatches amortize this boundary cost. |
| Full-reshard FSDP | Parameter shards materialized into full wrapped weights, around each wrapped microbatch execution | Repetition makes remote placement costly even when each collective is regular. |
| EP-MP | Routed token rows and returned expert outputs, at every routed layer in forward and backward for every microbatch | Fast locality, low latency, route balance, and small-message behavior all matter. |
| PP | Hidden activations and their gradients, at every remote stage boundary for every microbatch | Physical adjacency and independent-work windows determine the exposed P2P cost. |
| Operations | Effect | Why |
| Large dense and grouped GEMMs | Builds | Much device work per launch |
| Scaled dot-product attention | Builds | Same |
| Fused kernels and compiled regions | Builds | Many operations submitted as one launch |
| Autograd-engine backward | Builds | Roughly twice forward’s device work, submitted from C++ with low per-launch overhead |
| Python call stacks and dispatcher overhead | Consumes | Host time passes while little or no device work is queued |
| Compilation-cache lookups and metadata bookkeeping | Consumes | Same |
| Pattern | Host wait | Reason |
| x.item() | Yes | Materializes a CUDA scalar as a Python value. |
| x.cpu() followed by host use | Yes | The host-visible copy must wait for the required source data. |
| torch.nonzero(x) on CUDA | Yes | PyTorch documents a host–device synchronization for this dynamic-output operator. |
| torch.empty([int(x)]) | Yes, before allocation | The conversion of the CUDA scalar to a Python integer synchronizes; empty itself is not the cause. |
| Device indexing, gather, or scatter | Not inherently | These remain GPU operations when indices, shapes, and results stay on the device. |
| Mechanism | Host dependency removed | Cost accepted |
| Device-resident route metadata: route maps, counts, and offsets stay CUDA tensors | Python split lists interpreted per microbatch | Metadata kernels in every routed layer |
| Capacity-shaped buffers with a valid-row prefix | Data-dependent allocation shapes | Reserved capacity memory; possible route drops |
| Device-side cumulative offsets for grouped GEMM | item() per expert count and a host GEMM loop | Backend must accept device offsets |
| Phase | Left operand | Right operand | Contract | Result | Role of |
| Forward | Output rows | ||||
| Dgrad | Output rows | ||||
| Wgrad | Contraction length |
| Quantity | Created by | Consumed by |
| Router score or logit | Router projection and gating function | The configured score or logit drives top- ; selected unbiased affinities or logits determine mixture weights; normalized all-expert scores form LBL’s differentiable term |
| Requested count | Top- expert identifiers | LBL’s detached hard term, post-batch bias feedback, and capacity admission; on the rowwise path, all-gathered requested splits form the keep matrix that determines dispatch destination ranks and row offsets |
| Kept count | Capacity keep/drop policy | Dispatch and combine payload; expert grouped GEMM row counts on real-count paths; and executed-work diagnostics |
| Counts | Over experts | Over destination ranks |
| Requested | Selection concentration: the training signal the router asks for | Demand on the hottest rank: capacity pressure and drop risk |
| Kept | Executed expert shapes presented to grouped GEMM | Admitted receive, compute, and combine load on the rank that can straggle |
| Policy | Allocation unit | Unused capacity and overflow | Expert compute |
| Independent per expert | One limit for each local expert | Rows reserved by a cool expert cannot admit routes for a hot neighbor; overflow is expert-local | Padded BMM may execute capacity rows; grouped GEMM can still use real kept counts |
| Shared rank-wide | One aggregate limit for the destination rank | Local experts draw from one budget; overflow begins only when their aggregate fills the rank budget | Olmo’s rowwise path packs kept rows and uses real cumulative offsets; another sync-free path can execute capacity-padded splits |
| Scope | Statistics and semantics | Coordination and cost | Olmo status |
| Instance | One per-sequence hard histogram, reduced across tensor- and context-parallel shards as applicable, paired with each shard’s local differentiable soft-score contribution | Reductions induced by sequence sharding | Implemented as instance |
| Rank-local microbatch | Current rank’s current microbatch; unrelated local sequences can compensate, while CP shards remain local | No additional count communication | Implemented as local_batch |
| Synchronized current microbatch | Current hard counts reduced over the stage-local token balance group; each rank contributes its local soft term | One -element count all-reduce per routed-layer forward | Design only |
| GA-buffered online global | Synchronized counts accumulated through gradient accumulation; microbatch uses the prefix through | Same all-reduce plus an -element per-router buffer | Design only; analogous to Megatron global_aux_loss |
| Exact optimizer-batch global | Final hard counts and soft-score sums over all ranks and all accumulation microbatches | Delayed backward, retained graphs, or a routing prepass; stale statistics instead define an approximation | Not implemented |
| High coefficient ( ) | Low-coefficient branch ( ) | |||||
| Median over window | Early | Late | Change | Early | Late | Change |
| CE loss | 2.473 | 2.523 | 2.469 | 2.358 | ||
| Unscaled LBL sum | 46.864 | 41.278 | stable | |||
| Worst-case load imbalance | 2.748 | 6.317 | 2.655 | 2.544 | ||
| Dense peak (TFLOP/s) | ||||||
| GPU | CC | BF16 | FP8 | FP4 | Native MXFP8 | Matrix-instruction family |
| H100 SXM | 9.0 | 989.5 | 1979 | — | No | wgmma.mma_async |
| HGX B200, per GPU | 10.0 | 2250 | 4500 | 9000 | Yes | tcgen05.mma block_scale |
| HGX B300, per GPU | 10.3 | 2250 | 4500 | 13500 | Yes | tcgen05.mma block_scale |
| RTX PRO 6000 Blackwell, 600 W | 12.0 | 503.8 | 1007.6 | 2015.2 | Yes | mma.sync block_scale |
| Projection family | Prequantized GEMM faster from | Cached RHS, LHS Q included |
| Rectangular projections | ||
| , | — | No crossover in sweep |
| Projection | Raw MXFP8 GEMM | Shared, cached RHS | Routed, prequantized LHS | Routed, LHS Q charged |
| 1.00 | 0.25 | 0.52 | 0.31 | |
| 1.76 | 0.84 | 1.31 | 0.91 | |
| 1.84 | 1.45 | 1.70 | 1.43 | |
| 1.84 | 1.53 | 1.71 | 1.49 | |
| 1.83 | 1.62 | 1.76 | 1.46 | |
| 1.92 | 1.52 | 1.76 | 1.46 |
| OCP v1.0: floor | NVIDIA: round-ceiling ( rceil ) |
| a = max(abs(float(v[0:32]))) e = floor(log2(a)) - 8 e = clamp(e, -127, 127) s = 2**e for i in 0..31: q[i] = sat_E4M3( RNE(float(v[i]) / s)) return q, E8M0(e + 127) | a = max(abs(float(v[0:32]))) e = ceil(log2(a / 448)) e = clamp(e, -127, 127) s = 2**e for i in 0..31: q[i] = sat_E4M3( RNE(float(v[i]) / s)) return q, E8M0(e + 127) |
| Operation | Current path | Reason or alternative |
| SwiGLU + activation Q | Integrated fusion | Avoid a separate BF16 intermediate and Q pass |
| Route weighting + gradient Q | Integrated fusion | Avoid a route-expanded BF16 gradient before dispatch |
| Qdata + scale transport | Separate transfers | Paired and packed transport alternatives are outside the evaluated recipe |
| Routed grouped Wgrad | DQ, then BF16 grouped GEMM | The natural MXFP8 path was slower on the measured development shapes |
| Setting | Value |
| Model | 8 layers; 3.172B active and 13.037B total parameters |
| Experts and routing | 32 routed experts, top-4, one shared expert |
| Hardware and parallelism | Four B300 GPUs, EP |
| Global step | 524,288 tokens in eight gradient-accumulation microbatches |
| Matched across modes | Fixed-seed initialization and initialized weights, generated input batches, routing decisions, fused attention implementation, optimizer precision, compilation, and recompute policy |
| Timing window | Steps 1–14 untimed; throughput from total work and wall time over steps 15–24 |
| Saved item | Format | Size | 166.7B example (GiB) | What it restores |
| FP32 main weights | FP32 | bytes | 621.10 | The source-of-truth parameter values updated by the optimizer. |
| FP32 first moment | FP32 | bytes | 621.10 | Adam’s exponential moving average of gradients. |
| FP32 second moment | FP32 | bytes | 621.10 | Adam’s exponential moving average of squared gradients. |
| Optimizer bookkeeping | FP32 scalars | Negligible | Per-parameter step counters and rolling windows of recent losses and gradient norms. | |
| Trainer and run metadata | Mixed | Negligible | Step, token and epoch counters; data-loader position; random-number-generator state; callbacks; configuration; and data manifest. | |
| Total | bytes | 1,863.33 predicted 1,863.29 | BF16 compute weights and MXFP8 compute-weight caches are reconstructed after loading. |
| Policy | Checkpointed region | Retained at the boundary |
| Per-block | Every transformer block | Each block’s input; used by the measured endpoint below and, per layer, by the Ultra production recipe of Section 14 |
| Chunk | The whole block stack as one region | Only the embedding output between forward and backward; backward regenerates the whole stack’s activations at once |
| Selected blocks | Blocks named by key | The named blocks’ inputs; other blocks save normally |
| Attention region | The attention sub-region of a block | The sub-region input; expert compute and EP transport save normally |
| MoE expert region | The routed-expert compute and the unpermute that follow dispatch, on the block all-to-all EP path | The permuted expert input; dispatch and combine transport save normally |
| Setting | Value |
| Model | One dense block followed by seven MoE blocks; ; expert intermediate width 4096 |
| Experts and routing | 32 routed experts, top-4, one shared expert; uniform routes and zero token drops |
| Hardware and parallelism | Four B300 GPUs, EP |
| Batch | Two 8192-token sequences per rank microbatch; 524,288 tokens per optimizer step |
| Matched across variants | BF16 execution, identical normally initialized weights, identical input batches |
| Timing window | Steps 1–14 untimed (compilation and caches); throughput over steps 15–24 |
| Operating point | Active @ total Parameter activation | GBS | EP / PP | Precision | Recompute | TFLOP/s/GPU MFU (BF16 ref.) |
| Tiny-64E | 1.59B @ 12.91B | 8 Mi | 1 / 1 | BF16 | No | 903 40.1% |
| Small-64E | 4.29B @ 40.86B | 24 Mi | 8 / 1 | BF16 | No | 841 37.4% |
| Medium-64E | 7.37B @ 70.22B | 16 Mi | 8 / 2 | BF16 | No | 853 37.9% |
| Medium-96E | 7.38B @ 103.75B | 24 Mi | 8 / 2 | BF16 | No | 855 38.0% |
| Medium-128E | 7.38B @ 137.27B | 24 Mi | 8 / 4 | BF16 | No | 843 37.5% |
| Large-128E | 15.14B @ 295.99B | 32 Mi | 8 / 4 | BF16 | No | 768 34.1% |
| Direction | Intended benefit | Why it was not adopted | Limiting requirement |
| CPU activation offload | Lower saved-activation memory | The GPU produces activation bytes about six times faster than PCIe can move them to and from host memory; pinned host memory and prefetch scheduling add substantial machinery. | Host bandwidth or compression sufficient to match activation production. |
| Wave overlap | Hide dispatch and combine under expert compute | Waves reduce expert-GEMM size, while concurrent GPU communication stretches both the communication and GEMM regions on many tested shapes. | Enough hidden communication to offset smaller GEMMs and concurrent-kernel slowdown. |
| Two-batch overlap | Pipeline two half-batches through adjacent MoE blocks | The rowwise NVSHMEM kernels and the other lane’s compute compete for the same GPU resources; the extra lane also increases host scheduling. | Lower communication interference or less host work through replay. |
| MegaMoE megakernel | Coordinate communication and expert compute inside persistent CUDA kernels | The kernel grew into a distributed training-runtime project before it had a production backward and inter-node path. | Production-shaped forward scheduling, full backward, and inter-node transport. |
| Natural-layout MXFP8 Wgrad | Eliminate global dequantization before Wgrad | The naturally produced scales do not match Wgrad’s reduction layout; the prototype lost to global dequantization plus tuned BF16 grouped GEMM. | Wgrad-oriented requantization plus native block-scaled GEMM faster than the converted path. |
| Static CUDA Graphs | Remove repeated host launch work | Stable graph addresses conflict with several live pipeline microbatches; safe staging adds copies and memory. | Schedule-indexed saved-context storage with replay savings that exceed staging and retention costs. |
| One routed block, one rank microbatch | Size | Value |
| Saved activations multiples of BF16 elements | ||
| Attention: block input, normed input, grouped-query attention / / , attention output | 0.60 GB | |
| MoE norm inputs: residual and normed | 0.27 GB | |
| Routed experts: received rows, up/gate output, SwiGLU output, expert outputs ( ) | 2.68 GB | |
| Shared expert: up/gate output, SwiGLU output | 0.40 GB | |
| Candidate bytes | 3.96 GB | |
| Exact-natural prototype | Global DQ + BF16 grouped Wgrad | |
| 2,048 | 0.380 ms | 0.118 ms |
| 4,096 | 1.449 ms | 0.334 ms |
| 8,192 | 5.522 ms | 1.232 ms |
| Weight values | Grouped GEMM | Dense GEMM |
| Normal random | 31.059 ms | 24.724 ms |
| Uniform random | 31.682 ms | 25.829 ms |
| Random sign, fixed magnitude | 25.979 ms | 24.160 ms |
| All zeros | 18.896 ms | 19.424 ms |
| Constant fill | 20.606 ms | 24.350 ms |
| Uninitialized storage | 30.213 ms | 24.321 ms |
| Observation | What was measured | Source |
| Operand values change dynamic power | Random values drew 67% more average power than fixed values in a DGEMM; value distributions moved GEMM power by up to about 40% | A100, prior work |
| A power-limited GPU lowers its clock | Power-cap limiter active in 99.3% of telemetry samples; SM clock 1174 to 960 MHz | B300, Section 9 , Appendix A.12 |
| Lower throughput tracks the lower clock | Throughput 1366 to 1163 TFLOP/s at a nearly constant 1.16–1.19 TFLOP/s per MHz (per-shape medians), same kernel template | B300, Appendix A.12 |
| Operand values change time | Grouped-GEMM time fell by 16–39% and dense-GEMM time by up to 21% at fixed shape, layout, and dtype | B300, Table 34 |
| Architecture | Expert structure | Gate and evaluation | Why experts stay distinct |
| Adaptive local experts ( Jacobs et al., 1991 ) | Common expert bank; every expert receives the input | Normalized, input-dependent responsibilities for individual experts | Each expert receives a different dynamic coefficient |
| Hierarchical MoE ( Jordan and Jacobs, 1994 ) | Experts organized beneath a tree of gates | Input-dependent coefficients composed along each path | The effective coefficient varies by expert and input |
| Sparsely-gated MoE ( Shazeer et al., 2017 ) | Globally available feed-forward expert bank | Noisy top- routing evaluates and weights a selected subset | Both the coefficients and executed experts vary |
| One-gate Mixture-of-Experts (OMoE) ( Ma et al., 2018 ) | One expert pool supplies all task towers | One input-dependent vector gate forms a common dense mixture | One gating network still emits one coefficient per expert |
| Multi-gate Mixture-of-Experts (MMoE) ( Ma et al., 2018 ) | Expert parameters are shared across tasks | Each task has its own input-dependent softmax vector | The same input can expose a different mixture to each task |
| CGC / PLE ( Tang et al., 2020 ) | Multiple shared experts coexist with task-specific experts | Task gates separately weight both groups; PLE stacks these units | Every shared expert remains separately addressable |
| Convention | Coefficients | Immediate effect |
| Olmo | Unit-average routed coefficients and a unit-gain shared branch. | |
| Routed unit sum | The routed mixture receives one unit in total, while the shared branch independently receives unit gain. | |
| Equal branch mass | Routed and shared contributions split one unit of coefficient mass. | |
| Width-proportional unit sum | A single unit of coefficient mass is divided in proportion to active intermediate width. |
| Component in workload | ||||||
| Component | SFT | DPO | GRPO | PPO | Transfer | What transfers |
| Policy learner : the trained MoE | ✓ | ✓ | ✓ | ✓ | Learner mechanisms | The mechanisms in Parts II and III apply to the learner: distributed data parallel (DDP) synchronization and the sharded optimizer (Section 4 ), expert parallel (EP) placement and sync-free rowwise movement (Section 5 ), pipeline parallel (PP) layer placement (Section 6 ), grouped GEMM (Section 9 ), MXFP8 (Section 11 ), recompute (Section 13 ), and topology-agnostic checkpoints (Section 12 ). Response-only SFT changes the learner’s label tensor and loss denominator while retaining these mechanisms. |
| Reference policy : frozen, forward only | – | ✓ | ✓ | ✓ | Forward path only | The forward path only: sync-free dispatch, grouped GEMM, and the MXFP8 forward caches. DPO can cache reference log-probabilities before updates, as open-instruct does, avoiding a separate resident reference model. A resident reference adds parameter storage and forward activations, but no gradient buffers or optimizer states. |
| Critic : trained value model | – | – | – | ✓ | Second learner | A trained value model. If it uses the same MoE backbone as the policy, the learner mechanisms apply to both; its actual architecture determines the added memory. GRPO removes this model by using group-relative advantages. |
| Reward : verifier or reward model | – | – | None needed, or forward path only | A CPU-based checker needs no learner GPU, but verification cost depends on the task and execution environment. A learned reward model adds a separate forward-only workload. | ||
| Rollout generation : inference engine | – | – | Insight only, not code | An inference system outside this report’s measurements. At small decode batches, rows per expert can fall below the grouped-GEMM crossover (Section 9 ); batching and routing determine that count. Host-visible route metadata (Section 8 ), load imbalance (Section 10 ), and operand-dependent GEMM time (Section 19 ) are relevant costs, but the training measurements do not quantify them in a serving engine. | ||
| System | Weight materialization | Route representation | Update tensor precision | Overlap resource control |
| Olmo-core DDP (this report) | Rank-local compute weights stay resident across microbatches; experts partitioned by EP | Fixed-capacity, route-indexed rowwise movement with device-side agreement inside a node; optional DeepEP v2 across nodes; no host split lists | FP32 main weights, optimizer states, and gradients are the source of truth; BF16 and MXFP8 are derived caches and payloads | Phase-sequential by default; wave and two-batch overlap measured and not adopted |
| Megatron-Core MoE | Distributed optimizer: resident local compute weights; FSDP is a separate sharded-weight option | Dispatch as a backend choice, collective and optimized, drop-and-pad or dropless, including a sync-free dropless path | Configurable; example: FP32 main weights and gradients, BF16 moments; FP32 update arithmetic | Hiding conditioned on sufficient independent work; SM carve-out with a reported GEMM cost |
| PyTorch FSDP | Full-reshard policy: gather weights before forward and backward; release gathered copies after each | Not MoE-specific | FSDP2: original-dtype parameter shards for optimizer; compute and reduction dtypes configurable | Parameter prefetch; not an MoE dispatch concern |
| DeepSpeed-MoE | Gather/retention policy not specified in cited MoE paper | MoE-specific collectives at scale | Main-weight, gradient, and moment dtypes not specified in cited MoE paper | Not its emphasis |
| MegaBlocks | Gather/retention policy not specified in cited paper | Block-sparse expert kernels keep every assignment without capacity padding | Megatron-LM mixed precision; update-tensor dtypes not specified in cited paper | Not its emphasis |
| DeepSeek-V3 and DeepEP | V3 training: ZeRO-1 retains full local compute weights; experts partitioned by EP | V3 training: optimized dispatch and combine kernels; DeepEP v2 lowers communication’s SM footprint | V3 training: FP32 master weights and accumulation gradients; BF16 AdamW moments | V3 training: communication SMs assigned explicitly; cache and compute interference acknowledged |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Level | Term and responsibility |
| PyTorch operator | scaled_mm : accepts qdata and scale tensors, selects a backend, and returns the requested output dtype. |
| Device kernel | Block-scaled GEMM: tiles the matrices, moves operands and scales, runs the reduction over , and applies the output epilogue. |
| Tensor Core instruction | tcgen05.mma : computes one matrix tile update into an accumulator. |
| PyTorch entry point | CC major 9 or 10 | Other CUDA architectures |
| BF16 torch._grouped_mm | Device-offset fast path; no offset copy to the host | Host-loop fallback; offsets are copied to the CPU before the per-group GEMMs |
| Scaled torch._scaled_grouped_mm | Architecture gate passes | Rejected by the architecture gate rather than routed to the host-loop fallback |
| Rank-local microbatch | Expert layout | Projection | Routed rows | |||||||
| Scale | ||||||||||
| Small-2 | 3 | 8K | 4 | 8 | 64 | 8 | 2048 | 2560 | 96K | 12K |
| Small-3 | 4 | 4K | 8 | 8 | 64 | 8 | 2048 | 2560 | 128K | 16K |
| Small-4 | 3 | 8K | 8 | 8 | 128 | 16 | 1536 | 1536 | 192K | 12K |
| Medium-2 | 3 | 8K | 4 | 8 | 128 | 16 | 3584 | 4096 | 96K | 6K |
| Large-2 | 1 | 8K | 4 | 8 | 128 | 16 | 6144 | 8192 | 32K | 2K |
| HBM traffic category | Dense bytes | MoE bytes | MoE / dense |
| Weight reads | |||
| FFN input reads | |||
| Output writes, before combine | |||
| Up/gate writes + down input reads |
| Dense | 4 experts | 4 / dense | 8 experts | 8 / dense | |
| 1 | 7 | 4 | 0.588 | 4 | 0.620 |
| 16 | 111 | 67 | 0.601 | 68 | 0.608 |
| 128 | 854 | 520 | 0.608 | 548 | 0.643 |
| 256 | 1126 | 1086 | 0.964 | 1085 | 0.963 |
| 512 | 1293 | 1236 | 0.956 | 1230 | 0.951 |
| 1024 | 1381 | 1306 | 0.945 | 1298 | 0.941 |
| Boundary | Current path | Prototype or remaining cost |
|---|---|---|
| Weighted FP8 combine backward | Integrated weighting plus row quantization avoids a BF16 [N, K, D] intermediate; qdata and scales then use separate PUTs | A direct weighted-quantize+NVSHMEM-PUT kernel exists only as an unwired benchmark prototype |
| Qdata/scale transport | Separate regular transfers in the integrated path | Paired and packed transfers are benchmark prototypes, not live training |
| Natural-layout MXFP8 grouped Wgrad | Operands dequantize and use BF16 F.grouped_mm | Native MXFP8 grouped Wgrad exists only as a prototype; this report makes no end-to-end claim about it. |
| Scale swizzling and RHS prequantization | FP8 reduces payload size but adds separate quantization and scale-layout time, which can dominate small profiles | An RHS cache amortizes only the weight-side conversion; activation-side quantization and layout recur every microbatch |
| Item | Audited value |
| Compute | Four homogeneous nodes; eight NVIDIA B300 SXM6 AC GPUs per node; 32 GPUs total |
| GPU memory | 275,040 MiB reported per GPU |
| Driver/toolkit | NVIDIA driver 590.48.01; driver CUDA compatibility 13.1; installed CUDA toolkit 13.0.88 |
| Intra-node fabric | Every GPU pair reported as NV18; 18 links per GPU at 53.125 GB/s per link |
| Inter-node fabric | Eight active nominal 800-Gb/s XDR InfiniBand rails per node, one PXB-local rail aligned with each GPU; additional 100-Gb/s HDR ports present |
| Shared storage | Olmo-core checkout on WekaFS, visible identically from all four nodes |
| Role | Model and batch | Topology/precision | Experiment purpose |
| Profiling | 8 blocks, 32 routed experts, top-8, , sequence 8192, 0.75-Mi-token global batch | EP-MP 8; 1–4 nodes; BF16 rowwise by default; no PP, CP, TP, TBO, or recompute | Host-visible route-shape profile; synchronized/rowwise/DeepEP topology comparison; initial multi-node scaling |
| 30-block reference | 30 blocks, 128 routed experts, top-8, , sequence 8192, 18-Mi-token global batch | EP-MP 8; BF16 rowwise; no PP, CP, TP, TBO, or recompute in the base recipe | Full-reshard FSDP versus DDP-based training; production-scale throughput, efficiency, traffic, variance, and reliability |
| Quantity | 30-block reference | Profiling workload |
| Blocks and width | 30 blocks; first 2 dense, remaining 28 routed; | 8 blocks; first 2 dense, remaining 6 routed; same width and mixer family |
| Routed layer | , , routed intermediate width 1536, one shared expert | , , with the same expert widths and shared branch |
| Routing and capacity | Softmax top- , normalized route weights, capacity factor 1.25 | Same |
| Sequence and global batch | 8192 tokens; 18 Mi tokens per optimizer step | 8192 tokens; 0.75 Mi tokens per optimizer step |
| Configuration-derived size | 26.688B total, 2.906B active | 2.014B total, 0.995B active |
| Parallel/precision path | EP-MP 8; BF16 rowwise; no PP, CP, TP, TBO, or recompute | Same core path, with profiling callbacks and deliberately reduced model and batch |
| Prequantized input | Including Q | |||
| Rows | Eager | Graph | Eager | Graph |
| 128 | ||||
| 4,096 | ||||
| 32,768 | ||||
| Configuration | Model | Batch and scale | Parallel topology | Numeric/runtime path |
|---|---|---|---|---|
| Tiny-64E, no EP | 16 layers; 64 routed experts; 1.59B active, 12.91B total | GBS 8 Mi; rank microbatch 65,536; 2 nodes, 16 B300s | 16 / 16 / 1 / 1 | BF16; no recompute; local expert execution |
| Tiny-64E, EP8 | Same model as Tiny-64E, no EP | GBS 8 Mi; rank microbatch 65,536; 2 nodes, 16 B300s | 16 / 2 / 8 / 1 | BF16; rowwise NVSHMEM; no recompute |
| Small-64E, 24 Mi | 32 layers; 64 routed experts; 4.29B active, 40.86B total | GBS 24 Mi; rank microbatch 24,576; 16 nodes, 128 B300s | 128 / 16 / 8 / 1 | BF16; rowwise NVSHMEM; no recompute |
| Small-64E, 16 Mi | Same Small-64E model | GBS 16 Mi; rank microbatch 16,384; 16 nodes, 128 B300s | 128 / 16 / 8 / 1 | BF16; rowwise NVSHMEM; no recompute |
| Medium-64E | 39 layers; 64 routed experts; 7.37B active, 70.22B total | GBS 16 Mi; rank microbatch 16,384; 16 nodes, 128 B300s | 64 / 8 / 8 / 2 | BF16; rowwise NVSHMEM; no recompute |
| Medium-96E | 39 layers; 96 routed experts; 7.38B active, 103.75B total | GBS 24 Mi; rank microbatch 16,384; 16 nodes, 128 B300s | 64 / 8 / 8 / 2 | BF16; rowwise NVSHMEM; no recompute |
| Configuration | Routing | TFLOP/s /GPU | Observation and use |
|---|---|---|---|
| Tiny-64E, no EP | Learned | 895 | Began near 870 and stabilized after roughly 100 steps; learned-routing companion |
| Tiny-64E, no EP | Random | 903 | Headline; random-routing curve was nearly stable from the first step |
| Tiny-64E, EP8 | Learned | 820 | Began near 799, then stabilized; learned-routing EP8 run with no random-routing pair |
| Small-64E, 24 Mi | Learned | 768 | Learned-routing companion |
| Small-64E, 24 Mi | Random | 841 | Headline |
| Small-64E, 16 Mi | Random | 821 | Alternate smaller GBS and rank microbatch; not a repeat of the 24-Mi run |
| Metric | Definition |
| Steady-state step time | Wall time from the start of an optimizer step’s first forward to completion of optimizer update and required cache refresh, after declared startup/warmup. Depending on the retained samples, summaries may include median, mean, standard deviation, and tails |
| Useful tokens/s/GPU | Global non-padding tokens contributing to the normalized loss per measured step, divided by synchronized step time and GPU count; capacity-dropped routes are a separate quantity rather than silently removed |
| Useful-model MFU | Useful active-model FLOPs divided by elapsed time, GPU count, and the stated per-GPU compute peak. Sections 2 and 14 use the B300 dense BF16 peak of 2250 TFLOP/s/GPU for every row, including MXFP8, and label this common BF16-reference normalization explicitly. It is not utilization of the FP8 peak. Recompute and inactive experts add no useful FLOPs to the numerator |
| D2H route-shape wall time | Count and duration of host waits/copies causally between router metadata production and dependent dispatch/compute launches, plus the corresponding CPU launch gap and GPU idle interval from the same timeline |
| Communication exposure | Dispatch/combine or weight/gradient communication time not hidden by useful compute, with bytes, primitive, process group, and forward/backward scope identified |
| Route/drop statistics | Offered and accepted assignments, total and per-source drop rate, per-expert load distribution, capacity utilization, and rank skew |
| Term | Meaning |
|---|---|
| / PP | Pipeline degree. PP partitions layers and changes execution order, not which data-parallel ranks replicate each parameter. |
| / DP | Dense data-parallel degree. With CP, ordinary dense gradients reduce over the folded DP CP replica pool; dense optimizer state is sharded or replicated over DP and duplicated across CP. |
| / CP | Context-parallel degree. CP shards sequence before the PP schedule and participates in the folded expert pool. |
| / EP-DP | Expert replica degree. Ranks on this axis hold the same local EP-MP expert shard and reduce its ordinary gradients together. |
| / EP-MP | Expert-sharding degree. Ranks on this axis hold different routed-expert shards; it is not a gradient all-reduce group. |
| EP | Expert parallelism, used generically for distributing routed experts and moving token rows to ranks that store the selected experts. |
| Symbol or term | Meaning |
|---|---|
| / d_model | Transformer model width. |
| Routed expert intermediate width. | |
| Total shared-expert intermediate width (Section 20 ); distinct from the microbatch count . | |
| / num_experts | Total routed-expert count in an MoE layer. |
| / top_k | Number of routed experts selected per token. |
| Sequence length before context-parallel sharding. |
| Term | Meaning |
|---|---|
| persistent training state | Compute weights, accumulated gradients, FP32 main weights, and optimizer states that must remain available across microbatches or update boundaries. |
| execution working memory | Activations, routing and communication buffers, and workspaces used by execution. A preallocated fixed-address buffer can still belong to this class. |
| static placement / shape | A stable placement, buffer identity or address, capacity, shape, or graph structure. Static values need not be immutable. |
| symmetric memory | Collectively established storage that ranks in one team can address through one-sided operations. |
| Olmo symmetric memory | Current rowwise backend built around Olmo’s CUDA/NVSHMEM wrapper rather than the legacy Torch symmetric-memory path. |
| dispatch / combine | Movement from token order to expert-buffer order / return and router-weighted accumulation into token order. |
| Term | Meaning |
|---|---|
| MultiGroupDistributedDataParallel | Olmo reducer assigning a process group per parameter family and keeping persistent bucket views. |
| dense bucket | Gradient bucket reduced over the folded DP CP replica group. |
| expert bucket | Routed-expert bucket reduced over EP-DP, the group holding replicas of the same expert shard. |
| OLMoDDPOptimizer | Olmo optimizer that manages FP32 main weights, optimizer states, gradient intake, global gradient norm, copy-back, and checkpoint views. |
| compute weight | Weight representation consumed by forward or backward computation: a BF16 tensor for an ordinary parameter or a derived, orientation-specific MXFP8 qdata/scale cache for a selected linear. |
| FP32 main weight | FP32 parameter value updated by the optimizer and used to refresh its compute weights. |
| Model or implementation | Routed coefficients | Shared branch | Interpretation |
| Olmo, this report | Selected normalization followed by ; hence . | Unit gain, . | Unit-average selected coefficients; matches the additive dense-block reference under uniform selected routing. |
| Megatron-Core default, when shared experts are enabled | Post-top- softmax with by default; is configurable. | Unit gain by default; an optional learned sigmoid gate is available. | A configurable framework rather than one prescribed model recipe ( Yan et al., 2026 ; NVIDIA, 2026d ) . |
| DeepSeek-V3 released reference | Selected sigmoid affinities are renormalized and multiplied by . | Unit gain. | A model-specific routed scale between one and ( DeepSeek-AI et al., 2024 ; DeepSeek-AI, 2025a ) . |
| Qwen1.5-MoE and Qwen2-MoE released models | Full softmax over all experts followed by top- without selected renormalization; routed mass is . | Learned sigmoid gate. | An upcycled recipe in which routed mass and shared gain are both token dependent ( Qwen Team, 2024a ; Yang et al., 2024 ; Qwen Team, 2024b ; Qwen Team, 2024c ; Qwen Team et al., 2024 ) . |
| Qwen3.5 MoE public implementation | Selected top- probabilities are renormalized to sum to one. | Learned sigmoid gate. | Unit routed mass with a token-dependent shared contribution ( Qwen Team, 2026b ; Hugging Face, 2026 ) . |
| Mixtral | Softmax over the selected logits; routed mass is one. | No shared expert. | A routed-only weighted-average convention ( Jiang et al., 2024a ; Mistral AI, 2026 ) . |
Mixture-of-Experts Models
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
See all 488 papers