Multimodal model training breaks the loose coupling between data loading and model execution assumed by conventional training pipelines. Large per-sample variation in decoding cost, token/latent count, and tensor size causes both CPU preprocessing time and GPU compute time to vary substantially across data-parallel ranks (training processes). In synchronous training, these two sources of heterogeneity interact tightly: GPU execution determines how long the input pipeline has to prepare the next batch, while delayed preprocessing in any rank in turn stalls all GPUs at synchronization. As a result, optimizing CPU throughput or GPU load balance in isolation cannot eliminate pipeline bubbles. We present DataHelix, a metadata-first CPU-GPU co-scheduling data plane for multimodal training. DataHelix jointly reasons about preprocessing demand and GPU execution progress, treating each GPU compute window as the readiness budget for the next batch and coordinating preprocessing across ranks to minimize deadline misses rather than CPU makespan. To avoid expensive data movement, placement is decided while samples remain lightweight descriptors, before decoding and collation amplify them into large tensors. Descriptor-level token-latent-aware batching performs coarse pre-balancing, while cost-gated collective rebalancing corrects residual skew only when its expected benefit exceeds transfer cost. Across 11 video and multimodal workloads on 8 Ascend 910B accelerators, DataHelix reduces mean deadline-miss rate by 67.8% and mean p95 input-wait latency by 87.0%, while achieving 2.72x geometric-mean throughput over rank-local DataLoaders and 1.82x over a Ray actor pool under the same CPU budget.
Figures & tables
Figure 1. CPU–GPU coupling: preprocessing must meet per-rank readiness deadlines set by GPU execution.
Figure 2. Qwen-VL rank skew and long waits in native synchronous training (batch size 2, eight ranks). The upper panel shows Qwen-VL step-time p95 across eight ranks, with a maximum-to-minimum ratio of 1.44. The lower panel shows separately supplied Qwen-VL steps 42 and 50 with long data waits on Rank 3 and long synchronization waits on Rank 6. Segment widths are schematic; normal-weight labels report seconds.
Figure 3. Data path of multimodal training.
Figure 4. DataHelix design: metadata-only scheduling with DataServer preprocessing and tensor delivery.
Figure 5. Runtime architecture: descriptor routing, direct payload delivery, and scheduling feedback.
Table 1. Evaluation workload classes and concrete workloads. Data-only experiments measure the data plane in isolation; e2e experiments additionally include model forward, backward, and optimizer updates when a paired training model is available.
Figure 8. Qwen-VL rank-0 latency summaries. Compute includes gradient sync; bar totals sum components, not measured end-to-end step latencies.
Figure 9. Data-only throughput on Kinetics profiles.
Figure 10. End-to-end throughput across workloads and remote DataServer counts.
Figure 11. E2E baseline comparison across Native, Ray, and DataHelix .
Metric
Native
Ray actor96
DataHelix rd16
Tput. GM ↑
6.86 (1.00 × )
10.24 (1.49 × )
18.66 (2.72 × )
Wait p95 ↓
70.848s (1.00 × )
7.902s (0.11 × )
9.216s (0.13 × )
Ready p95 ↓
0.219s (1.00 × )
0.311s (1.42 × )
0.129s (0.59 × )
DDL miss ↓
0.506 (1.00 × )
0.554 (1.10 × )
0.163 (0.32 × )
Table 2. Aggregate end-to-end comparison across 11 workloads. Throughput uses a geometric mean; other metrics use arithmetic means. Ratios are relative to Native. Bold / underline values denote the best/2 nd -best results, respectively.
Figure 12. Multi-node throughput and DDL misses: Native versus DataHelix on mixed-profile workloads.
Figure 13. Batch-size scaling. Darker is better; each metric shares a color scale across methods. Heatmaps compare Native and DataHelix throughput, deadline miss rate, and input-wait p95 for per-GPU batch sizes 2, 4, and 8 and micro-batch sizes 1 and 2. DataHelix maintains nearly stable throughput as batch size grows and has lower deadline miss rates and input-wait p95 in every tested configuration.
Feature
Tput. ↑ (samples/s)
Input wait ↓ p95 (s)
DDL miss ↓ (%)
Cross-DS ↓ (GB/100 steps)
Video: Kinetics / Video-1B
Base DS
8.83
7.792
18.9
199.6
+Transport
8.03
7.908
20.2
203.5
+Prefetch/token
9.43
7.877
18.9
178.5
+Cost/DDL
9.87
7.430
16.5
233.0
Video: OpenVid / Video-1B
Table 3. End-to-end ablation. Bold / underline : best/second-best per workload, before rounding.
While ML model training and inference are both GPU-intensive, CPU-based data processing is often the bottleneck. Existing distributed data processing systems assume homogeneous resource requirements. They excel at CPU-based computation but either under-utilize heterogeneous resources or impose high overheads on failure and reconfiguration. We introduce Ray Data, a scalable system for heterogeneous workloads spanning batch inference and ML training. The key idea is to combine dynamic task creation for memory efficiency with dynamic task placement for operator elasticity. Making both decisions dynamically allows the system to adapt to heterogeneous pipelines where the memory and compute requirements per operator are unknown and variable over time. Ray Data addresses two challenges: (1) leveraging dynamic task creation for memory-efficient pipelining, and (2) designing a task placement policy that jointly allocates heterogeneous compute and memory resources to maximize throughput while limiting memory usage. Ray Data improves throughput on heterogeneous batch inference pipelines by 2.6-12× compared to traditional data processing systems. By leveraging heterogeneous clusters, Ray Data improves end-to-end training time for multimodal models such as Stable Diffusion by 31% compared to popular ML data loaders.
Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context windows grow, and encoder LLM scales diverge, a single LLM-centric TP/CP/PP/DP/EP layout increasingly limits throughput. This coupling forces encoders to inherit LLM-driven sharding and placement choices that can add communication, limit encoder parallelism, or constrain the LLM schedule; the mismatch is most pronounced at long contexts, where LLM context parallelism is needed for the fused multimodal sequence but encoder inputs remain bounded. We present heterogeneous parallelism for multimodal large language model training, an abstraction that lets modules in one end-to-end graph use independent layouts and rank placements, supporting colocated execution on shared GPUs and non-colocated execution on disjoint rank sets. The key challenge is preserving boundary tensor semantics across independent layouts: forward activations must be materialized for the destination layout, while backward gradients must be routed back to the source layout. We address this with boundary communicators that implement forward and backward layout transforms, plus scheduling extensions for both placement modes. We evaluate optimized homogeneous, colocated heterogeneous, and non-colocated heterogeneous configurations across multimodal workloads and GPU scales to characterize when added layout and placement freedom exposes a better operating point. Across this sweep, colocated heterogeneity improves TFLOPS/GPU by up to 49.3%, while non-colocated heterogeneity improves aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%. We validate loss convergence parity against homogeneous baselines and release the system as an open-source Megatron-LM extension.
Pipeline parallelism is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication. Existing pipeline systems typically consume static, profiled, or adaptively generated schedules as pre-committed execution orders. When realized task readiness diverges from the pre-committed order, stages may wait for not-yet-ready work even though other executable work is available, creating stage misalignment, idle bubbles, and reduced utilization. We present Runtime-Readiness-First Pipeline (RRFP), a readiness-driven runtime for pipeline-parallel training. RRFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint order for ranking currently ready work. To support this model, RRFP combines message-driven asynchronous communication, lightweight tensor-parallel coordination for collective consistency, and ready-set arbitration for low-overhead dispatch. We implement RRFP in a Megatron-based training framework and evaluate it on language-only and multimodal workloads at up to 128 GPUs. RRFP improves over fixed-order pipeline baselines across all settings. Using the BFW hint, RRFP achieves up to 1.77× speedup on language-only workloads and up to 2.77× on multimodal workloads. In cross-framework comparisons, RRFP with the default BF hint outperforms the faster available external system by up to 1.84× while preserving training correctness.