Ripple: Deadline-Aware CPU-GPU Sharing for Embodied Inference
Organizations: MBZUAI.
Abstract
Unified-memory edge devices increasingly run LLM/VLM inference alongside time-critical CPU tasks for perception and control. GPU memory traffic can overwhelm these tasks, yet platforms such as DGX Spark expose neither bandwidth counters nor direct bandwidth regulation. We present Ripple, a CPU-GPU co-scheduler that combines model-derived GPU memory volumes with CPU completion feedback to regulate interference. Ripple divides inference into pausable, layer-based units and translates each CPU job's deadline slack into a GPU-byte budget, admitting work within the budgets of all overlapping jobs. Online updates track changing interference, while deadline-aware pacing and judicious SM footprint narrowing preserve inference progress. Implemented with SGLang engine hooks and a central scheduler, we evaluate Ripple on DGX Spark with four robotic CPU task mixes and LLM/VLM inference. It is the only evaluated approach that consistently meets the 2% CPU deadline-miss target and VLM deadlines across the tested deadline settings while retaining best-effort LLM throughput.
Figures & tables
| Device | Memory | Size | BW | BW | Power | Shared |
| (GB) | (GB/s) | /GB | /GB | w/ CPU | ||
| Edge, UMA | ||||||
| M4 Max | LPDDR5 | 128 | 546 | 4.3 | – | yes |
| Ryzen AI Max+ | LPDDR5X | 128 | 256 | 2.0 | 0.9 | yes |
| Jetson Orin | LPDDR5 | 64 | 205 | 3.2 | 0.9 | yes |
| Jetson Thor | LPDDR5X | 128 | 273 | 2.1 | 1.0 | yes |
| Task | Description |
|---|---|
| Costmap | Maps nearby obstacles from LiDAR scans. |
| KDNN | Finds nearby points to help align scans and track position. |
| SGM | Estimates object distances from two camera images. |
| Control | Computes quadrotor control commands using model-predictive control (OSQP). |
| ORB | Extracts visual features from camera images. |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Definition |
|---|---|
| 16.4 GB/token | Qwen3-8B weight bytes per batch-1 decode token used in the co-run calculation |
| GPU rate | Decode tokens/s 16.4 GB/token |
| CPU rate | Benchmark bytes processed divided by elapsed time |
| Combined rate | Sum of GPU and CPU rates |
| 273 GB/s | Nominal GB10 memory bandwidth |
| Means | Effect on CPU | Shortcoming |
|---|---|---|
| Uncore / memory-controller PMU | – | Not exposed by the firmware |
| DCGM and Nsight DRAM metrics | – | None for this GPU |
| CPU core PMU, GPU L2 counters | – | One side’s cache traffic only |
| MPAM, MIG, OS regulation | – | Not enabled, or blind to the GPU |
| GPU clock 2.4 1.1 GHz | p50 | Too small; locking halves decode |
| CPU clock | – | Limit accepted, not applied |
| Model / phase | Whole forward (ms) | Single operation (ms) |
|---|---|---|
| Qwen3-8B Prefill | 76.61 | 1.83–7.37 |
| Qwen3-8B Decode | 69.00–74.04 | 1.68–8.40 |
| Qwen3-VL Image extend | 285.23 | 0.05–4.72 |
| Qwen3-VL Decode | 19.48–21.10 | 0.02–4.69 |
| Engine and phase | Operations | Boundary groups | Execution mode |
|---|---|---|---|
| Qwen3-8B Prefill | 38 | Input preparation; 36 decoder layers; output head | Eager |
| Qwen3-8B Decode | 37 | 36 decoder layers (preparation included with the first); output head | Eager |
| Qwen3-VL Image/prompt | 108 | Preparation; vision input, attention/MLP regions, and merger; language input, 48 layers, norm, and output | Eager |
| Qwen3-VL Decode | 52 | Preparation; language input, 48 layers, norm, and output | Breakable CUDA graphs |
| Mix | Scheduler | |||||||
|---|---|---|---|---|---|---|---|---|
| Ripple | 4.06/0.82 | 4.59/0.85 | 5.45/0.55 | 6.10/0.45 | 7.00/0.78 | 8.08/0.10 | 8.44/0.08 | |
| Sereno UI-First | 7.25/85.50 | 7.25/80.47 | 7.25/76.15 | 7.25/71.67 | 7.25/66.95 | 7.25/26.60 | 7.25/4.57 | |
| Sereno Balanced | 9.53/100.00 | 9.53/100.00 | 9.53/100.00 | 9.53/100.00 | 9.53/99.85 | 9.53/63.95 | 9.53/24.78 | |
| SiGAMMA | 6.16/99.98 | 6.16/99.80 | 6.16/98.83 | 6.16/62.92 | 6.16/21.50 | 6.16/0.18 | 6.16/0.02 | |
| Pause | 8.03/98.92 | 8.03/98.65 | 8.03/91.25 | 8.03/86.33 | 8.03/84.20 | 8.03/28.93 | 8.03/1.08 | |
| Cap- | 8.21/99.98 | 8.37/99.88 | 8.21/99.55 | 8.21/93.42 | 8.21/84.05 | 8.21/19.83 | 8.21/1.28 |
| Setting | Mean tokens/s | Change |
|---|---|---|
| Default | 7.130 | — |
| ms | 7.130 | |
| Lease 26 ms | 7.010 | |
| 7.070 |
| Start | First entry (s) | Misses, 10 s |
|---|---|---|
| Profiled | 57.4 / 97.3 / 45.6 | 0 |
| No profile | 8.4 / 60.5 / 63.6 | 0 |
| Run | Whole-hour DMR (%) | Worst window (%) |
|---|---|---|
| 1 | 0.281 / 0.625 / 0.708 | 1.1 |
| 2 | 0.253 / 0.719 / 0.572 | 1.1 |
LLM Inference Scheduling
Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.
See all 195 papers