Edge SoCs pack CPUs, GPUs, and NPUs, yet vision--language model (VLM) runtimes keep generation on one engine and leave the others idle. Streaming VLMs expose further parallelism: new frames arrive while earlier queries are still being answered. Engines differ in compute and memory access and contend when they co-run, so no single placement of stages wins across SoCs and workloads. Backend-local KV state keeps other engines from continuing the work. \name coordinates heterogeneous accelerators to execute stateful VLM inference concurrently. It separates a stream's versioned context from each engine's KV state, so any engine can continue the stream by reusing, importing, or reconstructing that state. Stateful Streaming Generation ingests new frames on one engine while another answers, and Concurrent Sequence Generation serves different ready requests in parallel. State-aware dispatch sends each request to the engine expected to finish it first. Each answer reads a fixed, published version of its history and matches its serial reference. \name beats the strongest single-engine baselines on Apple M4 Max, Rockchip RK3588, and Qualcomm QCS6490, even when they reuse context. On two RK3588 video streams, it delivers 19.6% more throughput than the best single engine with context reuse and 27.2% more than parallel engines without shared state. Cross-engine KV import cuts first-token latency by a factor of 4.6 over prefix recomputation. Stateful Streaming Generation delivers up to 2.48× the throughput of serial execution, and state-aware dispatch cuts query completion time by up to 72.7%.
Figures & tables
Figure 1: Overview of CrossInfer. (a) Stateful Streaming Generation: One engine ingests new observations while another generates responses from published context, using state handoff. (b) Concurrent Sequence Generation: CPU and GPU execute ready requests in parallel, with state-aware dispatch based on queued work, context preparation, and co-running performance.
Figure 2: CrossInfer’s request throughput when one engine ingests new frames while another answers, relative to the same engines run serially; every answer matches its serial reference. On QCS6490, the serial run already reuses context.
Figure 3: Per-stage VLM latency on (a) RK3588 and (b) Apple M4 Max for one image (ViT Dosovitskiy et al. (2021) ) and 32 tokens (prefill and decode), shown on a log scale. RK3588 runs CLIP ViT-B/32 and Qwen2.5-0.5B-Instruct Yang et al. (2024b) ; M4 Max runs Qwen2.5-VL-3B (Q4_K_M) with Core ML Apple Inc. (2026) vision on the ANE. (c) QCS6490 effective read throughput for 256 MiB inputs, with the dashed line marking CPU-only throughput (Appendix F ).
Figure 4: Execution modes in CrossInfer. (a) The CPU builds Ct+1 while the GPU answers Qt from the published Ct . (b) Two lanes answer different queries over Ct : the CPU reuses it in place and the GPU imports it.
Figure 5: Single request on Apple M4 Max for FP16 Qwen2.5-VL-3B. CrossInfer uses Core ML on the ANE for vision and ggml on Metal for language; the baselines run on the GPU.
MLX
CrossInfer
Input
Time ↓
E2E ↑
GPU
Time ↓
E2E ↑
GPU
ANE
(s)
(tok/s)
(%)
(s)
(tok/s)
(%)
(%)
2 @ 0.5
2.21
67.87
85
2.34
64.10
80
27
4 @ 1
2.85
52.63
89
3.23
46.44
70
43
8 @ 2
3.86
38.86
93
4.19
35.80
55
56
16 @ 4
6.33
23.70
93
6.10
24.59
37
76
Table 1: CrossInfer versus GPU-only execution on Apple M4 Max. (a) Video-MME Fu et al. (2025) with 150 generated tokens per run; input is frames @ fps. (b) FastVLM-1.5B on an eight-frame, eight-query stream with context reuse; Serial is CrossInfer’s path without overlap. (c) All four configurations serve 0.126 req/s. Bold marks the better result.
Figure 6: Stateful Streaming Generation on RK3588. A–D: neither mechanism, ingestion during generation only, context reuse only, and both (CrossInfer). (a) Session completion time, with mean TTFT listed below; error bars show SD. (b) Execution trace with each query’s bound version; triangles mark context readiness.
Figure 7: Concurrent Sequence Generation against serial execution on M4 Max (top) and GPU-only execution on RK3588 (bottom). (a,b) Speedup and median TTFT, with dashed lines marking serial execution. (c,d) Eight-stream request throughput and p95 latency; error bars show SD.
Figure 8: Adding a second language lane. (a, b) Gain over the best single lane across prefill and generation lengths. (c) Co-running decode throughput, with per-lane slowdowns and the predicted M4 Max rate; the RK3588 CPU lane runs on cores 0–4, mostly Cortex-A55. (d) Gain per model for both execution modes, with precomputed visual embeddings on RK3588 and on-device vision on QCS6490 (Appendices G.7 and G.8 ).
Figure 9: State-aware dispatch on RK3588. (a) The faster fixed policy (color) and the W+P+S choice (circle). (b) Query completion time under short (S), medium (M), and long (L) backlogs. (c, d) Request throughput and mean TTFT on two concurrent video streams with 112-token answers; error bars show SD. Naive Parallelism runs both lanes without shared state.
Figure 10: Visual-token transport on Apple M4 Max: descriptor handoff versus Python tensor serialization and reconstruction. (a) Complete TABM transport round trip for an 800 KB visual payload. (b) Average frame throughput under long-prompt backlog.
Figure 11: (a) Qwen2.5-0.5B Yang et al. (2024b) decode throughput on the RK3588 CPU and GPU, alone and together. (b) Round trip of a 768 KB buffer by mapping versus copying. (c) Preparation plus query time for three OneVision-0.5B queries, CPU replay versus importing GPU-built KV.
CPU local
Metal local
CPU → Metal
Accuracy (%)
75.0
74.5
75.0
Table 2: Task accuracy (%) after cross-engine state handoff on 200 single-frame OVO queries per model. (a) Qwen2.5-VL-3B with 4-bit language weights on M4 Max. (b) INT4 language weights on QCS6490, with GPU-built context imported for CPU generation.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
System
GPU (J)
CPU+GPU+ANE (J)
Power (W)
CrossInfer
18.57
20.74
29.49
llama.cpp
25.80
27.13
38.94
MLX
18.50
21.52
30.34
Appendix
Table 3: Per-request energy and mean power on M4 Max for Qwen2.5-VL-3B in FP16 with 32 output tokens.
Figure 12: Single-request TTFT for Qwen2.5-VL-3B on M4 Max with separate and combined prefill (CrossInfer). Vision runs through Core ML on the ANE, language through ggml on Metal.
Weights
Correct / 200
Accuracy (%)
FP16
117
58.5
INT8
123
61.5
INT4
106
53.0
Appendix
Table 4: OneVision-0.5B language-weight precision on RK3588 OpenCL, using the same 200 single-frame OVO queries.
32-token answers
128-token answers
Peak RSS
Model
Serial
Overlap
Gain
Serial
Overlap
Gain
(GiB)
Qwen3.5-0.8B
3.618
6.009
66.1%
1.614
1.906
18.1%
1.261
Qwen3.5-2B
2.278
3.927
72.4%
1.069
1.324
23.9%
1.917
Qwen3.5-4B
1.434
1.963
36.9%
0.603
0.682
13.1%
3.394
Qwen3.5-9B
1.010
1.304
29.1%
0.417
0.486
16.6%
6.094
Appendix
Table 5: Inter-stage overlap across Qwen3.5 model sizes on Apple M4 Max: request throughput (req/s, three runs per configuration) of the same retained-context pipeline run serially and with overlap, the gain from overlap, and peak process RSS during overlapped execution.
Figure 13: Second-lane throughput and buffer handoff on RK3588. (a) Makespan and p50/p95 response latency over a 16-request stream. (b) Buffer handoff latency versus transfer size, medians of 50 iterations. The vertical line marks the 2.5 MiB that CrossInfer moves per request.
64 MiB
256 MiB
Configuration
CPU
GPU
NPU
Sum
CPU
GPU
NPU
Sum
CPU
15.20 ± 0.06
–
–
15.20
15.52 ± 0.04
–
–
15.52
GPU
–
12.20 ± 0.10
–
12.20
–
13.93 ± 0.11
–
13.93
NPU
–
–
10.61 ± 0.10
10.61
–
–
11.68 ± 0.06
11.68
CPU+GPU
4.63 ± 0.09
11.07 ± 0.09
–
15.71
2.44 ± 0.11
13.28 ± 0.11
–
15.72
CPU+NPU
8.63 ± 0.10
–
6.91 ± 0.08
15.53
8.10 ± 0.10
–
7.81 ± 0.09
15.90
Appendix
Table 6: Effective read throughput on QCS6490 Qualcomm Technologies, Inc. (2025) (GB/s, mean ± SD over five repeats) for engines reading alone and together.
Figure 14: Per-query TTFT for the four matched configurations in Figure 6 . A: neither mechanism; B: ingestion during generation; C: context reuse; D: both (CrossInfer). Each query uses the same bound version in all four configurations. Points and error bars show five-run means and sample standard deviations.
Figure 15: CPU-to-Metal execution on M4 Max: 2.48× request throughput with context reuse and ingestion during generation (D vs. A), and 12.2% additional throughput after reuse (D vs. C). Qwen2.5-VL-3B with 4-bit language weights processes eight frames and four queries, generating 128 tokens per query. A–D use the mechanism settings in Figure 6 .
Configuration
Completion (s)
Mean TTFT (s)
A
60.219
21.345
B
57.259
19.837
C
27.265
7.760
D (CrossInfer)
24.293
6.234
Appendix
Table 7: M4 Max CPU-to-Metal execution on the eight-frame stream. A–D use the same mechanism settings as Figure 15 .
Figure 16: Cross-query context reuse, including the initial cache miss. Four requests share a visual prefix and generate eight tokens each. Mean TTFT falls by 68.6% on RK3588 and 67.0% on M4 Max; reuse delivers 2.79× and 2.34× the end-to-end request throughput of rebuilding context, respectively. Subsequent requests avoid visual encoding and common-prefix construction. RK3588 bars show five-run means with sample-SD error bars.
Figure 17: RK3588 concurrent-stream sweep. Dashed lines mark the GPU-only baseline shared by all four concurrency settings.
Configuration
Throughput (req/s)
Mean TTFT (s)
CPU-only
0.15811
23.877
GPU-only
0.11250
31.741
Idle-first
0.14699
22.985
Reuse-first
0.10976
32.738
CrossInfer
0.15773
21.732
Appendix
Table 8: Online dispatch on QCS6490: request throughput and mean TTFT over five runs of the dense two-stream trace. Both single-lane baselines retain context.
OneVision-7B
FastVLM-0.5B
Configuration
Req/s
Mean TTFT (s)
Req/s
Mean TTFT (s)
Concurrent Sequence Generation
CPU-only
0.04664
46.592
0.53015
4.074
GPU-only
0.04727
44.653
0.32787
6.565
CrossInfer
0.06021
33.267
0.72757
3.073
Stateful Streaming Generation
Appendix
Table 9: RK3588 execution modes with OneVision-7B and FastVLM-0.5B, using precomputed visual embeddings. Each concurrent request generates 32 tokens; each stateful query generates 64. CPU+GPU serial uses the same ingestion, state-transfer, and answer paths as CrossInfer, but waits for the previous answer before ingesting the next frame.
OneVision
FastVLM
Configuration
0.5B
0.5B
1.5B
Concurrent Sequence Generation
CPU-only
0.2145
0.2143
0.0765
GPU-only
0.1690
0.1622
0.0744
CrossInfer
0.2453
0.2491
0.0897
Stateful Streaming Generation
Appendix
Table 10: QCS6490 request throughput (req/s) with on-device visual processing and 128 output tokens per query. CPU+GPU serial is defined as in Table 9 .
CPU
GPU → CPU
Model
INT4
INT8
FP16
INT4
OneVision-0.5B
47.0
59.0
58.0
50.5
FastVLM-0.5B
69.5
75.5
74.0
70.5
FastVLM-1.5B
74.0
72.0
73.0
72.5
Appendix
Table 11: QCS6490 accuracy (%) on 200 single-frame OVO questions per model. CPU columns vary language-weight precision with the same visual frontend. The last column uses INT4 language weights and GPU-built context imported for CPU generation.
Figure 18: Both execution modes in the earlier two-stream RK3588 run with EOS termination. Colors identify the streams. Solid language intervals ingest frames; hatched intervals process queries, including query prefill and generation. Query labels identify the bound context version. Vision and projection process all sixteen arriving frames.
112 tokens
EOS, 128-token cap
Configuration
req/s
TTFT
req/s
TTFT
Naive Parallelism
0.1261 (0.0016)
17.01
0.118
16.96
CPU, reuse
0.1341 (0.0004)
17.27
0.134
15.11
GPU, reuse
0.1191 (0.0007)
19.37
0.119
18.30
CrossInfer
0.1604 (0.0013)
11.25
0.147
12.14
Appendix
Table 12: Two-stream RK3588 request throughput (req/s) and mean TTFT (s). Parentheses give throughput SD over three runs for the fixed-112-token setting.
Measurement
CPU
Metal
Metal/CPU
Sequential read, 1 GiB (GB/s)
286.82
463.77
1.62×
Prefill, 1024 tokens (tok/s)
270.91
2163.41
7.99×
Decode, 32-token prefix (tok/s)
90.51
152.51
1.69×
Appendix
Table 13: CPU and Metal on Apple M4 Max for the Qwen2.5-VL-3B language model.
Prefix
CPU
Metal
CPU+Metal
Gain
32 tokens
90.51
152.51
208.16
36.5%
1024 tokens
82.89
149.09
199.64
33.9%
Appendix
Table 14: Decode throughput (tok/s) on Apple M4 Max with each lane decoding from a retained prefix, alone and co-running. The gain is over Metal alone.
Vision-language-action (VLA) models have demonstrated exceptional performance in natural language-driven perception and control. However, the high computational cost of VLA models poses significant efficiency challenges, particularly for resource-constrained edge platforms in real-world deployments. However, since different stages of VLA (observation, action generation and execution) must proceed sequentially, and wait for the completion of the preceding stage, the system suffers from frequent halting and high latency. To address this, We conduct a systematic analysis to identify the challenges for fast and fluent generation, and propose enabling VLAs with the ability to asynchronously parallelize across VLA stages in a "streaming" manner. First, we eliminate the reliance on action chunking and adopt action flow matching, which learns the trajectory of action flows rather than denoising chunk-wise actions. It overlaps the latency of action generation and execution. Second, we design an action saliency-aware adaptive observation mechanism, thereby overlapping the latency of execution and observation. Without sacrificing performance, StreamingVLA achieves substantial speedup and improves the fluency of execution. It achieves a 2.4 × latency speedup and reduces execution halting by 6.5 ×.
Continuous inference over concurrent video streams imposes substantial compute and memory demands on vision-language model (VLM) serving. Streaming inference uses sliding windows to maintain a bounded context of recent video, but processing each window independently repeats visual encoding and large language model (LLM) prefilling for similar and overlapping content. Existing optimizations provide limited coordination across these stages and often rely on model-specific training, profiling, or model-generated signals. We present CodecSight, a streaming VLM serving system that uses codec metadata as shared runtime guidance across visual encoding and LLM prefilling, without model-specific training or offline profiling. Codec-derived change signals guide patch pruning before visual encoding, reducing both visual computation and the number of downstream visual tokens. Codec-defined frame types guide selective key-value (KV) refresh across windows, while positional correction enables reuse of the remaining cached keys. Across three VLMs and four video workloads, our vLLM-based implementation supports up to 3.3× as many concurrent streams and achieves up to a 5.3× speedup in average time-to-first-token relative to the state-of-the-art baselines. It also reduces executed FLOPs by up to 93%, with a maximum task-quality decrease of 4.64 percentage points.
Yulin Zou, Wenyan Chen, Yan Chen +6
NTU Singapore Singapore · Beihang University China · A*STAR IAIC Singapore +1
On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a streaming speculate-and-refine framework for efficient on-device LLM inference. LeanStream progressively refines computation, loading, and cache-retention priorities using partial GPU results, enabling fine-grained overlap between GPU execution and storage I/O. We implement LeanStream on both mobile and embedded platforms. Compared with prior on-device LLM inference systems, LeanStream reduces memory usage by 4.8× to 7.5× at the best throughput achieved by prior work, while further improving token generation throughput by 1.6× to 2.1×.
Renyuan Liu, Yuyang Leng, Kaiyan Liu +7
Richard · Global Technology Applied Research, JPMorganChase · University of Kansas +1