Text-based model collaboration requires a large model to generate intermediate responses before a smaller model can use its guidance, adding large-model decoding cost to inference. We introduce POND (Prefill-Only, No-Decode), a method that uses a frozen large sender for prefill and a small receiver for all autoregressive decoding. A trainable bridge maps the sender's hidden states to additional attention keys and values, allowing the receiver to use these representations throughout generation. POND also supports periodic review, in which the sender processes the receiver's partial response and updates its guidance through additional prefill passes while preserving the receiver's existing KV cache. Across reasoning and coding benchmarks, POND improves receiver performance with senders from different model families. In particular, GLM-5.3 provides gains comparable to those of Qwen3.8-Max when paired with a Qwen receiver, supporting the effectiveness of representation transfer between independently trained model families. POND further demonstrates its ability to generalize across domains. When trained on math and coding tasks, it transfers directly to general knowledge tasks without additional domain-specific training. These results demonstrate that large frontier model prefill representations can improve small-model generation without requiring large-model autoregressive decoding.
Figures & tables
Figure 1: Overview of POND . A frozen sender prefills the input, and a trainable bridge compresses its last layer hidden states into a shared latent message, which layer-specific adapters project into KV entries for the receiver. The receiver, fine-tuned jointly with the bridge, generates the full response. At review boundaries, the sender prefills the newly generated segment and the resulting entries are appended to the latent memory, leaving the receiver’s KV cache untouched.
Tasks
Metrics
Receiver alone
Sender: Qwen3.8-Max
Sender: GLM-5.3
Sender: DeepSeek-V4-Flash
POND
+Review
Improve
POND
+Review
Improve
POND
+Review
Improve
Acc.
80.0
85.4
86.2
↑6.2
84.9
85.5
↑5.5
84.6
85.1
↑5.1
Token
982.4
836.7
768.9
↓21.7%
861.5
801.4
↓18.4%
879.2
829.7
↓15.5%
AIME
Speed
219.4
190.7
199.6
↓13.1%
196.8
205.1
↓10.3%
202.3
209.7
↓7.8%
Acc.
50.0
52.9
53.7
↑3.7
52.6
53.1
↑3.1
52.2
52.8
↑2.8
Token
1437.8
1211.6
1094.3
↓23.9%
1248.9
1152.7
↓19.8%
1281.4
1197.6
↓16.7%
Table 1: Math and code results. Senders are frozen and prefill-only. POND uses one latent transfer; +Review adds periodic prefill updates. Acc.: accuracy/pass@1 (%, higher is better); Token: total decoded receiver tokens, including both thinking and final-response tokens (thousands per 100 runs, lower is better); Speed: end-to-end latency (s/run, lower is better). Bold marks the better of POND and +Review for each metric, including ties. Improve compares the bold value with the receiver alone: accuracy gain in percentage points, or relative reduction in tokens/latency; an upward latency arrow indicates an increase. Avg averages the five tasks.
Tasks
Metrics
Receiver alone
Sender: Qwen3.8-Max
Sender: GLM-5.3
Sender: DeepSeek-V4-Flash
POND
+Review
Improve
POND
+Review
Improve
POND
+Review
Improve
Acc.
76.77
79.29
80.81
↑4.04
78.79
79.80
↑3.03
78.28
79.29
↑2.52
Token
660.3
568.9
504.7
↓23.6%
585.1
529.8
↓19.8%
601.4
548.6
↓16.9%
GPQA Diamond
Speed
149.2
128.6
136.9
↓13.8%
134.8
142.1
↓9.7%
139.7
146.5
↓6.4%
Acc.
89.58
90.47
90.86
↑1.28
90.22
90.63
↑1.05
90.04
90.39
↑0.81
Token
102.4
95.6
89.7
↓12.4%
97.1
92.4
↓9.8%
98.5
94.9
↓7.3%
Table 2: Knowledge benchmark results. Configurations, metrics, and metric-wise bolding follow Table 1 . Improve reports accuracy gains (percentage points) or relative token/latency reductions (%) against the receiver; upward latency arrows denote increases. Avg averages all four tasks. Differences and reductions are computed before rounding.
Method
Math/code
Knowledge
Frozen receiver
63.94
84.77
Receiver-only training
64.2
83.45
POND
66.9
86.44
POND + Review
68.02
87.18
Table 3: Training and review controls. Suite-average accuracy. Receiver-only training uses math/code data; POND uses Qwen3.8-Max as the sender.
Figure 2: Accuracy gains and sender compute savings. Left: Math and code scores for the receiver (Qwen3.5-4B), the receiver with POND , and the sender (GLM-5.3); labels give the gain in points. Center: Estimated sender FLOPs for GLM-5.3 (40B active) with 1k input and 10k output tokens. Prefill-only guidance reduces sender compute from 880 to 80 TFLOPs. Right: Fraction of sender compute saved, T/(L+T) , as a function of output length T with L=1 k. Estimates use 2Pactive FLOPs per token and exclude attention, routing, communication, and bridge or receiver inference. The panels are not a matched end-to-end comparison of accuracy and cost.
Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.
Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.
Zhenyi Shen, Junru Lu, Wei Liu +8
King’s College London, UK · Tencent Youtu Lab, China · National Physical Laboratory, UK
Diffusion language models (DLMs) offer substantial speed advantages through parallel decoding, but the lack of token dependencies limits generation quality compared to autoregressive (AR) models. Recent progress attempts to bridge the gap via importance sampling, with DLM being the proposal and AR being the target. However, due to the huge gap between their distributions, the sampling requires a large number of particles and is thus expensive to compute. In this paper, we introduce PoE-Bridge, a novel decoding framework that drastically improves generation speed and accuracy by introducing an intermediate distribution to bridge the gap. The distribution is constructed as a Product-of-Experts (PoE) of the DLM proposal and the AR target. With the intermediate distribution, we first use the DLM to draft multiple continuations in parallel, then apply rejection sampling to verify the drafted tokens and move the resulting candidates toward the PoE. We then use importance sampling to further correct the PoE-aligned candidates toward the AR target. We further propose several improved techniques, including mixed-temperature sampling for enhanced diversity and elastic rejection windows for reducing wasted verification. Empirically, PoE-Bridge achieves significantly improved accuracy with 5× speedup over the standard DLM decoding approach, and recovers at least 95% of the target AR model's performance, efficiently advancing most of the quality gap on challenging mathematical reasoning and coding tasks. Our code is available at https://github.com/juntongshi48/poe-bridge.
Long chain-of-thought reasoning has made autoregressive decoding the dominant inference cost of modern large language models. Existing methods target either the input side (latent compression) or the output side (speculative decoding and multi-token prediction, MTP), but the two lines of work have been pursued independently. Moreover, output-side methods must incur an expensive verifier pass to validate the unreliable draft tokens predicted by MTP. To address these issues, we propose Pair-In, Pair-Out (PIPO), which unifies both sides by viewing a latent compressor and an MTP head as mirror-image operations: the compressor folds two input tokens into one latent representation, while the MTP head unfolds one hidden state into one additional output token. To remove the verifier cost without sacrificing reliability, PIPO trains a lightweight confidence head that decides whether draft tokens should be accepted. We observe that On-Policy Distillation (OPD) naturally matches the rejection-sampling criterion of speculative decoding, so the confidence head can be trained alongside OPD with negligible extra cost. Experiments on AIME 2025, GPQA-Diamond, LiveCodeBench v6, and LongBench v2 with Qwen3.5-4B and 9B backbones show that PIPO improves pass@4 over regular decoding by up to +7.15 points, while delivering up to 2.64× first-token-latency and 2.07× per-token-latency speedups. Project Page: GitHub.com/RedAI-Infra/PIPO.
Wenhui Tan, Minghao Li, Xiaoqian Ma +5
Gaoling School of Artificial Intelligence, Renmin University of China · AI Platform, Xiaohongshu Inc. · University of Electronic Science and Technology of China +1