Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.
Figures & tables
Figure 1: Opportunities and challenges of Prefill–to-Autoregressive Cross-model Transfer (PACT). (a) PLatour consistently outperforms the open-source C2C ( Fu et al., 2026 ) across all generation tasks and surpasses the closed-source Mostik on math and coding, despite using a much smaller sender. (b) Superior benchmark performance does not necessarily imply effective information transfer from the sender. PACT-Check examines whether these gains depend on sender information through inference-time interventions and separately trained controls. On Chemistry, PLatour substantially outperforms all four sanity checks, providing evidence of meaningful information transfer. In contrast, on GSM8K, all four checks achieve comparable performance, suggesting that PLatour’s large gain there cannot be attributed to task-specific information from the sender.
Figure 2: Overview of PLatour . During prefilling, independently parameterized, zero-initialized linear projectors Wi map hidden states from matched sender layers into the receiver’s residual stream via additive fusion. The receiver subsequently generates the response without further sender involvement. Both model backbones remain frozen; only the projectors are optimized using GRPO. Orange dashed arrows schematically illustrate backpropagation through the frozen receiver to the trainable projectors.
Answer-Only / Without CoT
Generation / With CoT
Method
ARC-C
MMLU-Redux
Chemistry
Avg.
Chemistry
GSM8K
MATH500
Avg.
Receiver-Alone
74.23
53.96
39.05
55.75
41.43
80.86
73.40
65.23
Qwen3-235B → Qwen3-1.7B
C2C
92.75
83.68
60.95
79.13
41.01
75.09
64.60
60.23
KV-Lingo
91.21
77.88
40.95
70.01
40.95
17.06
11.95
23.32
PLatour-SFT
95.56
82.90
78.57
85.68
57.80
85.63
70.65
71.36
Table 1: Cross-model communication results (accuracy, %) using Qwen3 senders and a Qwen3-1.7B receiver . Arrows denote sender → receiver. Avg. denotes the arithmetic mean of the three scores in each group. Bold scores indicate the best PACT result for each sender. TextComm and Sender-Alone are references outside PACT.
Mathematics (DAPO-17k)
Coding (DeepCoder-24k Clean)
Method
Sender
AIME-80
AIME-135
HMMT
IMO-Ans.
Avg.
LCB v6
HumanEval
MBPP
Avg.
Receiver-Alone
–
59.22
59.12
46.35
41.13
48.90
40.36
83.16
65.08
62.87
PLatour
397B
71.88
72.73
61.77
49.25
60.97
52.50
93.67
85.43
77.20
PLatour
35B
75.63
74.63
64.58
49.75
63.32
54.29
94.97
89.93
79.73
Mostik
753B
–
70.74 ∗
–
–
–
54.14 ∗
86.13 ∗
82.00 ∗
74.09
Sender-Alone
397B
85.86
86.39
82.29
61.06
76.40
63.79
95.43
86.78
82.00
Table 2: Mathematical reasoning and code generation accuracy (%) with a Qwen3.5-4B receiver . Senders are Qwen3.5 models except for Mostik, which uses GLM-5.2 (753B). AIME-80 excludes the 55 problems overlapping with DAPO-17k; AIME-135 is the full set. IMO-Ans. denotes IMO-AnswerBench v2. Math Avg. is the unweighted mean of AIME-80, HMMT, and IMO-Ans.; coding Avg. is the unweighted mean of the three coding benchmarks. DeepCoder-24k Clean is decontaminated against all three coding test sets. Bold scores indicate the best result in each column, excluding Receiver-Alone and Sender-Alone. ∗ Mostik scores are reported results.
Figure 3: Inference efficiency. (a) Prefill latency, measured as time to first token (TTFT), versus input length. (b) Decoding time and (c) total inference time versus output length, with input length fixed at 4K tokens.
Figure 4: PACT-Check results for PLatour and PLatour-SFT . Panels (a)–(c) show Qwen3 results, and panel (d) shows Qwen3.5 domain averages. Each benchmark is normalized independently, with Receiver Alone at 0% and the evaluated PACT system at 100%. Bars show the fraction of the system’s gain reproduced by each control (DT, IT, SC, and TR); darker bars mark the strongest control. Gain and PACT-Score report absolute accuracy differences in percentage points: Gain compares the system with Receiver Alone, while PACT-Score compares it with the strongest control. A positive PACT-Score indicates performance beyond all tested controls, providing evidence for a benefit from sender information. In panel (b), TR uses the better of full-parameter SFT and LoRA-SFT for each benchmark to provide an SFT-matched comparison. MATH500 is marked “no-gain” because PLatour-SFT underperforms Receiver Alone. Per-benchmark accuracies are reported in Tables A20 and A21 .
Answer-Only / Without CoT
Generation / With CoT
Method
ARC-C
MMLU-Redux
Chemistry
Chemistry
GSM8K
MATH500
Sender-Alone gain (235B)
+22.02
+30.95
+17.14
+16.61
+13.49
+16.30
Gain over Receiver-Alone (pp)
C2C
+18.52
+29.72
+21.90
−0.42
−5.01
−5.50
KV-Lingo
+16.98
+23.92
+4.28
−0.48
−2.51
−3.55
PLatour-SFT
+21.33
+28.94
+39.52
+16.37
+4.77
−1.30
Table 3: Gains over Receiver-Alone and PACT-Scores, both in percentage points (pp), with a Qwen3-1.7B receiver. The first row reports the gain of the strongest standalone sender (Qwen3-235B) over Receiver Alone. For each method and benchmark, we select the sender (235B or 8B) yielding the highest original cross-model accuracy and compute both metrics for that setting. Gain is the selected accuracy minus Receiver-Alone accuracy. PACT-Score subtracts the strongest applicable control among Dummy Transfer (DT), Irrelevant Transfer (IT), Trained Receiver (TR), and Self-Communication (SC), using DT and IT from the selected sender setting and method-matched SC where available. TR uses the better of full-parameter SFT and LoRA-SFT for answer-only tasks, and the better of full-parameter GRPO and LoRA-GRPO for generation tasks, consistently across methods. Positive PACT-Scores indicate performance above all included controls. Unreported results are denoted by –.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Runtime adapter
GSM8K avg@2
Truncation rate
Sample-wise, unpadded
77.29
0.38
Batched, intermediate padding
76.42
0.30
Batched, tail-only padding
77.33
0.23
Appendix
Table A1: Comparison of our C2C runtime adapters with Qwen3-4B → 1.7B on GSM8K. All values are percentages. Runs share the same data, split, layer mapping, three-epoch schedule, batch size of 64, learning rate of 10−4 , and response limit of 2,048 tokens.
Dataset
Train
Holdout
Test
Qwen3
ARC-C
1,107
12
1,172
MMLU
9,900
100
5,632
Chemistry (no-CoT)
1,871
19
210
Chemistry (CoT)
1,871
19
210
GSM8K
7,398
75
1,319
Appendix
Table A2: Data partitions. Qwen3 rows show supervised training splits; Qwen3 GRPO uses the complete source training pools. Qwen3.5 rows show GRPO training splits. DAPO and DeepCoder use the external test benchmarks described in the text.
Method
Sender information
Trainable components
Training objective
Standalone models
Receiver Alone
None
None
—
Sender Alone
—
None
—
Existing communication methods
C2C ( Fu et al., 2026 )
KV caches
Communication modules
SFT
KV-Lingo ( Castin et al., 2026 )
KV caches
Bridge
Linear alignment + KL-based training
Appendix
Table A3: Comparison of baseline methods and PLatour. The table distinguishes the source of information, trainable components, and optimization objectives.
Setting
SFT
KV-Lingo
GRPO
Epochs
3
3
3
Global batch size
64
64
64 / 32
Learning rate
10−4
10−4
10−5
LR schedule
Linear
Cosine
Constant
Warmup ratio
10%
5%
0
Appendix
Table A4: Training hyperparameters. GRPO batch sizes count questions, each with eight sampled responses; Chemistry uses 32 questions per batch.
Task
Decoding
k
Token limit
Qwen3
ARC-C (val/test)
Greedy
1
2,048
MMLU (val/test)
Greedy
1
64
Chemistry (no-CoT) (val/test)
Greedy
1
64
Chemistry (CoT) (val/test)
Sampled
8
2,048
GSM8K (val/test)
Sampled
2
2,048
Appendix
Table A5: Evaluation settings. k denotes responses per question; token limits apply to individual responses. Rows marked “val” denote evaluations on the validation partitions; reported GRPO checkpoints are selected by mean validation accuracy. AIME includes both the full 135-problem set and the 80-problem subset excluding DAPO overlaps.
Method
2022
2023
2024
2025
2026 I
AIME-135
Receiver-Alone
60.00
57.29
67.92
52.50
56.67
59.12
PLatour (397B)
73.54
73.33
77.92
65.63
73.75
72.73
PLatour (35B)
73.13
72.08
81.25
70.21
78.33
74.63
Full-GRPO
75.83
76.88
82.08
72.08
79.58
77.04
LoRA-GRPO
74.38
73.75
78.96
65.21
77.50
73.56
Self-Communication
73.13
70.21
75.83
68.33
76.67
72.41
Appendix
Table A6: AIME-135 accuracy (%) by year, measured as avg@16 . Each of 2022–2025 contains 30 questions (480 responses), while 2026 contains 15 AIME I questions (240 responses). The full-set column pools all 135 questions (2,160 responses) and is not the unweighted mean of the five displayed yearly percentages. All rows use the 32K response limit and the original Qwen3.5 mathematical answer scorer.
Method
2022
2023
2024
2025
2026 I
AIME-80
Receiver-Alone
68.75
59.38
67.92
52.50
53.57
59.22
PLatour (397B)
78.13
70.31
77.92
65.63
71.88
71.88
PLatour (35B)
71.88
71.88
81.25
70.21
76.79
75.63
Full-GRPO
68.75
75.00
82.08
72.08
78.13
76.95
LoRA-GRPO
62.50
76.56
78.96
65.21
75.89
72.73
Self-Communication
62.50
73.44
75.83
68.33
75.00
72.42
Appendix
Table A7: AIME-80 accuracy (%) by year on the fixed, decontaminated subset of AIME-135. The retained question counts for 2022–2026 are 2, 4, 30, 30, and 14 (32, 64, 480, 480, and 224 responses at k=16 ). The last column pools 80 questions (1,280 responses); it is not an equal-weight average of years. In particular, the 2022 and 2023 entries describe only two and four questions, not their complete annual AIME competitions. Scores retain the original 32K generation and answer-equivalence protocol.
avg@16 (%)
pass@16 (%)
Method
Feb.
Nov.
All 60
Feb.
Nov.
All 60
Receiver-Alone
39.79
52.92
46.35
80.00
83.33
81.67
PLatour (397B)
54.38
69.17
61.77
86.67
90.00
88.33
PLatour (35B)
58.75
70.42
64.58
90.00
90.00
90.00
Full-GRPO
60.63
69.79
65.21
90.00
86.67
88.33
LoRA-GRPO
56.46
62.50
59.48
90.00
86.67
88.33
Appendix
Table A8: HMMT 2025 results by competition, using the same 30 February and 30 November problems for every method. Each competition has 480 responses, so avg@16 is the fraction of correct responses and pass@16 is the fraction of questions answered correctly at least once; “All 60” pools both competitions rather than treating them as separate benchmarks in the three-test math average. Sender-Alone 397B reaches 82.29% avg@16 and 93.33% pass@16 locally; the two metrics must not be conflated. For Receiver-Alone and Sender-Alone 35B, only responses previously truncated at 16K were independently regenerated at 32K; their other saved responses were reused. All other rows use the original full 32K test runs.
Sender
Prompt
MATH500
AIME
Tok.
–
Receiver-Alone
85.65
58.61
–
Qwen3.5-35B-A3B
T2T
86.10 (+0.45 [ − 0.85, 1.75])
58.98 (+0.37 [ − 1.81, 2.55])
47–58
Step-Back
85.75 (+0.10 [ − 1.50, 1.65])
–
138
Plan
86.15 (+0.50 [ − 1.20, 2.20])
–
134
Hint
86.05 (+0.40 [ − 1.25, 2.05])
61.48 (+2.87 [ − 0.09, 5.79])
102–265
Understand
87.30 (+1.65 [0.25, 3.05])
61.76 (+3.15 [0.60, 5.74])
274–299
Appendix
Table A9: TextComm prompt selection via Qwen3.5-4B receiver. Each cell gives avg@ k , followed by the gain over Receiver-Alone and its 95% confidence interval. Tok. is the average number of message tokens seen by the receiver (MATH500–AIME). Step-Back and Plan are evaluated only on MATH500 with the Qwen3.5-35B-A3B sender.
Sender
Prompt
AIME
HMMT
IMO-Ans.
–
Receiver-Alone
61.25
44.27
38.63
397B
T2T
60.46 ( − 0.79)
46.56 (+2.29)
41.69 (+3.06)
Understand
63.15 (+1.90)
49.06 (+4.79)
42.06 (+3.44)
35B
T2T
61.30 (+0.05)
45.63 (+1.35)
39.88 (+1.25)
Understand
62.92 (+1.67)
46.77 (+2.50)
41.94 (+3.31)
4B
Understand
57.36 ( − 3.89)
43.96 ( − 0.31)
39.63 (+1.00)
Appendix
Table A10: TextComm on the Qwen3.5 math benchmarks via Qwen3.5-4B receiver ( Answer: format, 32K tokens). Gains over Receiver-Alone on the same hardware are in parentheses. IMO-Ans. denotes IMO-AnswerBench v2.
Table A11: Ablation studies with training on the MATH dataset and evaluation on MATH500, reporting avg@4. Within each ablation group, all other experimental settings are held fixed. PLatour’s final configurations are bolded.
Answer-only
CoT Generation
Method
Layer Matching
ARC-C
MMLU-Redux
Chemistry
Chemistry
GSM8K
Qwen3-235B → Qwen3-1.7B
PLatour-SFT
Equal-width
89.25
77.22
79.05
50.18
82.52
Last- n†
95.56
82.90
78.57
57.80
85.63
PLatour-GRPO
Equal-width †
–
–
–
–
89.35
Last- n
–
–
–
–
86.50
Appendix
Table A12: Layer-matching ablation for PLatour-SFT and PLatour-GRPO with a Qwen3-1.7B receiver. All scores are percentages, with evaluation settings specified in Section B.4 . Unreported results are denoted by –. † marks the configurations used in Table 1 .
avg@4 (%)
Step
Last- n
Relative-depth (full-to-full)
0
83.65
83.25
30
88.60
91.25
59
82.95
89.15
89
89.20
90.00
118
90.75
91.10
Appendix
Table A13: Comparison of last- n and relative-depth (full-to-full) layer matching across training steps. All scores are avg@4, reported as percentages.
Method
Projection
Init.
Propagation
Accuracy (%)
Bridge Weight Norm
Receiver alone
–
–
–
80.86
–
C2C
Original
Random
No
75.85
16.61
Linear C2C
Separate K/V
Random
No
75.63
18.48
Linear C2C
Separate K/V
Zero
No
79.87
1.17
Linear C2C
Separate K/V
Zero
Yes
82.22
1.10
Linear C2C
Shared K/V
Random
No
74.68
18.48
Appendix
Table A14: GSM8K avg@2 accuracy with a Qwen3-8B sender and a Qwen3-1.7B receiver, using a 2,048-token response limit. Shared K/V uses the same linear map for keys and values; separate K/V uses independently parameterized maps. Propagation indicates whether lower-layer fused KV caches influence higher-layer ones.
GRPO step
Initialization
0
30
60
90
120
Zero
84.7
85.6
87.9
87.5
88.1
Random
79.3
82.4
83.7
84.4
85.1
Appendix
Table A15: MATH500 avg@4 accuracy (%) during direct GRPO training with zero and random projector initialization. Step 0 denotes evaluation before training.
Method / Recipe
ARC-C
MMLU-Redux
Chemistry
Chemistry (Gen)
GSM8K
MATH500
Receiver-Alone
74.23
53.96
39.05
41.43
80.86
73.40
Qwen3-235B → Qwen3-1.7B
C2C
92.75
83.68
60.95
41.01
75.09
64.60
PLatour-SFT
95.56
82.90
78.57
57.80
85.63
70.65
Original
91.21
77.88
40.95
40.95
17.06
11.95
PLatour Recipe
90.19
77.70
39.05
–
–
–
Appendix
Table A16: KV-Lingo training-recipe ablation on answer-only and generation tasks (accuracy, %). Original denotes KV-Lingo with cosine decay and 5% warmup; PLatour Recipe uses linear decay with 10% warmup. All other KV-Lingo settings remain unchanged. Italicized methods are included as reference baselines. Unreported results are denoted by –.
Method
Supervision Source
Chemistry (CoT)
GSM8K
MATH500
LoRA-SFT
Qwen3-235B
48.10
81.16
70.00
Qwen3-8B
38.99
81.69
72.55
Qwen3-1.7B
40.83
81.08
72.05
Full-SFT
Qwen3-235B
51.13
83.59
70.55
Qwen3-8B
40.65
82.56
73.40
Qwen3-1.7B
42.62
81.61
71.85
Appendix
Table A17: Fine-tuning results (%) with training responses generated by different Qwen3 models. All variants adapt the Qwen3-1.7B receiver. Bold values indicate the best supervision source within each method and benchmark, corresponding to the variants reported in Table A20 .
Sender → Receiver
SFT
GRPO
SFT / GRPO
GSM8K
235B → 1.7B
21.128
2.634
8.02×
8B → 1.7B
19.132
2.299
8.32×
Chemistry (CoT)
235B → 1.7B
11.568
1.855
6.24×
8B → 1.7B
9.400
2.178
4.32×
Appendix
Table A18: Aggregate Frobenius norms of the projection matrices across 28 receiver layers at the final training checkpoints. Each configuration contains 234,881,024 projection parameters.
Step
PLatour-GRPO
LoRA-GRPO
Accuracy
Avg. Length
Clipping
Accuracy
Avg. Length
Clipping
0
73.10%
878
2.30%
73.35%
872
2.45%
59
78.20%
1,275
9.65%
78.20%
1,269
10.50%
118
76.45%
992
5.55%
79.60%
1,433
13.20%
177
0.00%
2,438
59.40%
80.10%
1,486
14.45%
236
0.00%
2,246
54.75%
80.60%
1,505
14.40%
Appendix
Table A19: MATH500 avg@4 accuracy, average response length (tokens), and clipping rate during GRPO training. Clipping denotes responses reaching the 4,096-token generation limit. Both methods use a Qwen3-1.7B receiver; PLatour additionally uses a Qwen3-235B sender.
Figure A1: Training dynamics of PLatour on DAPO. We compare Qwen3.5-397B and Qwen3.5-35B senders paired with a Qwen3.5-4B receiver. Panels show (a) validation accuracy (avg@4), (b) mean response length, (c) the percentage of clipped responses, and (d) mean critic score. For training metrics, faint curves show raw values and solid curves show trailing 10-step means; validation accuracy is shown at evaluated checkpoints. The 397B-sender run exhibits a transient clipping spike around step 180, together with increased response length and a sharp drop in critic score. These training metrics subsequently recover, but validation accuracy remains below its earlier peak.
Answer-Only / Without CoT
Generation / With CoT
Setting
Method
ARC-C
MMLU-Redux
Chemistry
Avg.
Chemistry
GSM8K
MATH500
Avg.
Receiver-Alone
74.23
53.96
39.05
55.75
41.43
80.86
73.40
65.23
Cross-Model Transfer
PLatour-SFT
95.56
82.90
78.57
85.68
57.80
85.63
70.65
71.36
PLatour
–
–
–
–
69.29
89.35
78.20
78.95
Dummy Transfer (DT)
PLatour-SFT
75.68
57.90
39.05
57.54
38.04
80.71
70.90
63.22
PLatour
–
–
–
–
52.44
87.76
78.20
72.80
Appendix
Table A20: Comparison of cross-model communication with control settings (accuracy, %). All settings use a Qwen3-1.7B receiver. Cross-model rows use a Qwen3-235B sender; Self-Communication uses Qwen3-1.7B as both sender and receiver; Virtual Sender uses trainable virtual sender representations. LoRA-SFT reports the best result for each benchmark across the configurations in Table A17 . Avg. denotes the arithmetic mean of the three scores in each group and is reported only when all three scores are available. Underlined scores mark the strongest of Dummy Transfer, Irrelevant Transfer, Trained Receiver, and Self-Communication for PLatour-SFT or PLatour in each column, from which PACT-Score is computed in Table 3 .
Mathematics (DAPO-17k)
Coding (DeepCoder-24k Clean)
Method
AIME-80
AIME-135
HMMT
IMO-Ans.
Avg.
LCB v6
HumanEval
MBPP
Avg.
Receiver-Alone
59.22
59.12
46.35
41.13
48.90
40.36
83.16
65.08
62.87
Cross-model transfer
PLatour (35B)
75.63
74.63
64.58
49.75
63.32
54.29
94.97
89.93
79.73
Mostik (753B)
–
70.74 ∗
–
–
–
54.14 ∗
86.13 ∗
82.00 ∗
74.09
Controls for PLatour
Appendix
Table A21: Comparison of cross-model communication with control settings (accuracy, %). All settings use a Qwen3.5-4B receiver. Cross-model PLatour uses a Qwen3.5-35B sender; Mostik uses GLM-5.2 (753B) and is shown for comparison. Trained Receiver fine-tunes the receiver with GRPO, either with full parameters or with LoRA. AIME, IMO-Ans., and Avg. are as in Table 2 ; the controls and underlines are as in Table A20 . Our coding experiments use DeepCoder-24k Clean, decontaminated against all three coding test sets. ∗ Mostik scores are reported results.
Answer-only
CoT Generation
Method
Inference Setting
ARC-C
MMLU-Redux
Chemistry
Chemistry
GSM8K
MATH500
Receiver alone (1.7B)
Receiver-only
74.23
53.96
39.05
41.43
80.86
73.40
Best trained receiver
Per-task best
81.06
61.54
77.62
65.95
88.53
82.25
Qwen3-235B → Qwen3-1.7B
PLatour-SFT
Original
95.56
82.90
78.57
57.80
85.63
70.65
Dummy Transfer
75.68
57.90
39.05
38.04
80.71
70.90
Appendix
Table A22: Original, dummy-transfer, and irrelevant-transfer results (%) with a Qwen3-1.7B receiver. Original denotes standard inference with the trained communication modules. Controls follow Section B.7 . PLatour-GRPO results use the checkpoints selected by mean validation accuracy. DT and IT retain the corresponding original checkpoint without additional training. Unreported values are denoted by –.
Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.
Diffusion language models are a promising and more efficient alternative to autoregressive models. However, most RL post-training techniques are not easily adapted to these models. One issue is that diffusion language models generate answers through many intermediate states, deciding at each step which masked positions to reveal and which tokens to place in them. Many existing RL methods used in this setting approximate intractable sequence likelihoods or weight every denoising step by the same advantage, without explicitly distinguishing how these intermediate states of the denoising trajectory affect the expected reward. To address this issue of trajectory blindness, we introduce PACT (Potential-Anchored Consistency of Transitions). Motivated by a reward-tilted reference distribution, PACT jointly learns the policy and a scalar potential that scores each intermediate state, anchored by terminal reward. For each sampled denoising transition, a consistency loss matches the policy-to-reference log-probability ratio of revealed tokens to the change in this potential. This formulation requires neither sequence-likelihood estimation nor importance-ratio clipping. On four tasks, PACT improves LLaDA-8B-Instruct by +9.9 points on GSM8K, +12.0 on MATH500, +55.7 on Countdown, and +72.6 on Sudoku, and reaches the best average accuracy across the four tasks reported for this model, +3.7 points over the next best method. Further compared to AGRPO, it reaches that baseline's best accuracy in about half the training compute on GSM8K.
Large language models (LLMs) have demonstrated remarkable capabilities in solving complex tasks in coding, mathematical logic, and iterative reasoning, especially by coordinating effectively within multi-agent systems (MAS). However, language agents traditionally communicate via text exchange, which introduces a bottleneck in information transfer and adds prefill latency for downstream agents. Recent work proposes that directly sharing the key-value (KV) cache between language models can enable efficient latent collaboration, entirely bypassing standard text-based communication. To enable latent communication between heterogeneous models of different sizes, prior work introduces trained fusion adapters for each model pair, which limits practical deployment. We introduce CacheFuse, a training-free method which demonstrates that augmenting a downstream model with an upstream model's KV cache using linear interpolation consistently enables superior performance than the standalone downstream model can achieve. We systematically analyze CacheFuse across four reasoning and knowledge benchmarks and demonstrate up to 8.6 percentage points higher accuracy compared to single-model baselines and up to 12.2x TTFT speedup compared to text-based communication in multi-model inference.
Prompt length is a major bottleneck in agentic large language model (LLM) workloads, where repeated inference steps and multi-call loops incur substantial prefill cost. Recent work on speculative prefill demonstrates that attention-based token importance estimation can enable training-free prompt compression, but this assumes the existence of a draft model that shares the same tokenizer as the target model. In practice, however, agentic pipelines frequently employ models without any smaller in-family draft model. In this work, we study cross-family speculative prefill, where a lightweight draft model from one model family is used to perform prompt compression for a target model from a different family. Using the same speculative prefill mechanism as prior work, we evaluate a range of cross-family draft-target combinations, including Qwen, LLaMA, and DeepSeek models. Across a broad diversity of tasks, we find that attention-based token importance estimation transfers reliably across different model families despite differences in model architectures and tokenizers between draft and target models. Cross-model prompt compression largely retains 90~100% of full-prompt baseline performance and, in some cases, slightly improves accuracy due to denoising effects, while delivering substantial reductions in time to first token (TTFT). These results suggest that speculative prefill depends mainly on task priors and semantic structure, thus serving as a generalizable prompt compression primitive. We discuss the implications of our findings for agentic systems, where repeated long-context inference and heterogeneous model stacks make cross-model prompt compression both necessary and practical.