Despite the growing capabilities of large language models (LLMs), prompt design remains largely heuristic and ad hoc. This project will explore prompt minimization, the process of reducing prompts to their smallest, most information-dense form while preserving output fidelity. Practically, shorter prompts reduce computational overhead and inference latency, especially when large contexts, such as entire documents or codebases, are included unnecessarily. Further, longer prompts can damage LLM reasoning and accuracy. Theoretically, the existence of multiple prompts yielding equivalent outputs suggests a high degree of redundancy in the input space, raising fundamental questions about what information is essential to elicit specific model behaviors. We propose three variant frameworks to identify and evaluate minimal prompts and demonstrate that minimal prompts often produce outputs comparable to those of their longer counterparts. These findings suggest new directions for efficient prompt engineering and deepen our understanding of input compression in LLMs.
Figures & tables
Figure 1: Flowchart illustrating proposed approach 1 (Zero-Shot Learning). We use an LLM to minimize a long prompt and then compare that minimized prompt’s output to the expected output. This variant generates one candidate minimal prompt and uses zero-shot learning.
Figure 2: Flowchart illustrating proposed approach 2 (In-Context Learning). We use an LLM to minimize a long prompt, then compare the resulting prompt’s output to the expected output. This variant generates multiple candidate minimal prompts and uses in-context learning.
Figure 3: Flowchart illustrating proposed approach 3 (RL-Fine Tuning). We use an LLM to minimize a long prompt and then compare that minimized prompt’s output to the expected output. This variant fine-tunes the LLM using LoRA and PPO RL.
Llama 3.1 8B
Qwen2.5 32B
Algorithm
Comp
BERT
Comp
BERT
Zero-Shot Learning
0.08 ± 0.05
0.88 ± 0.03
0.17 ± 0.08
0.89 ± 0.02
In-Context Learning
0.04 ± 0.02
0.89 ± 0.02
0.19 ± 0.10
0.90 ± 0.01
RL Fine-Tuning
0.22 ± 0.11
0.89 ± 0.02
0.25 ± 0.09
0.89 ± 0.02
Table 1: Aggregate statistics for the different algorithms using the different LLMs
Figure 4: Best score trajectory of the 60 long context prompts using the meta-llama/Llama-3.1-8B-Instruct LLM. BERTScore and compression score are weighted equally at 0.5.
Figure 5: Best score trajectory of the 60 long context prompts using the Qwen/Qwen2.5-32B-Instruct-AWQ LLM. BERTScore and compression score are weighted equally at 0.5.
Figure 6: Best compression score compared between Qwen/Qwen2.5-32B-Instruct-AWQ and meta-llama/Llama-3.1-8B-Instruct for the same prompt. Lower is better.
Figure 7: Best BERT score compared between Qwen/Qwen2.5-32B-Instruct-AWQ and meta-llama/Llama-3.1-8B-Instruct for the same prompt. Lower is better.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Milestone Discoveries in Prompt Minimization using Zero-Shot Learning. The top panel shows the verbose initial prompt. The bottom panels compare the minimization trajectory of Qwen-32B (Left) and Llama-3.1-8B (Right) using Algorithm from Section 3.1 . Metrics indicate total score, BERT-score, and compression ratio. Running with Temperature 0.
Figure 9: Milestone Discoveries in Prompt Minimization using Zero-Shot Learning. The top panel shows the verbose initial prompt. The bottom panels compare the minimization trajectory of Qwen-32B (Left) and Llama-3.1-8B (Right) using Algorithm from Section 3.1 . Metrics indicate total score, BERT-score, and compression ratio. Running with Temperature 0.
Figure 10: Milestone Discoveries in Prompt Minimization using In-Context Learning. The top panel shows the verbose initial prompt. The bottom panels compare the minimization trajectory of Qwen-32B (Left) and Llama-3.1-8B (Right) using Algorithm from Section 3.2 . Metrics indicate total score, BERT-score, and compression ratio. Running with Temperature 0.9.
Figure 11: Milestone Discoveries in Prompt Minimization using In-Context Learning. The top panel shows the verbose initial prompt. The bottom panels compare the minimization trajectory of Qwen-32B (Left) and Llama-3.1-8B (Right) using Algorithm from Section 3.2 . Metrics indicate total score, BERT-score, and compression ratio. Running with Temperature 0.9.
Figure 12: Milestone Discoveries in Prompt Minimization using RL Fine-Tuning. The top panel shows the verbose initial prompt. The bottom panels compare the minimization trajectory of Qwen-32B (Left) and Llama-3.1-8B (Right) using Algorithm from Section 3.3 . Metrics indicate total score, BERT-score, and compression ratio. Running with Temperature 0.8.
Figure 13: Milestone Discoveries in Prompt Minimization using RL Fine-Tuning. The top panel shows the verbose initial prompt. The bottom panels compare the minimization trajectory of Qwen-32B (Left) and Llama-3.1-8B (Right) using Algorithm from Section 3.3 . Metrics indicate total score, BERT-score, and compression ratio. Running with Temperature 0.8.