Template Gap Reconstruction for Shell Output Simulation with Small Language Models
Organizations: Department of Computer Science, University of Texas at San Antonio, USA.
Abstract
Small language models offer a compact basis for shell simulation without executing submitted commands, but full-output decoding requires them to predict recurring utility structure alongside content that varies across executions. We introduce template gap reconstruction (TGR), which supplies reusable output literals interspersed with sentinels to be interleaved with autoregressively generated gaps. A template selector chooses which template to be reconstructed, with the empty template retaining full-output decoding. We construct a template bank from 18.7 million outputs collected across heterogeneous execution environments and train a template selector and five reconstruction models initialized from various small language model checkpoints on a target environment excluded from bank construction. Evaluation demonstrates that TGR improves output edit similarity (O-ES) over full-output decoding (FOD) when the optimal template covers more than 25% of the template, while performing similarly for lower-coverage outputs. Additionally, TGR outperforms FOD for shorter gap sequences, with O-ES values of 0.964 and 0.812, respectively, while maintaining similar performance for longer sequences. TGR reduces the number of tokens generated by 26.3--38.3% across used tokenizer families. These results support using a reusable output structure as a practical design choice for small shell simulators, improving response fidelity and shortening generation targets while keeping template selection and residual prediction central to response quality.
Figures & tables
| User | N CPUs | Kernel | Memory | Disk |
| root | 1 | 5.15.220 | 0.5G | 0.5G |
| non-root | 2 | 6.1.128 | 1G | 1G |
| 6.8.0 | 2G | 2G |
| Method | O-ES | BLEU-4 | ROUGE-L F1 |
| Cowrie | 0.354 | 29.12 | 35.52 |
| GPT-4o-mini | 0.488 | 42.63 | 50.07 |
| FOD | 0.789 | 76.43 | 79.67 |
| Selected TGR | 0.808 | 78.21 | 81.82 |
| Oracle TGR | 0.824 | 79.93 | 83.31 |
| Gap length (characters) | |||||
| Condition | 0 | 1–16 | 17–64 | 65–256 | 257+ |
| O-ES | |||||
| FOD | 0.799 | 0.855 | 0.783 | 0.827 | 0.790 |
| Selected TGR | 0.900 | 0.938 | 0.870 | 0.843 | 0.765 |
| Oracle TGR | 0.998 | 0.958 | 0.935 | 0.872 | 0.772 |
| BLEU-4 | |||||
| Coverage (%) | Gap ES | BLEU-4 | ROUGE-L F1 |
| 0.472 | 44.98 | 52.02 | |
| 0.865 | 83.97 | 87.16 | |
| 0.828 | 73.29 | 80.32 | |
| 0.797 | 73.90 | 78.92 |
| Gap characters | Gap ES | BLEU-4 | ROUGE-L F1 |
| 1–16 | 0.860 | 83.31 | 85.34 |
| 17–64 | 0.774 | 74.89 | 77.69 |
| 65–256 | 0.699 | 60.14 | 70.33 |
| 257+ | 0.510 | 50.61 | 55.07 |
| Tokenizer | FOD | TGR | Reduction % |
| Qwen3 | 42.89 | 31.45 | 26.7 |
| ByT5 | 129.98 | 80.16 | 38.3 |
| CodeT5 | 48.16 | 35.49 | 26.3 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Condition | O-ES | BLEU-4 | ROUGE-L F1 |
| Cowrie | Full response | 0.354 | 29.12 | 35.52 |
| GPT-4o-mini | Full response | 0.488 | 42.63 | 50.07 |
| Qwen3-0.6B | FOD | 0.775 | 75.50 | 78.41 |
| Selected TGR | 0.811 | 78.48 | 82.12 | |
| Oracle TGR | 0.828 | 80.66 | 83.78 | |
| ByT5-base | FOD | 0.784 | 75.62 | 79.22 |
| Template coverage (%) | |||||||
| Model | Condition | 0 | 100 | ||||
| O-ES | |||||||
| Cowrie | Full response | 0.329 | 0.173 | 0.505 | 0.475 | 0.441 | 0.520 |
| GPT-4o-mini | Full response | 0.403 | 0.280 | 0.451 | 0.360 | 0.343 | 0.309 |
| Qwen3-0.6B | FOD | 0.682 | 0.623 | 0.818 | 0.706 | 0.743 | 0.743 |
| Selected TGR | 0.680 | 0.650 | 0.844 | 0.877 | 0.854 | 0.910 | |
| Template coverage (%) | |||||||
| Model | Condition | 0 | 100 | ||||
| BLEU-4 | |||||||
| Cowrie | Full response | 24.50 | 10.10 | 34.04 | 36.89 | 28.77 | 41.98 |
| GPT-4o-mini | Full response | 32.61 | 16.08 | 41.13 | 29.11 | 24.67 | 20.94 |
| Qwen3-0.6B | FOD | 65.87 | 59.23 | 79.97 | 67.13 | 70.48 | 71.48 |
| Selected TGR | 65.83 | 60.41 | 79.65 | 80.75 | 80.33 | 90.66 | |
| Template coverage (%) | |||||||
| Model | Condition | 0 | 100 | ||||
| ROUGE-L F1 | |||||||
| Cowrie | Full response | 32.08 | 19.87 | 48.80 | 46.80 | 41.27 | 53.09 |
| GPT-4o-mini | Full response | 40.65 | 31.18 | 47.21 | 38.54 | 35.10 | 32.67 |
| Qwen3-0.6B | FOD | 68.93 | 65.26 | 83.05 | 70.82 | 74.97 | 75.97 |
| Selected TGR | 68.71 | 68.17 | 84.14 | 87.98 | 87.56 | 92.21 | |
| Template gap length | ||||||
| Model | Condition | 0 | 1–16 | 17–64 | 65–256 | 257+ |
| O-ES | ||||||
| Cowrie | Full response | 0.520 | 0.693 | 0.403 | 0.194 | 0.149 |
| GPT-4o-mini | Full response | 0.309 | 0.520 | 0.336 | 0.307 | 0.245 |
| Qwen3-0.6B | FOD | 0.743 | 0.860 | 0.680 | 0.763 | 0.700 |
| Selected TGR | 0.910 | 0.927 | 0.846 | 0.836 | 0.686 | |
| Template gap length (characters) | ||||||
| Model | Condition | 0 | 1–16 | 17–64 | 65–256 | 257+ |
| BLEU-4 | ||||||
| Cowrie | Full response | 41.98 | 52.09 | 27.75 | 11.09 | 12.95 |
| GPT-4o-mini | Full response | 20.94 | 45.04 | 23.21 | 21.36 | 16.60 |
| Qwen3-0.6B | FOD | 71.48 | 84.09 | 65.53 | 72.51 | 67.26 |
| Selected TGR | 90.66 | 88.57 | 79.68 | 78.07 | 64.08 | |
| Template gap length (characters) | ||||||
| Model | Condition | 0 | 1–16 | 17–64 | 65–256 | 257+ |
| ROUGE-L F1 | ||||||
| Cowrie | Full response | 53.09 | 65.75 | 42.39 | 19.23 | 18.31 |
| GPT-4o-mini | Full response | 32.67 | 53.50 | 34.69 | 33.12 | 29.33 |
| Qwen3-0.6B | FOD | 75.97 | 86.49 | 68.24 | 77.14 | 73.31 |
| Selected TGR | 92.21 | 92.88 | 85.89 | 85.35 | 71.69 | |
| Model | Gap ES | BLEU-4 | ROUGE-L F1 |
| Qwen3-0.6B | 0.597 | 55.91 | 61.67 |
| ByT5-base | 0.614 | 56.11 | 63.61 |
| ByT5-small | 0.601 | 54.84 | 62.57 |
| CodeT5-base | 0.684 | 65.76 | 70.56 |
| CodeT5-small | 0.683 | 63.54 | 69.34 |
| Mean | 0.636 | 59.23 | 65.55 |
| Template coverage (%) | ||||
| Model | ||||
| Gap ES (0–1) | ||||
| Qwen3-0.6B | 0.433 | 0.875 | 0.804 | 0.728 |
| ByT5-base | 0.443 | 0.869 | 0.808 | 0.781 |
| ByT5-small | 0.409 | 0.838 | 0.824 | 0.806 |
| CodeT5-base | 0.546 | 0.857 | 0.848 | 0.827 |
| Reference-gap length (characters) | ||||
| Model | 1–16 | 17–64 | 65–256 | 257–1,024 |
| Gap ES (0–1) | ||||
| Qwen3-0.6B | 0.829 | 0.708 | 0.660 | 0.436 |
| ByT5-base | 0.863 | 0.769 | 0.625 | 0.547 |
| ByT5-small | 0.878 | 0.747 | 0.687 | 0.472 |
| CodeT5-base | 0.856 | 0.790 | 0.760 | 0.589 |
Language Model Decoding
Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.
See all 179 papers