Retrieval Supersedes Scale: Efficient Log Level Prediction via Multiplex Socio-Technical Context
Organizations: School of Computing, Queen’s University, Kingston, ON, Canada. · École de Technologie Supérieure, Université de Québec, Montreal, QC, Canada.
Abstract
Developers insert logging statements in source code to record its execution, and the resulting logs are essential for detecting and troubleshooting system anomalies. A key part of these statements is the log level (e.g., DEBUG, INFO, WARN, ERROR), which determines whether a message is recorded. Choosing it is a trade-off: levels set too low mask important information in production, while levels set too high create noisy logs and extra I/O and storage overhead. Recent work uses Large Language Models (LLMs) with In-Context Learning (ICL) to predict log levels, but typically augments the prompt with randomly sampled examples. We argue that this overlooks the socio-technical nature of logging, where conventions vary by component and development team. Off-the-shelf LLMs using random examples are reported to achieve median AUCs of only 0.56 to 0.76, leaving room for improvement without fine-tuning. In this paper, we propose a retrieval approach for LLM-powered log level prediction that prioritizes context relevance over random selection and basic similarity. Our method retrieves logging examples relevant along two socio-technical dimensions: (i) functionality (code performing similar operations) and (ii) ownership (code maintained by the same developers). Evaluating four LLMs across ten popular Java projects, we find that this retrieval improves AUC over random baselines by 0.07 (median), and combining both signals yields median AUCs of 0.83 to 0.97. Our ablation studies show that each signal improves on random retrieval, and that combining them outperforms the stronger of the two on all ten projects. Our results also significantly exceed state-of-the-art supervised predictors (e.g., DeepLV) by a median AUC of 0.12 to 0.19. Our findings highlight the importance of socio-technical signals such as code functionality and ownership for LLM-powered log level prediction.
Figures & tables
| Project | Versions | # SLOC | # Contributors | # Logging statements |
|---|---|---|---|---|
| Hadoop | 3.3.6 | 1.8M | 635 | 21,910 |
| HBase | 2.5.9 | 841K | 483 | 10,449 |
| Elasticsearch | 8.9.2 | 2.5M | 2,042 | 25,187 |
| Cassandra | 4.1.9 | 595K | 464 | 4,808 |
| Flink | 2.3.0 | 1.7M | 1,671 | 3,800 |
| Kafka | 4.3.1 | 998K | 1,442 | 3,933 |
| Model | Metric | Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Multiplex retrieval (5) | AUC | ||||||||||
| Accuracy | |||||||||||
| AOD | |||||||||||
| Flat similarity retrieval (5) | AUC | ||||||||||
| Accuracy | |||||||||||
| AOD |
| Model | Cassandra | HBase | Hadoop | Elasticsearch | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket | Average | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | AUC | AOD | |
| CodeLlama-34B (Multiplex) | 0.914 | 0.884 | 0.926 | 0.880 | 0.932 | 0.894 | 0.969 | 0.913 | 0.887 | 0.850 | 0.860 | 0.826 | 0.877 | 0.857 | 0.879 | 0.840 | 0.879 | 0.873 | 0.836 | 0.811 | 0.896 | 0.863 |
| CodeLlama-7B (Multiplex) | 0.907 | 0.877 | 0.914 | 0.868 | 0.921 | 0.883 | 0.966 | 0.912 | 0.875 | 0.844 | 0.851 | 0.828 | 0.871 | 0.843 | 0.866 | 0.857 | 0.879 | 0.868 | 0.827 | 0.785 | 0.888 | 0.857 |
| GPT-4o (Zero-shot) | 0.806 | 0.798 | 0.776 | 0.807 | 0.809 | 0.814 | 0.760 | 0.780 | 0.832 | 0.828 | 0.765 | 0.766 | 0.858 | 0.843 | 0.792 | 0.836 | 0.759 | 0.857 | 0.755 | 0.708 | 0.791 | 0.804 |
| DeepSeek-V3.2 (Zero-shot) | 0.669 | 0.779 | 0.693 | 0.802 | 0.679 | 0.794 | 0.595 | 0.775 | 0.807 | 0.823 | 0.746 | 0.763 | 0.840 | 0.835 | 0.788 | 0.815 | 0.749 | 0.850 | 0.741 | 0.769 | 0.731 | 0.801 |
| Model Strategy | Inference Method | Cost | vs. DeepSeek |
|---|---|---|---|
| CodeLlama-7B | Local (Batched) | $0.18 | Competitive |
| CodeLlama-34B | Local (Batched) | $0.76 | +1,051% |
| DeepSeek-V3.2 | API (Token Rate) | $0.066 | Reference |
| GPT-4o | API (Token Rate) | $2.44 | +3,598% |
| Model Size | AUC | Cost per 1k Logs | Performance Gain (AUC) | Cost Increase |
|---|---|---|---|---|
| 7B (Base) | 0.89 | $0.18 | – | – |
| 34B (Large) | 0.90 | $0.76 | +0.9% | +322% |
| Model | Provider | Architecture | Parameters | Context | Access |
| (Total / Active) | (Window) | ||||
| Open-Weight Models | |||||
| CodeLlama-7B-Instruct | Meta | Dense Decoder | 7B / 7B | 16k | Open Weights |
| CodeLlama-34b-Instruct | Meta | Dense Decoder | 34B / 34B | 16k | Open Weights |
| DeepSeek-V3.2 | DeepSeek | MoE (Sparse Attn) | 685B / 37B | 164k | Open Weights |
| Proprietary Models | |||||
| Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket | |
| Step 1: multiplex cluster retrieval | 100.0% | 100.0% | 99.4% | 99.8% | 95.1% | 99.1% | 100.0% | 100.0% | 91.7% | 99.1% |
| Ranked by the socio-technical score | 90.7% | 67.3% | 97.3% | 95.3% | 85.8% | 95.0% | 95.9% | 93.8% | 91.7% | 64.3% |
| Sampled from the multiplex cluster † | 9.1% | 32.7% | 0.5% | 4.0% | 1.4% | 0.9% | 0.9% | 6.2% | 0.0% | 0.0% |
| Fewer than candidates in the cluster ‡ | 0.2% | 0.0% | 1.7% | 0.5% | 7.9% | 3.2% | 3.2% | 0.0% | 0.0% | 34.8% |
| Step 2: single-signal retrieval | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Step 3: project-wide random retrieval | 0.0% | 0.0% | 0.6% | 0.2% | 4.9% | 0.9% | 0.0% | 0.0% | 8.3% | 0.9% |
| Project | Adjacent Level Confusion | Files Covering 50% of Mispredictions |
|---|---|---|
| Cassandra | 64.1% | 13.3% |
| Elasticsearch | 64.7% | 16.5% |
| Flink | 78.6% | 21.8% |
| Hadoop | 72.8% | 14.7% |
| HBase | 67.0% | 13.9% |
| JMeter | 73.4% | 19.8% |
| Project | Modularity ( ) | Median ARI | Intra/Inter Cosine Similarity | Coverage |
|---|---|---|---|---|
| Hadoop | 0.798 | 0.524 | 0.851 / 0.123 | 99.0% |
| HBase | 0.805 | 0.286 | 0.618 / 0.018 | 67.0% |
| Elasticsearch | 0.850 | 0.556 | 0.720 / 0.268 | 99.5% |
| Cassandra | 0.919 | 0.588 | 0.651 / 0.093 | 96.0% |
| Flink | 0.933 | 0.447 | 0.733 / 0.037 | 95.9% |
| Kafka | 0.862 | 0.158 | 0.630 / 0.032 | 95.9% |
| Project | Files | Best Param. (mcs / ms) | # Clusters | Silh. | DBCV | Noise | Bootstr. ARI |
|---|---|---|---|---|---|---|---|
| Hadoop | 7,395 | 50 / 25 | 31 | 0.533 | 0.230 | 49% | |
| HBase | 2,691 | 25 / 12 | 27 | 0.566 | 0.325 | 41% | |
| Elasticsearch | 14,976 | 100 / 25 | 31 | 0.553 | 0.219 | 45% | |
| Cassandra | 3,158 | 25 / 12 | 30 | 0.606 | 0.347 | 37% | |
| Flink | 14,691 | 50 / 25 | 59 | 0.561 | 0.330 | 39% | |
| Kafka | 5,864 | 50 / 25 | 22 | 0.541 | 0.387 | 30% |
| Model | Metric | Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ownership-based (5) | AUC | ||||||||||
| Accuracy | |||||||||||
| AOD | |||||||||||
| Few-random shot (5) | AUC | ||||||||||
| Accuracy | |||||||||||
| AOD |
| Model | Metric | Hadoop | HBase | Elasticsearch | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Functionality-based (5) | AUC | ||||||||||
| Accuracy | |||||||||||
| AOD | |||||||||||
| Few-random shot (5) | AUC | ||||||||||
| Accuracy | |||||||||||
| AOD |
| Retrieval | Metric | Hadoop | HBase | Elastic. | Cassandra | Flink | Kafka | JMeter | ZooKeeper | Karaf | Wicket |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Functionality | AUC | 0.875 0.001 | 0.871 0.005 | 0.889 0.037 | 0.859 0.009 | - | - | - | - | - | - |
| Accuracy | 0.616 0.008 | 0.580 0.012 | 0.644 0.004 | 0.569 0.026 | - | - | - | - | - | - | |
| AOD | 0.850 0.004 | 0.838 0.007 | 0.865 0.001 | 0.844 0.011 | - | - | - | - | - | - | |
| Ownership | AUC | 0.824 0.002 | 0.812 0.006 | 0.905 0.001 | 0.838 0.012 | - | - | - | - | - | - |
| Accuracy | 0.544 0.004 | 0.508 0.018 | 0.582 0.005 | 0.504 0.020 | - | - | - | - | - | - | |
| AOD | 0.822 0.001 | 0.809 0.007 | 0.843 0.003 | 0.821 0.010 | - | - | - | - | - | - |
Software Engineering
Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.
See all 290 papers