Revealing Domain-Spatiality Patterns for Configuration Tuning: Domain Knowledge Meets Fitness Landscapes
Organizations: IDEAS Lab, University of Birmingham, United Kingdom · University of Electronic Science and Technology of China, China · University of Birmingham, United Kingdom
Abstract
Configuration tuning for better performance is crucial in quality assurance. Yet, there has long been a mystery on tuners' effectiveness, due to the black-box nature of configurable systems. Prior efforts predominantly adopt static domain analysis (e.g., static taint analysis), which often lacks generalizability, or dynamic data analysis (e.g., benchmarking performance analysis), limiting explainability. In this work, we embrace Fitness Landscape Analysis (FLA) as a bridge between domain knowledge and difficulty of the tuning. We propose Domland, a two-pronged methodology that synergizes the spatial information obtained from FLA and domain-driven analysis to systematically capture the hidden characteristics of configuration tuning cases, explaining how and why a tuner might succeed or fail. This helps to better interpret and contextualize the behavior of tuners and inform tuner design. To evaluate Domland, we conduct a case study of nine software systems and 93 workloads, from which we reveal several key findings: (1) configuration landscapes are inherently system-specific, with no single domain factor (e.g., system area, programming language, or resource intensity) consistently shaping their structure; (2) the core options (e.g., pic-struct of x264), which control the main functional flows, exert a stronger influence on landscape ruggedness (i.e. the difficulty of tuning) compared to resource options (e.g., cpu-independent of x264); (3) Workload effects on landscape structure are not uniformly tied to type or scale. Both contribute to landscape variations, but their impact is system-dependent.
Figures & tables
| Landscape | Metric | Interpretation | Implications for tuner design/selection |
| Global structure | FDC ( ) ( Jones et al., 1995 ) | Guided: | A high correlation indicates that better configurations lie closer to the optimum, revealing predictable structural characteristics that favour exploitation-driven strategies (e.g., hill climbing ( Li et al., 2014 ; Xi et al., 2004 ) ). In contrast, weak or negative correlations illustrate that the landscape provides little or misleading structural signal, requiring exploration-driven methods (e.g., SWAY ( Chen et al., 2018a ) , GA ( Olaechea et al., 2014 ; Bäck and Schwefel, 1993 ) ). |
| Irregular: | |||
| Deceptive: | |||
| Basin of Attraction ( Malan, 2021 ) | : easy to reach optimum | A large global basin implies that even simple local search can easily reach the optimum. A small basin, by contrast, calls for exploration-enhancing mechanisms such as restarts or diversity injection to avoid premature convergence (e.g., ParamILS ( Hutter et al., 2009 ) ). | |
| : hard to reach optimum | |||
| Local structure | Number ( ) | Large: high multimodality | A large number of local optima increases the risk of premature convergence, favoring diversity-preserving methods (e.g., multi-objectivization ( Chen et al., 2024 ; Chen and Li, 2021 ) ). Fewer optima allow effective use of deterministic local search (e.g., hill climbing ( Li et al., 2014 ; Xi et al., 2004 ) ). |
| Type | Descriptions | Exampled Option | |
| Functional | Core ( ) | Governs the system’s core logic, directly impacting execution behavior. | MVSTORE in H2 |
| Utility ( ) | Provides auxiliary functionalities applicable across multiple modules. | anyScriptOrigin in Batik | |
| Resource | CPU ( ) | Controls CPU usage, parallel processing, or multi-threading. | threads in Dconvert |
| Storage ( ) | Manages persistent data storage. | VBRRange in Jump3r | |
| Memory ( ) | Handles cache, memory pools, and buffer. | blocksize in Kanzi | |
| Queue ( ) | Regulates scheduling, timeouts, and delays. | FlushTimeout in Xz | |
| System | Language | Area | Performance | Version | Resource Intensity | LOC | #O | #C | #W |
| Jump3r ( , 2023 ) | Java | Audio Encoder | Runtime | 1.0.4 | CPU | 29685 | 13 | 4196 | 6 |
| Kanzi ( , 2023 ) | Java | File Compressor | Runtime | 1.9 | I/O; Memory | 28614 | 18 | 4112 | 9 |
| Dconvert ( , 2023 ) | Java | Image Scaling | Runtime | 1.0.0- | I/O; Memory | 6888 | 10 | 6764 | 12 |
| H2 ( , 2023 ) | Java | Database | Throughput | 1.4.200 | I/O; Memory; Storage | 333539 | 16 | 1954 | 8 |
| Batik ( , 2023 ) | Java | SVG Rasterizer | Runtime | 1.14 | CPU | 360924 | 8 | 1919 | 11 |
| Xz ( , 2023 ) | C/C++ | File Compressor | Runtime | 5.2.0 | CPU | 43130 | 12 | 1999 | 13 |
| System | 7z | LLVM | |||||||
| Method | Data Scale | Basin of Attraction | Basin of Attraction | ||||||
| Large (50%) | 0.00758 | 0.85737 | 0.02092 | 1 | 0.01230 | 0.32979 | 0.08344 | 1 | |
| Medium (25%) | 0.03963 | 0.54504 | 0.00804 | 1 | 0.12177 | 0.15062 | 0.02307 | 1 | |
| Fixed | Small (5%) | 0.43036 | 0.11389 | 0.00000 | 1 | 0.66575 | 0.02754 | 0.00000 | 1 |
| Large (50%) | 0.00758 | 0.85737 | 0.02092 | 1 | 0.00009 | 0.92532 | 0.68311 | 2 | |
| Medium (25%) | 0.03963 | 0.54504 | 0.00804 | 1 | 0.00018 | 0.90927 | 0.82104 | 2 | |
| Lrzip | Xz | Z3 | Dconvert | Batik | Kanzi | X264 | H2 | Jump3r | ||||||||||
| Option | Type | RSD (%) | Type | RSD (%) | Type | RSD (%) | Type | RSD (%) | Type | RSD (%) | Type | RSD (%) | Type | RSD (%) | Type | RSD (%) | Type | RSD (%) |
| O1 | 107.254 | 2.118 | 1.333 | 7.422 | 0.14 | 1.136 | 1.162 | 6.703 | 1.695 | |||||||||
| O2 | 210.332 | 7.29 | 3.881 | 1.105 | 0.376 | 121.468 | 6.467 | 5.301 | ✗ | |||||||||
| O3 | 83.073 | 19.927 | 0.276 | 0.2 | 0.705 | 0.081 | 11.448 | 5.947 | 1.412 | |||||||||
| O4 | 499.477 | 20.944 | 1.182 | 0.099 | 1.224 | 164.22 | 8.785 | 0.133 | ✗ | |||||||||
| O5 | 306.08 | 2.042 | 2.047 | 0.615 | 0.476 | 32.901 | 13.159 | 0.152 | 9.008 | |||||||||
| Workload | Lrzip | Xz | Z3 | Dconvert | Batik | Kanzi | X264 | H2 | Jump3r |
| W1 | 0.025 | 0.381 | 0.488 | 0.461 | 0.319 | 0.207 | -0.083 | 0.637 | 0.439 |
| W2 | 0.186 | 0.460 | 0.535 | 0.524 | 0.280 | 0.328 | -0.015 | 0.627 | 0.067 |
| W3 | -0.037 | -0.198 | 0.545 | 0.468 | 0.520 | 0.123 | -0.014 | 0.641 | 0.363 |
| W4 | -0.078 | -0.099 | 0.492 | 0.511 | 0.550 | 0.181 | 0.065 | 0.257 | 0.267 |
| W5 | -0.050 | 0.566 | 0.098 | 0.523 | 0.360 | 0.123 | 0.031 | 0.637 | 0.443 |
| W6 | 0.019 | 0.612 | 0.018 | 0.524 | 0.438 | 0.192 | 0.155 | 0.653 | 0.117 |
| Wrokload | Lrzip | Xz | Z3 | Dconvert | Batik | Kanzi | X264 | H2 | Jump3r |
| W1 | 0.037 | 0.017 | 0.643 | 0.079 | 0.162 | 0.35 | 0.052 | 0.079 | 0.037 |
| W2 | 0.565 | 0.449 | 0.735 | 0.107 | 0.11 | 0.239 | 0.004 | 0.06 | 0.081 |
| W3 | 0.063 | 0.04 | 0.753 | 0.051 | 0.077 | 0.081 | 0.004 | 0.033 | 0.074 |
| W4 | 0.052 | 0.121 | 0.142 | 0.107 | 0.09 | 0.301 | 0.013 | 0.045 | 0.109 |
| W5 | 0.131 | 0.116 | 0.542 | 0.146 | 0.176 | 0.331 | 0.02 | 0.061 | 0.101 |
| W6 | 0.304 | 0.102 | 0.642 | 0.085 | 0.109 | 0.11 | 0.003 | 0.079 | 0.032 |
| Lrzip | Xz | Z3 | Dconvert | Batik | Kanzi | X264 | H2 | Jump3r | ||||||||||
| Workload | ||||||||||||||||||
| W1 | 0.120 | 0.972 | 0.025 | 0.767 | 0.015 | 0.628 | 0.006 | 0.787 | 0.017 | 0.754 | 0.086 | -0.333 | 0.180 | 0.096 | 0.026 | 0.244 | 0.041 | 0.376 |
| W2 | 0.194 | 0.992 | 0.203 | 0.592 | 0.125 | 0.784 | 0.005 | 0.753 | 0.009 | 0.906 | 0.105 | -0.234 | 0.164 | 0.072 | 0.025 | 0.071 | 0.042 | 0.534 |
| W3 | 0.099 | 0.987 | 0.025 | 0.749 | 0.024 | 0.967 | 0.026 | 0.569 | 0.010 | 0.788 | 0.147 | 0.421 | 0.176 | 0.098 | 0.024 | 0.232 | 0.044 | 0.510 |
| W4 | 0.105 | 0.945 | 0.021 | 0.818 | 0.016 | 0.966 | 0.005 | 0.739 | 0.011 | 0.890 | 0.084 | -0.338 | 0.173 | 0.075 | 0.028 | 0.316 | 0.048 | 0.450 |
| W5 | 0.099 | 0.830 | 0.025 | 0.633 | 0.012 | 0.992 | 0.008 | 0.718 | 0.011 | 0.879 | 0.086 | -0.247 | 0.176 | 0.092 | 0.030 | 0.101 | 0.042 | 0.396 |
| Workload | Lrzip | Xz | Z3 | Dconvert | Batik | Kanzi | X264 | H2 | Jump3r |
| W1 | -0.012 | 0.630 | 0.659 | 0.781 | 0.688 | 0.339 | 0.254 | 0.779 | 0.411 |
| W2 | 0.167 | 0.196 | 0.697 | 0.779 | 0.734 | 0.365 | 0.263 | 0.828 | 0.320 |
| W3 | 0.017 | 0.653 | 0.604 | 0.441 | 0.796 | 0.329 | 0.258 | 0.770 | 0.400 |
| W4 | -0.034 | 0.491 | 0.729 | 0.772 | 0.825 | 0.316 | 0.244 | 0.129 | 0.420 |
| W5 | -0.007 | 0.626 | 0.605 | 0.754 | 0.736 | 0.316 | 0.211 | 0.807 | 0.442 |
| W6 | 0.026 | 0.597 | 0.617 | 0.591 | 0.725 | 0.336 | 0.282 | 0.803 | 0.438 |
| Dconvert | H2 | ||
| Workload (Type-Scale) | Autocorrelation | Workload (Type-Scale) | Autocorrelation |
| jpeg-large | 0.781 | smallbank-1 | 0.779 |
| jpeg-medium | 0.779 | smallbank-10 | 0.828 |
| jpeg-small | 0.441 | tpcc-2 | 0.770 |
| png-large | 0.772 | tpcc-8 | 0.129 |
| png-medium | 0.754 | voter-16 | 0.807 |
| Lrzip | Xz | Z3 | Dconvert | Batik | Kanzi | X264 | H2 | Jump3r | ||
| Workload | Tuner | Mean (Std) | Mean (Std) | Mean (Std) | Mean (Std) | Mean (Std) | Mean (Std) | Mean (Std) | Mean (Std) | Mean (Std) |
| Transfer GA | 3.131 (0.014) | 3.773 (0.955) | 5.860 (0.016) | 1.797 (0.009) | 0.903 (0.009) | 1.129 (1.529) | 0.929 (0.258) | 26336.284 (1348.381) | 2.376 (0.478) | |
| W1 | Restarted GA | 3.133 (0.017) | 4.732 (1.299) | 5.907 (0.226) | 1.860 (0.092) | 0.924 (0.019) | 1.832 (1.368) | 1.097 (0.348) | 25721.608 (816.896) | 2.768 (0.658) |
| Transfer GA | 0.030 (0.000) | 0.011 (0.003) | 2.333 (0.656) | 1.078 (0.040) | 1.334 (0.006) | 0.133 (0.048) | 3.778 (1.110) | 18266.502 (1838.219) | 0.753 (0.131) | |
| W2 | Restarted GA | 0.030 (0.000) | 0.014 (0.005) | 1.761 (0.004) | 1.110 (0.040) | 1.360 (0.021) | 0.160 (0.044) | 4.005 (1.024) | 18378.785 (822.054) | 0.934 (0.272) |
| Transfer GA | 3.301 (0.003) | 4.014 (1.188) | 0.402 (0.695) | 0.371 (0.003) | 4.177 (0.020) | 0.427 (0.311) | 1.322 (0.300) | 932.575 (51.553) | 1.270 (0.357) |