A Flexible and Generic Approach for Explainable Landscape Analysis and the pyXla Toolbox
Authors: Tony Ombaso, Anna S. Bosman, Arnaud Liefooghe, Katherine M. Malan, Sébastien Verel
Organizations: Department of Computer Science, University of Pretoria, Pretoria, South Africa · African Institute for Data Science and Artificial Intelligence (AfriDSAI), University of Pretoria, Pretoria, South Africa · Univ. Littoral Cˆote d’Opale, UR 4491, LISIC, F-62100 Calais, France · Department of Decision Sciences, University of South Africa, Pretoria, South Africa
Landscape analysis has been successfully applied to understand complex optimisation problems, gain insights into algorithm behaviour, and automate algorithm selection and configuration. Although many landscape analysis techniques have been developed over the last decades, it remains difficult for researchers and practitioners to decide which approaches are appropriate and to implement them in practice. Some tools are available, but these are either restricted to particular problem domains (e.g., unconstrained black-box continuous optimisation), or are limited in what they model and measure. In addition, output from landscape analysis is often not easily interpretable, especially when computed landscape features do not correspond with aspects of problems that practitioners are familiar with. In this paper, we introduce a principled approach for explainable landscape analysis (XLA) with an associated Python package called pyXla. The approach is generic in that it applies to problems with different representations (continuous or combinatorial), with single or multiple objectives, with or without constraints. The extent of analysis provided by the XLA framework depends on the data available, with richer analysis offered as additional information is provided by the user. We demonstrate the explainable output produced by pyXla on a selection of hand-crafted problems with diverse landscape characteristics.
Figures & tables
Figure 1: A high-level diagram of the proposed XLA framework. Data loading and sampling are separated from analysis.
Input
Description
X
Solutions in the sample
Example headings for five decision variables: id x1 x2 x3 x4 x5
If not available as input data, X can be generated by pyXla using: specification of the domain of each variable with sampling strategy.
F
Objective values of the sample of solutions
Example headings for three objectives: id f1 f2 f3
If not available as input data, F can be generated by pyXla using: specification of objective function(s) and X .
Table 1: Description of the input data in pyXla . The id in X , F , V , I , and id1 and id2 in D and N link the data between files.
Function
Description
sample_X
Takes a domain specification for each variable with a sampling strategy as input and generates an X file.
compute_F
Takes an objective function as input and generates an F file. The X file is required.
compute_V
Takes a violation function as input and generates a V file. The X file is required.
compute_D
Takes a distance metric function as input and generates a D file. The X file is required.
compute_N
Takes an arbitrary neighbourhood function that determines if two solutions are neighbours and generates an N file. The X file is required.
Table 2: Description of the auxiliary functions in pyXla .
Figure 2: Comparison of the different sampling algorithms in 2D for 100 samples. Dots represent the sampled points. For ordered sampling methods, the connections between them indicate the sampling order. All samples lie within the bounding box [-5, 5]. Random walk ( Figure 2(c) ) and adaptive walk sampling ( Figure 2(d) ) are carried out with a step size of 0.5. The adaptive walk, which is guided by the objective function, is based on the sphere function ( Equation 1 ) in Figure 2(d) .
Figure 3: A diagrammatic summary of the landscape features currently available in pyXla . The transition diagram illustrates how richer analysis is provided as more input components are added. The Venn diagram depicts the various input component combinations, where the intersection between two circles indicates that both input components are present. Both the states of the transition diagram and sections of the Venn diagram are labelled with numbers. Each number label corresponds to a set of landscape features applicable to a given input component combination, as shown in the key, which is located to the right of the Venn diagram.
Feature
Requires
Description
distr_F
F
Name: Objective distribution
Visual output: Histograms of function values and function ranks for each objective
Numerical output: Descriptive statistics
distr_V
V
Name: Violation distribution
Visual output: Histogram of violation values and violation ranks for each constraint
Numerical output: Feasibility rate overall and per constraint, descriptive statistics
Table 3: Statistical landscape features included in pyXla . The symbols ∨ and ∧ refer to conjunction and disjunction, respectively, of the presence of their operands, i.e., input types.
Feature
Requires
Description
distr_Par
F ∨ V
Name: Pareto rank distribution
Visual output: Histogram of Pareto ranks ( Goldberg, 1989 ) for objective values, violations, and the combination (treating violations as objectives)
Numerical output: Descriptive statistics
distr_Deb
F ∧ V
Name: Deb’s feasibility rule ( Deb, 2000 ) ranks distribution
Visual output: Histogram of feasibility rules’ ranking
Numerical output: Descriptive statistics
Table 4: Rank-based landscape features included in pyXla . The symbols ∨ and ∧ refer to conjunction and disjunction, respectively, of the presence of their operands, i.e., input types.
Feature
Requires
Description
FDC
F ∧ D
Name: objective-distance correlation
Visual output: Scatter plots of objective values against distance to the nearest best solution in sample per objective, for all solutions or for feasible solutions only
Numerical output: Spearman’s correlation coefficients per objective
VDC
V ∧ D
Name: violation-distance correlation
Visual output: Scatter plots of violation values against distance to the nearest feasible solution in sample per constraint, for infeasible solutions only
Numerical output: Spearman’s correlation coefficients per violation
Table 5: Distance-based landscape features included in pyXla . The symbols ∨ and ∧ refer to conjunction and disjunction, respectively, of the presence of their operands, i.e., input types.
Visual output: A scatter plot of objective values between neighbours with a regression line for all solutions or feasible solutions only. The plot is divided by a broken line through the origin such that, assuming minimisation, points above the line are deteriorating neighbours, those below the line are improving neighbours, and those on the line are neutral neighbours.
Visual output: A scatter plot of violation values between neighbours for each constraint, for infeasible solutions only, with a regression line. The plot is divided by a broken line through the origin such that points above the line are deteriorating neighbours, those below are improving neighbours, and those on the line are neutral neighbours.
Table 6: Neighbourhood-based landscape features included in pyXla . The symbols ∨ and ∧ refer to conjunction and disjunction, respectively, of the presence of their operands, i.e., input types.
Problem
Domain
Objective(s)
Constraint(s)
Sampling
Sphere ( n=1 )
Cont.
eq. 1
eqs. ( 2 ), ( 3 )
Hilbert curve
Deceptive ( n=1 )
Cont.
eq. 4
eq. 5
Hilbert curve
Easom ( n=2 )
Cont.
eq. 6
None
Hilbert curve
Easy knapsack ( n=10 )
Combin.
Profit
Weight
Exhaustive
Hard knapsack ( n=10 )
Combin.
Profit
Weight
Exhaustive
NK ( n=14,k=1 )
Combin.
g
None
Exhaustive
Table 7: Selected problem instances, with n denoting the problem size. For NK landscape instances, g are random lookup tables that draw real numbers uniformly from the range [0, 1], see Section 4.1.5 for details. For Hilbert curve sampling, the sample size is 100 .
Figure 4: Line plots of the constrained sphere problem objective and constraints violation values.
Figure 5: Line plots of the constrained deceptive problem’s objective ( f0 ) and constraint ( v0 ).
Figure 6: Surface plot of the Easom function.
Figure 7: Neural network simulating the XOR gate. The weights and biases are labelled as wi and bi , respectively. The variables in1 and in2 correspond to columns Input A and Input B in Table 8 . The biases are multiplied by a constant input of 1.
Input A
Input B
Output ( A⊕B )
False
False
False
False
True
True
True
False
True
True
True
False
Table 8: XOR gate truth table.
Figure 8: Output of distr_F : histogram of objective values for the constrained sphere problem and the Easom function.
Statistic
Sphere
Easom
Minimum
0.0015
-1.5046
Maximum
25.0
0.0277
Mean
8.7426
-0.0606
Median
6.5597
6.5705e-33
Q1
1.8607
-1.4481e-16
Q3
14.873
1.0137e-08
Table 9: Numerical output of distr_F for the constrained sphere problem and the Easom function.
Figure 9: Output of distr_F : histogram of objective values for simulating the XOR gate using a neural network: ( 9(a) ) corresponds to the distribution of loss values for an SGD trajectory in parameter space, ( 9(b) ) shows the distribution of loss values when the neural network parameters are randomly sampled.
Figure 10: Output of distr_V : histogram of violation values (top row) and ranks (bottom row) for the easy and hard knapsack instances.
Figure 11: Output of corr : scatter plots of pairs of objectives for an NK instance with 3 objectives
Figure 12: Output of corr_ranks : scatter plots of pairs of ranks for the constrained sphere problem, split by feasibility; feas. is an abbreviation for feasible.
Figure 13: (Continued) Output of corr_ranks : scatter plots of pairs of ranks for the constrained sphere problem, split by feasibility; feas. is an abbreviation for feasible.
Figure 14: Output of X_imp for scenarios simulating the XOR gate using a neural network: ( 14(a) ) corresponds to the SGD trajectory in parameter space, and ( 14(b) ) corresponds to the scenario where neural network parameters are randomly sampled. The bar charts in the top row show the correlation values between each variable and the objective. The bar charts in the bottom row show the permutation feature importance scores.
Figure 15: Visual output of the X_imp feature applied to the NK problem instances. The bar charts in the top row show the correlation values between each variable and the objective. The bar charts in the bottom row show the permutation feature importance scores.
Figure 16: Histogram of Pareto ranks of objective and violation functions.
Table 10: Output of distr_Par : descriptive statistics of the Pareto rank of objective and violation values for the constrained sphere problem.
Statistic
Value
paretoFV minimum
1
paretoFV maximum
100
paretoFV mean
50.5
paretoFV median
50.5
paretoFV Q1
25.25
paretoFV Q3
75.75
Table 11: Output of distr_Par : descriptive statistics of the Pareto rank of objective and violation values for the constrained deceptive problem.
Figure 17: Histograms of Deb’s feasibility rule ranks.
Figure 18: Histograms of objective ranks.
Statistic
Easy knapsack instance
Hard knapsack instance
Minimum
1
1
Maximum
463
928
Mean
205.0576
464.5
Median
197.0
464.5
Q1
126.0
238.0
Q3
274.0
691.0
Table 12: Output of distr_Deb Descriptive statistics of Deb’s feasibility rule ranks for the easy and hard knapsack instances.
Figure 19: FDC visual output for the constrained sphere and constrained deceptive problems.
Figure 20: FDC visual output for NK landscape instances.
Figure 21: Output of FDC for scenarios simulating the XOR gate using a neural network: ( 21(a) ) corresponds to the SGD trajectory in parameter space, and ( 21(b) ) corresponds to the scenario where neural network parameters are randomly sampled.
Figure 22: Visual output of the VDC feature for each constraint of the constrained sphere problem, where v0 and v1 are Constraints ( 2 ) and ( 3 ).
Figure 23: Visual output of the RDC feature for each rank computed for the constrained sphere problem.
Figure 24: Visual output of the PDC feature for the constrained sphere problem.
Figure 25: Visual output of the PDC feature for the constrained deceptive problem.
Figure 26: Visual output of the PDC feature for the Easom problem.
Value/Rank space
Sphere
Objective space( F )
0.4393
Violation space ( V )
0.4116
Equation 2 constraint
0.4566
Equation 3 constraint
0.0098
Objective rank
0.3433
Violation rank
0.4233
Table 13: Numerical output of PDC for the constrained sphere problem.
Value/Rank space
Deceptive
Objective space ( F )
0.6251
Violation space ( V )
0.3155
Objective rank
0.6413
Violation rank
0.3274
Deb rank
0.6413
paretoFV
0.6413
Table 14: Numerical output of PDC for the constrained deceptive problem.
Figure 27: Visual output of disp_best for the constrained sphere problem; disp. is an abbreviation for dispersion.
Figure 28: Visual output of disp_best for the constrained deceptive problem; disp. is an abbreviation for dispersion
Figure 29: Visual output NFC .
Figure 30: Visual output of the NVC feature for the constrained sphere problem.
Figure 31: Visual output of NRC for the constrained sphere problem.
Figure 32: Visual output of NRC for the Easom function problem.
Figure 33: Bar chart showing output of NCF for the constrained sphere problem.
A3Data, BH MG Brazil · CEFET-MG - Centro Federal de Educação Tecnológica de Minas Gerais, BH MG Brazil · Aston University, Birmingham, West Midlands, U.K.