Published June 10, 2024 | Version v3
Dataset Open

MA-BBOB - Reproducibility and Additional Data

Creators

  • 1. ROR icon Leiden University
  • 2. ROR icon Chinese Academy of Sciences
  • 3. ROR icon Centre National de la Recherche Scientifique
  • 4. ROR icon Laboratoire de Recherche en Informatique de Paris 6

Description

Reproducibility files for the paper:

MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts

 

This document details the reproduction steps for the paper "MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts"

 

This readme is structured as follows:
- We first showcase a code-snippet highlighting the way in which the MA-BBOB generator can be used as a stand-alone feature
- Afterwards, we walk through the steps required for reproducing the results presented in the paper
- Next, we have a brief description of each file included in this repository
- Finally, we provide some background on the main dependencies used in this project

Accessing the MA-BBOB generator

 
Since it is integrated into the IOHexperimenter, accessing the MA-BBOB functions can be done easily from Python. As a default, the generator uses the function creation procedure described in section 3 of the paper to sample new functions. To ensure the functions are reusable, the 'instance id' is used to seed the generation procedure, so using the same id multiple times leads to the same function.
 
import ioh
f = ioh.problem.ManyAffine(1, n_variables = 5)

 

However, the code used in the remainder of the repository uses specific settings of weights, instances and optima to compare different settings. These are specified instead of the instance ID as follows:

 

xopt = np.random.uniform(size=(2), low=-5, high=5)
weights = np.ones(24)/24
iids = list(np.repeat(1,24))
f2 = ioh.problem.ManyAffine(xopt = list(xopt), weights = list(weights), instances = iids, n_variables = 2)

Reproducing the paper's results

 

The core file for reproducing the results from this project is the notebook 'Visualization.ipynb'. This notebook is interrupted at times to run scripts and collect data, which is explained both in the notebook as well as in the sections below. The data collection is related to Section 4, since the results of Section 3 are entirely contained in the notebook, and Section 5 uses available data from this Zenodo (data included here as well for convenience).

 

Determine the settings used

Within the notebook, the section 'Setup data collection' is used to generate the used weights, instance number and optima location for all pairwise experiments. Each of these is stored in its corresponding csv-file.
These files fully specify the pairwise instances of the MA-BBOB suite we use throughout the paper. These files are used in the scripts for the following parts (ELA + Performace)

 

Calculating ELA features

The ELA-feature computation is based on pflacco as described in the dependencies.
The script: 'ela_calculation.py' runs the computation and stores the results as csv files (Note: the 'dirname' parameter should be changed before running this script). The file loops over all selected BBOB and MA-BBOB instances and gets the following sets of ELA features:
*    meta data
*    distribution
*    level set
*    principal component analysis
*    linear model
*    nbc
*    dispersion
*    information_content
For more information on these feature sets, please see
"Mersmann et al. (2011), “Exploratory Landscape Analysis”, in Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, pp. 829—836. ACM (http://dx.doi.org/10.1145/2001576.2001690)"

 

Performance data

 

Collect performance data
The performance data collection script is 'collect_performance.py', which makes use of nevergrad and IOHprofiler to benchmark the selected algorithms. Note that the 'rootname' parameter should be modified when running this script.

 

The data from running this script is available in the zenodo repository as 'data.zip'.

 

Process performance data
The previous script generates IOHanalyzer-compatible data. This can be processed via the script 'auc_calculation.py', resulting in the file 'auc.csv' if the 'dirname' parameter is set to the one used in the previous script.

 

Visualizations

All remaining analyses and visualizations are part of the notebook. Note that the corresponding directory and filenames should be updated according to the ones used in the respective scripts. Parts of the notebook make use of data from other repositories, which is included in this repository as well for convenience.

Please note that to fully run the notebook, the zip-files 'ELA' and 'ELA_BBOB' need to be unzipped in the working directory of the notebook. 

Included Files

This repository contains the following files:
- All_2d_info2.csv and All_5d_info2.csv: these files contain ELA and performance data which is used in section 5 (and originates from this repository: https://zenodo.org/records/7826036)
- auc.csv: contains the performance data used in section 4
- auc_calculation.py: the script which is used to process the IOH-generated files into the corresponding performance metric
- auc_w_info.csv: an intermediate file created in the notebook which combines instance-information and the performance data from auc.csv
- collect_performance.py: The script used to generate the performance data (IOH-based files) used in section 4
- data_example.zip: a subset of the performance data as generated by 'collect_performance.py'
- dt_ela.csv: the ELA features used in section 4, calculated in the visualization notebook by concatenating files from the ELA.zip and ELA_BBOB.zip
- ELA.zip and ELA_BBOB.zip: ELA features collected by 'ela_calculation.py'
- ela_calculation.py: script used to generate all landscape features used for section 4
- ela_normalized.csv: normalized version of dt_ela, generated in the visualization notebook
- iids, opt_locs, weights.csv: information about the used instances in section 5
- Visualization.ipynb: the main reproducibility file which documents the process of using all files above to result in the figures and analysis presented in the paper

 

Dependencies

This project relies on a few different libraries to achieve the presented results. Note that each of these packages (except IOHanalyzer) is available directly from pip, with the used version mentioned in the requirements file. These are as follows:

 

IOHexperimenter

Part of the IOHprofiler environment, IOHexperimenter provides an interface between optimization algorithms and problems, and adds detailed logging functionality to this pipeline.
We use the python-version of IOHexperimenter, available on pip as 'ioh' (we used version 0.3.14). From this package, we use the logging component, as well as the interface to the BBOB problem suite and the MA-BBOB problem generator.

 

PFlacco

To analyze the function's low-level properties, we make use of Exploratory Landscape Analysis (ELA), which gives access to a wide range of features. To calculate these, we use the python-based pflacco library (version 1.2.2, note that this requires python 3.8 or higher).  
We make use of only the features which don't require sampling additional points from the function, from the 'classical_ela_features' module.

 

Nevergrad

To access a variety of optimization algorithms, we make use of the Nevergrad (version 0.4.3.post8).
We make use of the following algorithms from Nevergrad's optimizers module:  'DifferentialEvolution', 'DiagonalCMA', 'RCobyla'

 

Modular CMA-ES + DE

In addition to the Nevergrad algorithms, we make use of two modular algorithm frameworks in our portfolio. The first is Modular CMA-ES, 'modcma' on pip (version 1.0.2). The second is Modular DE, 'modde' on pip (version 0.0.1).

 

IOHanalzyer

While not used directly in any of the presented figures in the paper, we did make use of the IOHanalyzer for data exploration throughout the analysis. This is an R-based library for analyzing and visualizing optimization algorithm performance. We use version 0.1.8.4.

 

 

Files

All_2d_info2.csv

Files (3.7 GB)

Name Size Download all
md5:4e88223526001c7f3706a4773fa93fbc
1.3 MB Preview Download
md5:11dfc72b70ce0c66b971136edf2ccb52
1.4 MB Preview Download
md5:08d981433dc3c1137abca6611ed7b999
1.3 GB Preview Download
md5:8caa6a020228436e92c0a71f794a8a37
2.0 kB Download
md5:6c459500c1ed61cf8a7fe304a4d25f95
1.8 GB Preview Download
md5:4b858bf41c8f42bef61da35b6de29a2a
5.8 kB Download
md5:34368b6be7e3eef43424d61a428fbf91
530.7 MB Preview Download
md5:7f00a3661e05f3a4f56494a719c8a278
35.7 MB Preview Download
md5:05b59707a966e211160477f47308ea92
1.3 MB Preview Download
md5:9bc7092f4d11ae5c8f77d54c5d6da744
36.2 MB Preview Download
md5:0007be7dfb4065f29c2b56fa1db10ccd
1.3 MB Preview Download
md5:294412ea22b56a8993eb4d558ad73207
4.3 kB Download
md5:7879e093447b5507cb8d3077177a3074
39.5 MB Preview Download
md5:2cbbaba3077e981d227a639a0f4244e8
3.8 MB Preview Download
md5:44e9d08069209d2ee3716c3794e7be17
3.8 MB Preview Download
md5:dfacdbf34208c15388020ca56caa3a22
7.8 kB Preview Download
md5:36967ef4e0aa2e082d7e03b107aea2f6
161 Bytes Preview Download
md5:d1647478250d1d99aae97f72cfbd2322
5.6 MB Preview Download
md5:371ff64ced52729346d3bc6cc11527e0
4.2 MB Preview Download