Simple script for optimizing the rusty snakes hyperparameters with SMAC3 or DEHB.
Install the requirements.txt. Python 3.9 is recommended.
Run the script:
hpo_optimize.py [-h] [--optimizer {DEHB,SMAC}] [--num_games_per_eval NUM_GAMES_PER_EVAL] [--runcount_limit RUNCOUNT_LIMIT] [--walltime WALLTIME] [-j N_JOBS] [-o OUTPUT_DIR]There are additional options available, run
python hpo_optimize -hto view them together with additional usage information. NOTE: When using DEHB--runcount_limitis priotized over--walltime!
The output directory is used for SMAC-like output from both DEHB and SMAC, it is a good idea to keep runs produced by different optimizers separated.
Since SMAC-like output is produced, the runhistories can be analyzed using PyImp. Just pass the runhistory and trajectory files from the output.
NOTE: DEHB currently produces no scenario-file which is easiest generated by using SMAC first (for example with a minimal walltime) and then copying the generated scenario.txt and configspace.pcs files, the filepath to and the parameter-prefixes in the configspace.pcs will have to be adjusted).
Perform Hyperparameter search using SMAC (pSMAC to be more precise) on 8 cores for 4 hours. Only the Flood agent will be explored, output will be written to hpo/smac_output.
cd hpo
python hpo_optimize.py --optimizer SMAC --walltime 14400 -o ./smac -j 8To then analyze those results, use PyImp:
pimp --scenario_file smac_output/scenario.txt --history smac_output/runhistory.json --trajectory smac_output/traj.json --fanova_no_pairs --table True --modus all
Which performs all the different types of analysis available to PyImp but doesn't produce pairs for the fANOVA method. This results in a lot of graphs and a table like this one for example:
| Ablation | Forward-Selection | fANOVA | LPI | |
|---|---|---|---|---|
| food_distance_offset | 31.820 | 0.163 | 17.847 | 12.216 |
| board_control_decay | 0.000 | 0.058 | 0.685 | 2.131 |
| board_control_flip | 0.000 | - | 0.029 | 0.000 |
| board_control_offset | 0.000 | 0.126 | 8.851 | 35.516 |
| food_distance_decay | 0.000 | 0.077 | 0.112 | 11.054 |
| food_distance_flip | 0.000 | - | 0.066 | 0.000 |
| health | 0.000 | - | 0.274 | 0.408 |
| health_flip | 0.000 | - | 0.008 | 0.000 |
| health_offset | 0.000 | - | 0.177 | 0.067 |
| len_advantage | 0.000 | - | 0.719 | 0.027 |
| len_advantage_flip | 0.000 | - | 0.000 | 0.000 |
| food_distance | -0.012 | 0.067 | 6.770 | 1.181 |
| len_advantage_offset | 8.973 | 0.089 | 4.390 | 33.512 |
| board_control | 22.752 | - | 2.628 | 1.490 |
| len_advantage_decay | 19.313 | - | 3.606 | 1.783 |
| health_decay | 17.156 | - | 0.002 | 0.614 |
| None | - | 0.199 | - | - |
| agent | - | - | 0.000 | - |
Parameter Importance values, obtained using the PIMP package. Ablation values are percentages of improvement a single parameter change obtained between the default and an incumbent configuration.
fANOVA values are percentages that show how much variance across the whole ConfigSpace can be explained by that parameter.
Forward Selection values are RMSE values obtained using only a subset of parameters for prediction.
fANOVA and Forward Selection try to estimate the importances across the whole parameter space, while ablation tries to estimate them between two given configurations.