This is cool! A few questions.

1. It appears that when a hyperparameter setting has already been tested, the previous objective value will be used. This is fine I think if the objective is noiseless, but when it's noisy I think it needs to be handled carefully.
2. How are you converting int at cat parameters to an optimizable parameter space? If you can point me to where in the code you do this, that would be cool.
3. I believe Spearmint does a simple hack where they try out points near the best point so far. It seems to help a lot in practice. https://github.com/HIPS/Spearmint/blob/c97959ae0449a6594226fe7d18d1384d9463d215/spearmint/choosers/default_chooser.py#L337 You might consider adding that.
4. Sometimes people want to optimize things where each trial takes hours, or days. Is there a way to do a partial fit?
5. It'd be interesting to compare this to existing packages, which at their core do the same thing but may contain useful hacks/different parameters which help in practice.

Thanks for working on this.
It would be great to have an example where this is actually practical, that is improves over GridSearchCV.

doctests are failing.

Are you using this in practice and does it help?
I think the way spearmint gets a new point is evaluate the GP on a grid and does local search.
It would be interesting to compare against that.

I'd like to look into this during the Paris sprint (October 19-23)

That's nice !
Do you plan on starting from my code ? If yes, I can spend some time on it during this week to fix some issues.

As @craffel pointed out, I think it would be nice to handle cases where a single hyper-parameter test takes a long time. My idea for that is to sub-sample the dataset before performing a test. This introduces some noise but the process stays reliable if the sub-sample size is not too small.

subsampling is a good idea but I would make it optional and use the same subsample for all test runs. I think otherwise things will get too noisy.

@fabianp I just stumbled upon https://github.com/mpearmain/BayesBoost maybe it can be of help. Also, do you plan to make use of the new GP implementation? #4270

Sorry for the late reply (for some reason I didn't get the github notification).@sds-dubois my plan is to start from your code and take it from there.

My opinion is that data subsampling is an interesting idea but it is not specific to this PR, i.e., basically any grid search method could use this technique, so it should not be part of this PR.

Thanks @glouppe for the links, I'll definitely look into BayesBoost. For simplicity I'll stick with the current GP implementation. Since the GP is accessed only through its fit and predict interface, I suppose this implementation will not pose a compatibility problem to the new GP implementation.

Ok @fabianp so let me know if you have any question.
I think you're right regarding the data subsampling, but we may want to think about that. In particular you should fix a mistake in my code, at line 402 when we update the score of a hyperparameter that has already been tested. We should in some way keep track of the number of times it was tested before to weight properly each score (old and new ones). Of course it does not matter if the score is always the same for a given hyperparameter, but if we use noisy evaluations that's not good.
Good luck for the week !

@fabianp I talked to Jasper Snoek and the auto-sklearn guys quite a bit ;) What will be your approach?

spearmint seeds on a grid and does local gradient descent on the EI

Thanks all for your feedback :-). I've been working on this today with @Djabbz, adapting it to the new gaussian process and adding some tests. I hope we soon make a PR with these additions.

@sds-dubois I hope you don't feel offended if we modify your code rather than give you feedback, I think this way things get done faster (and in any case you will be able to comment on our modifications real soon).

@amueller right now we are focusing on getting the tests and the API right, i.e. not changing the logic of this code. Hopefully we will compare with the spearmint approach as soon as possible, and get to implement those tricks.

@fabienp of course not ! I'm glad to see this continuing and you should take the most advantage of the sprint ! Also this is my first attempt to contributing so I'm looking forward to see the results and what improvements you made, and learn that way. By the way I don't know if you can push your work to this PR or if I should give some writing access.

@sds-dubois if you can give me access to your branch then that would be easier

I opened a new pull request with the code that we build upon this: #5491

@fabianp you now have access to my branch

Thanks. At the end we thought it would be easier to open a new PR, but I've given you write access to my repo. Feel free to comment on that or take one of the TODO items.

I would also appreciate your opinion on some of the issues I comment, in particular in the format of the hyperparameters. I was quite surprised that {'foo_param': [1, 2, 3]} is accepted and works as in the GridSearchCV(). Is this intended?

@fabianp I answered in PR #5491

closing, this is in scikit-optimize now.