FIX Convergence Warnings in Gaussian process examples #18019

marenwestermann · 2020-07-28T12:03:42Z

Reference Issues/PRs

Fixes convergence warnings mentioned in #17946
See also #17943 and #12673

What does this implement/fix? Explain your changes.

This implementation fixes convergence warnings in examples/gaussian_process/* introduced through #12673

Any other comments?

I tried to fix all convergence warnings but I wasn't successful. I will detail what I have done so far below.

marenwestermann · 2020-07-28T12:20:10Z

Description of the warnings:

The following modules under examples/gaussian_process/ throw the following convergence warnings since the introduced changes in #12673 (the user warnings in plot_gpr_prior_posterior.py must have existed beforehand):

python examples/gaussian_process/plot_gpc_xor.py

/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/kernels.py:411: ConvergenceWarning: The optimal value found for dimension 0 of parameter k1__constant_value is close to the specified upper bound 100000.0. Increasing the bound and calling fit again may find a better value.
  warnings.warn("The optimal value found for "
/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/kernels.py:411: ConvergenceWarning: The optimal value found for dimension 0 of parameter k1__constant_value is close to the specified upper bound 100000.0. Increasing the bound and calling fit again may find a better value.
  warnings.warn("The optimal value found for "

#################################################

python examples/gaussian_process/plot_gpr_noisy.py

/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/kernels.py:402: ConvergenceWarning: The optimal value found for dimension 0 of parameter k1__k1__constant_value is close to the specified lower bound 1e-05. Decreasing the bound and calling fit again may find a better value.
  warnings.warn("The optimal value found for "

###################################################

python examples/gaussian_process/plot_gpr_on_structured_data.py

/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/kernels.py:402: ConvergenceWarning: The optimal value found for dimension 0 of parameter baseline_similarity is close to the specified lower bound 1e-05. Decreasing the bound and calling fit again may find a better value.
  warnings.warn("The optimal value found for "

##################################################

python examples/gaussian_process/plot_gpr_prior_posterior.py

/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/kernels.py:411: ConvergenceWarning: The optimal value found for dimension 0 of parameter k2__alpha is close to the specified upper bound 100000.0. Increasing the bound and calling fit again may find a better value.
  warnings.warn("The optimal value found for "
/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/_gpr.py:372: UserWarning: Predicted variances smaller than 0. Setting those variances to 0.
  warnings.warn("Predicted variances smaller than 0. "
/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/kernels.py:402: ConvergenceWarning: The optimal value found for dimension 0 of parameter k1__constant_value is close to the specified lower bound 0.01. Decreasing the bound and calling fit again may find a better value.
  warnings.warn("The optimal value found for "
/home/maren/Documents/GitHub/scikit-learn/sklearn/gaussian_process/_gpr.py:372: UserWarning: Predicted variances smaller than 0. Setting those variances to 0.
  warnings.warn("Predicted variances smaller than 0. "```

marenwestermann · 2020-07-28T12:50:20Z

Detailed description of my work:

examples/gaussian_process/plot_gpc_xor.py :
I fixed one ConvergenceWarning by setting length_scale=1.15 in RBF. However, I wasn't able to fix the warning caused by DotProduct. Regarding DotProduct: The convergence warning is triggered in sklearn/gaussian_process/kernels.py in the init method of ConstantKernel (constant_value_bounds=(1e-5, 1e5)). I first tried different values for sigma_0 and noticed that the ConvergenceWarning flips from an upper bound warning to a lower bound warning when sigma_0 is changed from 1e-4 to 1e-5. I also tried to change the variance of the random numbers but this also didn't fix the warning. In summary, it seems to me as if using DotProduct on the XOR data seems to be unstable but that this previously went unnoticed because there was no convergence warning message,

examples/gaussian_process/plot_gpr_noisy.py:
As above, the convergence warning is triggered in sklearn/gaussian_process/kernels.py in the init method of ConstantKernel (constant_value_bounds=(1e-5, 1e5)). I checked if I can change the constant_value_bounds from plot_gpr_noisy.py (and similarly from plot_gpc_xor.py which would probably have fixed the convergence warning described above) but I couldn't find out if this is possible.

examples/gaussian_process/plot_gpr_on_structured_data.py:
I gradually decreased the baseline_similarity_bounds in SequenceKernel by orders of magnitude but I still got the same ConvergenceWarning. I then gradually increased the number of iterations (max_iter) by orders of magnitudes but this also didn't resolve the problem. The error occurs under the section "Classification" which made me think whether it actually makes sense to use Gaussian process with such a small dataset. It might be better to either remove this example or use a more comprehensive dataset for it but please correct me if I am wrong.

examples/gaussian_process/plot_gpr_prior_posterior.py:
I was able to fix these convergence warnings. :)

thomasjpfan

Thank you for the PR @marenwestermann !

thomasjpfan · 2021-04-08T17:01:02Z

examples/gaussian_process/plot_gpr_prior_posterior.py

+           1.0 * RationalQuadratic(length_scale=1.0, alpha=0.1,
+                                   alpha_bounds=(1e-5, 1e6)),


I am still getting the same number of convergence warnings with this change. Lets revert this for now and keep the other change?

examples/gaussian_process/plot_gpc_xor.py

thomasjpfan

LGTM

…8019) Co-authored-by: Maren Westermann <maren.westermann@free-now.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>

Co-authored-by: Maren Westermann <maren.westermann@free-now.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>

…8019) Co-authored-by: Maren Westermann <maren.westermann@free-now.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>

marenwestermann added 2 commits July 27, 2020 10:42

fix warning on XOR dataset

8078bd3

fix warning for prior/ posterior gaussian process example

8da04e8

marenwestermann changed the title ~~Adjust bounds~~ FIX Convergence Warnings in Gaussian process examples Jul 28, 2020

Base automatically changed from master to main January 22, 2021 10:52

thomasjpfan reviewed Apr 8, 2021

View reviewed changes

Maren Westermann and others added 4 commits May 8, 2021 19:54

revert change in plot_gpr_prior_posterior.py

251a990

Merge branch 'main' into adjust-bounds

228fce4

Merge remote-tracking branch 'upstream/main' into pr/18019

97a651f

STY Fix formatting

3fa7092

thomasjpfan approved these changes Aug 2, 2022

View reviewed changes

thomasjpfan merged commit dbfb219 into scikit-learn:main Aug 2, 2022

marenwestermann deleted the adjust-bounds branch August 3, 2022 10:13

glemaitre pushed a commit that referenced this pull request Aug 5, 2022

FIX Convergence Warnings in Gaussian process examples (#18019)

f5b82c9

Co-authored-by: Maren Westermann <maren.westermann@free-now.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FIX Convergence Warnings in Gaussian process examples #18019

FIX Convergence Warnings in Gaussian process examples #18019

marenwestermann commented Jul 28, 2020 •

edited

Loading

marenwestermann commented Jul 28, 2020

marenwestermann commented Jul 28, 2020 •

edited

Loading

thomasjpfan left a comment

thomasjpfan Apr 8, 2021

thomasjpfan left a comment

		1.0 * RationalQuadratic(length_scale=1.0, alpha=0.1,
		alpha_bounds=(1e-5, 1e6)),

FIX Convergence Warnings in Gaussian process examples #18019

FIX Convergence Warnings in Gaussian process examples #18019

Conversation

marenwestermann commented Jul 28, 2020 • edited Loading

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

marenwestermann commented Jul 28, 2020

marenwestermann commented Jul 28, 2020 • edited Loading

thomasjpfan left a comment

Choose a reason for hiding this comment

thomasjpfan Apr 8, 2021

Choose a reason for hiding this comment

thomasjpfan left a comment

Choose a reason for hiding this comment

marenwestermann commented Jul 28, 2020 •

edited

Loading

marenwestermann commented Jul 28, 2020 •

edited

Loading