I'm having trouble reproducing this issue. Running the Iterative Imputer example with sklearn 0.21.0 (the earliest release to include it), then updating to 0.22.dev0 an re-running the example throws no errors for me. Could you clarify how you originally got stale data? Was there a python version change, or some change among your sub-dependencies?

$ python -c "import sklearn; print(sklearn.__version__)"
0.21.0
$ pip freeze | grep -v scikit-learn
Click==7.0
cycler==0.10.0
Cython==0.29.11
joblib==0.13.2
kiwisolver==1.1.0
matplotlib==3.1.0
numpy==1.16.4
pandas==0.24.2
pip-tools==3.8.0
pyparsing==2.4.0
python-dateutil==2.8.0
pytz==2019.1
scipy==1.2.1
six==1.12.0
$ python examples/impute/plot_iterative_imputer_variants_comparison.py
# ...
# Lots of output and ConvergenceWarnings
# ...
$ sha1sum ~/scikit_learn_data/cal_housing_py3.pkz
41bac263fe97ef4d681395c42af14e1fa2c72ce5  ~/scikit_learn_data/cal_housing_py3.pkz

$ python -c "import sklearn; print(sklearn.__version__)"
0.22.dev0
$ python examples/impute/plot_iterative_imputer_variants_comparison.py
# ...
# Lots of output and ConvergenceWarnings ... no ModuleNotFoundError
# ...

Hm... I can get close. If you clear out your data cache

rm -r ~/scikit_learn_data

then install an older version of sklearn

pip install "scikit-learn==0.19.0"

download california housing

python -c "from sklearn.datasets import fetch_california_housing;fetch_california_housing()"

then install a more recent version of sklearn

pip install "scikit-learn==0.21.2

and access california housing again, you see:

/Users/dillon/anaconda/envs/scikit-test/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)

If you do this in the reverse direction, it fails with

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/dillon/anaconda/envs/py35/lib/python3.5/site-packages/sklearn/datasets/california_housing.py", line 106, in fetch_california_housing
    cal_housing = joblib.load(filepath)
  File "/Users/dillon/anaconda/envs/py35/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 575, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/Users/dillon/anaconda/envs/py35/lib/python3.5/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 507, in _unpickle
    obj = unpickler.load()
  File "/Users/dillon/anaconda/envs/py35/lib/python3.5/pickle.py", line 1043, in load
    dispatch[key[0]](self)
  File "/Users/dillon/anaconda/envs/py35/lib/python3.5/pickle.py", line 1342, in load_global
    klass = self.find_class(module, name)
  File "/Users/dillon/anaconda/envs/py35/lib/python3.5/pickle.py", line 1392, in find_class
    __import__(module, level=0)
ImportError: No module named 'joblib'

Presumably, I had originally downloaded the california data some release before 0.19.0, when sklearn.externals.joblib was called sklearn.externals._joblib.

Okay, great. I can reproduce your DeprecationWarning.

Digging through the github history here, it appears this situation wasn't altogether unexpected. @rth took on the task of unvendoring joblib and right at the top of the PR mentions:

A major concern is backward compatibility of pickles.

Hence, the inclusion of the DeprecationWarning. The behavior is unattractive, but I'm not sure what action can be taken. You've pointed out that clearing the dataset cache after updating does work. Is there a smarter approach than the status quo?

Interested in hearing input.

So, one option would be to tag the downloads with the version of scikit-learn that pickled them, and check that tag before loading the data. If the version doesn't match, download a new thing.

I don't think we'd want to support the case where people would downgrade the package. And also for the warning, it should automatically fix the issue with #14197

I'd say this is a "won't fix" issue. Closing, happy to have it opened if the other core devs disagree.

And also for the warning, it should automatically fix the issue with #14197

Doesn't that PR actually solve this issue by refreshing local cache for legacy pickles? Not just the warning?

That PR only refreshes the cache if the warning is raised, The issue here is that going back from a future version of sklearn raises an error, and I think it should.

The issue here is that going back from a future version of sklearn raises an error, and I think it should.

I think that was only done to reproduce the original issue. The original issue, unless I misunderstood something, is that pickles done with older scikit-learn version fail to load with newer scikit-learn versions. Maybe I shouldn't have removed sklearn.externals._joblib during joblib unvendoring.

In any case, since the only complaint about it we got from datasets loading, it should be fairly straightforward to modify the _refresh_cache function to also refresh cache if pickles failed to load with ModuleNotFoundError: No module named 'sklearn.externals._joblib', I think? I'll look into it.

it should be fairly straightforward to modify the _refresh_cache function to also refresh cache if pickles failed to load

OK that statement was too optimistic. _refresh_cache is oblivious to the download URL, so another mechanism would be needed something like,

filepath = _pkl_filepath(data_home, 'cal_housing.pkz')

cal_housing = None
if exists(filepath):
    try:
        cal_housing = _refresh_cache([filepath], 6)
    except (ImportError,):
        # whitelist exceptions that are expected when loading
        # outdated pickles, otherwise raise exception

if cal_housing is None:
    # download it

which would require modifying all the fetchers. Not sure if we really want to do that, or if adding a message in _refresh_cache to manually remove the cache folder (or even remove the problematic pickle and ask the user to re-run the example again) would be better.

I think that was only done to reproduce the original issue.

Correct.

As for fixes, printing a warning about stale cache data sounds reasonable to me.

I would say "not worth fixing for now", let's close this one.

This was an issue a while ago when joblib was unvendored, but I would guess that a similar issue is unlikely to happen. In case I am wrong, another issue will be created and we can have a closer look ...