SimpleImputer strategy "most_frequent": infinite loop for categorical variables as strings #14270
Description
Description
Infinite loop when applying SimpleImputer with strategy "most_frequent" to data frame or pandas Series. Right from the first column, Spyder (the IDE I use) gets stuck and nothing happens.
Steps/Code to Reproduce
Each column of the data frame contains about 900,000 rows. With the code below, I am basically looping through the categorical columns of the dataframe, that has been created with the train_test_split method of sklearn. I tried to reset the index of the X_train data frame but it still did not change anything.
for cat in df_cat_attributes:
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent',verbose=10)
X_train[cat] = imp.fit_transform(X_train[[cat]]).ravel()
Edit: This code works with numerical categories (e.g. 0 and 1) but not with string categories (e.g. American states).
Expected Results
The columns of the data frame should have no more missing values.
Actual Results
The function creates an infinite loop: nothing shows up.
Versions
Could not locate executable g77
Could not locate executable f77
Could not locate executable ifort
Could not locate executable ifl
Could not locate executable f90
Could not locate executable DF
Could not locate executable efl
Could not locate executable gfortran
Could not locate executable f95
Could not locate executable g95
Could not locate executable efort
Could not locate executable efc
Could not locate executable flang
don't know how to compile Fortran code on platform 'nt'
System:
python: 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]
executable: C:\ProgramData\Anaconda3\pythonw.exe
machine: Windows-10-10.0.17134-SP0
BLAS:
macros:
lib_dirs:
cblas_libs: cblas
Python deps:
pip: 19.1.1
setuptools: 41.0.1
sklearn: 0.21.2
numpy: 1.16.4
scipy: 1.2.1
Cython: 0.29.10
pandas: 0.24.2
C:\ProgramData\Anaconda3\lib\site-packages\numpy\distutils\system_info.py:639: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
self.calc_info()
C:\ProgramData\Anaconda3\lib\site-packages\numpy\distutils\system_info.py:639: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
self.calc_info()
C:\ProgramData\Anaconda3\lib\site-packages\numpy\distutils\system_info.py:639: UserWarning:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
self.calc_info()