Skip to content

[MRG+1] Fix #10229: check_array should fail if array has strings #10495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Feb 22, 2018
Merged

[MRG+1] Fix #10229: check_array should fail if array has strings #10495

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
eadae7c
Add deprecation warning to check_array for flexible array w/ dtype=nu…
rtlee9 Jan 18, 2018
ccd1ced
Use assert_warns_message to test check_array deprecation warning
rtlee9 Jan 19, 2018
9d6946e
Add deprecation starting version to warning message
rtlee9 Jan 19, 2018
19b2bea
NumPy may convert string list to dtype U1 or S1 so need to check both…
rtlee9 Jan 19, 2018
2e851ce
Make warning message more clear
rtlee9 Jan 29, 2018
8a4b74e
Update check_array warning: reject byte arrays if numeric dtype
rtlee9 Feb 20, 2018
7734056
Update test to account for numpy conversion of string list to byte array
rtlee9 Feb 20, 2018
d12630f
No differentiation between arrays of strings and arrays of bytes
rtlee9 Feb 22, 2018
5849f5c
Raise FutureWarning instead of DeprecationWarning
rtlee9 Feb 22, 2018
4489841
Add recommendation to convert array to np.float64 to warning
rtlee9 Feb 22, 2018
e377cb3
Add whats_new bug fix entry
rtlee9 Feb 22, 2018
8c3bcca
Merge branch 'master' into check_array
rtlee9 Feb 22, 2018
ffb1448
Tweak tests and whats_new entry
lesteve Feb 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions doc/whats_new/v0.20.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,12 @@ Feature Extraction
(words or n-grams). :issue:`9147` by :user:`Claes-Fredrik Mannby <mannby>`
and `Roman Yurchak`_.

Utils

- :func:`utils.validation.check_array` yield a ``FutureWarning`` indicating
that arrays of bytes/strings will be interpreted as decimal numbers
beginning in version 0.22. :issue:`10229` by :user:`Ryan Lee <rtlee9>`

Preprocessing

- Fixed bugs in :class:`preprocessing.LabelEncoder` which would sometimes throw
Expand Down
36 changes: 36 additions & 0 deletions sklearn/utils/tests/test_validation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,42 @@ def test_check_array():
result = check_array(X_no_array)
assert_true(isinstance(result, np.ndarray))

# deprecation warning if string-like array with dtype="numeric"
X_str = [['a', 'b'], ['c', 'd']]
assert_warns_message(
FutureWarning,
"arrays of strings will be interpreted as decimal numbers if "
"parameter 'dtype' is 'numeric'. It is recommended that you convert "
"the array to type np.float64 before passing it to check_array.",
check_array, X_str, "numeric")
assert_warns_message(
FutureWarning,
"arrays of strings will be interpreted as decimal numbers if "
"parameter 'dtype' is 'numeric'. It is recommended that you convert "
"the array to type np.float64 before passing it to check_array.",
check_array, np.array(X_str, dtype='U'), "numeric")
assert_warns_message(
FutureWarning,
"arrays of strings will be interpreted as decimal numbers if "
"parameter 'dtype' is 'numeric'. It is recommended that you convert "
"the array to type np.float64 before passing it to check_array.",
check_array, np.array(X_str, dtype='S'), "numeric")

# deprecation warning if byte-like array with dtype="numeric"
X_bytes = [[b'a', b'b'], [b'c', b'd']]
assert_warns_message(
FutureWarning,
"arrays of strings will be interpreted as decimal numbers if "
"parameter 'dtype' is 'numeric'. It is recommended that you convert "
"the array to type np.float64 before passing it to check_array.",
check_array, X_bytes, "numeric")
assert_warns_message(
FutureWarning,
"arrays of strings will be interpreted as decimal numbers if "
"parameter 'dtype' is 'numeric'. It is recommended that you convert "
"the array to type np.float64 before passing it to check_array.",
check_array, np.array(X_bytes, dtype='V1'), "numeric")


def test_check_array_pandas_dtype_object_conversion():
# test that data-frame like objects with dtype object
Expand Down
9 changes: 9 additions & 0 deletions sklearn/utils/validation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,15 @@ def check_array(array, accept_sparse=False, dtype="numeric", order=None,
# To ensure that array flags are maintained
array = np.array(array, dtype=dtype, order=order, copy=copy)

# in the future np.flexible dtypes will be handled like object dtypes
if dtype_numeric and np.issubdtype(array.dtype, np.flexible):
warnings.warn(
"Beginning in version 0.22, arrays of strings will be "
Copy link
Member

@lesteve lesteve Feb 22, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The standard user will very likely have little idea what check_array is. This is the wording I am proposing, better suggestions more than welcome:

"Beginning in version 0.22, arrays of bytes/strings will be "
"interpreted as decimal numbers if dtype='numeric'. "
"It is recommended that you convert the array to "
"a float dtype before using it in scikit-learn, "
"for example by using "
"your_array = your_array.astype(np.float64)."

"interpreted as decimal numbers if parameter 'dtype' is "
"'numeric'. It is recommended that you convert the array to "
"type np.float64 before passing it to check_array.",
FutureWarning)

# make sure we actually converted to numeric:
if dtype_numeric and array.dtype.kind == "O":
array = array.astype(np.float64)
Expand Down