Skip to content

ENH CountVectorizer does not check for lowercase in vocabulary #19401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 12, 2021
Merged

ENH CountVectorizer does not check for lowercase in vocabulary #19401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
61209a3
Check vocab for uppercase chars when lowercase is True
zitorelova Feb 7, 2021
9367641
Add test to make sure proper warning is thrown
zitorelova Feb 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions sklearn/feature_extraction/tests/test_text.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,18 @@ def test_countvectorizer_custom_token_pattern_with_several_group():
vectorizer.fit(corpus)


def test_countvectorizer_uppercase_in_vocab():
vocabulary = ['Sample', 'Upper', 'Case' 'Vocabulary']
message = ("Upper case characters found in"
" vocabulary while 'lowercase'"
" is True. These entries will not"
" be matched with any documents")

vectorizer = CountVectorizer(lowercase=True, vocabulary=vocabulary)
assert_warns_message(UserWarning, message,
vectorizer.fit_transform, vocabulary)


def test_tf_idf_smoothing():
X = [[1, 1, 1],
[1, 1, 0],
Expand Down
9 changes: 9 additions & 0 deletions sklearn/feature_extraction/text.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1107,6 +1107,15 @@ def _count_vocab(self, raw_documents, fixed_vocab):
j_indices = []
indptr = []

if self.lowercase:
for vocab in vocabulary:
if any(map(str.isupper, vocab)):
warnings.warn("Upper case characters found in"
" vocabulary while 'lowercase'"
" is True. These entries will not"
" be matched with any documents")
break

values = _make_int_array()
indptr.append(0)
for doc in raw_documents:
Expand Down