Skip to content

[MRG] tfidfvectorizer documentation #12204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[MRG] tfidfvectorizer documentation #12204

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c15d5c8
tfidfvectorizer documentation
Sep 29, 2018
93ae2d1
adding line breaks
Sep 29, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion sklearn/feature_extraction/text.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1286,7 +1286,27 @@ def idf_(self, value):
class TfidfVectorizer(CountVectorizer):
"""Convert a collection of raw documents to a matrix of TF-IDF features.

Equivalent to CountVectorizer followed by TfidfTransformer.
Equivalent to CountVectorizer followed by TfidfTransformer.

CountVectorizer converts a collection of text documents to a matrix of
token counts.

TfidfTransformer then converts the count matrix from CountVectorizer to a
normalized tf-idf representation. Tf is term frequency, and idf is inverse
document frequency. This is a common way to calculate the count of a word
relative to the appearance of a ducument.

The formula that is used to compute the tf-idf of term t is
tf-idf(d, t) = tf(t) * idf(d, t), and the idf is computed as
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the other documentation has some formatting for these, can you make sure to copy the code, not the rendering?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

idf(d, t) = log [ n / df(d, t) ] + 1 (if ``smooth_idf=False``),
where n is the total number of documents and df(d, t) is the
document frequency; the document frequency is the number of documents d
that contain term t. The effect of adding "1" to the idf in the equation
above is that terms with zero idf, i.e., terms that occur in all documents
in a training set, will not be entirely ignored.
(Note that the idf formula above differs from the standard
textbook notation that defines the idf as
idf(d, t) = log [ n / (df(d, t) + 1) ]).

Read more in the :ref:`User Guide <text_feature_extraction>`.

Expand Down