1FfUrl0

Chapter 3
0.0
>>> print(tfidf("b", abb, D))
0.270310072072
>>> print(tfidf("a", abc, D))
0.0
>>> print(tfidf("b", abc, D))
0.135155036036
>>> print(tfidf("c", abc, D))
0.366204096223
We see that a carries no meaning for any document since it is contained everywhere.
b is more important for the document abb than for abc as it occurs there twice.
In reality, there are more corner cases to handle than the above example does.
Thanks to Scikit, we don't have to think of them, as they are already nicely packaged
in TfidfVectorizer, which is inherited from CountVectorizer. Sure enough, we
don't want to miss our stemmer:
>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> class StemmedTfidfVectorizer(TfidfVectorizer):
... def build_analyzer(self):
... analyzer = super(TfidfVectorizer,
self).build_analyzer()
... return lambda doc: (
english_stemmer.stem(w) for w in analyzer(doc))
>>> vectorizer = StemmedTfidfVectorizer(min_df=1,
stop_words='english', charset_error='ignore')
The resulting document vectors will not contain counts any more. Instead, they will
contain the individual TF-IDF values per term.
Our achievements and goals
Our current text preprocessing phase includes the following steps:
1. Tokenizing the text.
2. Throwing away words that occur way too often to be of any help in detecting
relevant posts.
[ 61 ]