Building Machine Learning Systems with Python - Richert, Coelho

Chapter 6
for d in documents:
allcaps.append(np.sum([t.isupper() \
for t in d.split() if len(t)>2]))
exclamation.append(d.count("!"))
question.append(d.count("?"))
hashtag.append(d.count("#"))
mentioning.append(d.count("@"))
result = np.array([obj_val, pos_val, neg_val,
nouns, adjectives, verbs, adverbs,
allcaps, exclamation, question,
hashtag, mentioning]).T
return result
Putting everything together
Nevertheless, using these linguistic features in isolation without the words themselves
will not take us very far. Therefore, we have to combine TfidfVectorizer with
the linguistic features. This can be done with scikit-learn's FeatureUnion class. It
is initialized the same way as Pipeline, but instead of evaluating the estimators
in a sequence and each passing the output of the previous one to the next one,
FeatureUnion does it in parallel and joins the output vectors afterwards:
def create_union_model(params=None):
def preprocessor(tweet):
tweet = tweet.lower()
for k in emo_repl_order:
tweet = tweet.replace(k, emo_repl[k])
for r, repl in re_repl.iteritems():
tweet = re.sub(r, repl, tweet)
return tweet.replace("-", " ").replace("_", " ")
tfidf_ngrams = TfidfVectorizer(preprocessor=preprocessor,
analyzer="word")
ling_stats = LinguisticVectorizer()
all_features = FeatureUnion([('ling', ling_stats), ('tfidf',
tfidf_ngrams)])
clf = MultinomialNB()
[ 145 ]