Building Machine Learning Systems with Python - Richert, Coelho

Classification II – Sentiment Analysis
r"\bwouldn't\b": "would not",
r"\bcan't\b": "can not",
r"\bcannot\b": "can not",
}
def create_ngram_model(params=None):
def preprocessor(tweet):
global emoticons_replaced
tweet = tweet.lower()
#return tweet.lower()
for k in emo_repl_order:
tweet = tweet.replace(k, emo_repl[k])
for r, repl in re_repl.iteritems():
tweet = re.sub(r, repl, tweet)
return tweet
tfidf_ngrams = TfidfVectorizer(preprocessor=preprocessor,
analyzer="word")
# ...
Certainly, there are many more abbreviations that could be used here. But already
with this limited set, we get an improvement for sentiment versus not sentiment of
half a point, which comes to 70.7 percent:
== Pos vs. neg ==
0.804 0.022 0.886 0.011
== Pos/neg vs. irrelevant/neutral ==
0.797 0.009 0.707 0.029
== Pos vs. rest ==
0.884 0.005 0.527 0.025
== Neg vs. rest ==
0.886 0.011 0.640 0.032
Taking the word types into account
So far our hope was to simply use the words independent of each other with the
hope that a bag-of-words approach would suffice. Just from our intuition, however,
neutral tweets probably contain a higher fraction of nouns, while positive or negative
tweets are more colorful, requiring more adjectives and verbs. What if we could
use this linguistic information of the tweets as well? If we could find out how many
words in a tweet were nouns, verbs, adjectives, and so on, the classifier could maybe
take that into account as well.
[ 138 ]