Building Machine Learning Systems with Python - Richert, Coelho

Classification II – Sentiment Analysis
To find out which of the synsets to take, we would have to really understand the
meaning of the tweets, which is beyond the scope of this chapter. The field of
research that focuses on this challenge is called word sense disambiguation. For
our task, we take the easy route and simply average the scores over all the synsets
in which a term is found. For "fantasize", PosScore would be 0.1875 and NegScore
would be 0.0625.
The following function, load_sent_word_net(), does all that for us, and returns a
dictionary where the keys are strings of the form "word type/word", for example "n/
implant", and the values are the positive and negative scores:
import csv, collections
def load_sent_word_net():
sent_scores = collections.defaultdict(list)
with open(os.path.join(DATA_DIR,
SentiWordNet_3.0.0_20130122.txt"), "r") as csvfile:
reader = csv.reader(csvfile, delimiter='\t',
quotechar='"')
for line in reader:
if line[0].startswith("#"):
continue
if len(line)==1:
continue
POS,ID,PosScore,NegScore,SynsetTerms,Gloss = line
if len(POS)==0 or len(ID)==0:
continue
#print POS,PosScore,NegScore,SynsetTerms
for term in SynsetTerms.split(" "):
# drop number at the end of every term
term = term.split("#")[0]
term = term.replace("-", " ").replace("_", " ")
key = "%s/%s"%(POS,term.split("#")[0])
sent_scores[key].append((float(PosScore),
float(NegScore)))
for key, value in sent_scores.iteritems():
sent_scores[key] = np.mean(value, axis=0)
return sent_scores
[ 142 ]