Lexicon-Based Methods for Sentiment Analysis - Simon Fraser ...

Computational Linguistics Volume 37, Number 2
The “Maryland” dictionary (Mohammad, Dorr, and Dunne 2009) is a very large
collection of words and phrases (around 70,000) extracted from the Macquarie Thesaurus.
The dictionary is not classified according to part of speech, and only contains
information on whether the word is positive or negative. To integrate it into our system,
we assigned all positive words an SO value of 3, and all negative words a value of −3. 33
We used the same type of quantification for the General Inquirer (GI; Stone et al.
1966), which also has only positive and negative tags; a word was included in the
dictionary if any of the senses listed in the GI were polar.
The Subjectivity dictionary is the collection of subjective expressions compiled by
Wilson, Wiebe, and Hoffmann (2005), also used in our Mechanical Turk experiments in
the previous section. The Subjectivity dictionary only contains a distinction between
weak and strong opinion words. For our tests, weak words were assigned 2 or −2
values, depending on whether they were positive or negative, and strong words were
assigned 4 or −4.
The SentiWordNet dictionary (Esuli and Sebastiani 2006; Baccianella, Esuli, and
Sebastiani 2010), also used in the previous section, was built using WordNet (Fellbaum
1998), and retains its synset structure. There are two main versions of SentiWordNet
available, 1.0 and 3.0, and two straightforward methods to calculate an SO value: 34
Use the first sense SO, or average the SO across senses. For the first sense method,
we calculate the SO value of a word w (of a given POS) based on its first sense f as
follows:
SO(w) = 5 × (Pos( f ) − Neg( f ))
For the averaging across senses method, SO is calculated as
SO(w) =
5
|senses| x∈senses (Pos(x) − Neg(x))
that is, the difference between the positive and negative scores provided by SentiWord-
Net (each in the 0–1 range), averaged across all word senses for the desired part of
speech, and multiplied by 5 to provide SO values in the −5 to 5 range. Table 9 contains a
comparison of performance (using simple word averaging, no SO-CAL features) in our
various corpora for each version of SentiWordNet using each method. What is surprising
is that the best dictionary using just basic word counts (1.0, first sense) is actually
the worst dictionary when using SO-CAL, and the best dictionary using all SO-CAL
features (3.0, average across senses) is the worst dictionary when features are disabled.
We believe this effect is due almost entirely to the degree of positive bias in the various
dictionaries. The 3.0 average dictionary is the most positively biased, which results in
degraded basic performance (in the Camera corpus, only 20.5% of negative reviews
are correctly identified). When negative weighting is applied, however, it reaches an
almost perfect balance between positive and negative accuracy (70.7% to 71.3% in the
Camera corpus), which optimizes overall performance. We cannot therefore conclude
definitively that any of the SentiWordNet dictionaries is superior for our task; in fact,
33 We chose 3 as a value because it is the middle value between 2 and 4 that we assigned to strong and weak
words in the Subjectivity Dictionary, as explained subsequently.
34 A third alternative would be to calculate a weighted average using sense frequency information;
SentiWordNet does not include such information, however. Integrating this information from other
sources, though certainly possible, would take us well beyond “off-the-shelf” usage, and, we believe,
would provide only marginal benefit.
298