Building Machine Learning Systems with Python - Richert, Coelho

Chapter 6
Determining the word types
Determining the word types is what part of speech (POS) tagging is all about. A
POS tagger parses a full sentence with the goal to arrange it into a dependence tree,
where each node corresponds to a word and the parent-child relationship determines
which word it depends on. With this tree, it can then make more informed decisions;
for example, whether the word "book" is a noun ("This is a good book.") or a verb
("Could you please book the flight?").
You might have already guessed that NLTK will also play a role also in this area.
And indeed, it comes readily packaged with all sorts of parsers and taggers. The POS
tagger we will use, nltk.pos_tag(), is actually a full-blown classifier trained using
manually annotated sentences from the Penn Treebank Project (http://www.cis.
upenn.edu/~treebank). It takes as input a list of word tokens and outputs a list of
tuples, each element of which contains the part of the original sentence and its part of
speech tag:
>>> import nltk
>>> nltk.pos_tag(nltk.word_tokenize("This is a good book."))
[('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('good', 'JJ'), ('book',
'NN'), ('.', '.')]
>>> nltk.pos_tag(nltk.word_tokenize("Could you please book the
flight?"))
[('Could', 'MD'), ('you', 'PRP'), ('please', 'VB'), ('book', 'NN'),
('the', 'DT'), ('flight', 'NN'), ('?', '.')]
The POS tag abbreviations are taken from the Penn Treebank Project (adapted from
http://americannationalcorpus.org/OANC/penn.html):
POS tag Description Example
CC coordinating conjunction or
CD cardinal number 2 second
DT determiner the
EX existential there there are
FW foreign word kindergarten
IN
preposition/subordinating
conjunction
on, of, like
JJ adjective cool
JJR adjective, comparative cooler
JJS adjective, superlative coolest
LS list marker 1)
MD modal could, will
[ 139 ]