Learning Data Mining with Python

Social Media Insight Using Naive Bayes
Putting it all together
Now comes the moment to put all of these pieces together. In our IPython
Notebook, set the filenames and load the dataset and classes as we have done
before. Set the filenames for both the tweets themselves (not the IDs!) and the
labels that we assigned to them. The code is as follows:
import os
input_filename = os.path.join(os.path.expanduser("~"), "Data",
"twitter", "python_tweets.json")
labels_filename = os.path.join(os.path.expanduser("~"), "Data",
"twitter", "python_classes.json")
Load the tweets themselves. We are only interested in the content of the tweets, so
we extract the text value and store only that. The code is as follows:
tweets = []
with open(input_filename) as inf:
for line in inf:
if len(line.strip()) == 0:
continue
tweets.append(json.loads(line)['text'])
Load the labels for each of the tweets:
with open(classes_filename) as inf:
labels = json.load(inf)
Now, create a pipeline putting together the components from before. Our pipeline
has three parts:
• The NLTKBOW transformer we created
• A DictVectorizer transformer
• A BernoulliNB classifier
The code is as follows:
from sklearn.pipeline import Pipeline
pipeline = Pipeline([('bag-of-words', NLTKBOW()),
('vectorizer', DictVectorizer()),
('naive-bayes', BernoulliNB())
])
[ 128 ]