Building Machine Learning Systems with Python - Richert, Coelho

Classification II – Sentiment Analysis
The only missing thing is to define how GridSearchCV should determine the best
estimator. This can be done by providing the desired score function to (surprise!) the
score_func parameter. We could either write one ourselves or pick one from the
sklearn.metrics package. We should certainly not take metric.accuracy because
of our class imbalance (we have a lot less tweets containing sentiment than neutral
ones). Instead, we want to have good precision and recall on both the classes: the
tweets with sentiment and the tweets without positive or negative opinions. One
metric that combines both precision and recall is the F-measure metric, which is
implemented as metrics.f1_score:
Putting everything together, we get the following code:
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import f1_score
def grid_search_model(clf_factory, X, Y):
cv = ShuffleSplit(
n=len(X), n_iter=10, test_size=0.3, indices=True, random_
state=0)
param_grid = dict(vect__ngram_range=[(1, 1), (1, 2), (1, 3)],
vect__min_df=[1, 2],
vect__stop_words=[None, "english"],
vect__smooth_idf=[False, True],
vect__use_idf=[False, True],
vect__sublinear_tf=[False, True],
vect__binary=[False, True],
clf__alpha=[0, 0.01, 0.05, 0.1, 0.5, 1],
)
grid_search = GridSearchCV(clf_factory(),
param_grid=param_grid,
cv=cv,
score_func=f1_score,
verbose=10)
grid_search.fit(X, Y)
return grid_search.best_estimator_
[ 134 ]