1FfUrl0

Regression – Recommendations
reg.fit(xc, y[train]-x1)
# We need to perform the same normalization while testing
xc,x1 = movie_norm(x[test])
p = np.array(map(reg.predict, xc)).ravel()
e = (p+x1)-y[test]
err += np.sum(e*e)
We did not explain the movie_norm function. This function performs per-movie
normalization: some movies are just generally better and get higher average marks:
def movie_norm(x):
xc = x.copy().toarray()
We cannot use xc.mean(1) because we do not want to have the zeros counting for
the mean. We only want the mean of the ratings that were actually given:
x1 = np.array([xi[xi > 0].mean() for xi in xc])
In certain cases, there were no ratings and we got a NaN value, so we replace it with
zeros using np.nan_to_num, which does exactly this task:
x1 = np.nan_to_num(x1)
Now we normalize the input by removing the mean value from the non-zero entries:
for i in xrange(xc.shape[0]):
xc[i] -= (xc[i] > 0) * x1[i]
Implicitly, this also makes the movies that the user did not rate have a value of zero,
which is average. Finally, we return the normalized array and the means:
return x,x1
You might have noticed that we converted to a regular (dense) array. This has the
added advantage that it makes the optimization much faster: while scikit-learn
works well with the sparse values, the dense arrays are much faster (if you can fit
them in memory; when you cannot, you are forced to use sparse arrays).
When compared with simply guessing the average value for that user, this approach
is 80 percent better. The results are not spectacular, but it is a start. On one hand,
this is a very hard problem and we cannot expect to be right with every prediction:
we perform better when the users have given us more reviews. On the other hand,
regression is a blunt tool for this job. Note how we learned a completely separate
model for each user. In the next chapter, we will look at other methods that go
beyond regression for approaching this problem. In those models, we integrate the
information from all users and all movies in a more intelligent manner.
[ 162 ]