Learning Data Mining with Python

Chapter 4
When loading the file, we set the delimiter parameter to the tab character, tell pandas
not to read the first row as the header (with header=None), and set the column
names. Let's look at the following code:
all_ratings = pd.read_csv(ratings_filename, delimiter="\t",
header=None, names = ["UserID", "MovieID", "Rating", "Datetime"])
While we won't use it in this chapter, you can properly parse the date timestamp
using the following line:
all_ratings["Datetime"] = pd.to_datetime(all_ratings['Datetime'],
unit='s')
You can view the first few records by running the following in a new cell:
all_ratings[:5]
The result will come out looking something like this:
UserID MovieID Rating Datetime
0 196 242 3 1997-12-04 15:55:49
1 186 302 3 1998-04-04 19:22:22
2 22 377 1 1997-11-07 07:18:36
3 244 51 2 1997-11-27 05:02:03
4 166 346 1 1998-02-02 05:33:16
Sparse data formats
This dataset is in a sparse format. Each row can be thought of as a cell in a large
feature matrix of the type used in previous chapters, where rows are users and
columns are individual movies. The first column would be each user's review
of the first movie, the second column would be each user's review of the second
movie, and so on.
There are 1,000 users and 1,700 movies in this dataset, which means that the full
matrix would be quite large. We may run into issues storing the whole matrix in
memory and computing on it would be troublesome. However, this matrix has the
property that most cells are empty, that is, there is no review for most movies for
most users. There is no review of movie #675 for user #213 though, and not for most
other combinations of user and movie.
[ 65 ]