Learning Data Mining with Python

To do this, we will compute the test set confidence, that is, the confidence of each
rule on the testing set.
Chapter 4
We won't apply a formal evaluation metric in this case; we simply examine the rules
and look for good examples.
First, we extract the test dataset, which is all of the records we didn't use in the
training set. We used the first 200 users (by ID value) for the training set, and we will
use all of the rest for the testing dataset. As with the training set, we will also get the
favorable reviews for each of the users in this dataset as well. Let's look at the code:
test_dataset =
all_ratings[~all_ratings['UserID'].isin(range(200))]
test_favorable = test_dataset[test_dataset["Favorable"]]
test_favorable_by_users = dict((k, frozenset(v.values)) for k, v
in test_favorable.groupby("UserID")["MovieID"])
We then count the correct instances where the premise leads to the conclusion, in the
same way we did before. The only change here is the use of the test data instead of
the training data. Let's look at the code:
correct_counts = defaultdict(int)
incorrect_counts = defaultdict(int)
for user, reviews in test_favorable_by_users.items():
for candidate_rule in candidate_rules:
premise, conclusion = candidate_rule
if premise.issubset(reviews):
if conclusion in reviews:
correct_counts[candidate_rule] += 1
else:
incorrect_counts[candidate_rule] += 1
Next, we compute the confidence of each rule from the correct counts. Let's look at
the code:
test_confidence = {candidate_rule: correct_counts[candidate_rule]
/ float(correct_counts[candidate_rule] + incorrect_counts
[candidate_rule])
for candidate_rule in rule_confidence}
Finally, we print out the best association rules with the titles instead of the
movie IDs.
for index in range(5):
print("Rule #{0}".format(index + 1))
[ 77 ]