Learning Data Mining with Python

Chapter 4
- Train Confidence: 1.000
- Test Confidence: 0.971
Rule #5
Rule: If a person recommends Shawshank Redemption, The (1994), Toy
Story (1995), Twelve Monkeys (1995), Empire Strikes Back, The (1980),
Fugitive, The (1993), Star Wars (1977) they will also recommend Return
of the Jedi (1983)
- Train Confidence: 1.000
- Test Confidence: 0.900
The second rule, for instance, has a perfect confidence in the training data, but it
is only accurate in 60 percent of cases for the test data. Many of the other rules in
the top 10 have high confidences in test data though, making them good rules for
making recommendations.
If you are looking through the rest of the rules, some will have a
test confidence of -1. Confidence values are always between 0 and
1. This value indicates that the particular rule wasn't found in the
test dataset at all.
Summary
In this chapter, we performed affinity analysis in order to recommend movies based
on a large set of reviewers. We did this in two stages. First, we found frequent
itemsets in the data using the Apriori algorithm. Then, we created association rules
from those itemsets.
The use of the Apriori algorithm was necessary due to the size of the dataset.
While in Chapter 1, Getting Started With Data Mining, we used a brute-force
approach, the exponential growth in the time needed to compute those rules
required a smarter approach. This is a common pattern for data mining: we can
solve many problems in a brute force manner, but smarter algorithms allow us to
apply the concepts to larger datasets.
We performed training on a subset of our data in order to find the association rules,
and then tested those rules on the rest of the data—a testing set. From what we
discussed in the previous chapters, we could extend this concept to use cross-fold
validation to better evaluate the rules. This would lead to a more robust evaluation
of the quality of each rule.
[ 79 ]