Berry

Market Basket Analysis and Association Rules 313
In other words, minimum support pruning eliminates items that do not
appear in enough transactions. The threshold criterion applies to each step in
the algorithm. The minimum threshold also implies that:
A and B must appear together in at least 10,000 transactions, and,
A and C must appear together in at least 10,000 transactions, and,
A and D must appear together in at least 10,000 transactions,
and so on.
Each step of the calculation of the co-occurrence table can eliminate combinations
of items that do not meet the threshold, reducing its size and the number
of combinations to consider during the next pass.
Figure 9.11 is an example of how the calculation takes place. In this example,
choosing a minimum support level of 10 percent would eliminate all the combinations
with three items—and their associated rules—from consideration.
This is an example where pruning does not have an effect on the best rule since
the best rule has only two items. In the case of pizza, these toppings are all
fairly common, so are not pruned individually. If anchovies were included in
the analysis—and there are only 15 pizzas containing them out of the 2,000—
then a minimum support of 10 percent, or even 1 percent, would eliminate
anchovies during the first pass.
The best choice for minimum support depends on the data and the situation.
It is also possible to vary the minimum support as the algorithm progresses.
For instance, using different levels at different stages you can find
uncommon combinations of common items (by decreasing the support level
for successive steps) or relatively common combinations of uncommon items
(by increasing the support level).
The Problem of Big Data
A typical fast food restaurant offers several dozen items on its menu, say 100.
To use probabilities to generate association rules, counts have to be calculated
for each combination of items. The number of combinations of a given size
tends to grow exponentially. A combination with three items might be a small
fries, cheeseburger, and medium Diet Coke. On a menu with 100 items, how
many combinations are there with three different menu items? There are
161,700! This calculation is based on the binomial formula On the other hand,
a typical supermarket has at least 10,000 different items in stock, and more typically
20,000 or 30,000.