SEKE 2012 Proceedings - Knowledge Systems Institute

3.3. Evaluation Method
We use top-N precision to evaluate the performance of
algorithms in recommendation task. In order to measure
precision, we first separate the original gathered data set
which is mentioned in section 3.1 into two parts. The
training set contains 80 percent of data, and the testing set
20 percent. To satisfy the different recommendation algorithms
mentioned above, we apply two methods to separate
the data set. For content-based, item-to-item-CF and userbased-CF,
we cannot do any recommendation when the user
comes to visit our site for the first time. So when we randomly
select 20 percent of the data from the original gathered
data set, we exclude those records of the products user
visits when they come to our site at first. For the last three
algorithms, the recommendation is based on the overall data
and not specific to each user. So we select the 20 percent
testing data completely randomly.
The computation of precision proceeds as follows. For
any single test, if the test product is one of the recommended
N products it is a hit case. Then the overall precision can be
defined by follow formula:
hit − times
precision(N) = (3)
N ∗|U T |
where hit-times is in all test cases the times of recommended
N products contain test product, T is the test set
and |U T | is the number of users T contains.
Except for precision, we also evaluate the cost of each
recommendation method according to following five criterions:1.
require item feature;2. require rating information;3.
require user behavior history data;4. require statistical information
of the data set;5. require relatively complex calculation;
Each criterion adds 1 point to the total cost.
4. Result and Discussion
4.1. Experimental Result
In this section, we present the experimental results of
the recommender algorithms mentioned in above section on
our data set.We apply the evaluation method described in
section 3.3 to evaluate six recommender algorithms.
Figure 1 shows that item-to-item-CF gets the best performance
with precision of 19.50%, it outperforms other
methods quite a lot, and its 3 points cost is relatively high
but not the highest. The most-popular as benchmark method
gets precision of 8.86% which is the second best performance,
and its 2 points cost is relatively low compared
with other methods with similar performance. It shows that
most-popular has really high cost performance. Contentbased
with a 6.85% precision and user-based-CF getting a
7.39% precision do not outperformmost-popular which we
Figure 1. Top-5 precision and cost of methods
select as the benchmark. Also, these two methods have the
highest 4 points cost. This does not indicate that these two
main recommendation methods are not effective, but maybe
they have somelimitations under vertical B2C E-commerce
cased. Cheapest and newest have not brought us any surprise.
Although their cost is really low with 1 point, but
the performances of cheapest with a 6.12% precision and
newest with a 3.23% precision are not acceptable.
4.2. Discussion
Content-based: Content-based do not outperform the
benchmark in our experiment, and we think two main reasons
are the low quality of product feature information and
lack of item ratings to detect users’ preference. The low
quality feature information of bags mainly comes from two
reasons, one is some suppliers do not give all feature information
and the other is different suppliers use different
words as features. To preprocess the feature information
from different suppliers, we need to fill in some missing
features and unify different words which express the same
meaning. In our experiments, we assume all visited items
are of the same interest to user. But if we know the degree
of users’ interest in each item, we can weight most preferred
items’ features more.
Item-to-item-CF: Item-to-item-CF gets the best performance,
this is an algorithm successfully applied in Amazon’s
recommender system. In this algorithm, the main
cost is that we need to count all item pairs’ co-visited or
co-purchased times. Although the average number of items
visited by one single user is only about 5, we have sufficient
visiting history of different users to one single item,
and it is exactly what this algorithm needs to figure out similar
items from the item perspective. Also this method do
not need item feature information, it is one advantage to
141