1FfUrl0

Chapter 3
How to do it
More robust than edit distance is the so-called bag-of-word approach. It uses simple
word counts as its basis. For each word in the post, its occurrence is counted and
noted in a vector. Not surprisingly, this step is also called vectorization. The vector is
typically huge as it contains as many elements as the words that occur in the whole
dataset. Take for instance two example posts with the following word counts:
Word
Occurrences in
Post 1
disk 1 1
format 1 1
how 1 0
hard 1 1
my 1 0
problems 0 1
to 1 0
Occurrences in
Post 2
The columns Post 1 and Post 2 can now be treated as simple vectors. We could
simply calculate the Euclidean distance between the vectors of all posts and take
the nearest one (too slow, as we have just found out). As such, we can use them
later in the form of feature vectors in the following clustering steps:
1. Extract the salient features from each post and store it as a vector per post.
2. Compute clustering on the vectors.
3. Determine the cluster for the post in question.
4. From this cluster, fetch a handful of posts that are different from the post in
question. This will increase diversity.
However, there is some more work to be done before we get there, and before we
can do that work, we need some data to work on.
Preprocessing – similarity measured as
similar number of common words
As we have seen previously, the bag-of-word approach is both fast and robust.
However, it is not without challenges. Let's dive directly into them.
[ 51 ]