Building Machine Learning Systems with Python - Richert, Coelho

Clustering – Finding Related Posts
>>> num_samples, num_features = vectorized.shape
>>> print("#samples: %d, #features: %d" % (num_samples, num_features))
#samples: 3414, #features: 4331
We now have a pool of 3,414 posts and extracted for each of them a feature vector of
4,331 dimensions. That is what KMeans takes as input. We will fix the cluster size to
50 for this chapter and hope you are curious enough to try out different values as an
exercise, as shown in the following code:
>>> num_clusters = 50
>>> from sklearn.cluster import KMeans
>>> km = KMeans(n_clusters=num_clusters, init='random', n_init=1,
verbose=1)
>>> km.fit(vectorized)
That's it. After fitting, we can get the clustering information out of the members of
km. For every vectorized post that has been fit, there is a corresponding integer label
in km.labels_:
>>> km.labels_
array([33, 22, 17, ..., 14, 11, 39])
>>> km.labels_.shape
(3414,)
The cluster centers can be accessed via km.cluster_centers_.
In the next section we will see how we can assign a cluster to a newly arriving post
using km.predict.
Solving our initial challenge
We now put everything together and demonstrate our system for the following new
post that we assign to the variable new_post:
Disk drive problems. Hi, I have a problem with my hard disk.
After 1 year it is working only sporadically now.
I tried to format it, but now it doesn't boot any more.
Any ideas? Thanks.
[ 68 ]