Learning Data Mining with Python

Clustering News Articles
The labels variable now contains the cluster numbers for each sample. Samples
with the same label are said to belong in the same cluster. It should be noted that
the cluster labels themselves are meaningless: clusters 1 and 2 are no more similar
than clusters 1 and 3.
We can see how many samples were placed in each cluster using the Counter class:
from collections import Counter
c = Counter(labels)
for cluster_number in range(n_clusters):
print("Cluster {} contains {} samples".format(cluster_number,
c[cluster_number]))
Many of the results (keeping in mind that your dataset will be quite different to
mine) consist of a large cluster with the majority of instances, several medium
clusters, and some clusters with only one or two instances. This imbalance is quite
normal in many clustering applications.
Evaluating the results
Clustering is mainly an exploratory analysis, and therefore it is difficult to evaluate
a clustering algorithm's results effectively. A straightforward way is to evaluate the
algorithm based on the criteria the algorithm tries to learn from.
If you have a test set, you can evaluate clustering against it. For
more details, visit http://nlp.stanford.edu/IR-book/html/
htmledition/evaluation-of-clustering-1.html.
In the case of the k-means algorithm, the criterion that it uses when developing
the centroids is to minimize the distance from each sample to its nearest centroid.
This is called the inertia of the algorithm and can be retrieved from any KMeans
instance that has had fit called on it:
pipeline.named_steps['clusterer'].inertia_
The result on my dataset was 343.94. Unfortunately, this value is quite meaningless
by itself, but we can use it to determine how many clusters we should use. In
the preceding example, we set n_clusters to 10, but is this the best value? The
following code runs the k-means algorithm 10 times with each value of n_clusters
from 2 to 20. For each run, it records the inertia of the result.
[ 226 ]