Building Machine Learning Systems with Python - Richert, Coelho

Topic Modeling
… for tj,v in t:
… dense[ti,tj] = v
Now, dense is a matrix of topics. We can use the pdist function in SciPy to compute
all pairwise distances. That is, with a single function call, we compute all the values
of sum((dense[ti] – dense[tj])**2):
>>> from scipy.spatial import distance
>>> pairwise = distance.squareform(distance.pdist(dense))
Now we employ one last little trick; we set the diagonal elements of the distance
matrix to a high value (it just needs to be larger than the other values in the matrix):
>>> largest = pairwise.max()
>>> for ti in range(len(topics)):
pairwise[ti,ti] = largest+1
And we are done! For each document, we can look up the closest element easily:
>>> def closest_to(doc_id):
return pairwise[doc_id].argmin()
The previous code would not work if we had not set the diagonal
elements to a large value; the function would always return the same
element as it is almost similar to itself (except in the weird case where
two elements have exactly the same topic distribution, which is very
rare unless they are exactly the same).
For example, here is the second document in the collection (the first document is
very uninteresting, as the system returns a post stating that it is the most similar):
From: geb@cs.pitt.edu (Gordon Banks)
Subject: Re: request for information on "essential tremor" and Indrol?
In article sundar@ai.mit.edu writes:
Essential tremor is a progressive hereditary tremor that gets worse
when the patient tries to use the effected member. All limbs, vocal
cords, and head can be involved. Inderal is a beta-blocker and is
usually effective in diminishing the tremor. Alcohol and mysoline are
also effective, but alcohol is too toxic to use as a treatment.
----------------------------------------------------------------Gordon
Banks N3JXP | "Skepticism is the chastity of the intellect, and
geb@cadre.dsl.pitt.edu | it is shameful to surrender it too soon."
----------------------------------------------------------------
If we ask for the most similar document, closest_to(1), we receive the
following document:
From: geb@cs.pitt.edu (Gordon Banks)
[ 82 ]