1FfUrl0

Chapter 4
If you are going to explore the topics yourself or build a visualization tool, you
should probably try a few values and see which gives you the most useful or most
appealing results.
However, there are a few methods that will automatically determine the number of
topics for you depending on the dataset. One popular model is called the hierarchical
Dirichlet process. Again, the full mathematical model behind it is complex and beyond
the scope of this book, but the fable we can tell is that instead of having the topics be
fixed a priori and our task being to reverse engineer the data to get them back, the
topics themselves were generated along with the data. Whenever the writer was going
to start a new document, he had the option of using the topics that already existed or
creating a completely new one.
This means that the more documents we have, the more topics we will end up with.
This is one of those statements that is unintuitive at first, but makes perfect sense upon
reflection. We are learning topics, and the more examples we have, the more we can
break them up. If we only have a few examples of news articles, then sports will be a
topic. However, as we have more, we start to break it up into the individual modalities
such as Hockey, Soccer, and so on. As we have even more data, we can start to tell
nuances apart articles about individual teams and even individual players. The same
is true for people. In a group of many different backgrounds, with a few "computer
people", you might put them together; in a slightly larger group, you would have
separate gatherings for programmers and systems managers. In the real world, we
even have different gatherings for Python and Ruby programmers.
One of the methods for automatically determining the number of topics is called
the hierarchical Dirichlet process (HDP), and it is available in gensim. Using it is
trivial. Taking the previous code for LDA, we just need to replace the call to gensim.
models.ldamodel.LdaModel with a call to the HdpModel constructor as follows:
>>> hdp = gensim.models.hdpmodel.HdpModel(mm, id2word)
That's it (except it takes a bit longer to compute—there are no free lunches). Now,
we can use this model as much as we used the LDA model, except that we did not
need to specify the number of topics.
Summary
In this chapter, we discussed a more advanced form of grouping documents, which is
more flexible than simple clustering as we allow each document to be present in more
than one group. We explored the basic LDA model using a new package, gensim, but
were able to integrate it easily into the standard Python scientific ecosystem.
[ 87 ]