Building Machine Learning Systems with Python - Richert, Coelho

Chapter 4
Subject: Re: High Prolactin
In article JER4@psuvm.psu.edu (John
E. Rodway) writes:
>Any comments on the use of the drug Parlodel for high prolactin in
the blood?
>It can suppress secretion of prolactin. Is useful in cases of
galactorrhea. Some adenomas of the pituitary secret too much.
------------------------------------------------------------------
Gordon Banks N3JXP | "Skepticism is the chastity of the
intellect, and geb@cadre.dsl.pitt.edu | it is shameful to surrender
it too soon."
----------------------------------------------------------------
We received a post by the same author discussing medications.
Modeling the whole of Wikipedia
While the initial LDA implementations could be slow, modern systems can work
with very large collections of data. Following the documentation of gensim, we are
going to build a topic model for the whole of the English language Wikipedia. This
takes hours, but can be done even with a machine that is not too powerful. With a
cluster of machines, we could make it go much faster, but we will look at that sort of
processing in a later chapter.
First we download the whole Wikipedia dump from http://dumps.wikimedia.
org. This is a large file (currently just over 9 GB), so it may take a while, unless your
Internet connection is very fast. Then, we will index it with a gensim tool:
python -m gensim.scripts.make_wiki enwiki-latest-pages-articles.xml.bz2
wiki_en_output
Run the previous command on the command line, not on the Python shell. After a
few hours, the indexing will be finished. Finally, we can build the final topic model.
This step looks exactly like what we did for the small AP dataset. We first import a
few packages:
>>> import logging, gensim
>>> logging.basicConfig(
format='%(asctime)s : %(levelname)s : %(message)s',
level=logging.INFO)
Now, we load the data that has been preprocessed:
>>> id2word =
gensim.corpora.Dictionary.load_from_text('wiki_en_output_wordids.txt')
>>> mm = gensim.corpora.MmCorpus('wiki_en_output_tfidf.mm')
[ 83 ]