Learning Data Mining with Python

Clustering News Articles
If there is an error in obtaining the website, we simply skip this website and keep
going. This code will work on 95 percent of websites and that is good enough for
our application, as we are looking for general trends and not exactness. Note that
sometimes you do care about getting 100 percent of responses, and you should
adjust your code to accommodate more errors. The code to get those final 5 to 10
percent of websites will be significantly more complex. We then catch any error that
could occur (it is the Internet, lots of things could go wrong), increment our error
count, and continue.
except Exception as e:
number_errors += 1
print(e)
If you find that too many errors occur, change the print(e) line to just type
raise instead. This will cause the exception to be called, allowing you to debug
the problem.
Now, we have a bunch of websites in our raw subfolder. After taking a look at these
pages (open the created files in a text editor), you can see that the content is there
but there are HTML, JavaScript, CSS code, as well as other content. As we are only
interested in the story itself, we now need a way to extract this information from
these different websites.
Putting it all together
After we get the raw data, we need to find the story in each. There are a few online
sources that use data mining to achieve this. You can find them listed in Appendix A,
Next Steps…. It is rarely needed to use such complex algorithms, although you can
get better accuracy using them. This is part of data mining—knowing when to use it,
and when not to.
First, we get a list of each of the filenames in our raw subfolder:
filenames = [os.path.join(data_folder, filename)
for filename in os.listdir(data_folder)]
Next, we create an output folder for the text only versions that we will extract:
text_output_folder = os.path.join(os.path.expanduser("~"), "Data",
"websites", "textonly")
[ 220 ]