Learning Data Mining with Python

Chapter 5
Thought should always be given to how to represent reality in the form of a model.
Rather than just using what has been used in the past, you need to consider the goal
of the data mining exercise. What are you trying to achieve? In Chapter 3, Predicting
Sports Winners with Decision Trees, we created features by thinking about the goal
(predicting winners) and used a little domain knowledge to come up with ideas for
new features.
Not all features need to be numeric or categorical. Algorithms have
been developed that work directly on text, graphs, and other data
structures. Unfortunately, those algorithms are outside the scope of
this book. In this book, we mainly use numeric or categorical features.
The Adult dataset is a great example of taking a complex reality and attempting to
model it using features. In this dataset, the aim is to estimate if someone earns more
than $50,000 per year. To download the dataset, navigate to http://archive.ics.
uci.edu/ml/datasets/Adult and click on the Data Folder link. Download the
adult.data and adult.names into a directory named Adult in your data folder.
This dataset takes a complex task and describes it in features. These features describe
the person, their environment, their background, and their life status.
Open a new IPython Notebook for this chapter and set the data's filename and
import pandas to load the file:
import os
import pandas as pd
data_folder = os.path.join(os.path.expanduser("~"), "Data",
"Adult")
adult_filename = os.path.join(data_folder, "adult.data")
Using pandas as before, we load the file with read_csv:
adult = pd.read_csv(adult_filename, header=None,
names=["Age", "Work-Class", "fnlwgt",
"Education", "Education-Num",
"Marital-Status", "Occupation",
"Relationship", "Race", "Sex",
"Capital-gain", "Capital-loss",
"Hours-per-week", "Native-Country",
"Earnings-Raw"])
Most of the code is the same as in the previous chapters.
[ 83 ]