Learning Data Mining with Python

Extracting Features with Transformers
Next, we need to load the dataset with pandas. First, we set the data's filename
as always:
import os
import numpy as np
import pandas as pd
data_folder = os.path.join(os.path.expanduser("~"), "Data")
data_filename = os.path.join(data_folder, "Ads", "ad.data")
There are a couple of issues with this dataset that stop us from loading it easily.
First, the first few features are numerical, but pandas will load them as strings.
To fix this, we need to write a converting function that will convert strings to
numbers if possible. Otherwise, we will get a NaN (which is short for Not a
Number), which is a special value that indicates that the value could not be
interpreted as a number. It is similar to none or null in other programming languages.
Another issue with this dataset is that some values are missing. These are
represented in the dataset using the string ?. Luckily, the question mark doesn't
convert to a float, so we can convert those to NaNs using the same concept. In further
chapters, we will look at other ways of dealing with missing values like this.
We will create a function that will do this conversion for us:
def convert_number(x):
First, we want to convert the string to a number and see if that fails. Then, we will
surround the conversion in a try/except block, catching a ValueError exception
(which is what is thrown if a string cannot be converted into a number this way):
try:
return float(x)
except ValueError:
Finally, if the conversion failed, we get a NaN that comes from the NumPy library
we imported previously:
return np.nan
Now, we create a dictionary for the conversion. We want to convert all of the
features to floats:
converters = defaultdict(convert_number
[ 94 ]