Learning Data Mining with Python

Chapter 5
A transformer is akin to a converting function. It takes data of one form as input
and returns data of another form as output. Transformers can be trained using some
training dataset, and these trained parameters can be used to convert testing data.
The transformer API is quite simple. It takes data of a specific format as input and
returns data of another format (either the same as the input or different) as output.
Not much else is required of the programmer.
The transformer API
Transformers have two key functions:
• fit(): This takes a training set of data as input and sets internal parameters
• transform(): This performs the transformation itself. This can take either
the training dataset, or a new dataset of the same format
Both fit() and transform() fuction should take the same data type as input, but
transform() can return data of a different type.
We are going to create a trivial transformer to show the API in action. The
transformer will take a NumPy array as input, and discretize it based on the mean.
Any value higher than the mean (of the training data) will be given the value 1 and
any value lower or equal to the mean will be given the value 0.
We did a similar transformation with the Adult dataset using pandas: we took
the Hours-per-week feature and created a LongHours feature if the value was more
than 40 hours per week. This transformer is different for two reasons. First, the code
will conform to the scikit-learn API, allowing us to use it in a pipeline. Second, the
code will learn the mean, rather than taking it as a fixed value (such as 40 in the
LongHours example).
Implementation details
To start, open up the IPython Notebook that we used for the Adult dataset.
Then, click on the Cell menu item and choose Run All. This will rerun all of
the cells and ensure that the notebook is up to date.
First, we import the TransformerMixin, which sets the API for us. While Python
doesn't have strict interfaces (as opposed to languages like Java), using a mixin
like this allows scikit-learn to determine that the class is actually a transformer.
We also need to import a function that checks the input is of a valid type.
We will use that soon.
[ 99 ]