Learning Data Mining with Python

Authorship Attribution
Kernels
When the data cannot be separated linearly, the trick is to embed it onto a higher
dimensional space. What this means, with a lot of hand-waving about the details, is
to add pseudo-features until the data is linearly separable (which will always happen
if you add enough of the right kinds of features).
The trick is that we often compute the inner-produce of the samples when finding
the best line to separate the dataset. Given a function that uses the dot product, we
effectively manufacture new features without having to actually define those new
features. This is handy because we don't know what those features were going to
be anyway. We now define a kernel as a function that itself is the dot product of
the function of two samples from the dataset, rather than based on the samples
(and the made-up features) themselves.
We can now compute what that dot product is (or approximate it) and then just
use that.
There are a number of kernels in common use. The linear kernel is the most
straightforward and is simply the dot product of the two sample feature vectors,
the weight feature, and a bias value. There is also a polynomial kernel, which raises
the dot product to a given degree (for instance, 2). Others include the Gaussian
(rbf) and Sigmoidal functions. In our previous code sample, we tested between
the linear kernel and the rbf kernels.
The end result from all this derivation is that these kernels effectively define a
distance between two samples that is used in the classification of new samples in
SVMs. In theory, any distance could be used, although it may not share the same
characteristics that enable easy optimization of the SVM training.
In scikit-learn's implementation of SVMs, we can define the kernel parameter to
change which kernel function is used in computations, as we saw in the previous
code sample.
Character n-grams
We saw how function words can be used as features to predict the author of a
document. Another feature type is character n-grams. An n-gram is a sequence of n
objects, where n is a value (for text, generally between 2 and 6). Word n-grams have
been used in many studies, usually relating to the topic of the documents. However,
character n-grams have proven to be of high quality for authorship attribution.
[ 198 ]