CRANFIELD UNIVERSITY Eleni Anthippi Chatzimichali ...

2.3.2.3 Comparison of validation techniques for the optimisation of the (RBF)
SVM hyperparameters
As thoroughly explained thus far, the prediction power and accuracy of a classifier is
chiefly dependent on the optimisation (tuning) of its hyperparameters. The grid and
surface plots of Figure 2-8 and Figure 2-9 aim at assessing the outcome of the
optimisation process and defining the best technique. Simulations were initiated using
a coarse grid-search; the grid resolution was gradually refined to a grid-step of
, forming a final grid space of 6561 points as presented in Figure 2-8. For
each combination of hyperparameters in the grid, bootstrapping, 10-fold crossvalidation
and LOOCV were applied on the training set.
According to Section 1.6.3, even though LOOCV is a nearly unbiased technique, it
often presents cases of unacceptable high variance, especially when applied to
relatively small datasets such as the ones of case study 1. In addition, LOOCV is a
computationally expensive validation technique that may lead to long execution times
and computationally prohibitive solutions for relatively large datasets (Boardman and
Trappenberg, 2006). On the contrary, 10-fold cross-validation, proved to be the fastest
technique among the three. A great advantage of 10-fold cross-validation is that all
instances within a dataset are eventually used for both training and testing. However,
since the outcome highly depends on the random split into folds, this approach may
lead to training instabilities and relatively high variance (see Section 1.6.2). In
addition, cross-validation proved to be extremely prone to overfitting, especially
during the fine-tuning process. Therefore, this technique proved to be unreliable.
Ensembles of SVMs were also optimised using bootstrapping. Fine grid-search
proved to be extremely fruitful since all overall prediction accuracies increased by a
minimum of 2%. As presented in Section 1.6.4, the probability for any given instance
not being selected in a bootstrap set is approximately 36.8%; thus, bootstrapping
minimises the chances of overfitting (Kohavi, 1995). Indeed, overfitting was no
exception in this case either, however, it was only present in a miniscule number of
instances. Even though bootstrapping is a fairly straightforward method, it constitutes
a computationally demanding statistical procedure that leads to extremely long runs.
The execution times may increase exponentially for larger datasets.
53