CRANFIELD UNIVERSITY Eleni Anthippi Chatzimichali ...

Furthermore, metrics such as the bias and the variance are also very powerful tools for
the assessment of a machine learning model. The bias of a method can be defined as
“the difference between the expected and the estimated value” (Kohavi, 1995). In
addition, the variance indicates the variability of a classifier’s predictive power across
the different training sets (Bauer and Kohavi, 1999). Ideally, a good classifier presents
both low bias and low variance. According to Burges (1998), the generalisation
ability of a classifier is highly dependent on the “bias-variance trade-off” (Germal et
al., 1992).
1.6.1 The holdout method
The holdout method randomly partitions the entire input dataset into two mutually
exclusive subsets (Suykens et al., 2002). The two newly created sets are commonly
termed as the training and the test set, or holdout set. A common approach is to
randomly designate 1/3 of the initial data as the test set, whereas the remaining 2/3 of
the data are used to train the model (Kohavi, 1995; Brereton, 2009). The test set is
kept aside during the training process and is only used to evaluate the accuracy or the
error rate of the trained classifier. In order to assure strong classifier and optimal
prediction rates, there should be exactly a third of the instances for each available
class label included in the test set (Kohavi, 1995; Brereton, 2009); this approach is
often referred to as the stratified holdout method.
The main drawback of this method is the demand for an adequate amount of samples
in the test set. The prediction rate tends to increase as more instances are provided.
The more instances included in the test set, the higher the bias of the estimate.
However, for datasets that the initial number of samples is quite small, the results tend
to present high variance. Thus, alternative algorithms such as cross-validation and
bootstrapping are applied.
24