PhD Thesis Semi-Supervised Ensemble Methods for Computer Vision

64 Chapter 5. On-line Semi-Supervised Boosting
Discussion As we have seen, by training a prior classifier F P from labeled samples a
priori provided, it is possible to include unlabeled data into the on-line boosting framework
using pseudo-labels and pseudo-importances. Note that if some labeled data is given
in advance, this data can be used to train the prior (or even update the prior when it arrives
over time). However, it should also be clear that the incorporation of prior knowledge
is not limited to a prior classifier and in principle any source of prior knowledge can
be incorporated. Our on-line semi-supervised boosting algorithm for feature selection is
sketched in Algorithm 5.1. As can be seen, compared to the original on-line boosting
algorithm [Grabner and Bischof, 2006] only a few lines of code have to be changed in
order to incorporate unlabeled data.
5.2 On Robustness of On-line Boosting
Currently, boosting is one of the best and thus one of the mostly applied classification
methods in machine learning. This is also true for the on-line variant, which is frequently
applied in practice. However, boosting is proven, from both the theoretical and the experimental
point of view to be sensitive to label noise. For off-line boosting, this issue was
discovered relatively early and, hence, different more robust methods [Maclin and Opitz,
1997, Mason et al., 1999, Dietterich, 1998, Friedman et al., 2000, Freund, 2000, Domingo
and Watanabe, 2000,Long and Servedio, 2008b,Masnadi-Shirazi and Vasconcelos, 2008]
have been proposed.
Although for on-line boosting robustness is highly important because it is frequently
applied in autonomous learning problems, in contrast to the off-line case, up to now this
issue has not been studied. Thus, in the following, we study the robustness of on-line
boosting for feature selection in terms of label noise. In particular, we follow the work of
Long et al. [Long and Servedio, 2008b], who showed that the loss function has not only
high influence to the learning behavior but also on the robustness. Especially convex loss
functions (typically used in boosting) are highly susceptible to random noise. Hence, to
increase the robustness the goal is to find more suitable less noise sensitive loss functions.
For that purpose, we first introduce a generic boosting algorithm, which we call Online
GradientBoost, where arbitrary loss functions can be plugged in easily. In fact, this
method extends the GradientBoost algorithm of Friedman et al. [Friedman, 2001] and
is similar to the AnyBoost algorithm of Mason et al. [Mason et al., 1999]. Based on
this algorithm, we develop different on-line boosting methods using the loss functions
proposed for robust off-line boosting algorithms.
We talk about label noise, if a sample was assigned a wrong label during the labeling
process of the data. AdaBoost is highly susceptible to noise. This sensitivity comes from
the fact that AdaBoost increases the weight of a mis-classified sample in each iteration.