PhD Thesis Semi-Supervised Ensemble Methods for Computer Vision

Chapter 5
On-line Semi-Supervised Boosting
Although there have been proposed a large amount of approaches to the SSL task,
most of them operate in off-line or batch mode 1 . Recall from Chapter 2 that offline
methods assume having access to the entire training data at any time, which eases
optimization and typically yields good classifiers. In practice, however, on-line learning
capability is demanded because learners often have limited access to the problem domain
due to dynamic environments or streaming data sources. Additionally, on-line learning
methods usually require less memory to operate which makes them ideal for incremental
learning problems, interactive learning or in cases where limited hardware resources do
not allow for full data storage, for instance on mobile devices. Finally, in supervised
learning on-line methods are frequently applied to large scale data problems due to their
inherent good scaling characteristics.
These benefits make on-line approaches also interesting for semi-supervised learning
and leverage the usage of SSL in scenarios that require sequential data analysis. Furthermore,
the goal of SSL is to benefit from large amounts of unlabeled data. Yet, the bad
scaling behavior of current state-of-the-art SSL methods, which can be up to O(n 3 ) [Mann
and Mccallum, 2007], where n is the number of unlabeled samples, often limits their applicability
in practice. This seems to be in particular paradox in a field that claims to
benefit from the huge amounts of unlabeled data that are often available for free. Hence,
incorporating on-line learning techniques into SSL has also the potential to make them
operate in really large data scenarios. Although on-line learning can bring several advantages
to semi-supervised learning, in the literature there exist only a small amount of
approaches dealing with on-line SSL, e.g., methods based on deep neural networks [Weston
et al., 2008] or convex programming in kernel space [Goldberg et al., 2008].
In this chapter, we discuss on-line semi-supervised learning algorithms based on online
boosting [Oza, 2001]. The reasons for developing such algorithms are that boosting
1 Note that throughout this text we use the two terms interchangeably with equal meaning.
61