Project Proposal (PDF) - Oxford Brookes University

FP7-ICT-2011-9 STREP proposal
18/01/12 v1 [Dynact]
multiple actions performed by the same person.
Breakthrough #3: presence of covariance factors and recognition "in the wild". In order to
bring action recognition out of our research labs, and towards extensive deployment in the real world, the
issue of the manifold nuisance factors that affect human motion has to be somehow addressed. Again, we
propose two parallel research lines to tackle this problem, one based on imprecise-probabilistic graphical
models, classified in a flexible, adaptive way by learning classifiers from the actual distribution of the data in
the training set; the other, founded on flexible spatio-temporal discriminative models.
Breakthrough #4: inherent variability and overfitting due to the limited size of training sets. In
a machine learning framework, the ability to classify (in our case, to recognize actions or activities) is
acquired from a training set of (labelled or unlabeled) examples. Unfortunately, collecting (and labelling) a
large training set is a long and costly procedure: fully processed training sets for which ground truth is
provided (think action labels and the corresponding bounding boxes in the video sequence) are a scarce
resource. On the other hand a massive amount of unlabeled data is theoretically available, but in practise
cannot be used for supervised learning of motions. We propose to cope with the overfitting issue by
introducing a more robust, flexible class of "imprecise-probabilistic" graphical models which allow for
uncertainty in the determination of the probabilities which characterize the model.
Breakthrough #5: moving from elementary actions to complex activities. Finally, in order to
learn and recognize more complex, sophisticated activities, we need to incorporate some sort of description
of the activity itself: in this project we propose to do so by both developing new spatio-temporal
discriminative models, and by moving to a more flexible class of "imprecise" stochastic generative models.
1.3.2 Overview of the S/T methodology and project articulation into Work Packages
To achieve such a goal, we articulate our project into three parallel lines of research, which amount to the
development of novel, ground breaking methodologies in three distinct scientific fields: 1 – structured
discriminative models, i.e., collections of discriminative models able to describe the spatio-temporal
structure of a motion; 2 – a fully fledged theory of imprecise-probabilistic graphical models, capable of
retaining the good features of traditional graphical models, while allowing more flexibility in the modelling
the inherent variability of actions, especially considering the limited size of any realistic training set; 3 – a
novel theory of classification for both precise and imprecise graphical models, able to cope with the large
number of nuisance factors and push towards action recognition “in the wild”.
The project is split into six interdependent work packages (WPs), one for each of the three methodological
developments, plus one dedicated to the necessary analysis of novel algorithms for the extraction of suitable
features from video/range sequences, one explicitly devoted to the integration and validation in various
scenarios of the algorithms developed in the three research pipelines, and one for project administration.
The goal of WP1 is the development of a theory of imprecise-probabilistic graphical models, in
particular learning and inference techniques which generalize classical methods for models based on sharp,
“precise” assessments. The first step is the identification of a reliable technique to learn imprecise models
from incomplete data, via the generalisation of the classical Expectation-Maximization algorithm to the
imprecise-probabilistic framework, and its specialisation to imprecise hidden Markov models (iHMMs).
Later on these tools will be extended to general imprecise-probabilistic graphical models. This concerns both
the case of directed models, for which the class of “credal networks” has already been formalised and
studied, and the case of Markov Random Fields (MRFs), for which an extension to imprecise probabilities
represents a truly groundbreaking research challenge. This will give more flexibility to the description of the
motion's dynamics (e.g., by relaxing the Markovian assumption to support more complex topologies).
The aim of WP2 is to develop a theory of classification for both imprecise-probabilistic and classical
generative models, able to tackle the issue with nuisance factors. To classify a new instance of an action we
need to compare the corresponding generative model with those already stored in a labelled training set. This
requires the development of dissimilarity measures for generative models, both precise and imprecise. In
particular we will investigate techniques for learning the most appropriate distance measure given a training
set. In the second part of WP2, these measures will be applied to classification proper, by exploring the use
of innovative frameworks such as structured learning and compressed sensing.
The aim of WP3 is to develop a novel class of discriminative models able to explicitly capture the
spatio-temporal structure of human motions. We can decompose this task into two parts. First, we need to
identify the “most discriminative” (in the sense that they contribute the most to discriminating the presence
of an action of a certain class) parts of an action/activity: we propose to use Multiple Instance Learning
(MIL) techniques to learn such action parts from a weakly labelled training set. The second task is to arrange
these parts in a coherent spatio-temporal structure. We will build on recent advanced in the field of 2D object
Proposal Part B: page [12] of [67]