Learning manifolds of dynamical models for activity recognition

are run on the KTH and CMU databases.
Some efforts have been recently put into recognition
from single images too (e.g. [30], where actions are learnt
from static images taken from the web).
Dynamical models in action recognition. However, encoding
the dynamics of videos or image sequences by
means of some sort of dynamical model can be useful in situations
in which the dynamics is critically discriminative.
Furthermore, the actions of interest have to be temporally
segmented from a video sequence: we need to know when
an action/activity starts or stops. Actions of sometimes very
different lengths have to be encoded in a homogeneous fashion
in order to be compared (“time warping”). Dynamical
representations are very effective in coping with time warping
or action segmentation [51].
Furthermore, in limit situations in which a significant number
of people move or ambulate in the field of view (as it
is common in surveillance scenarios), the attention has necessarily
to move from single objects/bodies to approaches
which consider the monitored crowd some sort of fluid, and
describe its behavior in a way similar to the physical modeling
of fluids []. Dynamical models are well equipped to
deal with such scenarios [].
In these scenarios, action (or identity) recognition reduces
to classifying dynamical models. Hidden Markov
models [23] have been indeed widely employed in action
recognition [43, 51] and gait identification [54, 9]. HMM
classification can happen either by evaluating the likelihood
of a new sequence with respect to the learnt models, or by
learning a new model for the test sequence, measuring its
distance from the old models, and attributing to it the label
of the closest model.
Indeed, many researchers have explored the idea of encoding
motions via linear [5], nonlinear [25], stochastic [42, 24]
or chaotic [1] dynamical systems, and classifying them by
measuring distances in their space. Chaudry et al [10], for
instance, have used nonlinear dynamical systems (NLDS)
to model times series of histograms of oriented optical flow,
measuring distances between NLDS by means of Cauchy
kernels, while Wang and Mori [56], have proposed sophisticated
max-margin conditional random fields to address locality
by recognizing actions as constellations of local motion
patterns.
Sophisticated graphical models can be useful to learn
in a bottom-up fashion the temporal structure or plot of
a footage, or to describe causal relationships in complex
activity patterns [38]. Gupta et al [28] work on determining
the plot of a video by discovering causal relationships
between actions, represented as an AND/OR graph whose
edges are associated with spatio-temporal constraints. Integer
Programming is used for storyline extraction on baseball
footage.
Distance-based recognition. The use of distances and mani-
Figure 1: Datasets.
fold learning for action recognition is not limited to dynamical
models. Lin et al [36] think of actions as sequences
of prototype trees, learned by hierarchical k-means in a
joint shape and motion space. Prototype-to-prototype distances
are generated as a look-up table. The joint likelihood
of location/prototype is maximized to track actors in
the Wiezmann and KTH datasets, while actions are recognized
by prototype sequence matching. In an interesting
related work, Li at el [35] describe activities as discriminative
temporal interaction matrices, living in a Discriminative
Temporal Interaction Manifold. They set probability densities
on this manifold, and use a MAP classifier to recognize
new activities. Their data is a collection of NCAA American
football footage.
Distance function learning. A number of distance functions
between linear systems have been introduced in the past
(e.g., [52]), and a vast literature about dissimilarity measures
for Markov models also exists [20], mostly concerning
variants of the Kullback-Leibler divergence [33]. However,
as models (or sequences) can be endowed with different
labels (e.g., action, ID) while maintaining the same geometrical
structure, no single distance function can possibly
outperform all the others in every classification problem.
A reasonable approach when possessing some a-priori information
is therefore trying to learn in a supervised fashion
the “best” distance function for a specific classification
problem [3, 4, 48, 55, 61, 22]. A natural optimization criterion
consists on maximizing the classification performance
achieved by the learnt metric, a problem which has elegant
solutions in the case of linear mappings [50, 57].
However, as even the simplest linear dynamical models live
in a nonlinear space, the need for a principled way of learning
Riemannian metrics from such data naturally arises.
Pullback metrics. An interesting tool is provided by the formalism
of “pullback metrics”. If the models belong to a
Riemannian manifold M, any diffeomorphism of M onto
itself or “automorphism” induces such a metric on M. By
designing a suitable family of automorphisms depending on
a parameter λ, we obtain a family of pullback metrics on M
we can optimize on.
Pullback metrics [31] have been recently proposed in the
context of document retrieval [34], where a proper Fisher
metric is available: instead of optimization classification
rates, the inverse volume of the pullback manifold is there
maximized. In [13] pullback Fisher metrics for simple
scalar autoregressive models of order 2 are learned. Besides
considering only a very limited class (AR2) of models, [13]
only deals with scalar observations, making the approach
impractical for action, activity or identity recognition. As
[34, 13] choose to optimize a geometric quantity totally unrelated
to classification, the obtained metrics deliver rather
modest classification performances. Furthermore, for important
classes of dynamical models used in action recogni-