Learning manifolds of dynamical models for activity recognition

Learning manifolds of dynamical models for activity recognition
Abstract
Dr Fabio Cuzzolin
Department of Computing, Oxford Brookes University
OX33 1HX Oxford, United Kingdom
In action, activity or identity recognition it is sometimes
useful, rather than to extract some spatio-temporal features
from the volumes representing motions, to explicitly encode
their dynamics by means of dynamical systems, such as for
instance hidden Markov models, nonlinear dynamical systems,
or hierarchical HMMs. Actions can then be classified
by measuring distances in the appropriate space of models.
However, using a fixed, arbitrary distance to classify
dynamical models does not necessarily produce good classification
results. The present proposal is concerned with a
general differential-geometric framework for learning Riemannian
metrics or distance functions for dynamical models,
given a training set which can be either labeled or unlabeled.
Given a training set of models, the optimal metric
or distance function is selected among a family of pullback
metrics induced by a parameterized automorphism of the
space of models. The exploitation potential of the proposed
methodology for action and activity recognition for human
machine interaction, video indexing and summarization, is
enormous.
1 Previous research track record
The Proposer: Dr Fabio Cuzzolin graduated in 1997
from the University of Padua (Universitas Studii Paduani,
founded 1222, it is the seventh most ancient university in
the world) with a laurea magna cum laude in Computer Engineering
and Master’s thesis on “Automatic gesture recognition”.
He received the Ph.D. degree from the same institution
in 2001, for a thesis entitled “Visions of a generalized
probability theory”. He was first Visiting Scholar at the ES-
SRL laboratory at the Washington University in St. Louis,
currently 12th in the US universities ranking. He was later
appointed fixed-term Assistant Professor with the Politecnico
di Milano, Milan, Italy (consistently recognized as the
best Italian university), then moved as a Postdoctoral Fellow
to the UCLA Vision Lab, University of California at
Los Angeles, led by Professor Stefano Soatto. He later received
a Marie Curie Fellowship in partnership with INRIA
Rhone-Alpes, Grenoble, France. Since September 2008, he
is Lecturer and an Early Career Fellow with the Department
of Computing, School of Technology, Oxford Brookes University,
Oxford, U.K.
In addition, Dr Cuzzolin recently classified second in the
2007 Senior Researcher national recruitment at INRIA, and
had interviews with/offer from Oxford University, EPFL,
http://cms.brookes.ac.uk/staff/FabioCuzzolin/
June 21, 2010
1
Universitat Pompeu Fabra, UCSD, GeorgiaTech, U. Houston,
Honeywell Labs, Riya.
Dr Cuzzolin’s research interests span both machine
learning applications of computer vision, including gesture
and action recognition and identity recognition from
gait, and uncertainty modeling via generalized and imprecise
probabilities, to which he has contributed by developing
a systematic analysis of the geometry of random sets
and other uncertainty measures. His scientific activity goes
therefore under the heading of interdisciplinarity. His scientific
productivity is extremely high, as the thirty papers
he has published in the last three years attest. Dr Cuzzolin
is author of 53 peer reviewed scientific publications, 44 of
them as first or single author, including two book chapters
and 8 journal papers. Several more journals are currently
under review or revision. His work has recently won a best
paper award at the recent Pacific Rim Conference on AI
symposium (PRICAI’08).
Dr Cuzzolin has been recently elected member of the Board
of Directors of the “Belief Functions and Applications Society”.
He has been Guest Editor for “Information Fusion”,
and collaborates with several other international journals in
both computer vision and probability, such as: the IEEE Tr.
on Systems, Man, and Cybernetics, the Int. J. on Approximate
Reasoning, Computer Vision and Image Understanding,
the IEEE Trans. on Fuzzy Systems, Information Sciences,
the Journal of Risk and Reliability, the International
Journal on Uncertainty, Fuzziness, and Knowledge-Based
Systems, Image and Vision Computing. He has served in
the program committee of some 15 international conferences
in both imprecise probabilities (e.g. ISIPTA, EC-
SQARU, BELIEF) and computer vision (e.g. VISAPP).
He is reviewer for BMVC and ECCV. He has co-supervised
two Ph.D. and two MsC. students and is involved in the supervision
of the Oxford Brookes vision group’s cohort of
students.
Proposer’s Related Work. The proposer has a significant
record of research in human motion analysis and recognition.
After some early work on gesture recognition, he
moved to study marker-less motion capture in 3D. Using
a system of 12 synchronized cameras available at the Politecnico
di Milano, silhouettes of the moving person in
all cameras were extracted to recover its volumetric extension.
The final purpose was to recognize actions and behaviors
from sequences of volumes using Markov models
[18]. He has recently explored the use of bilinear and multilinear
models to identity recognition from gait [11, 14],
a relatively new but promising branch of biometrics. As

in this context performance is influenced by factors as diverse
as viewpoint, emotional state, illumination, presence
of clothes/occlusions, etcetera, that can be modeled through
tensor algebra. He has recently published a book chapter
[14] with IGI on this topic. In the wider field of articulated
motion analysis he has published several papers on spectral
motion capture techniques [40], focusing in particular on
the crucial issue of how to select and map eigenspaces generated
by two different shapes in order to track 3D points on
their surfaces or consistently segment bodyparts along sequences
of voxelsets [17], as a preprocessing step to action
recognition. In direct relation to the topic of this proposal,
he is now exploring manifold learning techniques for dynamical
models representing (human) motions, in order to
learn the optimal metric of the space they live in and maximize
classification performance. Another book chapter [16]
collecting his preliminary results on the topic has been recently
accepted by Springer.
Proposer’s Other Scientific Contributions. Dr Cuzzolin
is also recognized as one of the most prominent experts
in the field of non-additive probabilities and belief
functions. He has been recently elected member of the
Board of Directors of the newly founded “Belief Functions
and Applications Society”, and is member of the “Society
for Imprecise Probabilities and Their Applications”.
His most important contribution in the field of uncertainty
theory and imprecise probabilities is a general geometric
approach to uncertainty measures, in which probabilities,
possibilities and belief functions can all be represented as
points of a Cartesian space and there analyzed [12]. Evidence
aggregation operators (the analogues of Bayes’ rule
in the Bayesian formalism) can also be seen as geometric
operators. The issues of how to approximate a belief function
with an additive probability or a possibility measure, or
what probability transformation is appropriate for decision
making can be all solved by geometric means.
In his recent award-winning paper [15], Dr Cuzzolin has
also investigated alternative combinatorial foundations for
the theory of belief functions, and their algebraic properties.
He is currently working on the generalization of the
total probability theorem to finite random sets, as a key contribution
to the field of non-additive probabilities. He is in
the process of finalizing a book entitled “The geometry of
uncertainty” which will collect all his contributions to the
mathematics of uncertainty.
Past Collaborations and Professional Links. Dr Cuzzolin
acquired considerable international experience by
working in the past for some of the most prominent research
laboratories in both the US and Europe. He gave seminar
and invited talks at several world-leading institutions such
as MIT, EPFL, GeorgiaTech, Microsoft Research Europe,
INRIA. His network of collaborations with groups of researchers
in both Europe and the United States is quite large
and expanding.
Dr Cuzzolin is currently arranging meetings with several
groups of researchers all around Europe to set up active
collaborations to support his goal of establishing a fairly
large group of five-ten people in a few years time in perspective
of reaching a professorial position in the medium
term. He is in talks with M. Zaffalon (IDSIA, Switzerland)
for a STREP on imprecise Markov chains for gesture recognition.
He is setting up with INRIA’s Radu Horaud, Alejan-
dro Frangi (Pompeu Fabra) and Technion (R. Kimmel, M.
Bronstein) an interdisciplinary Future Emerging Technology
(FET) EU proposal on large scale manifold learning,
with applications to scene understanding. He is discussing
a collaborative project on uncertainty theory at UK level
with J. Lawry (Head of Department of Bristol’s Engineering
Mathematics) and F. Coolen (Durham’s Dept of Statistics),
and exploring the opportunity of a European Network
of Excellence in the same field. He plans to apply for the
European Research Council Starting Grant in October 2010.
Dr Cuzzolin enjoys personal links with several world class
companies (many of them with research divisions in the
UK) such as Microsoft Research, Honeywell Labs (I. Cohen),
Boston’s MERL (M. Brand, S. Ramalingam), GE (G.
Doretto), Google (A. Bissacco), Riya.
The host organization: Oxford Brookes University,
School of Technology, (OBU). In the department there
are around 30 academic staff, these include, in computer
graphics, Prof. David Duce (co-chair of Eurographics
2003, 2006 conference), Bob Hopgood OBE, Prof. M.K.
Pidcock, world leader in Electrical Impedance Tomography,
and in AI and image processing, Prof. William
Clocksin. The Computer Science department had the following
break down in the recent RAE: 4* 15%, 3* 35%,
2* 35% and 1* 15%, which means that 85% of output was
deemed internationally leading and that no research output
was considered unclassified. The School of Technology
has recently established a new doctoral training programme
with the theme of intelligent transport systems (
http://tech.brookes.ac.uk/research/), which includes
many computer vision problems, with a set of
courses and associated infrastructure which will be directly
beneficial to this project.
Dr Fabio Cuzzolin belongs to the Oxford Brookes
vision group founded by Professor Philip Torr (
cms.brookes.ac.uk/research/visiongroup/),
which comprises some seventeen staff, students and
post-docs who will add value to this project. Professor Torr
was awarded the Marr Prize, the most prestigious prize
in computer vision, in 1998. Members of the group have
recently received awards in 4 other conferences, including
best paper at CVPR 08 and honourary mention at NIPS, the
top machine learning conference.
The group was mentioned four times in the UKCRC
Submission to the EPSRC International Review Sep.
20064. It enjoys ongoing collaborations with companies
such as 2d3, Vicon Life, Yotta, Microsoft Research Europe,
Sharp Laboratories Europe, Sony Entertainments Europe.
The group’s work with the Oxford Metrics Group in a
Knowledge Transfer Partnership 2005-9 won the National
Best Knowledge Transfer Partnership of the year at the
2009 awards, sponsored by the Technology Strategy Board,
selected out of several hundred projects.
Oxford Brookes also has close links with Oxford University,
with both the Active Vision Group and Prof.
Zissermans Visual Geometry group, including a joint EP-
SRC grant and EU collaboration as well as co-supervision.
Members of the all groups regularly attend each others
reading groups and seminars.
Dr Cuzzolin holds a Early Career Fellow position with
minimal undergraduate teaching duties and hence has sufficient
time to conduct the research listed in this proposal.

2 Proposed research and its context
2.1 Background
Topic of research. Recognizing human activities from
video is a natural application of computer vision. Since
the classic experiment of Johansson [] showing that moving
light displays were enough for people to recognize motions
or even identities, the matter has been subject of increasing
interest from the vision community. The problem
consists on telling, given one or more image sequences capturing
one or more people performing an activity, what category
(among those previously learned) this activity belongs
to. A useful even though quite fuzzy distinction is that between
“actions”, meant as simple (usually stationary) motion
patterns, and “activities” as more complex sequences
of actions, sometimes overlapping in time as they are performed
by different parts of the body or different agents in
the scene.
State of the art. The activity recognition problem involves,
as we have seen, numerous challenges at different
levels of representation. Accordingly, activity recognition
frameworks can be effectively described in terms of three
layers: 1. feature extraction from single images o videos;
2. action description or modeling; 3. high-level semantic
activity modeling.
Critical issues of activity recognition. Though the formulation
of the problem is simple and intuitive, activity
recognition is a much harder problem than it may look [?].
Motions inherently possess an extremely high degree of
variability. Movements quite different from each can in
fact carry the same meaning, or represent the same gesture.
Even in perfectly controlled situations (constant illumination,
static background) this inherent variability makes
recognition hard (e.g., the trajectory followed by a walking
person is irrelevant to the classification of the action they
perform).
In addition, motions are subject to a large number of nuisance
or “covariate” factors [37], such as: illumination,
background, viewpoint. The list goes on and on. The
combination of inherent variability and nuisance factors
explains why experiments have been so far conducted in
small, controlled environments to reduce their complexity.
Attempts have been recently done to go beyond such controlled
environments. Liu, Luo and Shah [37] has taken
on the challenge of recognizing actions “in the wild” and
coping with the tremendous nuisance variations of unconstrained
videos. Motion statistics are used to prune motion
and static features. Adaboost is chosen to integrate all
those single-feature classifiers. The algorithm is tested on
the KTH dataset and on YouTube videos. Hu et al [59]
have also investigated action detection in complex, cluttered
scenes, representing candidates regions are bags of instances.
In their SMILE-SVM framework human action detectors
are learnt based on imprecise action locations. They
test their approach on both the CMU dataset and videos shot
in a shopping mall. Yao and Zhu [58] learn deformable action
templates from cluttered real-world videos. Such action
templates are sequences of image templates consisting
of a set of shape and motion primitives. Templates are
deformed to match KTH and CMU videos by space-time
warping.
If we remove the assumption that a single motion of interest
is present in our video, locality emerges too as a critical factor.
For instance, a person can walk and wave at the same
time. Different actions performed by different agents can go
on in different regions of the image sequence, without being
necessarily synchronized or coordinated. Gilbert et al [27],
for instance, perform multi-action recognition using very
dense spatio-temporal corner features in a hierarchical classification
framework, testing it on the multi-KTH dataset
and the Real-World Movie dataset. Reddy et al [45] propose
to cope with incremental recognition using feature trees that
can handle multiple actions without intensive training. Local
features are recognized by NN, while actions are later
labeled using a simple voting strategy. They use the KTH
and IXMAS datasets.
If we also remove the assumption of having a single body
moving in the field of interest, problems like occlusion assume
a critical importance []. On the other hand, the presence
of one of more (static) objects in the vicinity of the motion
can effectively help to disambiguate the activity class
(recognition “in context”). Marszalek, Laptev and Schmid
[39] apply a joint SVM classifier of action and scene to bags
of features, using movie scripts as means of automatic supervision
for training (as proposed in [21]) on the newly
proposed “Hollywood movies” database. Han et al [29],
for instance, introduce bag-of-detectors scene descriptors
to encode such contextual presence, and use Gaussian processes
to select and weight multiple features for classification
on the KTH and Hollywood dataset. Sun et al [53] use
the Hollywood and LSCOM databases to model and test
spatio-temporal context in a hierarchical way, using SIFT
for point-level context and Markov processes to model trajectory
proximity and transition descriptors, on top of which
they apply multi-channel non-linear SVM.
In opposition, sometimes very few instances of each action
category are available. Seo and Milanfar [49] detect actions
from single examples by computing sense space-time local
regression kernels, which are used as descriptors and passed
through PCA to extract salient features. A matrix generalization
of cosine similarity is then used to compare videos,
on the Irani database.
Bag-of-features on spatio-temporal volumes methods.
Recently, recognition methods which neglect action dynamics
(typically extracting spatio-temporal [6] features
from the 3D volume associated with a video [32]) have
proven very effective (http://www.wisdom.weizmann.ac.il/ vision/SpaceTimeActions.html).
Kim and Cipolla [32] do
spatio-temporal pattern matching, representing volumes as
tensors. They propose an extension of canonical correlation
analysis to tensor to detect actions on a 3D window search
with exemplars on the KTH dataset. Bregonzio et al [7] also
use a global spatio-temporal distribution of interest points.
They extract and select “holistic” features from clouds of
interest points over multiple temporal scales. Rapantzikos
et al [44] also dense spatio-temporal features detected using
saliency measures, which incorporate color, motion and
intensity, in a multi-scale volumetric representation. Yuan
at el [60] describe actions as collections of spatio-temporal
invariant features, and propose a naive Bayes mutual information
maximization method for multi-class. A search
algorithm is also proposed to locate the optimal 3D subvolume
in which the action can be detected. Experiments

are run on the KTH and CMU databases.
Some efforts have been recently put into recognition
from single images too (e.g. [30], where actions are learnt
from static images taken from the web).
Dynamical models in action recognition. However, encoding
the dynamics of videos or image sequences by
means of some sort of dynamical model can be useful in situations
in which the dynamics is critically discriminative.
Furthermore, the actions of interest have to be temporally
segmented from a video sequence: we need to know when
an action/activity starts or stops. Actions of sometimes very
different lengths have to be encoded in a homogeneous fashion
in order to be compared (“time warping”). Dynamical
representations are very effective in coping with time warping
or action segmentation [51].
Furthermore, in limit situations in which a significant number
of people move or ambulate in the field of view (as it
is common in surveillance scenarios), the attention has necessarily
to move from single objects/bodies to approaches
which consider the monitored crowd some sort of fluid, and
describe its behavior in a way similar to the physical modeling
of fluids []. Dynamical models are well equipped to
deal with such scenarios [].
In these scenarios, action (or identity) recognition reduces
to classifying dynamical models. Hidden Markov
models [23] have been indeed widely employed in action
recognition [43, 51] and gait identification [54, 9]. HMM
classification can happen either by evaluating the likelihood
of a new sequence with respect to the learnt models, or by
learning a new model for the test sequence, measuring its
distance from the old models, and attributing to it the label
of the closest model.
Indeed, many researchers have explored the idea of encoding
motions via linear [5], nonlinear [25], stochastic [42, 24]
or chaotic [1] dynamical systems, and classifying them by
measuring distances in their space. Chaudry et al [10], for
instance, have used nonlinear dynamical systems (NLDS)
to model times series of histograms of oriented optical flow,
measuring distances between NLDS by means of Cauchy
kernels, while Wang and Mori [56], have proposed sophisticated
max-margin conditional random fields to address locality
by recognizing actions as constellations of local motion
patterns.
Sophisticated graphical models can be useful to learn
in a bottom-up fashion the temporal structure or plot of
a footage, or to describe causal relationships in complex
activity patterns [38]. Gupta et al [28] work on determining
the plot of a video by discovering causal relationships
between actions, represented as an AND/OR graph whose
edges are associated with spatio-temporal constraints. Integer
Programming is used for storyline extraction on baseball
footage.
Distance-based recognition. The use of distances and mani-
Figure 1: Datasets.
fold learning for action recognition is not limited to dynamical
models. Lin et al [36] think of actions as sequences
of prototype trees, learned by hierarchical k-means in a
joint shape and motion space. Prototype-to-prototype distances
are generated as a look-up table. The joint likelihood
of location/prototype is maximized to track actors in
the Wiezmann and KTH datasets, while actions are recognized
by prototype sequence matching. In an interesting
related work, Li at el [35] describe activities as discriminative
temporal interaction matrices, living in a Discriminative
Temporal Interaction Manifold. They set probability densities
on this manifold, and use a MAP classifier to recognize
new activities. Their data is a collection of NCAA American
football footage.
Distance function learning. A number of distance functions
between linear systems have been introduced in the past
(e.g., [52]), and a vast literature about dissimilarity measures
for Markov models also exists [20], mostly concerning
variants of the Kullback-Leibler divergence [33]. However,
as models (or sequences) can be endowed with different
labels (e.g., action, ID) while maintaining the same geometrical
structure, no single distance function can possibly
outperform all the others in every classification problem.
A reasonable approach when possessing some a-priori information
is therefore trying to learn in a supervised fashion
the “best” distance function for a specific classification
problem [3, 4, 48, 55, 61, 22]. A natural optimization criterion
consists on maximizing the classification performance
achieved by the learnt metric, a problem which has elegant
solutions in the case of linear mappings [50, 57].
However, as even the simplest linear dynamical models live
in a nonlinear space, the need for a principled way of learning
Riemannian metrics from such data naturally arises.
Pullback metrics. An interesting tool is provided by the formalism
of “pullback metrics”. If the models belong to a
Riemannian manifold M, any diffeomorphism of M onto
itself or “automorphism” induces such a metric on M. By
designing a suitable family of automorphisms depending on
a parameter λ, we obtain a family of pullback metrics on M
we can optimize on.
Pullback metrics [31] have been recently proposed in the
context of document retrieval [34], where a proper Fisher
metric is available: instead of optimization classification
rates, the inverse volume of the pullback manifold is there
maximized. In [13] pullback Fisher metrics for simple
scalar autoregressive models of order 2 are learned. Besides
considering only a very limited class (AR2) of models, [13]
only deals with scalar observations, making the approach
impractical for action, activity or identity recognition. As
[34, 13] choose to optimize a geometric quantity totally unrelated
to classification, the obtained metrics deliver rather
modest classification performances. Furthermore, for important
classes of dynamical models used in action recogni-

tion such as HMMs or variable length Markov models [26]
(VLMMs) a proper metric has not yet been identified. In
order to learn optimal pullback distances for such important
classes of models we necessarily need to relax the constraint
of having a proper manifold structure, extending the
pullback learning technique to mere distance functions or
divergences.
Industrial and societal context. The growing market
for action and gesture recognition applications, activity
recognition or human-computer interface is just
too big to be described extensively here. It is perhaps
worth citing the case of motion-based video
games interfaces. Microsoft has recently launched
its Project Natal, which with its controller-free gaming
experience is probably destined to revolutionize the
whole field of interactive video games and consoles
(see http://www.xbox.com/en-us/live/projectnatal/ for some
amazing demos). The Oxford Brookes vision group enjoys
continuing strong links with Microsoft through its founder
Professor Torr, and has recently acquired a range camera
(http://en.wikipedia.org/wiki/Range imaging) in order
to kickstart cutting-edge research in motion analysis.
FIX Historically, the first intended application of activity
recognition was human machine interaction. Gesturing can
be seen a much more natural way of interacting with a
computer endowed with a simple webcam, and people in
the 1990’s started envisaging the replacement of mouse
and keyboard as main interfaces between computers and
their users. This later lead to research efforts focused on
near-future scenarios in which numerous devices possessing
some degree of intelligence would interact with people
in so called ”smart rooms”.
Nowadays, as videos have become part of everyday life,
methods to store or index or summarize video footage are
of increasing commercial interest. Content-based video retrieval
from repositories such as youtube or ... has to rely
on extraction and labeling of significant motion patterns in
the video.
Another application field of growing importance is security
and surveillance, where motion classification techniques
can be employed to either detect anomalous behavior in
surveillance video (in order to require the attention of a
human supervisor), or to recognize people’s identity from
their walking gait in uncooperative scenarios, in one of the
most promising approaches to behavioral biometrics. Most
such companies focus at the moment on cooperative biometrics
such as face or iris recognition: investing in behavioral,
non-cooperative biometrics before the rest of the
market could provide them with a significant competitive
advantage. “In both the identity management and security
arenas, the use of biometric technology is increasing apace
... the world biometrics market has expanded exponentially.
Annual growth is forecast at 33% between the years 2000
and 2010. Europe is expected to have the fastest growing
biometrics market by 2010 ... The Intellect Association for
Biometrics (IAfB) is the UK body that represents companies
developing these technologies ... has fostered close ties
with the UK Border Agency and Home Office.” (Biometrics,
November 2008).
2.2 Research hypotheses and objectives
Research idea.
The goal of the present proposal is to present and test a
general differential-geometric framework for learning Riemannian
metrics or distance functions for dynamical models,
given a training set which can be either labeled or unlabeled.
Given a training set of models, the optimal metric
or distance function is selected among a family of pullback
metrics induced by a parameterized automorphism of the
space of models. Such function is arguably the most appropriate
for the collection of motions at hand, and can subsequently
be used to classify new movements.
The available information (in the form of a training set of
recorded actions/activities) is used to learn the “best” way
to recognize new actions/activities. The proposed approach
can be straightforwardly applied to action/identity recognition,
identity recognition from gait, and video content summarization.
Novelty and contributions.
- contribution to distance learning in nonlinear spaces
- classification of complex, structured objects -¿ direct
competition with structured learning
Timeliness.
- commercial applications of computer vision are spreading
rapidly - action recognition one of hottest topic right
now
Goals of the project.
Milestones.
2.3 Programme and methodology
2.3.1 Methodology
Preliminary results on pullback metric learning. The
proposer [13] has recently investigated the use of pullback
metrics for simple scalar autoregressive models of order 2,
and their use for identity recognition from gait. Besides
considering only a very limited class (AR2) of models, [13]
only deals with scalar observations, making the approach
impractical for real-world action or activity recognition. An
first extension to multi-dimensional AR models have been
proposed by Dr Cuzzolin [16]. More to the point, in previous
work [34, 13] a geometric quantity totally unrelated to
classification was optimized, leaving the obtained metrics
with rather modest classification performances. A framework
in which classification rates are directly optimized by
the obtained distance function is sorely needed. Furthermore,
for important classes of dynamical models used in action
recognition such as HMMs or variable length Markov
models [26] (VLMMs) a proper metric has not yet been
identified. In order to learn optimal pullback distances for
such important classes of models we necessarily need to
relax the constraint of having a proper manifold structure,
extending the pullback learning technique to mere distance
functions or divergences. This is all the more critical for
the applicability of distance learning to the kind of sophisticated
graphical or dynamical models necessary in activity
recognition.
Pullback formalism. Let us suppose a data-set D =
{m1, ..., mN} of dynamical models is available. Suppose
also that such models live on a Riemannian manifold M of
some sort, i.e, a Riemannian metric is defined in any point

of the manifold. Consider then an automorphism (invertible
differentiable map) between M and itself: F : M → M,
m ↦→ F (m), m ∈ M. Let us denote by TmM the tangent
space to M in m. Each tangent vector v ∈ TmM maps any
function f on M to the derivative of f along the direction
v: v(f) = ∂f/∂v.
Any automorphism F is associated with a push-forward
map of tangent vectors: F∗ : TmM → T F (m)M, v ∈
TmM ↦→ F∗v ∈ T F (m)M defined as F∗v(f) = v(f ◦ F )
for all smooth functions f on M, which maps f to its partial
derivative ∂f/∂v in F (m).
Consider now a Riemannian metric 1 g : T M × T M → R
on M. The automorphism F induces a pullback metric
on M: g∗m(u, v) . = g F (m)(F∗u, F∗v) such that the scalar
product of two tangent vectors u, v in m ∈ M according to
the pullback metric g∗ is the scalar product with respect to
the original metric g of the push-forward vectors F∗u, F∗v
in F (m). The pullback geodesic (shortest path) between
two points is the lifting of the geodesic connecting their images
with respect to the original metric. A pullback distance
between two points on M (in our case, two dynamical models)
can be computed along such pullback geodesic.
By defining a class of such automorphisms {Fλ, λ ∈ Λ}
depending on some parameter λ, we get a corresponding
family of pullback metrics {g∗λ, λ ∈ Λ} on M. We can
then define an optimization problem over such family in order
to select an “optimal” metric. The nature of the resulting
manifold will obviously depend on the objective function
we choose to optimize.
Figure 2: The push-forward map associated with an automorphism
on a Riemannian manifold M.
Spaces of dynamical models. To apply the pullback
metric framework to dynamical models we first need to define
a structure of Riemannian manifold on them. Even
though a Fisher Riemannian metric has been computed for
several manifolds of linear MIMO systems [?], and work on
pullbacks of Fisher information metrics has been recently
conducted [31], for important classes of dynamical models
(such as hidden Markov models or variable length Markov
models) no manifold structure is analytically known. Standard
methods for measuring distances between HMMs, for
instance, rely on the Kullback-Leibler divergence [33] (even
though several other distance functions have been proposed
[20]).
Learning pullback distances for dynamical models.
In this proposal the general framework for learning an optimal
pullback metric/distance from a training set of dynamical
models, outlined in Figure 3, is proposed.
1. given a data-set Y of observation sequences {yi =
[yi(t), t = 1, ..., Li], i = 1, ..., N} of variable length Li,
a dynamical model mi of a certain class C can be estimated
by parameter identification, yielding a set of models
1 Informally speaking, g determines how to compute scalar products of
tangent vectors v ∈ TmM.
D = {m1, ..., mN};
2. such models of class C belong to a certain domain
MC: to measure distances between pairs of models on MC
we need either a distance function dM or a proper Riemannian
metric gM;
3. a family {Fλ, λ ∈ Λ} of automorphisms from MC
onto itself (parameterized by a vector λ) is then designed to
provide a search space of metrics/distances (the variable in
our optimization scheme) from which to select the optimal
one;
4. Fλ induces a family of pullback metrics {g∗λ, λ} or
distances {d∗λ, λ} on M, respectively;
5. optimizing over this family of pullback distances/metrics
(according to some sensible objective function)
yields an optimal pullback metric 2 ˆg∗ or distance function
ˆ d∗. The learnt optimal distance function can finally be
used to cluster or classify new “test” models/sequences.
Objective function. When the data-set of models is labeled,
we can determine the optimal metric/distance function
by maximizing the classification performance of the
metric. As the classification score is hard to describe analytically,
in preliminary work [13, ?] we extracted a number
of samples from the parameter space and pick the maximal
performance sample.
Image feature representation. ADAPT Historically, silhouettes
have been often (but by no means always [41])
used to encode the shape of the walking person along the sequence,
but are widely criticized for their sensitivity to noise
and the fact that they require solving the (inherently ill defined)
background subtraction problem. In the perspective
of a real-world deployment of behavioral biometrics it is
essential to move beyond silhouette-based representations,
as a crucial step to improve the robustness of the recognition
process. An interesting feature descriptor, for instance,
called “action snippets” [47] is based on motion and shape
extraction within rectangular bounding boxes which, contrarily
to silhouettes, can be reliably obtained in most scenarios
by using person detectors [?] or trackers [19]. Our final
goal is to adopt a discriminative feature selection stage,
such as the one proposed in [46], where discriminative features
are selected from an initial bag of HOG-based descriptors.
In this sense the expertise of the Oxford Brookes vision
group in this area will be extremely valuable to the final
success of the project.
Crucial issues and further developments.
In perspective, the proposed methodology can be extended
to cope with more complex classes of non-linear dynamical
models [], allowing classification of more complex
activities rather than simple stationary actions.
Similarly, other important tasks in vision such as face
and object recognition, so long as they involve the classification
of objects living on a manifold endowed with a metric
or a distance function, can be treated in the same way.
2.3.2 Programme of work and milestones
2.4 Relevance to academic beneficiaries
Impact on activity recognition.
2 In the Riemannian case the geodesic path between any two models
has to be known to compute the associated geodesic distance: knowing the
geodesics of M we can calculate distances on M based on ˆg∗.

Figure 3: A bird’s eye view of our proposed framework for learning pullback metrics for dynamical models.
Impact on other classification problems. In addition,
the developed techniques can be applied in a rather straightforward
way to any classification problem involving complex
objects living in a structured, metric space. They can
therefore find natural applications in fields such as face
recognition, ...
Impact on manifold learning.
References
[1] S. Ali, A. Basharat, and M. Shah, Chaotic invariants for human
action recognition, ICCV’07.
[2] S.-I. Amari, Differential geometric methods in statistics,
Springer-Verlag, 1985.
[3] A. Bar-Hillel, T. Hertz, N. Shental, and D. Weinshall, Learning
distance functions using equivalence relations, ICML03,
2003, pp. 11–18.
[4] M. Bilenko, S. Basu, and R. J. Mooney, Integrating constraints
and metric learning in semi-supervised clustering,
Proc. of ICML’04, 2004.
[5] A. Bissacco, A. Chiuso, and S. Soatto, Classification and
recognition of dynamical models: The role of phase, independent
components, kernels and optimal transport, IEEE
Trans. PAMI 29 (2007), no. 11, 1958–1972.
[6] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri,
Actions as space-time shapes, IEEE Conference on Computer
Vision (2005), 1395–1402.
[7] M. Bregonzio, S. Gong, and T. Xiang, Recognising action
as clouds of space-time interest points, CVPR’09, pp. 1948–
1955.
[8] P. Burman, A comparative study of ordinary crossvalidation,
v-fold cross-validation and the repeated
learning-testing methods, Biometrika 76(3) (1989), 503–
514.
[9] N. L. Carter, D. Young, and J. M. Ferryman, Supplementing
Markov chains with additional features for behavioural
analysis, 2006, pp. 65–65.
[10] R. Chaudhry, A. Ravichandran, G. Hager, and R. Vidal, Histograms
of oriented optical flow and Binet-Cauchy kernels
on nonlinear dynamical systems for the recognition of human
actions, CVPR’09, pp. 1932–1939.
[11] F. Cuzzolin, Using bilinear models for view-invariant action
and identity recognition, CVPR’06, vol. 1, pp. 1701–1708.
[12] F. Cuzzolin, A geometric approach to the theory of evidence,
IEEE Transactions on Systems, Man, and Cybernetics - Part
C 38 (2008), no. 4, 522–534.
[13] F. Cuzzolin, Learning pullback metrics for linear models,
Workshop on Machine Learning for Vision-based Motion
Analysis MLVMA, 2008.
[14] F. Cuzzolin, Multilinear modeling for robust identity recognition
from gait, Behavioral Biometrics for Human Identification:
Intelligent Applications (Liang Wang and Xin Geng,
eds.), IGI Publishing, 2009.
[15] F. Cuzzolin, Three alternative combinatorial formulations of
the theory of evidence, Intelligent Decision Analysis (2010).

[16] F. Cuzzolin, Manifold learning for multi-dimensional autoregressive
dynamical models, Machine Learning for Visionbased
Motion Analysis (L. Wang, G. Zhao, L. Cheng, and
M. Pietikine, eds.), Springer, 2010.
[17] F. Cuzzolin, D. Mateus, D. Knossow, E. Boyer, and
R. Horaud, Coherent laplacian protrusion segmentation,
CVPR’08.
[18] F. Cuzzolin, A. Sarti, and S. Tubaro, Action modeling with
volumetric data, ICIP’04, vol. 2, pp. 881– 884.
[19] N. Dalai and B. Triggs, Histograms of oriented gradients for
human detection, CVPR’06, pp. 886– 893.
[20] M. N. Do, Fast approximation of Kullback-Leibler distance
for dependence trees and hidden Markov models, IEEE Signal
Processing Letters 10 (2003), no. 4, 115 – 118.
[21] O. Duchenne, I. Laptev, J. Sivic, F. Bach, and J. Ponce, Automatic
annotation of human actions in video, ICCV’09.
[22] C. F. Eick, A. Rouhana, A. Bagherjeiran, and R. Vilalta, Using
clustering to learn distance functions for supervised similarity
assessment, ICML and Data Mining, 2005.
[23] R. Elliot, L. Aggoun, and J. Moore, Hidden markov models:
estimation and control, Springer Verlag, 1995.
[24] J. M. Wang et. al., Gaussian process dynamical model,
NIPS’05.
[25] L. Ralaivola et. al., Dynamical modeling with kernels for
nonlinear time series prediction, NIPS’04.
[26] A. Galata, N. Johnson, and D. Hogg, Learning variablelength
Markov models of behavior, CVIU 81 (2001), no. 3,
398–413.
[27] A. Gilbert, J. Illingworth, and R. Bowden, Scale invariant
action recognition using compound features mined from
dense spatio-temporal corners, 2008, pp. I: 222–233.
[28] A. Gupta, P. Srinivasan, J. Shi, and L. S. Davis, Understanding
videos, constructing plots learning a visually
grounded storyline model from annotated videos., CVPR’09,
pp. 2012–2019.
[29] D. Han, L. Bo, and C. Sminchisescu, Selection and context
for action recognition, ICCV’09.
[30] N. Ikizler-Cinbis, R. G. Cinbis, and S. Sclaroff, Learning
actions from the web, ICCV’09.
[31] M. Itoh and Y. Shishido, Fisher information metric and Poisson
kernels, Differential Geometry and its Applications 26
(2008), no. 4, 347 – 356.
[32] T. K. Kim and R. Cipolla, Canonical correlation analysis of
video volume tensors for action categorization and detection,
31 (2009), no. 8, 1415–1428.
[33] S. Kullback and R. A. Leibler, On information and sufficiency,
Annals of Math. Stat. 22 (1951), 79–86.
[34] G. Lebanon, Metric learning for text documents, IEEE Tr.
PAMI 28 (2006), no. 4, 497–508.
[35] R. N. Li, R. Chellappa, and S. H. K. Zhou, Learning multimodal
densities on discriminative temporal interaction manifold
for group activity recognition.
[36] Z. Lin, Z. Jiang, and L. S. Davis, Recognizing actions by
shape-motion prototype trees, ICCV’09, pp. 444–451.
[37] J. G. Liu, J. B. Luo, and M. Shah, Recognizing realistic actions
from videos ’in the wild’, CVPR’09, pp. 1996–2003.
[38] C. C. Loy, T. Xiang, and S. Gong, Modelling activity
global temporal dependencies using time delayed probabilistic
graphical model, ICCV’09.
[39] M. Marszalek, I. Laptev, and C. Schmid, Actions in context,
CVPR’09.
[40] D. Mateus, R. Horaud, D. Knossow, F. Cuzzolin, and E.
Boyer, Articulated shape matching using Laplacian eigenfunctions
and unsupervised point registration, CVPR’08.
[41] C. Nandini and C. N. Ravi Kumar, Comprehensive framework
to gait recognition, Int. J. Biometrics 1 (2008), no. 1,
129–137.
[42] B. North, A. Blake, M. Isard, and J. Rittscher, Learning and
classification of complex dynamics, PAMI 22 (2000), no. 9.
[43] M. Piccardi and O. Perez, Hidden Markov models with kernel
density estimation of emission probabilities and their use in
activity recognition, VS’07, pp. 1–8.
[44] K. Rapantzikos, Y. Avrithis, and S. Kollias, Dense saliencybased
spatiotemporal feature points for action recognition,
CVPR’09, pp. 1454–1461.
[45] K. K. Reddy, J. Liu, and M. Shah, Incremental action recognition
using feature-tree, ICCV’09.
[46] G. Rogez, J. Rihan, S. Ramalingam, C. Orrite, and P. H. S.
Torr, Randomized trees for human pose detection, CVPR’08.
[47] K. Schindler and L. van Gool, Action snippets: How many
frames does human action recognition require?, CVPR’08.
[48] M. Schultz and T. Joachims, Learning a distance metric from
relative comparisons, NIPS’04.
[49] H. J. Seo and P. Milanfar, Detection of human actions from a
single example, ICCV’09.
[50] N. Shental, T. Hertz, D. Weinshall, and M. Pavel, Adjustment
learning and relevant component analysis, ECCV’02.
[51] Q. F. Shi, L. Wang, L. Cheng, and A. Smola, Discriminative
human action segmentation and recognition using semi-
Markov model, CVPR’08.
[52] A. J. Smola and S. V. N. Vishwanathan, Hilbert space embeddings
in dynamical systems, IFAC’03, pp. 760 – 767.
[53] J. Sun, X. Wu, S. C. Yan, L. F. Cheong, T. S. Chua, and J. T.
Li, Hierarchical spatio-temporal context modeling for action
recognition, CVPR’09, pp. 2004–2011.
[54] A. Sundaresan, A. K. Roy Chowdhury, and R. Chellappa,
A hidden Markov model based framework for recognition of
humans from gait sequences, ICIP’03, pp. II: 93–96.
[55] I. W. Tsang, J. T. Kwok, C. W. Bay, and H. Kong, Distance
metric learning with kernels, ICAI’03.
[56] Y. Wang and G. Mori, Max-margin hidden conditional random
fields for human action recognition, CVPR’09, pp. 872–
879.
[57] E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russel, Distance
metric learning with applications to clustering with side information,
NIPS’03.
[58] B. Yao and S.C Zhu, Learning deformable action templates
from cluttered videos, ICCV’09.
[59] Y. Hu, L. Cao, F. Lv, S. Yan, Y. Gong, and T. S. Huang,
Action detection in complex scenes with spatial and temporal
ambiguities, ICCV’09.
[60] J. S. Yuan, Z. C. Liu, and Y. Wu, Discriminative subvolume
search for efficient action detection, CVPR’09, pp. 2442–
2449.
[61] Z. Zhang, Learning metrics via discriminant kernels and
multidimensional scaling: Toward expected euclidean representation,
ICML’03.

A Justification of resources B Diagrammatic work plan

C Academic beneficiaries (4000 characters
max)
Please summarise how your proposed research will contribute
to knowledge, both within the UK and globally.
Please look broadly beyond narrow research fields. We
recognise that in generating new knowledge, a crossdisciplinary
or single-disciplinary approach may be the
most appropriate. Applicants are asked to clearly state their
chosen approach and provide justification for that choice.
List of beneficiaries
How the research will benefit other researchers in the
field
Whether there are any academic beneficiaries in other
disciplines and, if so, how they will benefit and what will be
done to ensure that they benefit
What other researchers, both within the UK and elsewhere
are likely to be interested in or to benefit from the
proposed research.
Relevance of the research to beneficiaries
Identify the potential academic impact of the proposed
work
Show how the research will benefit other researchers
(this might include methodological or theoretical advances
Identify whether the research will produce data or materials
of benefit to other researchers, and explain how these
will be stored, maintained and made available
D Impact Summary (4000 characters
max)
The Impact Summary (4000 characters max) should address
the three questions:
Who will benefit from this research?
How will they benefit from this research?
What will be done to ensure that they have the opportunity
to benefit from this research?
Who will benefit from this research
List any beneficiaries from the research, for example
those who are likely to be interested in or to benefit from
the proposed research both directly or indirectly. It may
be useful to think of beneficiaries as users of the research
outputs, both immediately, and in the longer term.
within the commercial private sector?
policy-makers, government and government agencies
within the public sector, third sector or any others? (museums,
galleries and charities)
within the wider public?
How will they benefit from this research
Describe the relevance of the research to these beneficiaries,
identifying the potential for impacts arising from the
proposed work. Please consider the following when framing
your response:
Explain how the research has the potential to impact on
the nations health, wealth or culture.
security, surveillance
What will these impacts be, and what is their importance?
What are the realistic timescales for the benefits to be
realised?
What research and professional skills will staff working
on the project develop which they could apply in all employment
sectors?
Actions taken to ensure this
Please detail how the proposed research project will be
managed to engage users and beneficiaries and increase the
likelihood of impacts
Communication and engagement plans
Collaboration arrangements
Plans for exploitation
Relevant experience and track record
Oxford Brookes Computer Vision Group has very well
established channels for making technology transfer from
research to product and a track record of achieving this. The
group has now well trodden paths for exploiting IP with a
track record of company interactions including Sony, Vicon,
2d3, HMGCC and Sharp. Prof. Torrs links with Yotta,
Microsoft, Sony in particular, and throughout the industry
in general will provide a fertile ground for exploitation.

E Impact plan
Please detail how the proposed research project will be
managed to engage users and beneficiaries and increase the
likelihood of impacts
Methods for communications and engagement
Collaboration and exploitation in the most effective and
appropriate manner
Track record in this area and the costs of these activities
E.1 Communications and engagement
Describe engagement with the identified beneficiaries, for
example:
How have beneficiaries been engaged to date, and how will
they be engaged moving forward?
How will the work build on existing or create new links?
Outline plans to work with intermediary organisations or
networks.
What activities will be undertaken to ensure good engagement
and communication?
Secondments of research or user community staff
Events aimed at a target audience
Workshops to provide training or information dissemination
Publications and publicity materials summarising main
outcomes in a way that beneficiaries will be able to understand
and use
Websites and interactive media
Media relations
Public affairs activities
E.2 Exploitation and application
Identify the mechanisms in place for potential exploitation,
both commercially and non-commercially
Do you have any specific partnership, collaborative or
exploitation agreements in place?
Oxford Brookes Computer Vision department already
has an established record for exploiting IP, and interactions
with company, including Sony, Vicon, 2d3, HMGCC and
Sharp.
How will the outputs with potential impact be identified?
What structure and mechanisms can you put in place to
exploit and protect the outputs from the research, during
and at the end of the grant lifecycle?
Intellectual Property Rights management and exploitation
will be managed by the Research and Business Development
Office (RBDO) at Oxford Brookes University,
which has access to financial and other resources to enable
Intellectual Property and its commercial exploitation to
be effectively managed, whilst maximizing the widespread
dissemination of the research results. This includes, as appropriate,
finance for patenting and proof of concept funding;
IP, technology and market assessment; resources for
defining and implementing a commercialization strategy
though licensing, start-up company or other routes.
E.3 Capability
Who is likely to be undertaking the impact activities? For
example: The Principal Investigator or Co-Investigator
PhD students and post-doctoral researchers who may be involved
in activities in addition to research; Specialised staff
employed to undertake communication and exploitation activities;
Technical experts to write publications, web pages
and user-friendly interfaces.
What previous and relevant experience do they have
in achieving successful knowledge exchange and impact?
How will they acquire the skills?
E.4 Resource for the activity
If there are any resource implications as a result of implementing
the knowledge exchange and/or impact activities,
please ensure these are documented in the financial summary
and also in the Justification of Resources section of
the proposal.
Research results will be fully disseminated and published
via international journals and conferences, and the
Oxford Brookes website. Currently the Oxford Brookes
vision group the proposer belongs to maintains a full programme
of publication at all the major conferences in addition
in the related fields of graphics and machine learning.
All published papers and the generated code will be made
available on a website that will be established specifically
for the project.
Many companies are active in the commercialization of
vision-based products, in particular in the fields of automatic
surveillance, gait biometrics, human computer interaction,
image-based web retrieval. I expect to exploit my
personal links in this sense with researchers in several world
class companies (many of them with research divisions in
Europe) like Microsoft Research, Honeywell Labs (I. Cohen),
Boston’s Mitsubishi Electric Research Lab (M. Brand,
S. Ramalingam), GE (G. Doretto), Google (A. Bissacco).