D2.1 Requirements and Specification - CORBYS

D2.1 Requirements and Specification
impact assessment or threat refinement (level 3), and lastly process refinement (level 4). This model,
however, does not incorporate human-in-the-loop, and another level called user refinement (level 5) has been
proposed to “delineate the human from the machine in the process refinement” allowing for the human to play
an important role in the fusion process. (Blasch and Plano, 2002)
• Level 0: Sub-object assessment
• Level 1: Object assessment
• Level 2: Situation assessment
• Level 3: Impact assessment
• Level 4: Process refinement
• Level 5: User refinement
Level 0 and 1 deal with sub-object and object assessment respectively, making use of information from
multiple sources to arrive at a representation of objects of interest in the environment. In level 2, a
relationship between the identified objects is established. At the end of this level, once the situation
assessment process is complete, the system has achieved situation awareness as the objects detected in the
environment and the various ways in which they are related or connected with each other is known by the
system. Level 3 allows the system to predict effects of actions or situations on the environment. Level 4
attempts to refine the outcomes of levels 1, 2 and 3.
Corradini et al. (2005) list a number of approaches and architectures for multimodal fusion in multimodal
systems such as carrying out multimodal fusion in a maximum likelihood estimation framework, using
distributed agent architectures (e.g. Open Agent Architecture OAA (Cheyer and Martin, 2001)) with intraagent
communication taking place through a blackboard, identification of individuals via “physiological
and/or behavioural characteristics” e.g. biometric security systems using fingerprints, iris, face, voice, hand
shape etc. (Corradini et al. 2005). It is stated by Corradini et al (2005) that modality fusion in such systems
involve less complicated processing as they fall largely under a “pattern recognition framework” and that this
process may use techniques for integrating “biometric traits” (Corradini et al. 2005) such as the weighted sum
rule as in Wang et al. (2003), Fisher discriminant analysis (Wang et al. 2003), decision trees (Ross and Jain,
2003), decision fusion scheme (Jain et al, 1999) etc.
Corradini et al (2005) also list a number of systems fusing speech and lip movements such as using
histograms and multivariate Gaussians (Nock et al. 2002), artificial neural networks (Wolff et al. 1994; Meier
et al. 2000) and hidden Markov models (Nock et al. 2002).
Some systems use independent individual modality processing modules such as speech recognition modules,
gesture recognition module, gaze localisation etc. Each module carries out mono-modal processing and
presents the output to the multimodal processing module which handles the semantic fusion. These systems
are ideal for introducing a shelf framework where various showcases may be developed for different
application domains applying re-usable off-the-shelf components each handling a single modality in full.
Other systems include “quick set” (Landragin, 2007) which offers the user the freedom to interact with a mapbased
application using a pen-and-speech cross-modal input capability. The system presented in Elting (2002)
enables the user to specify a command by way of speech, pointing gesture and the input from a graphical user
interface into a “pipelined architecture”. The system put forward by Wahlster et al (2001) is a multimodal
115