An Ontology for Video Human Movement Representation based on ...

parts of the body. Complex movement is a series of simplemovement and actions taking place over a period of time.Simple movement can also be divided into three kinds ofmovement: 1) Hand<strong>Movement</strong>, when the body part of amovement uses only Lefthand or RightHand elements. 2)Foot<strong>Movement</strong> when it uses only feet elements. 3)HandFeet<strong>Movement</strong>, which is <strong>based</strong> on the movement of handsand feet elements together.Figure 4. Part of the ontology related to different kinds of movement.Both concepts and concept instances (individuals) aredefined using OWL, which is considered as standard languageto represent knowledge and to facilitate exchanges and linksbetween ontologies.TABLE I.DESCRIPTION GLOSSARYFigure 3. Part of the ontology related to movement representation <strong>based</strong> onBenesh notationWe also developed Descriptors, which are used to annotatethe measurements of a movement, such as, the Distancebetween the different body parts, and the time <strong>for</strong> modeling thekeypose Duration in the sequences. We presented the differentmovement Actions by using the concepts (move, raise, etc.). aspictured on Figure 4.Now, we can define the glossary of the VMO. This glossarywill describe the most relevant concepts, and it is summarizedin Table I. In fact, visual data is included in our ontology asindividuals applied on the concepts. Each individual includes:object identifier, time label, content descriptors and link to theraw data (the keypose in the repository).Concept<strong>Movement</strong>Body-<strong>Movement</strong>BodyPartStaveLinesHeadLineShouldersLineWaistLineKneesLineFeetLineBasicSignInFrontOfLevelBehindPlaneSagittalCoronalTransverseDefinitionA way of putting body parts together to <strong>for</strong>m ahigher-level movementIt is the blocks of a body part that tells what eachpart should do at a specific point in time. TheBody-<strong>Movement</strong> element is the basic class <strong>for</strong> allmovementsContains one ore more areas of the body that areaffected by the movement. It has two elements(part-name, side)Each stave line describes a number of static bodypositionHeight of the top of the headHeight of the shouldersHeight of the waistHeight of the kneesRepresents the floor lineRepresents the position of the hands and feet,relative to the body. It has three signs values (infront,level, and behind)The position of body part is in the front of the bodywallPosition of body part is within the body wallPosition of body part is behind the body wallUsed to describe the anatomic position of the body.It has three values: sagittal, coronal, and transverse<strong>An</strong>y plane dividing the body vertically into rightand left portionsIt divides the body into front and back halvesIt divides the body into top and bottom halves

ConceptDescriptorsDurationDistance<strong>Video</strong>SequencekeyPoseDefinitionDescribe the measurements of a movementDescribe the time <strong>for</strong> modeling the keyposeduration in the sequencesRepresents an exact measure of distanceConssiste of moving images with or without soundConsists of two element (Keypose and duration)Represent the locations occupied by the fourextremities, the hands and feet, in relation to thebody of the person on the planeC. FormalizationAfter having studied the conceptualization step. We arecontinuing the VMO modeling of our by <strong>for</strong>malization step.The ontology was <strong>for</strong>malized using OWL and Protege.4.1 1 . So,it can be easily reused and shared. Figure 5 shows the hierarchydefined in Protégé.Figure 5. VMO hierarchy in ProtégéTo <strong>for</strong>malize in OWL the composition of movements (usedin Figure 3 and 4 respectively), we exploit the method definedby [18]. Figure 6 shows the <strong>for</strong>malization in OWL, which is apart of the ontology related to video human movement.Formal expressions in SWRL (Semantic Web RuleLanguage) were also defined to classify each kind of movementpresented in the Figure 4 (see the SWRL expressions in Figure9). Those rules are defined by expressing how to infer that aspecific movement is considered of a specific type. Forexample, the rule (Hand-Left-Moved-InFrontOf-SagittalPlane)states that a movement named a (<strong>Movement</strong> (?a)), is classifiedas a simple movement which is a (Hand-Left-Moved-InFrontOf-SagittalPlane (?a)), since it has a body part named(Hand(?y) ∧ hasPart(?a, ?y) ∧ hasSide(?y, Left) ∧Actios(Move) ∧ Sagittal(?z) ∧ hasPlane(?a, ?z) ). In this rule,the concepts are (<strong>Movement</strong>, Hand, Actions, and Sagittal), andthe relations between concepts are (hasPart, hasSide andhasPlane).111111…….Figure 6. OWL code <strong>for</strong> the <strong>for</strong>malization of movement part of VMO .IV.EXPERIMENTAL RESULTSFormalization of <strong>Movement</strong> ClassTo validate our VMO, we create several instances <strong>based</strong> onreal video human movements. For training and evaluation ourpurposes, 10 videos were used, and duration of the data is about2 minutes average per video of the data. We have applied thetemporal segmentation method (background subtractionaverage) [23], to cut the edge of the human body shape. To findthe keypose images, we have been verified that shapes do notchange their positions. Then, the low level video features havebeen extracted. Notice that, we have used the OpenCV imageprocessing library (version (2.3.1)) 2 , in all our implementationsteps.Extracted video features are recorded in XML file <strong>for</strong>producing BMN. The keypose is generated automaticallyinvolving text in<strong>for</strong>mation. This test is <strong>based</strong> on the keyposenumber, the time element (duration), the in<strong>for</strong>mation (part,1 http://protege.stan<strong>for</strong>d.edu2 http://source<strong>for</strong>ge.net/projects/opencvlibrary

side), and BMN in<strong>for</strong>mation (plane, position, and distance) .<strong>An</strong>example of a keypose in XML <strong>for</strong>mat is given at Figure 7:< sequence total key pose = ''3'' duration =''6,9''>1 footline11 footline11 kneeline11 kneeline1Figure 7. XML movement representation.During ontology instantiation, we verified that all conceptswere used and all required in<strong>for</strong>mation <strong>for</strong> supporting themovement annotation <strong>based</strong> BMN was represented. In fact, anumber of SWRL rules were created, as shown in the Figure 9.Each SWRL rule was constructed by using SWRLtab 1 , andwhich is a Protégé plug-in. The aim of inference is to derivethe new knowledge by applying the rules on the existingknowledge in a specific domain. Currently, various ruleengines <strong>for</strong> OWL reasoning have been proposed [9,15]. Weuse ‘Jess’ rule engine 2 , which integrated in Protégé, <strong>for</strong>reasoning and execution the SWRL rules.The key idea <strong>for</strong> detection of the semantic inference rules is<strong>based</strong> on the mappings between the visual features of videomovement presented in XML fragments, and ontology(concepts and rules). Figure 8, on the upper left side, shows theXML code that corresponds to the key pose image on theupper right side and its SWRL rule (bottom side).We annotate all our movements by defining the compositionof movement concepts, properties, and the relations betweenthose concepts. Then, we specify the inference rules that let usdraw conclusions from what we recognize in the data (XMLfile). All this in<strong>for</strong>mation is exploited to create severalinstances in the VMO. Afterwards, the system runs inferenceengine from Protégé. As a result the inferred movements areproposed <strong>for</strong> annotating video human movement. Figure 9illustrates the use of the VMO and the movement rules. Inorder to facilitate the definition of those rules, several instancesof the movement concept have been created in our ontology.Preliminary results have shown a good adequacy betweenour approach <strong>for</strong> semantic annotation and human interpretation.However, we have still to conduct larger experiments toquantify our method.< sequence total key pose = ''5'' duration =''8,21''>1 footline11 footline11 ShoulderLine 31 ShoulderLine3<strong>Movement</strong> (?a) ∧ Hand(?y) ∧ hasPart(?a, ?y) ∧ hasSide(?y, Left) ∧Hand(?x) ∧ hasPart(?a, ?x) ∧ hasSide(?x, Right) ∧ Actios(Move) ∧hasDirection(?a, horizontal) ∧ Coronal(?z) ∧ hasPlane(?a, ?z) →Hands-Moved-Horizontally-PlaneFrontal(?a)Figure 8. XML and SWRL movement representation.V. CONCLUSION AND FUTURE WORKIn this paper, we presented a VMO defined from the analysisof the main concepts of the multi-media domain consideringand BMN. We aimed to using this ontology to automate videomovement annotation. The main contribution of this paper isthe detailed description of the definition process of a VMO,from the definition of the concepts till the specification and<strong>for</strong>malization of axioms related to the domain. We describealso how this ontology can be used to support video annotationand retrieval by the inference of the axioms defined in theontology. We collected the instances of the concepts from lowlevelvideo data, which were recorded in XML file <strong>based</strong> onBenesh <strong>Movement</strong> Notation (BMN).We are now working to applying our proposal to develop aquery system that automatically captures the instances fromdifferent XML file, and to analyze the different possibilities ofcomplex movements inferences that we can get by using thisontology. Further validations have to be carried out to verifythe adequacy between our VMO approach and humanperception. We will also include the evolution of the VMO withother kind of movement’s types (Dance, Sport, etc.).1 http://protegewiki.stan<strong>for</strong>d.edu/wiki/SWRLTab2 http://www.jessrules.com/