New Developments in Artificial Intelligence and the Semantic Web

Publications of the Finnish Artificial Intelligence Society - 23New Developments in ArtificialIntelligence and the Semantic WebProceedings of the 12 th Finnish Artificial IntelligenceConference STeP 2006, Helsinki University ofTechnology, Espoo, Finland, October 26-27, 2006.Edited by Eero Hyvönen, Tomi Kauppinen, Jukka Kortela,Mikko Laukkanen, Tapani Raiko, and Kim ViljanenFinnish Artificial Intelligence Society – Helsinki University ofTechnology – University of Helsinki

Publications of the Finnish Artificial Intelligence SocietyThe Finnish Artificial Intelligence Society publishes national and internationalconference papers on theoretical and applied artificial intelligence research andpopular multidisciplinary symposium papers on intelligence and relatedstudies in Finnish and other languages.New Developments in Artificial Intelligence and the Semantic WebThe 12 th Finnish Artificial Intelligence Conference STeP 2006Helsinki University of Technology, Espoo, Finland, October 26-27, 2006.Organizers:Sponsors:Finnish Artificial Intelligence SocietyHelsinki University of TechnologyUniversity of HelsinkiSemantic Computing Research GroupTeliaSonera Finland OyjCity of EspooNokia OyjProgram Committee and Editors:Eero Hyvönen, Helsinki University of Technology, University of Helsinki, ChairTomi Kauppinen, University of Helsinki, Helsinki University of TechnologyJukka Kortela, Artificial Intelligence Technologies, University of OuluMikko Laukkanen, TeliaSonera Finland OyjTapani Raiko, Helsinki University of TechnologyKim Viljanen, Helsinki University of TechnologyOrganizing Committee:Tapani Raiko, Helsinki University of Technology, ChairTomi Kauppinen, University of Helsinki, Helsinki University of TechnologyJukka Kortela, Artificial Intelligence Technologies, University of OuluEero Hyvönen, Helsinki University of Technology, University of HelsinkiMikko Laukkanen, TeliaSonera Finland OyjKim Viljanen, Helsinki University of TechnologyIina Tarnanen, Helsinki University of TechnologyPentti Haikonen, Nokia Research CenterHarri Valpola, Helsinki University of TechnologyKrista Lagus, Helsinki University of TechnologyOtto Lappi, University of HelsinkiMatti Miettinen, RD-TechCover:Series Editor:Tomi Kauppinen, Kim Viljanen, and AITapani Raiko, Helsinki University of TechnologySeries: Conference Series – No 23 – Proceedings of STeP 2006Publisher:Printed by:Finnish Artificial Intelligence SocietyTSF Vallilan kopiopalvelu, HelsinkiISBN-13:ISBN-10:ISSN:ISBN-13:ISBN-10:ISSN:URL:978-952-5677-02-7 (paperback)952-5677-02-8 (paperback)1238-4658 (Print)978-952-5677-03-4 (PDF)952-5677-03-6 (PDF)1796-623X (Online)http://www.stes.fi/step2006/proceedings.pdfii

Sponsors and Partners of STeP 2006iii

Welcome to STeP 2006!These proceedings contain the research papers presented at the 12 th Finnish ArtificialIntelligence Conference (Suomen tekoälytutkimuksen päivät, STeP) that has been heldbiannually since 1984. In 2006, the twelth conference was organised alongside the ninthScandinavian Conference on Artificial Intelligence (SCAI 2006) at Helsinki University ofTechnology, Espoo. The first half of these proceedings is on the minisymposium “SemanticWeb at Work” whereas the second half contains topics from various areas of artificialintelligence.The Semantic Web is a research and application area where artificial intelligence techniques,such as knowledge representation and reasoning techniques are combined with webtechnologies, and applied to the domain of sharing and reusing knowledge and services onthe web. There are already many standards in use, such as RDF and OWL, and lots of toolsand methodologies are being developed for creating applications. However, at the momentthe number of practical semantic web applications on the web is not very large in comparisonwith the wide interest in the area.The idea of this mini symposium was to focus on the practical aspects of making the vision ofthe Semantic Web through in practice. The papers presented explain and discuss the benefitsand drawbacks of using semantic web technologies in real life applications. The theme isdiscussed from two viewpoints. From the application developers' viewpoint, best practicesand methodologies of creating ontologies and semantic web systems are in focus. From theend-user's viewpoint, the semantic services and innovative user interface designs arediscussed, as well as issues related to the content creation and harvesting processes on theweb.The symposium programme was opened with a keynote talk by Ora Lassila, Nokia ResearchCentre Cambridge, and was followed by two consecutive sessions containing thirteenpresentations. The second day of STeP 2006 was opened with a keynote talk by Tom Ziemke,University of Skövde, followed by again two consecutive sessions containing elevenpresentations. We thank all contributors and participants of the conference and authors ofthese proceedings for their activity in this exiting field of technology and for smooth cooperation.The occasion was organized by the Finnish Artificial Intelligence Society together with theNational Semantic Web Ontology Project (FinnONTO) 2003-2007, being conducted at theSemantic Computing Research Group at the Helsinki University of Technology (TKK),Laboratory of Media Technology, and at the University of Helsinki, Department of ComputerScience.TeliaSonera Finland Oyj has provided financial support for printing the proceedings.October 16, 2006Eero Hyvönen, Tomi Kauppinen, Jukka Kortela, Mikko Laukkanen,Tapani Raiko, and Kim Viljaneniv

Table of ContentsSemantic Web at WorkSemantic media application for combining and playing with user created andprofessional content....................................................................................................................1Asta Bäck, Sari Vainikainen, and Pirjo NäkkiDescribing and Linking Cultural Semantic Content by Using Situations and Actions............13Miikka Junnila, Eero Hyvönen, and Mirva SalminenCultureSampo—Finnish Culture on the Semantic Web: The Vision and First Results...........25Eero Hyvönen, Tuukka Ruotsalo, Thomas Häggström, Mirva Salminen, MiikkaJunnila, Mikko Virkkilä, Mikko Haaramo, Eetu Mäkelä, Tomi Kauppinen,and Kim ViljanenOntology-based Modeling and Visualization of Cultural Spatio-temporal Knowledge..........37Tomi Kauppinen, Riikka Henriksson, Jari Väätäinen, Christine Deichstetter,and Eero HyvönenImproving the Quality of Medication by Semantic Web Technologies...................................46Juha Puustjärvi and Leena PuustjärviRosettaNet and Semantic Web Services...................................................................................52Paavo Kotinurmi and Armin HallerCASCOM: Context-Aware Service Coordination in Mobile Computing Environments........60Heikki Helin and Ahti SyreeniDescribing Rich Content: Future Directions for the Semantic Web......................................143Timo Honkela and Matti PölläNew Developments in Artificial IntelligenceTowards reflective information management...........................................................................62Jari Yli-Hietanen and Samuli NiiranenElastic Systems: Role of Models and Control..........................................................................67Heikki HyötyniemiProlonged Spiking Periods Pattern Detection Method by Using EMFit and SCSB Signals...75Jarmo Alametsä, Antti Saastamoinen, Eero Huupponen, Alpo Värri, Atte Joutsen,Joel Hasan, Esa Rauhala, and Sari-Leena HimanenObject-oriented declarative model specification in C++..........................................................82Risto LahdelmaSolving and Rating Sudoku Puzzles with Genetic Algorithms................................................86Timo Mantere and Janne KoljonenUsing SOM based resampling techniques to boost MLP training performance......................93Jussi Ahola and Mikko HiirsalmiData-Based Modeling of Electroless Nickel Plating................................................................97Hans-Christian Pfisterer and Heikki HyötyniemiLähdekoodin symbolinen analysointi tekoälyn näkökulmasta...............................................103Erkki LaitilaScrap charge optimization using fuzzy chance constrained linear programming..................118Risto Lahdelma and Aiying RongSimulating processes of language emergence, communication and agent modeling.............129Timo Honkela, Ville Könönen, Tiina Lindh-Knuutila, and Mari-Sanna PaukkeriProgram comprehension theories and Prolog-based methodologies......................................133Erkki Laitilav

Semantic media application for combining and playing withuser created and professional contentAsta BäckVTT Media and InternetP.O. Box 1000, FI-02044 VTT, Finlandasta.back@vtt.fiSari VainikainenVTT Media and InternetP.O. Box 1000, FI-02044 VTT, Finlandsari.vainikainen@vtt.fiPirjo NäkkiVTT Media and InternetP.O.Box, FI-02044 VTT, Finlandpirjo.nakki@vtt.fiAbstractThere are two important trends bringing changes and new opportunities into media consumption:the emergence of user-created content and Semantic Web technologies. In this paper we present anapplication that shows how these technologies can be combined to create an enjoyable mediaconsumption experience. The application development work served also as a feasibility test of thematurity of Semantic Web technologies for media sector applications. The application containsmaterial relating to the historical Ox road of Häme. The results indicated that users enjoyed usingthe application. Several ontologies were utilised, most of which were based on existing ontologiesor taxonomies. With their help, it was possible to offer multiple views and exploratory routes intothe available content. Further development can, among other things, be made in improving searchstrategies and in utilising user-created metadata for both for enriching ontologies and as anindication of user interests.1 IntroductionMedia sector is undergoing huge changes as thecontinuously evolving electronic media gets astronger role in consumers' daily lives. Anotherimportant change is the more active role of mediaconsumers. The active role does not only meancommenting and discussing the content that mediacompanies publish but actively interacting with thecontent - publishing one’s own content andcombining self-created content with content fromother sources. Neither are users satisfied only withgood usability but they expect enjoyableexperiences with a considerable element of play andfun.Semantic Web is an important trend changing theWeb. The vision of the Semantic Web is to make theweb more intelligent. Semantic Web technologiessuch as standards and tools relating to ontologies arecurrently being developed to reach this goal.The text is originally published in Asta Bäck & al.Semantically supported media services with user participation.Report on the RISE-project. VTT Publications 612. Espoo 2006.Reprinted with permission from the publisher.The work that we describe here was made in aproject that wanted to research what kind of newopportunities these two trends bring to commercialmedia companies. In our view these trends connectto each other. Semantic Web technologies make itpossible to make more enjoyable media contentexperiences because applications can be made moreintelligent, and this way they require less effort fromthe users. The research approach of the project wasprototyping, and a prototype application calledStorySlotMachine was developed. The applicationhelps people in choosing a travel destination byletting them explore background informationrelating to sights. They can also combine differentmedia objects - both their own and others’ - intopresentations. The assembled material can be takenalong to enrich the actual visit. The aim was to makean application that offers interactivity opportunitiesfor the active users, but also gives an enjoyable userexperience for the less active ones. All this shouldbe built utilising rich metadata and Semantic Webtechnologies to test their applicability.1

2 Use scenarioOur initial use scenario was inspired by a slotmachine analogy: users are presented with somecontent in the topic of their interest, and if they arenot happy with the results, they can try their luckagain. An underlying assumption was that if aperson does not know so much about a topic,exploring and browsing a media object collection ismore pleasant than making explicit searches. Also,the results should not be shown as a typical list ofitems as usual in search engines, but as a page orcollection where different elements like images,videos and texts may be seen.The presented material may then be taken as astarting point to explore the topic more, a bit likewith a slot machine, where some of the items maybe locked and some redrawn to improve the result.The most active users may explore the topic frommany different points of view whereas the lessactive ones are satisfied with what is initially shownthem. This way both the more and less active usersare taken into consideration.Our scenario also includes the opportunity tostore the presentation and show it to other people,because an important feature that many peopleappreciate is the opportunity to get feedback fromother users. Other opportunities for utilisingpresentations are either exporting it into a personaldevise or printing the content. Different templatesmay be offered to take into consideration, whichmedia elements are emphasised and for whichdevice the presentation is generated for. If allowedby the original creator, other users may utilise thesepresentations and their components in their ownones.We chose location related content for our pilotapplication with the emphasis on travelling. Whenpreparing for a trip, people often are interested inexploring content to find out about their destination.During a trip, people take photos and videos, whichcan be used together with content from othersources.The use scenario can be divided into threeseparate cases: before, during and after the trip.Before the trip the user can familiarise withpotential destinations and their sights to find outmore about them. The material can be browsedtheme wise, and the user can select and combine themost relevant items into a collection that can beviewed either on the web or printed to be takenalong for the trip. After the trip, the user makes hisown travel story either utilising his or her ownmaterial or by combining it with the material offellow users or the content that the media companyprovides. Also after the trip, the user may maketheme stories like before the trip as well as alsonormal word based searches. The users areencouraged to add metadata in the form of keywordsor tags, which are utilised to propose additionalmaterial. The users may choose any words todescribe their content or choose from the ones thatthe system offers based on relevant ontologies.3 User interfacesThis chapter presents screen shots of the userinterfaces and describes their functionality. Moredetailed descriptions of the implementation andutilisation of underling ontologies are presented inchapter “Ontologies”.The first step is to select the places of interestfrom the map or list. The demonstration target areais the Ox road of Häme 1 , a historical route betweenHämeenlinna and Turku in the South of Finland.After selecting a place, the user is shown a list of thesights located there.The user can sort the sights by history, natureand culture, read short descriptions of them, viewboth commercial and user imported pictures and addthe sights he or she find most interesting into apersonal item list (see Figure 1).The user can search background information ofthe selected sights as theme stories (Figure 2). Atheme story is a collection of media content fromsome point of view (Figure 3). Our theme stories are“Life now and then”, “Life stories”, “Nature andanimals”, “Historical events”, “Fairytales andstories”, “Wars”, and “Art and culture”. Some of thethemes are divided into sub themes. For example,historical events are divided according to historicalperiods. Only the categories with some content areshown to the user. The user can play with thecontent: View commercial and user-created picturesand videos, and view and build theme stories. Theuser may include theme stories into the travel planto be created for the trip, as well as photos anddescriptions of the chosen sights. The travel plan isavailable as a slide show and as a web page suitablefor printing.1http://www.harkatie.net/english/index.htmlNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 2

Sort sights by history, nature, cultureView picturesShort description of the sightThe list of the chosensightsAdd sights that you are interested in into thelistFigure 1: Choosing sights to visit.After uploading the content, the user is asked toadd some metadata. As the first step, the photos areconnected to the sights by dragging and droppingthem to the correct sight. After that, additionalmetadata can be given in the form of keywords ortags and by indicating the genre of the photo (seeFigure 4). The keywords can be written freely or theuser may utilise those that are suggested by thesystem based on relevant ontologies. The user mayalso add free text to his or her photos and change thevisibility of the photos to other users.Users are offered commercial and other users’content, which they can combine with their own (seeFigure 5). There are several ways to search foradditional content. The user can browse through theavailable photos, videos and texts. Content can alsobe searched with the help of tags, both user’s owntags and those suggested by the application based onontologies, or by making a traditional free textsearch. The travel story is created automatically outof the content selected by the user. It can be viewedas a slide show or as a web page suitable forprinting.Theme storiesChoose themeAdditional metadataThemes:Life before and nowLife storiesNature and animalsHistorical eventsFairytales and storiesWarsArt and cultureFigure 2: Making theme stories to get backgroundinformation relating to the selected sightWho the photocan be shown to(all, family)Keywords• Tags suggested basedon the ontology• User's own freelyselected tagsGenreTheme label: Life storiesInclude a theme story intoyour travelling guideSaveFigure 4: Adding metadataPlay with the content:• View pictures andvideos (commercialand user created)• View and buildtheme stories• Try different themesFigure 3: An example of a theme story.After the trip, the user may create his or her owntravel story by utilising his/her own material and thematerials in the system. Photos can be uploadedafter selecting the visited sights. As part of theuploading process, the user determines whether thephotos can be viewed by other users, and accepts thelicensing terms.3

Combining owncontent withcommercial andother users'contentUser's owncontentTextFigure 5: Combining user-created content withcommercial and other users’ content. Additionalcontent can be found by browsing available photosand videos by media types or tags.4 ContentMedia contentThe user canbrowse through theavailable photosand videos.They can be addedto the travel story.We Different types of media content, such asfacts, stories and news, are needed in order to beable to create versatile travel plans, theme storiesand travel stories. Media content that is directlyrelated to the target area is preferred, but also moregeneral information is usable. A mixture of videos,photos, sounds and texts makes the presentationsmore appealing and interesting.The commercial media content of the pilotapplication consists of newspaper and encyclopaediaarticles with images, articles from the Häme Oxroad magazines, stories out of a book called“Hämeen Härkätiellä”, and photos from the HämeOx road website. In addition to the commercialcontent, the application has user-created photos. Thecontent is mostly general background informationand not specific travel information like openinghours or prices.This mixture of content and media formatsmeant that it was necessary to work with severalmetadata vocabularies. Different vocabularies areused to describe newspaper, magazine andencyclopaedia articles as well as short stories andusers’ own content. Also different media formats(text, photos, and videos) have different needs andvocabularies for metadata.The content was delivered for our prototype indifferent formats and the amount of metadata varieda lot. The project did not address automatic methodsfor creating semantic metadata, and adding metadataand converting it into RDF required manual work.The newspaper articles and images had somemetadata that had been generated in the normalnewspaper production process, and some moremetadata like genre, scene and IPTC subject codeswere added by the media company persons for theprototype. We received the metadata in text formatand it had a structure that helped us in converting itinto XML even though manual work could not beavoided completely.The encyclopaedia articles were delivered inXML and the structure of the articles could beutilised in converting their metadata into RDF. Theencyclopaedia content also has the potential forpopulating the target ontology, for example withpersons relating to the Finnish history.The content received from the Häme Ox road didnot contain any metadata so the metadata wascreated by hand. The articles of the Häme Ox roadmagazines were received in PDF format, and thatcaused also extra work.5 Ontologies5.1 The role of ontologiesThe prototype utilises a number of ontologies,each of which captures knowledge of some area thatis necessary to fulfil the required functionality.Ontologies are utilised when selecting content andalso to produce some basic information to be shownto the users. The ontologies are also utilised whenusers add metadata to their own content such assuggestions to keywords.The Target ontology describes the knowledgerelated to places, routes and sights, and containsinformation that has relevance to them such aspersons, events, objects and nature.The Media ontology describes the mediacontent. Relevant elements were selected out of theDublin Core (DC) and IPTC Newscodevocabularies. The Media ontology includes thetypical metadata fields, such as title, creator,publisher, date, media type, genre, scene, but alsorelations to the Time and Target ontologies, forexample relations to persons, sights, places, routes,events, objects, animals or plants. The subject ofmedia content was described with the YSA ontology(a general-purpose thesaurus in Finnish) wheneverpossible, but for news articles also the IPTC and forencyclopaedia articles the Facta ontologies wereused.The Presentation ontology contains theinformation on themes and their subcategories andwhat kind of content (subject, genre, scene, time) isto be searched for presentations. There are themeslike “Life now and then”, “Life stories”, “Natureand animals”, “Historical events”, “Fairytales andstories”, “Wars”, and “Art and culture”.An ontology based on YSA (a general-purposethesaurus in Finnish) is utilised as a kind of upperontology for classifying the knowledge. Target,Media and Presentation ontologies are connected toNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 4

each other via the concepts of this upper YSAontology. The YSA ontology was created only to alimited extent because the idea was to replace it withYSO (Finnish General Ontology), which was underdevelopment and not yet available during the timewhen the application was made.The Time ontology defines a taxonomy of timeeras and periods by time intervals, and it is based onthe ontology developed in the MuseumFinlandproject 2 . We added some time periods relating to theFinnish history as well as the seasons.The subject of the media content is determineddifferently for different content types: the IPTContology is used to determine the subject ofnewspaper articles. The ontology is based on theIPTC ontology 3 that was developed in the Neptunoproject.The content of encyclopaedia uses its owntaxonomy (Facta ontology). YSA-ontology is usableas a general subject ontology.5.2 Searching content for theme storiesTheme stories consist of elements like a title,text, image and fact box, and they are selected onthe fly based on the knowledge in the ontologies.The fact box shows information retrieved out of theontology. It may contain knowledge about howevents, like a war, are related to the sight or basicinformation about a person who has a connection tothe sight. Sometimes the user may wonder why acertain article was shown, and the role of the factbox is to give some indication about the connection.Media content is not linked directly to thevarious themes. The knowledge in the Targetontology and the search rules are utilised insearching and offering relevant media content. Thesearch rules are determined with the Presentationontology, Java application and SPARQL queries.The criteria for how the media content is connectedto a theme, such as the subject, genre or time, aredetermined in the Presentation ontology. Theadvantage is that the search criteria are not hiddeninside the Java code, but that they can be changedby modifying the instances of the ontology. Also,themes may be created, changed or deleted bymodifying the ontology classes or their instances.The Java application creates SPARQL queriesfor searching relevant media content based on theknowledge in the Presentation ontology. Searchesutilise the knowledge in the Target ontology (e.g.Life stories -> persons related to the sight) and/orsubjects related to themes (e.g. Every day life nowand before -> food, professions, clothing etc. orWars -> Great Northern War, World War I & II etc.).In addition to that, some restrictions may be used,2http://museosuomi.cs.helsinki.fi/3http://nets.ii.uam.es/neptuno/iptc/like time (e.g. Historical events), genre (e.g. Storiesand fairy tails), place or sight.The subjects of the different themes aredetermined as relations to the YSA ontology. Alsothe subjects of the IPTC and Facta ontologies areconnected to themes. Media content that is related tosame subjects is searched for. If content that isdescribed with some other ontology were broughtinto the system, the subjects of this new ontologywould need to be connected to the existing themes.5.3 Adding metadata to user generatedcontentUsers can add metadata to their own content.Users are free to use any words they want todescribe their content, but by utilising the availablecontextual information and the Target ontology,keywords are suggested. These suggestions relate toyearly events, objects, terms, other related sightsand seasons. It was made easy to use these terms– itis enough to click a word, and no writing is needed.Users are thus encouraged to use these words thatcan then be utilised to suggest additional relevantcontent from the system.In similar manner, users are encouraged to addkeywords relating to the genre based on theknowledge in the Media ontology. Genres have beendefined for all media types but only image genresare currently utilised. The genre information isuseful when the user generated media objects areutilised with future users.5.4 Searching commercial content tocomplement user’s own contentOffering media content to complement user’sown content is based on the user-given metadata andthe knowledge of the Target ontology. First, themedia content that is related directly to the sight issearched. After that, more general media contentrelating to events, persons and places is searchedfor. The relevance of the media content isdetermined with the help of the search order startingwith from exact searches and then proceeding tomore general searches.Additional content can be searched with the helpof tags. The tags suggested by the ontology mayalso be related persons or events in addition to tagsrelating to yearly events, objects, terms, other sightsrelating to sight and seasons. Already existing themestories made by earlier users might be an additionalway to search information also when creating one’sown travel story. Theme stories give ready-madetext and image/video combinations that can easilybe added to a new travel story.6 Software and architecture5

The ontology editor Protégé 3.1 was used fordeveloping ontologies. Ontologies were developedas RDFS-schema.The application is implemented as a Java client –server solution using Struts framework and JavaServer Pages (JSP). Tomcat 5.5 is used as the webserver. The user interfaces were implemented withAJAX (Asynchronous JavaScript and XML) inorder to offer good interactivity and usability, and toadd new features, like drag-and-drop. Differentviews of the travel plan and travel story aregenerated utilising CSS style sheets.The ontologies and RDF-based data are handledby Profium SIR (Semantic information router). Abeta version with support for SPARQL-queries wasused. Profium SIR saved the RDF data into aPostgres 7.4 database. Postgres 7.4 was used also formanaging user information.The Sparql4j-jdbc driver 4 was used for queringRDF-data. Profium SIR created the result accordingto the SPARQL Protocol for RDF specification 5 andforwarded it to a Sparql4j-jdbc driver, whichprovides the results via the Java ResultSetabstraction.It is not easy to choose the tools for applicationdevelopment with Semantic Web technologies.There are several open source tools, most of whichhave been created for research purposes. SemanticWeb related standards and recommendations are stillunder development, and different tools supportdifferent subsets of the standards. For example, weused one tool for developing the ontologies andanother for handling the RDF-data, and this causedsome extra work to overcome the incompatibilities.Protégé 3.1 is a versatile ontology editor withmany useful features. It also has features formanaging and developing multiple relatedontologies, but we had problems with this feature.Reopening a Protégé project file with connections toother ontologies caused error messages andsometimes even meshed up the instance data.The development of a standard query languageSPARQL for querying RDF repositories is a step tothe right direction. We wanted to use such a versionof Profium SIR that supported SPARQL eventhough it was in beta at the time. Java applicationdevelopment was speeded up by using the Sparql4jjdbcdriver with SIR, even though it supported onlyselect and ask type of queries at the time of theapplication development.Utilising AJAX made it possible to addimpressive features into the web user interface. Adownside was the lack of good development tools;debugging JavaScript code is troublesome. We alsoencountered the well-known problem for developingweb applications for different web browsers: what4http://sourceforge.net/projects/sparql4j5http://www.w3.org/TR/rdf­sparql­protocol/works in one browser does not necessarily work inanother.As a brief summary we can conclude that therealready are usable tools for developing semanticweb application, but currently many tools only havea partial support for the specifications. There isroom and need for further development to make theimplementation and management of Semantic Webapplications easier.7 Results7.1 User testsThe user experience of the application was testedin two phases in the context of real schoolexcursions. The test group consisted of 33schoolchildren (12–18 years old) and 4 teachersfrom four different schools. In the first phase, userneeds and expectations were studied using artefactinterviews, observation, collages, metadata test andprototype tests. The prototype tests were made usinga co-discovery method, where two participants usedthe prototype together and discussed about thedecisions they made. Some users tested the travelplanning part of the software before an excursion,whereas others had already made a trip and createdtravel stories with the prototype.At the end of the project the functionalapplication was tested again with the same usergroup but with a smaller number of participants (6schoolchildren, 12 years old). The test users hadmade a trip to the Häme Ox road and they used theapplication afterwards to store their own picturesand memories. The users were interviewed bothbefore and after testing. After the test session theyalso filled out a short questionnaire.As the result of the first interviews, observationand collages made by users, following requirementsfor the StorySlotMachine were found. Theapplication should- arouse interest and offer necessary factsbefore the trip- enable experiencing the stories during thetrip- give additional information about thethemes studied on the trip, as well as thethemes about which no information wasavailable on the trip- support creating a personalised travel story- enable storing rich metadata about pictures,e.g. memories and feelings, as well ascomments and hints for other travellers.A metadata test was made in order to gatherinformation about the meanings that the usersassociate with their travel photos. The aim was tofind out, how semantic information could be addedinto the pictures. The users were asked to addcaptions and keywords into their own travel photos,New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 6

as well as select applicable tags from a list ofkeywords. The written captions were generally veryshort, and the users did not necessarily rememberanything about the objects of their photos. Theintuitiveness of selecting keywords varied a lotamong the users. The users must understand whatthe purpose of adding metadata is in order to find iteasy to do. In addition, the user should seeimmediate advantage of adding metadata.The keywords used to describe their photos canbe divided into five groups: 1) description of anobject or an action, 2) memories and atmosphere, 3)background information about the place or object, 4)questions for additional information (history and/orpresent), and 5) hints for other travellers.The application functioned only partially in thefirst user tests. For that reason many of the test usersfound the user interface somewhat confusing and theidea of mixing own and media content did notbecome clear to everyone. In addition, mediacontents were not presented attractively enough torouse the interest of users. Nonetheless, half of theusers found the application engaging and useful.Schoolchildren appreciated the idea of finding thenecessary information easily in one place. Imageswere regarded as the most interesting part of thecontent. The complete report of the first user testscan be read in Näkki (2006).The prototype was developed further after thefirst user tests and tested again at the end of theproject. In the second test, user attitudes towards thefunctional application were very positive. Thesystem was found useful, quick and easy to use.Users found the StorySlotMachine more pleasantthan traditional search machines, because therelevant content could be found easily as stories.The users regarded photos as the core of theapplication and added both their own andcommercial pictures into their travel stories. Theusers were also eager to write short captions to thephotos. Adding metadata into their own pictures wasintuitive and did not burden the users. Other users’pictures from the same excursion were foundinteresting, as well.Some users wanted to create their travel storyquickly, whereas others were ready to use a lot oftime to finish their stories. Interestingly, theStorySlotMachine was found to be suitable for boththese user groups. All participants of the last testssaid that they would like to use the system again.Summary of user experiences in the both test phasescan be seen in Figure 6.Figure 6: User experiences of the system: a) afterthe first user test (N=22), b) after the second usertest (N=6).From the users’ point of view, the value ofsemantic content is in the quickness and easiness ofinformation retrieval. The way to find informationas stories was something new for the users, but mostof them found the idea easy to understand. However,the users did not necessarily want to collect, storeand print stories, when planning the trip. The systemshould therefore better support pure browsing of thecontent. After a trip it was seen more understandableand useful to create a travel story, where ownmemories are linked to editorial material.When users create semantic content, onechallenge lies in the process of adding metadata. Itwas discovered that the travel images and memoriesinclude a lot of meanings that are hard to put intowords as simple keywords. Users’ activeparticipation will be needed, even though automaticsemantic reasoning is used for creatingpresentations. It is the user who decides whichcontent is valuable for her. However, theStorySlotMachine can substantially help the user byoffering suggestions about the media content relatedto the theme of the user’s trip. The semanticprocessing makes it possible to discover interestingand surprising relations between contents that wouldbe hard to find otherwise.7.2 Ontologies and implementationCreating, updating and managing ontologies arenot easy tasks, but there are clear benefits in thistype of an application:- Ontologies make it possible to searchcontent from multiple directions (sights,events, persons etc.).- Also general media content can be utilised.- It is possible to make different thematicpresentations or views for people withdifferent interests.- For example, one user might be interested inthe historical places and events of the Oxroad during the 19th century and another isonly in churches during the trip. They caneasily be served with this kind of anapplication.7

- Ontologies contain knowledge that makes itpossible to create visualisations such astimelines, cause-effect diagrams, dialogues,trees, and maps of related resources.- Ontologies support generating aggregationsautomatically.- The benefits of being able to link thecontent automatically into different themesbecome significant as the number of contentitems increases and grows continuously.There already are usable tools for developingsemantic web applications, but currently many toolsonly have a partial support for the specifications.There is room and need for further development tomake the implementation and management ofSemantic Web applications easier.Theme stories were the most central part of theapplication for ontology deployment. Theme storiescould be easily generated for sights with a longhistory, but not so smaller sights. Theme storiesshould rather be offered at higher level like for atown or a village or as in our case, for the wholehistorical route, than for a single sight.There were challenges in creating general searchrules for the themes. Every theme had uniquerequirements and complicated the Presentationontology. Some examples are listed below:- “Every day life now and before” hassubcategories weekday, celebration andsociety. Subjects like food, professions,clothing, inhabitation, celebrations, laws,and source of livelihood relate to thistheme. These determine the mainframework for searching, but to get morerelevant content also the time periodsand the type of the sight should bedetermined to find relevant content for aparticular sight.- “Arts and culture” is divided into thefollowing subcategories: persons, art andbuildings, and environment. Whensearching for content for the subcategory‘Persons’, it is required to find personswho have a connection to the sight andhave or had a profession relating to artand culture, such as composer, writer,painter, or architect.- “Historical events” are divided intohistorical periods, and time restrictionsare needed in searches. There are severalways to do this: search the media contentthat relates to the time and thesight/place, utilise terms that describethe time period or events that happenedduring that time.- In the theme “Stories and fairy tails” thegenre is used to restrict the contentselection.When making ontology-based searches, severalsearch criteria can be used and their priority ordermust be determined. Here, it is important to find thecorrect balance between the number of differentsearch criteria to use and the speed of theapplication. First, the most relevant content issearched and after that, the search can be expandedto more general material. The challenge is to knowhow deep the searches should navigate into the netof relations of the ontology and still find content thatis relevant to the user. We encountered this problemboth when making theme stories and searching foradditional content to complement users’ owncontent.When the application has more content, this willprobably be less of a problem, and the challenge isthe ordering or grouping of the relevant mediaobjects in an interesting way. One of the challengesis to inform user of why certain content is offered toher/him in this context. For example, pictures ofhistorical persons might be confusing, if user doesnot know who the person is and how he/she isrelating to the sight. In connection to the themestories, a fact box was used to give some indicationabout the connection by utilising the knowledge inthe ontology, and a similar approach should be usedelsewhere in the application.The implementation uses an upper ontology thatcan be replaced with another one, if needed. Thisgives flexibility to the application development. Weturned selected parts of YSA (a general-purposethesaurus in Finnish) into ontology, and used it asour upper ontology. There is currently a project inFinland making a comprehensive ontology out ofthe YSA, and it should be used also here when itbecomes available.We had many different vocabularies fordescribing the subject of different type of mediacontent in the StorySlotMachine application. Ourfirst idea was to map the IPTC and Factavocabularies to the concepts of our top YSAontology.Mapping different vocabularies to eachother turned out to be complicated since it was notalways possible to find corresponding concepts inthe different ontologies. Also, there was a lack ofgood tools for ontology mapping.Instead of mapping ontologies to each other, wedecided to keep the different ontologies. Asdescribed earlier, the subjects of media objects weredescribed with the concepts of the YSA-ontologywhen ever possible, but we also stored the originalsubjects described with IPTC or Facta. The subjectsof the different themes were also described with theYSA, IPTC and Facta ontologies. Several searchesmay be done for media objects during a usersession: first utilising the YSA concepts and thesubjects from other ontologies. In other words,ontologies are not mapped to each other but to someextent, the mapping is done via the different themes.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 8

In order to better test the feasibility of this approachmore media content should be added to the system.This approach will probably work with mediacompanies’ own media services, where they candecide which themes are available. Of course, itmight be possible to offer users an interface wherethey can combine ontology concepts and create theirown themes. One idea for the future development ofthe StorySlotMachine is to let users create newthemes with the help of the tags they have used.In general it is good practice to use one commonupper vocabulary or ontology for describing themetadata inside the media house and also usestandardised vocabularies as much as it is possible.However it is realistic to assume that a media housewill have several vocabularies also in the future andone upper vocabulary or ontology cannot solve allissues and challenges. Better tools are needed tosupport ontology mappings and even better ifmappings could be made automatically by thesystem.8 Related workThe key idea in the StorySlotMachine is toaggregate content in a way that lets users explore thecontent in an enjoyable manner. Related work isbeing done in the various areas. The first distinctioncan be made between the aggregation level: is theaim a single story to be created of out the availablecontent, or a collection of independent resources.Geurts et al. (2003) and the Artequakt project (Kimet al. 2002) work at the first area. Geurts et al.(2003) describe the system where the knowledge ofontologies is used to create multimediapresentations like artists' bibliographies.Presentations vary based on the genre (e.g.Biography and CV) and output format that can beselected by the user. The basic idea of theirDiscourse ontology is same than our Presentationontology. The ontologies define rules for searchingcontent. They have different genres, whereas wehave themes. Our themes use more versatile datathan what is needed for artists’ bibliographies andwe also have more general content whichcomplicated the ontologies and rules. One differenceis that they focus more on ready-made multimediapresentations, which contain parts (e.g. address,private life and career) that are determined in theDiscourse ontology.Our work is more related to creating a collectionout of independent resources and turning them intopresentations. However, we let users combineimages and texts in new ways and we do not aim atproducing one collection for the user to view but astarting point for further exploration with thecontent.Mc Schraefel et al. (2005) have developed anopen source framework called mSpace, which isavailable at mspace.sourceforge.net. The startingpoint for the mSpace development as well as for ourStorySlotMachine is same: to offer an exploratoryaccess to content. The user should be able to browsecontent according to their interests and associations,to leave tracks on the way by storing the mostinteresting items, and to get multimedia as a resultrather than links to resources.The original mSpace demonstrator was aClassical Music explorer , and it has since beenutilised in other applications and domain.mSpace is based on the idea of associativeexploration of the content and user-defined andmanipulated hierarchies. mSpace lets the userexplore the material with the help of hierarchicalcolumns like periods, composers, arrangements andpieces: the selection in the first column constrainsthe selections of the following column. Users canarrange columns according to their preferences andalso add new dimensions or remove them.mSpace provides preview cues (for exampleaudio clips) of some representative example in thevarious dimensions to help in exploring anddeciding whether an area is interesting. This wayusers may find new interesting areas without priorknowledge of them. mSpace also has info views toshow related information like for example adescription of a composer. Interesting items may bestored in favourites for future reference.The preview cues in mSpace have the same aimas the themes in the StorySlotMachine: to give usersideas and hints as to what kind of content isavailable relating to a topic.An interesting feature of mSpace is to let userssort and swap dimensions according to theirinterests. In the current StorySlotMachine version,the users are tied to pre-made themes, but one ideafor future development is to let users create newthemes with the help of the tags they have used.Creation of different theme stories inStorySlotMachine is based on associations that areinferred automatically by utilising the knowledge inthe ontologies. For example, a theme story may tellabout a war that relates to a sight and story mayinclude important historical persons. New themestories are offered based on these relations and othertheme story may tell more about the person. Nowthis is made by the system, as our guiding idea wasto give the users the opportunity to try their luck andget surprised, but as an alternative, users could begiven the opportunity to guide the process based ontheir own associations.One difference between the StorySlotMachineand mSpace is that the StorySlotMachine offer usersthe possibility to make exportable packages out ofthe content and also utilise their own content. ThemSpace user interface is more formal in style than inthe StorySlotMachine, where emphasis has been putto offering a user interface with the element of play.The Bletchley Park Text application developedfor the Bletchley Park Museum (Mulholland et al.9

2005) concentrates on post-visitors of museum.During their visit, people may express their interestby sending text (SMS) messages containingsuggested keywords relating to displayed objects.After the visit, they can get a collection of contentrelating to the selected keywords as a personalisedweb site. The content can be explored and a numberof different views on the collection are provided.Bletchley Park Text application is made for aspecific museum and its specific content. In theStorySlotMachine application, we have severalplaces and sights, and the material is general bynature, since one of the major goals of our projectwas to study how the general media content can beutilised in new ways with the help of semanticmetadata. Both Bletchley Park Text application andthe StorySlotMachine share the similar ideas ofusing the application for learning, but the BletchleyPark Text does not include utilising users’ ownmaterial like we do.Bentley et al. (2006) have studied howconsumers use photos and music to tell stories. Theapplication is slightly different from ours, as theycompare using only music and photos, but there areimportant similarities and lessons to be learnt. Theyfind that different media formats, photos and musicin this case, and commercial and private contentshould not be stored in separate silos. Instead, theyshould be available with the help of similarmethods. Serendipitous finding utilising availableinformation like contextual cues should be utilisedto remind users of what is available and to helpthem in finding related resources. They also see aneed for systems that allow communication by usingmedia.9 Discussion and future workThis chapter discusses future developmentopportunities of StorySlotMachine and what are thebenefits, opportunities and main challenges of themedia companies in creating new semantic mediaservices, particularly in relation toStorySlotMachine type applications.Easy access to electronic content and users’participation opportunities into the media productioncycle are bringing about huge changes in the waythat media content is created, offered and consumed.The StorySlotMachine explores the possibilities ofletting people explore content in a playful and themewise way and letting them do the final touch inputting the presentation together. Semantic metadataand ontologies are utilised to offer multiple viewsinto the available content and help the users toexplore and learn in a pleasant way of topics thatmay not be so familiar to them. The application alsolets the users import their own content and mix itwith other people’s and commercial content.The application is most suited when the contentis available as relatively small units. The user testsindicated that photos and videos are important inraising interest, whereas particularly reading longtexts requires more effort and is less attractive in aplayful use scenario. This implies that text should bewritten in a way that makes it quick and easy to seewhat the text deals with to arouse users' interest. Inthis kind of a context, the user is not looking for aspecific answer to a specific question, but lookingaround and checking if something interesting comesup, and the content that is presented should supportthis approach.The application makes it possible for people tomake their own narratives about a topic. This isrelevant at least in connection to learning andhobbies, like travelling and making memorabilia.The speciality here is the opportunity to combineself-created content with content from other sources.The final result must have an attractive andprofessional look in order to motivate the use. Theelectronic format also makes it possible to add livefeatures; for example, a user-made narrative may beupdated with current news and developments.It was not possible to carry out user tests to suchan extent that we would have seen how peoplewould like to use the materials and what new ideascome up. More research is also needed to see, howpurposefully prepared content users want, or do theyenjoy with a mix of material and modifying itaccording to their own ideas.Serendipity is a concept that seems to bepopping up as a goal in search related application(Leong et al. 2005; Bentley et al. 2006; Lassila2006) and here Semantic Web technologies comeinto play. Particularly in consumer applications easy,intuitive and entertaining user interfaces andapplications are needed. Also our work aims atproviding serendipitous finding of interestingresources. We did not directly explore how thissensation emerges, and which factors contribute toit. One assumption is that if we know what theusers’ interests are, we’ll be able to suggestresources that he or she is interested in, and canoffer experiences of serendipitous finding.Many media companies have extensive archivesthat are not effectively utilised as end user services.The StorySlotMachine is an example of how contentcan be offered in a more interesting context than asmere searches and search result lists. If the contentis not already modular and if there is not muchmetadata about the content, an investment is neededto turn the content into more usable format. Thecurrent production processes and practices should bechanged so that the content is directly processed intoa format that supports the reuse in this kind ofapplications. The best starting point for offering thiskind of a new service is an area where the content isalready modular and where people may have longerterminterest and personal material. These includeareas like travelling, hobbies, encyclopaedia andnews.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 10

Other key questions that media companies needto agree on are descriptive metadata and terms forlicensing the content for creating the narratives, andhow strictly to guard their IPR. On the other hand,networking to freely available network resourcessuch as photos that are licensed under CreativeCommons licenses should be considered as a way toadd resources for users to choose.We can see at least following businessopportunities with this type of an application:- some basic features could be free, but accessto more content could be available forpaying customers- related materials could be available forbuying, such as books or maps, or additionalservices like back-up storing of images- co-operation with operators in theapplication area, for example with localtravel associations- targeted advertising, particularly if peoplecan be encouraged to longer term use, theninformation about their interests willaccumulate and opportunities for effectiveadvertising becomes better.There are several opportunities for utilising anddeveloping the StorySlotMachine applicationfurther. The StorySlotMachine application can beused as a platform to test user expectations andexperiences of mixing and playing with mediacontent, and sharing one’s own content with otherusers’ content more extensively. More contentshould be added for additional testing, and theconversion of metadata into RDF format should bemade automatically.The application could be developed into acommercial travel application or a learningapplication, e.g. for teaching history. For the travelapplication, some additional features are needed,like exact travel information (opening hours, prices),maps and mobile user interface, and collectingfeedback and recommendations from users. Alsonew features like collaborative storytelling e.g.creating one travel story from the contents of allgroup members, and real time travel story that iscontinuously updated with topical information,could be added.Similar applications could be built relating toother topics such as hobbies or collecting gatheringpersonal memories from past. A new Targetontology may be needed for a new application, if itdoes not make sense to expand the existing one. Thesearch criteria are not hidden inside the Java code,but they can be changed by changing the instancesof the ontology, which makes it easy to develop thecurrent application further, and to adapt it to newareas. Also, themes may be created, changed ordeleted by changing the classes of ontology or itsinstances.There is always room for improving the searchcriteria with help of the Presentation ontology, oreven a general tool for automatic generation oftheme stories could be created. In the future,RuleML (Rule Markup Language) or SWRL(Semantic web rule language) may be the solution touse.At the beginning of the application development,we considered using the CIDOC CRM culturalheritage ontology that is being developed speciallyfor describing the cultural heritage of museumcollections. We decided not to, because the ontologyseemed too specific and complicated for ourpurposes. CIDOC CRM is currently in the finalstage of the ISO process as ISO/PRF 21127, and itcould be reconsidered as an option in a travelrelatedapplication like StorySlotMachine todescribe cultural heritage. The challenge is to decidewhich level of the ontology to include and what toexclude as too detailed. Additional benefits could begained if content or information can be integratedfrom various museums with the help of a commonontology.Automatic methods for creating metadata andconverting it into RDF format were not addressed inthis project, but they are important particularly whenexisting media archives are utilised in semanticapplications. Once the number of concepts in ourYSA ontology has been increased, or the nationalYSO becomes available, utilising automaticmethods will be easier. Additionally, users should beutilised as metadata creators, where feasible.One of the future needs is to increase the amountof available content. In our application, the contentresources were moved to our server, but in acommercial application, content from differentsources should probably be utilised. This requiresthat there is an agreement on what metadata to useand the content should have this metadata.There are many opportunities to develop thesearches and the ways that search results are orderedfor presentation. Scene and genre information couldbe used for ordering images. Images from outsideand inside a building, and general images anddetailed close-ups could be alternated. New waysgrouping media objects could be developed inaddition to the current location-based presentation.User generated metadata could be utilised moreextensively. The words that people use to describetheir content could be stored in case they are notfound in the ontologies, and they could be offered tofuture users visiting the same sight. In the currentsystem, users can add tags only to describe theircontent, but tags could be utilised more widely, forexample, to describe media content, travel plans,and sights. If we had mechanisms to combine tagswith more formal semantics and to analyse thereliability of user generated knowledge, this couldbe one way of adding knowledge into theontologies.11

To summarise, we can conclude that theapplication lets users explore and combine varioustypes of media content, as well as virtual and reallife experiences. Utilising ontologies helps inmaking the application more intelligent and givesopportunities to offering enjoyable user experiences.AcknowledgementsThe StorySlotMachine was developed in the project“Rich Semantic Media for Private and ProfessionalUsers” (RISE). The two-year project was part of atechnology programme “Interactive ComputingTechnology Programme” (FENIX) run by theFinnish Funding Agency for Technology andInnovation (Tekes).Besides the main financier, Tekes and VTT, fourcompanies supported the project financially and bygiving expertise and tools for the project, and theywere represented in the project management group.The project steering group consisted of thefollowing persons: Eskoensio Pipatti(SanomaWSOY Oyj) and Tuomo Suominen (WernerSöderström Osakeyhtiö), Hannele Vihermaa (firsthalf of the project) and Teppo Kurki (the second halfof the project) (Alma Media Oyj), Jouni Siren (Yle),Janne Saarela (Profium Oy), Keith Bonnici (Tekes),Caj Södergård (VTT) and Marko Turpeinen (HIIT).The project group members want to thank thesteering group for their good support and feedback.We want to express our special thanks to the HämeOx Road project and project manager MattiKorolainen for their co-operation in the project - thematerials and user contacts provided by themcontributed to the successful completion of theproject.web. Proceedings of Semantic Authoring,Annotation and Knowledge Markup Workshopin the 15th European Conference on ArtificialIntelligence. Lyon, France.Lassila, O. 2006. Sharing Meaning BetweenSystems, Devices, Users and Culture.Symposium on Digital Semantic Contentacross Cultures. Paris, The Louvre. May 2–5,2006.http://www.seco.tkk.fi/events/2006/2006-05-04-websemantique/presentations/friday-0900-Lassila-Ora-DSCaC.pdf.Leong, T., Vetere, F., Howard, S. 2005. TheSerendipity Shuffle. Proceedings of OZCHI,Canberra, Australia. November 23–25, 2005.ISBN 1-59593-222-4. 4 p.Mulholland, P., Collins, T., Zdrahal, Z. 2005.Bletchley Park Text: Using mobile andsemantic web technologies to support the postvisituse of online museum resources.http://jime.open.ac.uk/2005/24/mulholland-2005-24-paper.html.Näkki, P. 2006. Käyttäjäkokemuksen suunnittelusemanttiseen mediapalveluun – tarkastelussakouluretkien tarinat (in Finnish). Master’sthesis. Espoo: Helsinki University ofTechnology.ReferencesBentley, F., Metcalf, C., Harboe, G. 2006. Personalvs. Commercial Content: The Similaritiesbetween Consumer Use of Photos and Music.In: CHI 2006. Montreal, Canada. April 22–27,2006. ACM. Pp. 667–676. ISBN 1-59593-178-3/06/0004.Geurts, J., Bocconi S., van Ossenbruggen, J.,Hardman, L. 2003. Towards Ontology-drivenDiscourse: From Semantic Graphs toMultimedia Presentations.http://homepages.cwi.nl/~media/publications/iswc2003.pdf.Kim, S., Harith, A., Hall, W., Lewis, P., Millard, D.,Shadbolt, N., Weal, M. 2002. Artequakt(2002): Generating tailored biographies withautomatically annotated fragments from theNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 12

Describing and Linking Cultural Semantic Content by UsingSituations and ActionsMiikka Junnila⋆ Semantic Computing Research GroupHelsinki Institute for Information Technology (HIIT)University of Helsinkihttp://www.seco.tkk.fimiikka.junnila@uiah.fiMirva Salminen‡ Semantic Computing Research GroupHelsinki Institute for Information Technology (HIIT)University of Helsinkihttp://www.seco.tkk.fi/mirva@pieni.netAbstractEero Hyvönen† Semantic Computing Research GroupHelsinki University of Technology (TKK)University of Helsinkihttp://www.seco.tkk.fi/eero.hyvonen@tkk.fiOntologies have been used to describe cultural objects, such as artifacts, by their physical or mediaspecific properties, or by the life cycle of the objects in collections. In contrast, this paper discusses theproblem of creating ontological descriptions that allow describing different kinds of cultural contentthrough the situations and actions that take place in the real world. This point of view is importantwhen semantic metadata is used as a basis for creating intelligent, educational, and entertaining linkingof content on the semantic web. The idea is addressed not only in theory but by presenting the firstprototype implementation of a cross-domain semantic portal “CultureSampo—Finnish Culture on theSemantic Web”. This system is able to automatically link together different kind of cultural resourcesin meaningful ways with explanations. The content types considered include paintings, artifacts, photographs,videos, cultural processes, and stories .1 IntroductionThis paper investigates possibilities of exploiting semanticcultural metadata in creating intelligent portals.The research is a continuation of the work behindthe semantic portal MuseumFinland 1 (Hyvönenet al., 2005a). This work showed that, based onontologies and associated metadata, semantic searchand browsing can be supported to enhance end-userservices. The content in MuseumFinland was homogenousin the sense that only artifact metadataconforming to a shared metadata schema and ontologieswas used. In this paper the main research problemis to create an ontology that can be used to describemany kinds of cultural resources, so that theycan still be searched with a unified logic and be linkedtogether semantically in insightful ways.We chose processes and actions (events) to be thebasis of our ontology. In the context of cultural re-1 This application is operational at http://www.museosuomi.fiwith an English tutorial.sources, this proved out to be a fruitful starting point.Many cultural objects of interest, such as paintings,stories, and artifacts have a connection to the processesof human life and the actions of people.To get a concrete grip of the problems involved,we created the first prototype I of the portal“CultureSampo—Finnish Culture on the SemanticWeb” system in 2005. This paper documents andsummarizes experiences and lessons learned in thiswork documented in more detail in (Junnila, 2006;Salminen, 2006). The vision and some other resultsof the CultureSampo project in the larger perspective2005-2007, including a later prototype CultureSampoII, is described in (Hyvönen et al., 2006). “Sampo”is a machine that fulfills all the needs of people inthe Finnish mythology. We try to fulfill the needs ofpeople interested in getting a good picture of Finnishculture through the semantic web.The CultureSampo I prototype was built on top ofthe MuseumFinland architecture and tools (Mäkeläet al., 2004; Viljanen et al., 2006; Mäkelä et al.,13

2006). The system is based on metadata of differentkind of cultural objects, such as process descriptions,mythic poems, paintings, old photos, artifacts,and educational videos about Finnish culture. Theseresources have been described with matching metadataschemas and shared ontologies, which enablessemantic search and browsing. We introduced a newtype of metadata for describing actions and situationsthat relate cultural objects with each other. The maingoal of this paper is to investigate, how such descriptionscan be used to bring the cultural context closerto the people looking for information in cultural semanticportals.2 Describing the Cultural Contextof ResourcesThe motivation for describing cultural resources semanticallyis to make it easier to search for and automaticallylink culturally related things together. Themore information there is on the semantic web, themore interesting and useful it can be to people. A prerequisiteof this is that the information is easy to find,and that the cross-links from resource to resource arereally semantically insightful and support the user infinding the right information and connections.2.1 Towards Event-based Content DescriptionsInformation that connects to other information is easierto grasp, as for example constructivist learningtheories show (Holmes et al., 2001). In fact, manyphilosophers from Aristotle to Locke and Hume havestated that all knowledge is actually in the form of associations(Eysenc and Keane, 2002). All this leadsus to observe the fact that links can be very useful fora person looking for information about a particularsubject.In our work, actions and processes in the real lifeand fiction were chosen as the key enabler for connectingdifferent resources semantically. There weretwo major reasons for this. Firstly, actions and processesconnect to many kinds of different resources.Human culture is much about action, about peopledoing things. People looking for information are alsolikely to be interested in the theme of action. Actionsand processes often tell us more about the life aroundcultural objects than other points of view. Life, stories,and emotions are things that everyone can easilyconnect to and are interesting for the end-user. Peoplewith varying specific needs are surely interestedin other points of view, too, but generally speakingactions are foundational in structuring our knowledgeabout the world. As a result, event-based representationhave been widely developed and applied in thefields of artificial intelligence and knowledge representation(Sowa, 2000). CultureSampo builds uponthis research tradition and ideas developed within semanticweb research.The second reason for using actions and processesis that the information about processes themselves isvery important to preserve. For example, informationabout cultural processes, such as “how to farmland with the traditional slash burn method procedure”,are important to preserve. By linking other resourcesthrough action, the processes themselves areresearched and the old know-how preserved in digitalformat for future generations (Kettula, 2005).Let us take an example of cultural resources andtheir connection to each other through their culturalcontext and actions. Consider a painting depictingpeople and burning trees. We know that the pictureis a part of an old cultural process, slash and burn,where trees are cut down and burned to make thesoil richer for nutrition of the crop. There is also apoem in the national Kalevala epic 2 , where Kullervo,a tragic hero, is forced to cut down trees in the slashand burn process, but he ends up with cursing thewhole forest. An axe displayed in the National Museumof Finland has been used in western Finland forslash and burn, and there may be other related tools,too. We also have some photos of people doing slashand burn, and an educational video about the subject.All these things can be linked together in insightfulways, if they have been annotated with metadata thattells about what is happening during the slash andburn procedure.To accomplish the needed semantic annotations,we defined a set of domain and annotation ontologies.We then selected a representative set of heterogenouscultural contents of different kinds and annotatedthem with metadata conforming to the designedontologies and annotation schemas. The resultwas a knowledge base in RDF(S) 3 , that was homogenizedbased on the shared action-based knowledgerepresentation scheme of the real world. Afterthis, the view-based semantic search engine and logicalrecommender system of MUSEUMFINLAND wasadapted and applied to the new content set, resultingin the prototype portal CultureSampo I.2 http://www.finlit.fi/kalevala/index.php?m=163&l=23 http://www.w3.org/2001/sw/New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 14

2.2 Stories as ProcessesIn addition to describing general process types, wewanted to be able to describe specific process instances,i.e., situations where something actually happens.This leads us to a very classic content mediumand type: stories. As processes are chains of actions,where something is done, leading to another action,and so on, stories are the same in many ways. Storiescan bring separate facts into life and place in the rightcontexts for the end-user, and in this way give a betterview to the culture that the resources are describing.Furthermore, many cultural content objects, such ashistorical events and biographies, are actually storiesand often relate to other stories.People want to hear stories (Kelly, 1999): they arean ancient method of spending time, educating andhaving fun. Stories bring new points of view to theprocess descriptions, which are on a general level. AsAristotle states in Poetics (Aristotle, 2006), drama isabout action. This suggests that describing actions isa good way to describe drama. A process descriptionjust needs less information, as it is less precise. Fora process description, it may be enough to tell whatis done and in what order. A story needs additionalinformation. For example, it may be important toknow who does what, what are the relations betweenthe actors, and what goes possibly wrong. Based onthis, stories can be seen as a kind of subclass of processes,that need more information and give a richerand more specific description of a situation or happening.Our main motivation to describe stories with metadatais not to represent the stories, but only to makethe actual stories better accessible through bettersearch and links from related resources. A story maybe in textual form or maybe depicted in a painting,a photo or a video. The stories lose much in contentwhen they are reduced to metadata, but since the storyresources themselves can be put on the semantic web,people searching for stories will find and experiencethe real stories.3 The OntologiesThree ontologies were used to create the action-basedmetadata for the resources. First, a situation ontologywas created in order to define how to describe one situationor moment, the smallest unit of a process or astory. Second, a process ontology was designed. Itwas used to put the moments in the right order in relationto each other: what is done first, what followsand so on. The third ontology was the content definitionontology, that included the concepts of the realworld, so that reasoning could be made also based onthe things that appear in the situations. For example,an axe is used in the slash and burn method, so theaxe has its own place in the content definition ontology,as a subclass of tools. Also the actions, the verbs,and their relations to each other are an important partof the content definition ontology.In the following, these three ontologies will nextbe discussed in some more detail. We concentrate onthe situation ontology that was the main focus of theresearch.3.1 The Situation OntologyThe situation ontology (cf. figure 1) is used to describemoments, where someone does something. Itis actually not much of an ontology in the senseof a hierarchical network, but more of a metadataschema implemented with semantic web technology.The idea is to use instantiated situations for semanticbrowsing and search.A situation, as defined in the situation ontology,is the moment starting with someone starting doingsomething, and it ends when the action ends, andanother begins, or there is a jump in time or space.These situations can be anywhere: in a process, astory, a painting, a video or in anything else that representsa situation. There is an actor who does anaction. In the same situation, the actor can have otheractions going on too, and there can be other actorspresent, doing their actions. Apart from the actor andthe action (that together form an event) there is alsothe surroundings, which consists of absolute and relativetime and place, and possibly other elements ofthe situation. Also the mood and theme of the situationcan be annotated.When using the situation ontology, the philosophyis to leave the properties open if they don’t existor one doesn’t know them. The ontology is meantmostly to be used for creating links and for helping insearch.Culture, for example art, is often much about interpretations(Holly, 1984). This means that whenbringing culture resources available to people on thesemantic web, interpretations cannot be left out of thesystem. For example, consider the painting “Kullervodeparts for the war” in figure 2 depicting an eventin Kalevala. In this case the interpretation that thepainting is in particular about Kullervo (and not aboutsome unknown man) departing for the war in Kalevalain order to take revenge on his enemies is of importance.Therefore we have made it possible to dis-15

Figure 1: The situation ontologyFigure 2: Kullervo departs for the war. A painting at the Finnish National Gallery. On the right, the original keywordsdescribing the content are given as the value of the CIDOC CRM (Doerr, 2003) property P129F is about:Kalevala, event, Kullervo, birch bark horn, sword, ornament, animal, horse, dog, landscape, winter, sky, stars,snow.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 16

tinguish between factual information and interpretationsin the annotations. Interpretations can be shownin a different way in the user interface or even left out,if the person looking for information wants to avoidinterpretations.3.1.1 Describing ActionIn one situation, there can be many events. An eventrefers to one whole, that consists of an actor, anaction,anobject of action and an instrument. The actor,action and instrument are quite straightforward intheir semantics, but the object of action can vary dependingon the situation. The object can vary frombeing an object (e.g., hitting a ball), or a predicate(e.g., try to get up) to being an adverbial (e.g., sit on acouch). Because of this, the object of action is somewhatsemantically ambiguous and cannot be used soeasily for machine reasoning. However, when peopleuse the system, the natural meaning of the relationscan usually be understood easily by the user.The action can also have an objective. The objectiveis an important part of the event, as what a persondoes may tell much less about what he is actuallydoing than knowing the objective. In a generic processdescription, the objective isn’t used, as there itis taken for granted that doing one part of the processalways has the objective of getting to the nextpart of the process. But when we look at stories,the objective becomes important. If we have a paintingwhere a man is riding a horse, it is important toknow that he is not only riding, but actually his objectiveis to depart for the war. We may know thisobjective from the name of the painting, like in thecase of the painting “Kullervo departs for the war”.When the objective has been annotated, this paintingis found also when looking for information about war,not only about horses and riding.The question about interpretation is quite apparentin the case of the objective. If the painter of “Kullervodeparts for the war” would have given the name asan ironic joke for example, it would already requiresome knowledge about this to not make the wrongannotation. The same is true if the painting wouldnot have a name at all, but still the objective is obviousby looking at the picture: there is a man with asword on his side, he’s blowing a big horn, etc. Especiallywhen describing pictures much of metadata isbased on the interpretations of the annotator. In textualstories, the objectives can actually be explainedin the text.Objectives can also exist on many levels. When weread the story about Kullervo in Kalevala we can seethat the objective of him to go to the war is actually totake revenge on the people who killed his family. Sohe is riding to get to war to get revenge. That’s threelevels already, and more can sometimes be found.3.1.2 Describing the SurroundingsEven though our main focus in this research is on theaction and the processes, sometimes the surroundingsof the situation are also interesting. They should notbe left out of the metadata description of the situation.In the situation ontology, the surroundings ofthe situation are described by the time and the placeof the situation (cf. figure 1). It may be important toknow, where and when something happens. This canbe relevant to generic process descriptions as well asto story situations.Time and place can be modeled both on the level ofabsolute facts and from the relative point of view. Forexample, a situation described in a news article mayhave the absolute time (date) September 11th 2001,and the absolute place (location) New York. However,in many kinds of resources the absolute timeand place are not known. For example, generic processdescriptions have no absolute time or place, andmany stories lack these facts, too. Thus the relativetime and space are important. Relative time describesthe time of year and time of day of the situation, andrelative place (space) describes the surroundings on amore general level. The relative time and space of theexample above could be “in autumn”, “in daytime”,and “in a city”. When describing the slash and burnprocedure, one could say that the “cutting down thetrees” -situation happens in spring time in a forest.The duration of the situation can also be modeled bycreating an instance of the class Time.3.2 More FeaturesApart from modeling events in time and space, thesituation ontology allows the annotation of a theme,a mood and other elements. The theme and the moodare properties that have to do with all the non-genericsituations, like parts of a story or a painting. Thetheme reflects the meaning of the situation from anintellectual point of view, whereas the mood reflectsthe emotional mood of the situation. The elementsmean any objects or things of interest in the situation,that are not directly involved in the events, but are stillsomehow important to mention. These properties addto the flexibility of the model, even though they don’thave to be used, and especially the two former needlots of interpretation from the annotator.To further deepen the model, we have includedsome more features in a wrapper class called Situa-17

tionElement. All the instances of the properties mentionedin the ontology are wrapped inside this class.In addition to the link to the content definition ontology,this class has the possibility to mark the informationinside as an interpretation. With this informationexplicitly shown, the part of the information thatis most subjective can be shown to the user in a differentway in the user interface, or it can be left outif someone chooses to only be interested in definitefacts.There is also a property called attribute in theSituationElement-class. The annotator can add attributesto any elements of a situation. This maybring interesting additional information about someelements and can be used for answering questionssuch as ”How?” or ”What kind of...?”. Still anotherproperty is symbolizes that makes it possible to expressthe fact that something symbolizes somethingelse. As a last addition, we have added the propertyfreeText, so that everything annotated can alsobe given a literal value that can be used in the userinterface.3.3 The Process OntologyThe situation ontology can be used to describe theparts of a process or a story, but there has to be a logicfor binding situations together in order to form largeprocesses or stories. The process ontology was createdfor this. The process ontology was strongly influencedby the OWL Services (Coalition, 2003) processespart. The atomic processes of OWL-S could beseen as a close relative to the situations in our terminology,as these are the units used to build up biggerwholes. It was needed for easy specification of the orderingof the situations. The class ControlConstructwith its subclasses made this possible.Some problems arose when annotating the order ofthe situations with Protege-2000 4 because this editordid not support the use of ordered lists, even thoughthey are a part of the RDF specification. Some hardcodingwas needed to get around this problem, takingaway the possibility to reuse subprocesses as a partof other processes, but this can probably be solved inlater versions of the ontology.3.4 The Content Definition OntologyThe content definition ontology is the ontology thatdefines how things in the world relate to each other.The actions, the objects, the places, indeed everythingthat is present in situations that are being annotated,4 http://protege.stanford.edushould be defined in a this large ontology that tellswhat are the semantic connections between the concepts.As the processes are made up of situations, the situationsare made up of different properties of the situation.All these properties have a value, that is a referenceto a resource of the content definition ontology.For example, consider the situation where the treesare cut down in the slash and burn method. In theaction-property of the situation, there will be a referenceto the concept of cutting down, and the object ofaction is a reference to the concept of tree. With allthe concepts and there relations defined, we can latercreate links between different processes that have todo with trees, even though in another process or situation,the annotation would have been oaks. Theontology tells the computer that oaks are a subclassof trees.3.5 Semantic Difficulties EncounteredSome theoretical problems of using these ontologieswere discovered during the ontology design process.One was that of interpretation. Even the same situationcan be seen in many ways, depending on thepoint of view of the annotator. Things get even moresubjective when properties such as mood are used.Still, we wanted to include these possibilities, as thismetadata is meant primarily for people and not formachines, that can’t as easily cope with the fact thatall facts are not really facts. We envision that interpretationsof cultural resources can be very interesting tomany end-users. It would be too a high price to payfor making the data totally fact-based.Another interesting dilemma encountered was therelation between the thing being modeled and reality.What is the relation between a story characterand a real historical character? There may be almostno differences, but the fact that the other one is onlyfictional. There is a relation between Alexander theGreat and Aragorn of the “Lord of the rings”, butshould the difference of the fictional world and thereal world be emphasized or not?The same problem exist with for example toys. Atoy aeroplane has lots of semantic connections to realaeroplanes, but still it’s a totally different thing. Andif there is an old tractor somewhere, that children useas a playing ground, should it be annotated as a tractor,a playing ground, or a space ship, as the childrenalways think of it as one?The impact of changes in time and space in themetadata and ontologies is also an interesting question.For example, an old axe may have been used in aNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 18

medieval war, but in modern wars axes are rarely usedas weapons. Thus the war related properties are onlyrelevant to axes during a certain time in history. Axesare an easy example because axes have been fairlysimilar objects all through the history. In contrast,some concepts may mean a totally different thing indifferent eras. For example, the process of healinga wound with modern technology and knowhow isvery different from what it was a thousand years ago.Also a change in place or culture can twist the meaningsof some concepts very much. Depending on theviewpoint, these differences can be seen either as aproblem or as a source of most interesting and complicatedsemantic connections.4 CultureSampo I PrototypeIn order to test whether the actions and processes indeedprovide a useful basis for linking different culturalresources, the theory has to be tested. The CultureSampoI prototype was built on top of the MuseumFinlandarchitecture to test the new point of view,and especially to see how different kinds of resourcescould be linked together.The content of MuseumFinland was semanticallyhomogenous, conforming to a shared metadataschema, consisting of mainly artifact metadata originatingfrom heterogenous and distributed museumcollections. In contrast, CultureSampo aims at publishingFinnish culture content of many kinds, conformingto different metadata schemas, on the semanticweb. The idea is to allow the search of the resourcesthrough the actions they are involved in orinvolve, and provide the end-user with automatic semanticlinking of the resources based on the content(Hyvönen et al., 2006).4.1 GoalsCultureSampo shares the main goals of MuseumFinland:1. Global view to distributed collections. It is possibleto use the heterogeneous distributed collectionsof the museums participating in the systemas if the collections were in a single uniformrepository.2. Content-based information retrieval. The systemsupports intelligent information retrieval basedon ontological concepts, not only simple keywordmatching as is customary with currentsearch engines.3. Semantically linked contents. A most interestingaspect of the collection items to the end-userare the implicit semantic relations that relate collectiondata with each other. In MuseumFinland,such associations are exposed to the end-user bydefining them in terms of logical predicate rulesthat make use of the underlying ontologies andcollection metadata.4. Easy local content publication. The portalshould provide the museums with a costeffectivepublication channel.CultureSampo I prototype was designed especiallyto deepen the third goal of the system, by bringingin the possibility to relate different kinds of culturalresources through semantically richer annotations,based on actions and processes. The need forthis became evident when making MuseumFinlandand was actually also echoed in the end-user feedbackof MuseumFinland. One of the feedback e-mails brought us this wish:“Are there any plans to give more detailed informationabout the objects? Now the items are laidon display: here they are, look at them. But it’s nottold to what, how and why the artifacts have beenused....This kind of extra information would serve meat least. Some of the objects and their use is familiarto me, but not all.”Also the professionals in the museum field are ponderingthese questions. Trilce Navarrete writes thefollowing (Navarrette, 2002):“As museums expand the definition of ’education’there is a need to consider alternative models of explanation,such as oral history, mythology, familyfolklore and other ways to create the context in whichstories are told — the story of each museum. How domuseums go about fostering their community’s narrativeconstruction? The question in museums is notonly how to better explain the story of the given object,but also how can museums better create and inspirethe context for the public to construct an interestto relate to these stories?”These two quotes outline nicely the field of problemsthat we tried to approach with the help of CultureSampoI, using the ideas described earlier in thispaper.4.2 A Use CaseTo illustrate the intended usage of CultureSampo,consider the following use case. John is a 12-yearoldpupil of a school in Helsinki. The teacher hasgiven him and his friend Michael the task of doing an19

exercise together. John and Michael should do theirwork about agriculture in Finland in the nineteenthcentury. The work should be broad in content, andcontain pictures and other materials, too.The objective of John is to get information aboutagriculture in the nineteenth century. Michael wantsto write about the farm animals, so John has to considerthe part about agriculture and people’s lives ingeneral. The teacher gives John a link to Culture-Sampo to find some information. John goes to thesite, looks at the search window, ends up with choosingthe search categories “Agriculture” and “Thenineteenth century”, and he gets a link to a scathethat has been used in Helsinki in the nineteenth century.He copies the picture, it’s good material. Thenhe finds a link to the slash and burn process, whichis a process where a scathe has been used. He getsinformation about this kind of farming, and decidesto write about that. He also finds a link to a paintingwhere a mythic figure from the Finnish epic Kalevalais using the slash and burn method, and from that hefinds a link to the right point in the poem itself, andgets exited about the dramatic twists. The poem alsogives him some insight to the life in ancient Finland,and he writes about the jobs he finds people were doingin the poem, like shepherding and fishing. On thebasis of the information of CultureSampo, John andMichael manage to make a good school essay aboutagriculture.4.3 Illustration of Semantic LinkingFigure 3 illustrates semantic linking in CultureSampoI by a screenshot. The page shows metadata about thepainting “Kullervo curses” by Akseli Gallen-Kalleladepicting a famous event in Kalevala. The paintingitself is shown on the left and its metadata in the middle.The semantic links appear on the right. They includedifferent kinds of links, depending on the logicalrules that have created them. The labels of thelinks and headlines explain why following this linkshould be of interest of the user. Three semantic recommendationlinks created by the system are visualizedon top of the screenshot. One link points toan artifact in the collections of the National Museum,one to bibliographic information of the artist, and oneto the actual point in Kalevala where the event takesplace. Different kinds of content items have pages oftheir own similar to this painting page. They look differentbecause of the differences in content types andcontent on them, but in all cases the content item itselfis shown on the left, the metadata in the middle,and the semantic links on the right.In addition to semantic browsing, the system alsosupports faceted semantic search in the same way asMuseumFinland.4.4 Cultural ContentIn the prototype seven kinds of content resourceswere used. We selected resources that were somehowimportant for the Finnish culture, and that related toeach other, so that links could emerge between themif described correctly with the annotation ontologies.We also wanted to have examples of different mediumspresent in the resources.1. Processes were one of the core things we wantedto have in the system, as they have a lot to dowith action and with culture. Preserving informationabout cultural processes is also importantin itself. We had the slash and burn procedureand seine fishing as examples of old Finnish processesthat are not any more very well known tomodern Finns. These two processes were builtup from about ten situations each.2. Stories were the other resource type that hadbeen much thought of when designing the CultureSampoontologies, and is an inspiring wayto describe the context of old Finnish culture.Kalevala was a natural choice when choosingwhat stories to include in the prototype. As theKalevala stories are old, the world they describeis close to the context of life in the past agrarianFinland. Two quite separate entities were chosenfrom the whole epic: the sad story of Kullervowhose family was killed, and another where themost famous hero of Kalevala, Väinämöinen,learns how to farm land by the slash and burnmethod. These stories had a start and an end oftheir own, and they had good references to theother resources we had.3. Paintings were the first resource where no textwas included, but the information was in pictureform. It was good to add paintings not only becauseof getting to test describing this medium,but also as visual resources give another kind oflife to the context of old Finnish culture. Weselected some paintings about Kalevala to fit thestories we had, and also some other paintings describingthe old days.4. Photographs are quite similar to paintings w.r.t.their content annotation. Still, the documentaryaspect of photographs is much stronger than inNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 20

Figure 3: An example of the interface of CultureSampo I illustrating a painting at the Finnish National Galleryabout an event in the epic Kalevala. Three semantic recommendation links created by the system are visualizedon top of the screenshot.paintings, so we believe they bring a good authenticpoint of view to old processes.5. Videos were an interesting and complicated caseof resources. At this stage of the process, wedidn’t yet implement very complicated metadatadescriptions of the videos we had, even thoughusing the situation ontology with moving imageand sound could be used in a more sophisticatedway. At this time it was enough to get some linksto the other resources through a simple metadatadescription, and the logics can later be improved.6. People were added as one kind of “resources”,as information about people who lived in the olddays can be relevant, when forming the contextof cultural resources. Since the core of our metadataschema is action, and the actor’s are usuallypeople, it was natural to use people as contentitems. Most of the persons in the system wereartists who had created the artworks included.However, there were also a couple of fictitiouspeople from the Kalevala epic.7. Artifacts were a natural kind of resource weneeded to have, because artifacts are importantin actions (e.g., as instruments), and we also hadlots of metadata about them in the MuseumFinlandknowledge base.The CultureSampo I prototype uses ontologies designedfor action-based description of cultural resources.The resources we used in the prototype were2 processes, which were built from 21 situations, 2stories of Kalevala in 32 situations, 8 paintings, 4photographs, 2 educational videos and 14 museumobjects. The annotation of the situations was quiteslow, as there were no existing action-based metadataavailable, and no special tools designed to help in theprocess. Most situations could be described in theway we wanted, so the expressive power of the situationontology proved out to be satisfactory.A couple of problems arose in describing the situationsof stories. One was a situation where a womanreads a spell to protect the cows when sending themto the forest. This was a long monologue, and thecontents of the monologue could not be made into situations,but we could only say the woman is readinga spell. This shows a problem of describing speechin stories and in general: the things that people talkabout are not necessarily true or real actions. In our21

scheme there is no notion of reification, although thisis in principle supported by the RDF recommendation.As a result, dialogues or monologues are difficultto describe with our action-based annotations asit is. When describing some parts of the story, weneeded to limit the level of detail. All the actionswere not described in situations, but only the onesthat were interesting and mattered in the whole plot.In describing processes, there were some problemswith representing time. As different situations of aprocess take different amounts of time, choosing whatis important in the process sometimes ended up withsituations that are very uneven in their lengths. Forexample, cutting the trees down is fast but waitingfor a year or letting the forest grow back take longertimes.These problems were not critical to the functionalityof the system. Most of the situations in differentresources were straightforward to annotate withthe ontologies. When using the prototype, it couldbe seen that actions really do link different culturalresources together. Lots of links were created withthe logical rules we defined, and the links really hadmeaning for the end-user. When looking at an axein a museum, you get a link to the process of slashand burn where axes were used, to a painting wheresomeone is using an axe, and to a poem of Kalevalawere Kullervo is sharpening his axe.The situation ontology was the main object of ourinterest, but also the other ontologies worked wellenough. The process ontology was expressive enoughto describe the order of the situations we had. Asthe content defining ontology we used a preliminaryversion of the General Finnish Upper Ontology YSO(Hyvönen et al., 2005b), that was being designed atthe same time. As it wasn’t totally ready, we made aversion of it that had the concepts we needed, and itworked at this stage of the process.5 Related WorkProcess models have been researched a lot in computerscience, though usually from the point of viewof machine-executable processes or business processes.As examples of these include the ProcessSpecification Language (Schlenoff et al., 2000), BusinessProcess Execution Language for Web Services(Andrews et al., 2003) and OWL Services (Coalition,2003), that we used as a starting point when designingour process ontology. However, these approachesto process modeling were not designed for describingcultural processes, so the end results are quite different,even though some similarities exist. Questionsabout actions happening in time are always somehowinvolved in processes.There are ontologies that have been designed fordescribing cultural resources. The CIDOC CRM(Doerr, 2003) is an upper level ontology that has beendesigned for making different metadata schemas andcontent interoperable. The ABC Harmony (Lagozeand Hunter, 2001) is a related effort intended to providea common conceptual model to facilitate interoperabilityamong application metadata vocabularies.Our approach is different from these ontologies andother metadata systems that concentrate on describingthe cultural resources in collections. Our focus ison describing the actions and processes that relate theresources in the real world (or in fiction), with goalof providing the user with insightful semantic recommendationsand enhancing search.6 ConclusionsThis paper is about describing the cultural context ofresources. As the common denominator of differentcultural resources, a situation including some actionswas chosen. Situations describe what happens in apainting, in a story, or in a cultural process. Such descriptionsprovide a semantic network linking relatedcultural content items together. This point of viewcomplements the more traditional idea of using classificationsand ontologies for defining cultural conceptsand metadata schemas.An experimental situation ontology was created toprovide the common annotation scheme to describesituations and their main properties, whatever themedium in which the situations appear is. Also a processontology and a content defining ontology weretaken as a part of model, in order to link different situationstogether into larger wholes, and in order todefine the concepts appearing in the situations unambiguously.It was clear from the beginning that describing culturalcontent in terms of real world actions and processesis more complicated than describing culturalobjects in collections. The prototype CultureSampoI was designed and implemented in order to find outboth the practical and theoretical problems of the approach,and also to see if the benefits of this kindof metadata in use could be as useful as we hoped.The first results seem promising. However, the contentset used in this experiment was very small and itwas pre-selected according to our goals. Experimentswith larger content sets and annotations are definitelyneeded.The final goal of the CultureSampo projectNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 22

(Hyvönen et al., 2006) is be become a demonstrationof a nation-wide cross-domain cultural publicationfor the semantic web. As a first step towardsthis ambitious goal, the work of this paper shows thata situation ontology and action-based metadata descriptionscan bind together different kind of culturalresources in meaningful ways, from paintings and objectsto processes and stories.ReferencesT. Andrews, F. Curbera, H. Dholakia, Y. Goland,J. Klein, F. Leymann, K. Liu, D. Roller, D. Smith,S. Thatte, I. Tricovic, and S. Weerawarana. Businessprocess execution language for web services -version 1.1, 2003.Aristotle. Poetics. Focus Philosophical Library,Pullins Press, 2006.The OWL Services Coalition. OWL-S: SemanticMarkup for Web Services, November 2003.http://www.daml.org/services/owl-s/1.0/owl-s.pdf.M. Doerr. The CIDOC CRM - an ontological approachto semantic interoperability of metadata. AIMagazine, 24(3):75–92, 2003.M. Eysenc and M. Keane. Cognitive PsychologyA Students Handbook. Psychology Press, Exeter,UK, 2002.M. Holly. Panofsky and the foudations of art history.Cornell University Press, Ithaca, 1984.B. Holmes, B. Tangney, A. FitzGibbon, T. Savage,and S. Meehan. Communal constructivism: Studentsconstructing learning for as well as with others.In Proceedings of SITE 2001, Florida, 2001.E. Hyvönen, E. Mäkela, M. Salminen, A. Valo,K. Viljanen, S. Saarela, M. Junnila, and S. Kettula.MuseumFinland – Finnish museums on the semanticweb. Journal of Web Semantics, 3(2):224–241,2005a.E. Hyvönen, T. Ruotsalo, T. Häggström, M. Salminen,M. Junnila, M. Virkkilä, M. Haaramo,T. Kauppinen, E. Mäkelä, and K. Viljanen. CultureSampo— Finnish culture on the semantic web.The vision and first results. In Semantic Webat Work - Proceedings of the 12th Finnish ArtificialIntelligence Conference STeP 2006, Volume 1.,Nov 2006.E. Hyvönen, A. Valo, V. Komulainen, K. Seppälä,T. Kauppinen, T. Ruotsalo, M. Salminen, andA. Ylisalmi. Finnish national ontologies for thesemantic web - towards a content and service infrastructure.In Proceedings of International Conferenceon Dublin Core and Metadata Applications(DC 2005), Nov 2005b.M. Junnila. Tietosisältöjen semanttinen yhdistäminentoimintakuvausten avulla (Event-based approachto semantic linking of data content). Master’s thesis,University of Helsinki, March 6 2006.L. Kelly. Developing access to collections through assessinguser needs, May 1999. Museums AustraliaConference, Albury.S. Kettula. Käsityöprosessit museossa semanttisentiedonhaun lähteenä ja kohteena. (Handicraft processesin museum as a source of object for semanticcontent and search). In L. Kaukinen and M. Collanus,editors, Tekstejä ja kangastuksia. Puheenvuoroijakäsityöstä ja sen tulavaisuudesta. (Viewsof handicraft and its future). Artefakta 17, JuvenesPrint, 2005.C. Lagoze and J. Hunter. The ABC Ontology andModel. Journal of Digital Information, 2(2), 2001.E. Mäkelä, E. Hyvönen, and S. Saarela. Ontogator— a semantic view-based search engine service forweb applications. In Proceedings of the 5th InternationalSemantic Web Conference (ISWC 2006),Nov 2006.E. Mäkelä, E. Hyvönen, S. Saarela, and K. Viljanen.Ontoviews – a tool for creating semantic web portals.In Proceedings of 3rd International SemanticWeb Conference (ISWC 2004), Hiroshima, Japan,November 2004.T. Navarrette. Museums in the street: Cultural creationin the community, October 2002. INTER-COM Conference Leadership in Museums: Areour core values shifting?, Dublin, Ireland.M. Salminen. Kuvien ja videoiden semanttinensisällönkuvailu (Semantic content description ofimages and videos). Master’s thesis, University ofHelsinki, May 2006.C. Schlenoff, M. Gruninger, F. Tissot, J. Valois,J. Lubell, and J. Lee. The process specificationlanguage(psl): Overwiev and version 1.0 specification,2000. NISTIR 6459, National Institute ofStandards and Technology, Gaithersburg, MD.23

J. Sowa. Knowledge Representation. Logical,Philosophical, and Computational Foundations.Brooks/Cole, 2000.K. Viljanen, T. Känsälä, E. Hyvönen, and E. Mäkelä.Ontodella - a projection and linking service forsemantic web applications. In Proceedings ofthe 17th International Conference on Databaseand Expert Systems Applications (DEXA 2006),Krakow, Poland. IEEE, September 4-8 2006.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 24

CultureSampo—Finnish Culture on the Semantic Web:The Vision and First ResultsEero Hyvönen, Tuukka Ruotsalo, Thomas Häggström,Mirva Salminen, Miikka Junnila, Mikko Virkkilä, Mikko Haaramo,Eetu Mäkelä, Tomi Kauppinen, and Kim Viljanen⋆ Semantic Computing Research GroupHelsinki University of Technology (TKK), Laboratory of Media TechnologyUniversity of Helsinki, Department of Computer Sciencehttp://www.seco.tkk.fi/first.last@tkk.fiAbstractThis paper concerns the idea of publishing heterogenous cultural content on the Semantic Web. Byheterogenous content we mean metadata describing potentially any kind of cultural objects, includingartifacts, photos, paintings, videos, folklore, cultural sites, cultural process descriptions, biographies,history etc. The metadata schemas used are different and the metadata may be represented at differentlevels of semantic granularity. This work is an extension to previous research on semantic culturalportals, such as MuseumFinland, that are usually based on a shared homogeneous schema, such asDublin Core, and focus on content of similar kinds, such as artifacts. Our experiences suggest that asemantically richer event-based knowledge representation scheme than traditional metadata schemasis needed in order to support reasoning when performing semantic search and browsing. The newkey idea is to transform different forms of metadata into event-based knowledge about the entitiesand events that take place in the world or in fiction. This approach facilitates semantic interoperabilityand reasoning about the world and stories at the same time, which enables implementation ofintelligent services for the end-user. These ideas are addressed by presenting the vision and solutionapproaches taken in two prototype implementations of a new kind of cross-domain semantic culturalportal “CULTURESAMPO—Finnish Culture on the Semantic Web”.1 Towards Semantic CrossdomainInteroperabilityA widely shared goal of cultural institutions is to providethe general public and the researchers with aggregatedviews to cultural heritage, where the usersare able to access the contents of several heterogenousdistributed collections of institutions simultaneously.In this way, the organizational and technicalobstacles for information retrieval between collectionsand organizations, even between countries andlanguages could be crossed.Content aggregation may occur at the syntactic orsemantic level. The basis for syntactic interoperabilityis sharing syntactic forms between differentdata sources, i.e., the metadata schemas such as theDublin Core Metadata Element Set 1 or the Visual Re-1 http://dublincore.org/documents/1998/09/dces/source Association’s (VRA) Core Categories 2 . Suchschemas make it possible to identify different aspectsof the search objects, such as the “author”, “title”, and“subject” of a document, and focus search accordingto these. Syntactic interoperabilty facilitates, for example,multi- or metasearch 3 . Here the user types ina query in a metaportal. The query is then distributedto a set of underlying systems and the results areaggregated for the end-user. For example, the AustralianMuseums and Galleries Online 4 and ArtefactsCanada 5 are multi-search engines over nation-widedistributed cultural collections. Here the content includesmetadata about museum artifacts, publicationsetc. represented using shared metadata schemas.Content aggregation at the semantic level meansthat not only the form of the data is shared and in-2 http://www.vraweb.org/vracore3.htm3 http://en.wikipedia.org/wiki/Metasearch engine4 http://www.amonline.net.au/5 http://www.chin.gc.ca/25

teroperable, but also the values used in the metadataschema, and that the meanings of the values are semanticallydefined in terms of ontological structures.The values of metadata fields, such as authors, materialtypes, and geographical locations are taken froma set of shared vocabularies, i.e., ontologies, or if differentvocabularies are used, then the mappings betweenthem are available. At this level of content aggregation,reasoning about the ontological relationsbetween content items becomes possible enablingsemantic search, semantic browsing, recommendations,explanations, and other “intelligent” services.A prototypical example of this approach is the portal“MUSEUMFINLAND—Finnish Museums on the SemanticWeb” 6 (Hyvönen et al., 2005a), where distributed,syntactically heterogeneous museum collectiondatabases are integrated by a set of seven sharedontologies, and semantic search and browsing servicesare provided to end-users based on the aggregatedknowledge base.Another distinctive feature between cultural contentaggregation systems is whether they deal withmetadata that conforms to a single metadata schemaor multiple schemas. For example, the Helmet librarysystem 7 aggregates public library collectionsof neighboring cities for the public by using a singlemetadata format. In the same vein, an online recordshop may deal with CD/DVDs whose metadata is representedin a homogeneous way. On the other hand,in a system such as Artefacts Canada, the underlyingdatabases contain items of different kinds, suchas art, furniture, photos, magazines etc. whose metadataconform to different schemas. For example, ametadata field representing physical the material ofan object is essential for a piece of furniture or artifactbut not for a publication.Semantic web portals have tackled the problem ofsemantic interoperability usually by sharing metadataschemas. For example, in MUSEUMFINLAND heterogeneousartifact collection databases were madesemantically interoperable, but the content was of asingle domain (artifacts), and the metadata was basedon a single, Dublin core like schema of artifacts.There are paintings and some other forms of art inMuseumFinland collections, but they have been catalogedas pieces of artifacts in the cultural museumsparticipating in the portal, and not as pieces of art.The reasoning routines were based on the annotationschema and the ontologies.In this paper we investigate the problem of seman-6 This application is operational at http://www.museosuomi.fiwith a tutorial in English.7 http://www.helmet.fitic cross-domain interoperability, i.e. how contentof different kinds conforming to multiple metadataschemas could be made semantically interoperable.The focus is the cultural domain and content typesstudied in our work include artifacts, paintings, photographs,videos, audios, narratives (stories, biographies,epics), cultural processes (e.g., farming, boothmaking), cultural sites, historical events, and learningobjects. In this case, the content is cross-domain innature and, as a result, comes in forms that may bequite different from each other. Mapping them intoa Dublin Core like generic metadata framework isproblematic. Instead, we propose content aggregationat a semantically more foundational and rich levelbased on events and thematic roles (Sowa, 2000). Theresearch is being carried out not only in theory, but byimplementing real portal prototypes. More specifically,we show how the idea of MUSEUMFINLANDcan be extended into a cross-domain semantic culturalportal called “CULTURESAMPO—Finnish Cultureon the Semantic Web”. Figure 1 illustrates thepositioning of CULTURESAMPO along the distinctionsdiscussed above and its relation to some otherportal systems mentioned.Multi-domainSingle-domainArtefacts CanadaHelmet librarysystemSyntacticinteroperabilityCultureSampoMuseumFinlandSemanticinteroperabilityFigure 1: Portals can be classified in terms of thenumber of metadata schemas used (vertical axis) andthe level of interoperability (horizontal axis).In the following we first state the vision and goalsof creating CULTURESAMPO. After this problemsof obtaining semantic cross-domain interoperabilityare discussed and the idea of using event-based descriptionsis proposed as a solution. The discussion isbased on experiences gathered in creating two experimentalprototypes of CULTURESAMPO. In conclusion,contributions of the work are summarized anddirections for further research are proposed.2 The Vision and Goals of CultureSampoCULTURESAMPO shares the general goals ofMUSEUMFINLAND:New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 26

Global view to heterogeneous, distributed contentsThe portal supports the usage of heterogenousand distributed collections and contents of theparticipating organizations as if there were asingle uniform repository.Intelligent end-user services The system supportssemantic search based on ontological conceptsand semantic browsing, where semantic associationsbetween search objects are exposed dynamicallyto the end-user as recommendationlinks with explicit explanations. These links aredefined in terms of logical rules that make use ofthe underlying ontologies and collection metadata.Shared content publication channel The portalshould provide the participating organizationswith a shared, cost-effective publicationchannel.CULTURESAMPO focuses, from the content perspective,especially on material related to the “GoldenEra” of the Finnish culture in the 19th century. Duringthis period the notion of Finland as a nationwith an original cultural background and history wasformed, and the development resulted in the independenceof the country in 1917. 8 A central componentof the Finnish cultural heritage has been thenational epic Kalevala 9 . It was published originallyin 1835 and has been translated into some 60 languages.This epic, based on large collections of folklore10 collected especially in the eastern parts of Finland,Karelia, has been a continuous source of inspirationin Finnish fine arts, music, sculpture, literature,and other branches of culture. The world of Kalevalaalso nicely relates to the original agrarian Finnish lifeand artifacts that are available in museums.In CULTURESAMPO the Karelian culture is centralalso because one goal of the work is to reuniteKarelian collections using semantic web techniques.These collections have now been distributed in severalmuseums due to the result of the World War IIwhere eastern parts of Finland were annexed to theSoviet Union. The semantic web provides means forre-uniting cultural entities virtually on the semanticweb. The problem of distributed cultural heritage dueto wars and other reasons is very common in Europe.We envision, that the ideas and techniques developedin CULTURESAMPO could later contribute to creation8 Before that Finland had been a part of Sweden (until 1809)and Russia (1809-1917).9 http://www.finlit.fi/kalevala/index.php?m=163&l=210 http://www.finlit.fi/english/kra/collections.htmof cross-national and multi-lingual cultural portals, akind “CultureEurope”.The system will also contribute, in a sense, to thetradition of Kaleva translations. It provides first excerptsof Kalevala translated, not into natural languagesfor the humans to use but for the machine to“understand” in the formal languages of the semanticweb, RDF and OWL. 11The latter part of the portal name “Sampo” is thename of the mythical machine-like entity of the Kalevalaepic. Sampo gives its owner power, prosperity,everything, but its actual construction and natureis semantically ambiguous and remains a mystery— tens of academic theories about its meaninghave been presented. CULTURESAMPO adds still anothermodern interpretation of what a “Sampo” couldbe based on the semantic web.3 Making a Cultural Portal MoreIntelligentA major focus of our work in CULTURESAMPO isto study how to provide the end-user with intelligentsearch and browsing services based on semanticallyrich cross-domain content originating from differentkind of cultural institutions. For example, considerthe painting “Kullervo departs for the war” in figure2 depicting an event in Kalevala. From the endusers’viewpoint, it could probably be interesting, ifthis piece of art could be linked with other paintingsand photos, with the war theme in art museums, withweapons and accessories in cultural museums, withacademic studies about Kalevala and Kullervo, withinformation about dogs and horses in the museum ofnatural history, with other (external) information onthe web about Kullervo, with the actual poem in Kalevalaand related pieces of folk poetry, with moviesand videos on a media server, with biographies of theartist, and so on. An interesting line of associationscould be created by considering the events, processes,and the Kalevala story that takes place in the picture.In this way, for example, the painting could be linkedwith content concerning the next or previous events inthe Kalevala story. Such associations and viewpointscould be insightful, useful, and even entertaining bothwhen searching for content and when browsing it.To investigate and test the feasibility of this ideain practise, we are extending the portal MUSEUM-FINLAND into CULTURESAMPO by a sequence ofnew prototypes. In 2005, the first prototype tobe called “CULTURESAMPO I” was designed and11 http://www.w3.org/2001/sw/27

Figure 2: Kullervo departs for the war. A painting at the Finnish National Gallery.Figure 3: The painting “Kullervo cursing” and its metadata from the Finnish National Gallery.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 28

implemented (Junnila et al., 2006; Junnila, 2006;Salminen, 2006). Figure 3 depicts a painting andits metadata in CULTURESAMPO I. The metadatashown originates from the Finnish National Gallery 12and describes the subject of the painting in thefollowing way: First, the CIDOC CRM 13 (Doerr,2003) property P129F is about lists the followingset of keywords (in Finnish): “Kalevala”, “event”,“Kullervo”, “knife”, “bread”, “knapsack”, “robe”,“animal”, “dog”, “forest”, “rowan”, “pine”, “lake”,and “mountain”. Second, the property “Iconclass”lists a set of ICONCLASS 14 (van den Berg, 1995)notations (categories) describing the subject. Thisdescription is partly redundant with the Finnish keywords.In figure 4 this painting is viewed in CULTURE-SAMPO I. On the right column, a set of semanticlinks to other search objects are recommended withexplanations created by the logical linking server Ontodella(Viljanen et al., 2006). The figure illustrates alink to a knapsack in the collections of the NationalMuseum of Finland 15 , a link to a biography of theartist, and a link to the point in the Kalevala epicwhere the event of the painting actually takes place.CULTURESAMPO I was implemented using thesame framework as MUSEUMFINLAND, i.e., the OntoViewsframework (Mäkelä et al., 2004) includingthe view-based semantic search engine Ontogator(Mäkelä et al., 2006) and Ontodella (Viljanen et al.,2006). However, in this case much richer crossdomainmetadata was used. The test material waslimited in size but included examples of artifacts,paintings, photos, videos, biographical information,and narratives such as poems of Kalevala, and descriptionsof traditional agrarian processes, such asfarming by the slash and burn method.During this experiment we identified two major obstaclesfor creating cross-domain semantic culturalportals:Semantic Interoperability of metadata schemas.The problem of integrating metadata schemasoccurs 1) horizontally when integrating schemasof different form semantically and 2) verticallywhen integrating content annotated at differentlevels of granularity.Expressive power of metadata schemas. A centralresearch hypotheses underlying CULTURE-SAMPO is that, from the end-user’s viewpoint,12 http://www.fng.fi13 http://cidoc.ics.forth.gr/14 http://www.iconclass.nl15 http://www.nba.fi/en/nmfdifferent processes and events that take placein the society and history should be used asa kind semantic glue by which “insightful semanticlinks” could be created for the user tobrowse. This idea was already tested to some extentin MUSEUMFINLAND by creating an artificialevent view for the end-user, and by mappingcontents of it using logical rules. However, itseemed that a richer and a more accurate knowledgerepresentation method was needed in annotatingthe contents than traditional metadataschemas.In the following, our approaches to addressingthese problems are outlined.4 Semantic Interoperability ofMetadata SchemasRe-consider the figure 2. Its metadata may tell e.g.that this painting was created by A. Gallen-Kallela in1901 in Helsinki. This metadata can be represented,by using RDF triples in Turtle notation 16 , in the followingway (this example is not based on the actualmetadata but is for illustration only)::Kullervo_departs_wardc:creator persons:A.Gallen-Kallela ;dc:date "1901" ;dc:spatial places:Helsinki .The metadata record in a biographical repository,such as the ULAN 17 of the Getty Foundation, may tellus more about the artist in a very different metadataformat, e.g.:persons:A.Gallen-Kallela:placeOfBirth places:Pori ;:timeOfBirth "1865" ;:placeOfDeath places:Stockholm ;:timeOfDeath "1931" .A problem here is that the number of differentproperties in metadata schemas easily getslarge in cross-domain applications. Furthermore,the meaning of many properties, such asdc:date and dc:spatial in the metadataschema of paintings and timeOfBirth/Deathand placeOfBirth/Death in the biographicalmetadata schema of persons may share some meaning,but are still different. We soon realized that whenusing the schemas for reasoning tasks, the logicalrules accounting properly all kinds of combinations16 http://www.dajobe.org/2004/01/turtle/17 http://www.getty.edu/vow/ULANSearchPage.jsp29

Figure 4: The painting of figure 3 viewed in the semantic portal CULTURESAMPO I. Three semantic recommendationlinks created by the system are visualized on top of the screenshot.of properties become complicated, and the numberof rules becomes large due to combinatorial explosion.It seems that a more primitive representationof knowledge than traditional metadata schemas isneeded for reasoning.A potential solution approach to solve the problemis to use the CIDOC CRM ontology. The system“provides definitions and a formal structure fordescribing the implicit and explicit concepts and relationshipsused in cultural heritage documentation” 18 .The framework includes some 80 classes, such as“E22 Man-Made Object”, “E53 Place”, and “E52Time-Span”, and a large set of some 130 propertiesrelating the entities with each other, such as “P4 HasTime-Span” and “P87 Is Identified By”. Interoperabilityof cultural content can be obtained by mappingmetadata standards to CIDOC CRM.The focus in CIDOC CRM is in modeling conceptsnecessary for representing the documentation semanticsof different metadata schemas used in the culturaldomain, such as Dublin Core. In contrast, inCULTURESAMPO our main focus is to represent realworld knowledge related to cultural heritage, e.g., thesubjects that the paintings in figures 2 and 3 depict.For this purpose, a different kind of knowledge repre-18 http://cidoc.ics.forth.gr/sentation scheme and large domain ontologies containingtens of thousands of domain concepts andevents are needed.Our solution to the problem of semantic interoperabilityis to transform different metadata schemasinto a shared, more primitive knowledge representationof the real world. In this way, the meaning ofdc:date, :timeOfBirth and :timeOfDeathcan be made interoperable. By basing reasoning onthe more primitive representation, more generic andfewer rules operating a smaller set of properties canbe devised. As for the knowledge representationscheme, the idea of representing knowledge in termsof actions and thematic relations between actions andentities was adopted. This general idea has been appliedwidely in computational linguistics and naturallanguage processing (cf. e.g. (Zarri, 2003)), inknowledge representation research (Sowa, 2000), andalso in CIDOC CRM, where events are of central importance,too.For example, in CULTURESAMPO the three timerelationsof the above examples are reduced intoonly one time-relation relating an instance of anevent type, such as “painting event”, “birth event”,or “death event” to a time entity. The meaning of semanticallycomplex properties in metadata schemasNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 30

is essentially represented in terms of different eventsand related entities. For example, the metadata aboutthe painting “Kullervo departs for the war” meansthat there was a painting event related with A. Gallen-Kallela, the year 1901, and Helsinki by the thematicroles “agent”, “time”, and “place”::painting_event_45rdf:type cs:painting_event ;cs:agent persons:A.Gallen-Kallela ;cs:time "1901" ;cs:place places:Helsinki .Information about the artist’s birth and death datescan be transformed in a similar manner into birth anddeath events, respectively. In this way, we can notonly eliminate various time-related properties fromthe knowledge base but also aggregate knowledgeform different sources on the more primite knowledgerepresentation level. In this case, for example, eventbasedbiographical knowledge about the life eventsof A. Gallen-Kallela can be enriched with the knowledgeabout the paintings he painted.Solving the semantic interoperability problem ofmetadata schemas by using a primitive event-basedknowledge representation scheme was one of the majorchallenges in creating the CULTURESAMPO IIprototype in 2006. This idea will be described, especiallyfrom the semantic browsing viewpoint, in moredetail in (Ruotsalo and Hyvönen, 2006).5 Extending Semantic RepresentationalPower of MetadataSchemasThe idea of using event-based knowledge representationsin annotation provides also a solution for creatingsemantically richer annotations. Event-basedannotations have been studied before, e.g., in thecontext of annotating the subject of photographs(Schreiber et al., 2001) and in representing narratives(Zarri, 2003).To illustrate this idea, re-consider the painting“Kullervo departs for the war” of figure 2. The subjectof content is here annotated by a set of keywords(in Finnish) including “Kullervo”, “horse” and “dog”.A problem from the knowledge representation viewpointis that the mutual relations of the subject annotationsare not known. For example, it is not knownwhether Kullervo rides a horse, a dog, both of them,or none of them. It is also possible that the dog ridesKullervo, and so on. Events can be used for elaboratingthe description, if needed, by specifying valuesfor their thematic roles. In this case, for example,Kullervo would be in the agent role and the horse inthe patient role in a riding event. This kind of informationcan be essential when searching the contents(e.g. to distinguish between riders and riding entities)or when providing the end-user with semantic linksand explanations (e.g. to distinguish links to otherriding paintings in contrast to other horse paintings).In CULTURESAMPO content comes not only in differentforms but is also annotated at different levels ofdetail “vertically”. For example, the metadata froma museum database is given as it is and may containonly minimal metadata while some other content maybe described in a very detailed manner by using lotsof Iconclass notations or manually annotated events.In our case, detailed semantic descriptions are beingcreated, for instance, when translating the Kalevalastory into RDF and OWL. Here each Kalevala partof potential interest to the end-user is annotated interms of events, thematic roles and other metadata.Furthermore, the events may constitute larger entitiesand have some additional semantic relations witheach other. In CULTURESAMPO I this idea was experimentedby representing processes and events oftwo Kalevala poems, in paintings, photos, and culturalprocesses (Junnila et al., 2006).In CULTURESAMPO II this work continues with anew modified event-based model. Furthermore, in thenew scheme, annotations can be given at three levelsof granularity in order to enable vertical interoperability:Keywords In some cases only keywords are availableas subject metadata. At this level the annotationis a set of literal values. Even if ontologicalannotations have been used (cf. below),literal keywords may be needed for free indexingwords.Keyconcepts Here the annotation is a set of URIsor other unique references to an ontology or aclassification system, such as Iconclass. The additionalknowledge introduced by keyconceptsw.r.t. using literal keywords is their ontologicalconnections. This enables semantic interoperability,as discussed earlier.Thematic roles At this level thematic roles betweenactivities and other entities can be specified. Theadditional knowledge w.r.t. using only keyconceptsis the distinction of the roles in which thekeyconcepts are at different metadata descriptions.Each new level of annotation granularity only adds31

new information with respect to the previous level.This means that semantically richer representationscan be easily interpreted at the lower level. Eventbaseddescriptions mean at the keyconcept level thatonly the entity resources that are used in the eventsare considered, not the properties. At the keywordlevel, only the literal labels of the annotations atthe keyconcept level are considered. This strategyenables, e.g., application and integration of traditionaltext-based search methods with ontologicalannotations—a useful feature since much of the contentin semantic portals is in textual form in any case(e.g., free text descriptions of collection items, biographicalarticles, poems etc.).The main ontology underlying CULTURESAMPOII is the General Finnish Upper Ontology YSO(Hyvönen et al., 2005b) of about 20,000 concepts.This lightweight ontology has been created based onthe widely used General Finnish Thesaurus YSA 19 .CULTURESAMPO also makes use of extended versionsof the ontologies used in MUSEUMFINLAND.6 The PortalCULTURESAMPO II provides the end-user with semanticsearch and browsing facilities in a way similarto MUSEUMFINLAND. Semantic multi-facet searchcan be used. Since the ontological model is eventcentric,the user is provided with a view classifyingverb-like event concepts in addition to more traditionalviews (persons, collections, etc.). Figure 5 illustratesthe search interface.When a search object is selected to viewing, recommendedsemantic links with explanations are providedfor browsing. Also here the event-centricmodel is evident: most recommendations are basedon sharing events and roles. Figure 6 shows a searchobject page of a photograph for illustration.In addition, CULTURESAMPO II includes manynew forms of semantic visualization, especially w.r.t.geographical information and time lines (Kauppinenet al., 2006). For visualizing search results on themap, Google Maps 20 service is used (cf. figure 7). Itwill be used as a search interface, too, later on. In thesame vein, the Simile Time Line 21 has been incorporatedin the user interface using Ajax-technology (cf.figure 8.CultureSampo I was implemented on our old OntoViewsarchitecture, based on Apache Cocoon 22 .19 http://www.vesa.lib.helsinki.fi20 http://maps.google.com/21 http://simile.mit.edu/timeline/22 http://cocoon.apache.org/However, when adding many more cross-linked componentsto the system in CULTURESAMPO II, such asthe timeline, map views, and the new recommendationsystem, severe limits in the old architecture becameapparent.A major guideline in our work has been to createapplications that can be configured to work with awide variety of RDF data. To accomplish this, wehave endeavored to build our applications out of modularcomponents that combine to provide advancedfunctionality. As CULTURESAMPO II became morecomplex and started to incorporate components fromother projects, there appeared a need for making theindividual components smaller and supporting a morecomplex multidirectional control and configurationflow between them. Apache Cocoon, however, isbased on a generally sequential pipeline architecture,which is very limited in its ability to support any multidirectionalcommunication. And while it was possibleto make use of modular component libraries onthe Java level, there was no architectural support forkeeping these components either universal or configurable,which in general resulted in them not beingsuch.To solve these problems, a new architecturewas developed for CultureSampo II based on thewell-known Service Oriented Architecture, Inversionof Control and Dependency Injection principles.Specifically, the new platform was built on top ofthe Apache HiveMind 23 services and configurationmicrokernel.7 DiscussionOur work on CULTURESAMPO suggests that usingevent-based annotations is a promising approach tocreating cross-domain semantic portals for severalreasons:1. By using more accurate semantic descriptionssemantic search and browsing (recommendation)can be made more accurate and explainedin more detail. The semantic accuracy of annotationscan be extended in a natural way by thenew layer of relational event annotations that explicatethe thematic roles between activities andother entities in the description. First tests onCULTURESAMPO I and II seem to indicate thatthis kind semantic knowledge is vital for semanticinformation retrieval tasks (search) and forcreating insightful semantic linking of contents23 http://jakarta.apache.org/hivemind/New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 32

Figure 5: CULTURESAMPO II search page. Views are on left and hits on the right.Figure 6: CULTURESAMPO II item page. Metadata is on the left and recommendation links on the right.33

Figure 7: Using Google Maps in CULTURESAMPO II for visualizing search items on the map. The items arepositioned based on a place ontology, and interactive to obtain additional information.Figure 8: Using Simile Time Line in CULTURESAMPO II for visualizing search items on the time line, and forselecting them for a closer look.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 34

automatically (Junnila et al., 2006; Ruotsalo andHyvönen, 2006).2. Event-based descriptions can be used for representingthe meanings in terms of happeningsand entities of the real world based on differentmetadata schemas. This enables semantic interoperability.3. The resulting knowledge representation schemeis simpler in terms of the number of propertiesthan the original set of metadata schemas. Thismakes it simpler to implement reasoning rulesneeded for the intelligent services for the enduser.The price for more accurate semantics is the extracost of creating the annotations. In CULTURESAMPOI all content was manually crafted. In CULTURE-SAMPO II a semi-automatic process has been used.At the schema level, the content has been enrichedautomatically by a separate, rule-based knowledgetransformation module. This system transforms, e.g.,the metadata of paintings into painting events. At thelevel of enriching the subject descriptions of the content,enriching has been mostly manual by adding thematicrole relations between the entities used in theoriginal databases. For example, to tell that Kullervorides a horse and not vice versa in figure 2, a ridingevent with Kullervo and an instance of horse in theproper thematic roles has to be created. In principle,the machine and ontologies could help the annotatorin her work, if it is known that usually humans ridehorses.The work of annotating narratives, such as theKullervo poem in Kalevala and the process of farmingby the slash and burn method in CULTURESAMPO I(Junnila et al., 2006) has been done completely manually.However, we are also investigating how languagetechnology can be applied to creating semiautomaticallyannotations for textual contents (Vehviläinenet al., 2006). It is clear, that development oftools that could help in creating annotations will beof outmost importance in the future.In some cases, like when annotating unique importantmaterials such as Kalevala, the price for detailedannotations can be paid, while in many other casesit is unrealistic to assume that such representationswill be available. In CULTURESAMPO this problemof dealing with materials annotated at different levelsof semantic accuracy is addressed by using three layersof annotations: keywords, keyconcepts and thematicroles.The success of the CULTURESAMPO will finallybe judged by the end-users. Empirical usability testsare needed in order to evaluate the added value of thesemantic approach. The first test, based on the currentCULTURESAMPO II, has been scheduled for theautumn 2006. The goal of this experiment is to testwhether the end-users really find the semantic recommendationsgenerated by the event-based model feasibleand helpful.CULTURESAMPO II is still a research prototypeand its current version contains only a few contenttypes and less that 10,000 search objects. For example,in contrast to CULTURESAMPO I, there are nonarratives in the system yet, only events. However,new types of content are being included in the schemeand in the system. Another line of development inthe system is designing additional conceptual visualizationtools. On the reasoning side, spatiotemporalreasoning under uncertainty is being studied (Kauppinenand Hyvönen, 2006) and is being implementedin the system.We plan to publish CULTURESAMPO on the publicweb in 2007.AcknowledgmentsOur research is a part of the National Semantic WebOntology Project in Finland 24 (FinnONTO) 2003-2007 funded mainly by the Finnish Funding Agencyfor Technology and Innovation (Tekes).ReferencesM. Doerr. The CIDOC CRM - an ontological approachto semantic interoperability of metadata. AIMagazine, 24(3):75–92, 2003.E. Hyvönen, E. Mäkela, M. Salminen, A. Valo,K. Viljanen, S. Saarela, M. Junnila, and S. Kettula.MuseumFinland – Finnish museums on the semanticweb. Journal of Web Semantics, 3(2):224–241,2005a.E. Hyvönen, A. Valo, V. Komulainen, K. Seppälä,T. Kauppinen, T. Ruotsalo, M. Salminen, andA. Ylisalmi. Finnish national ontologies for thesemantic web - towards a content and service infrastructure.In Proceedings of International Conferenceon Dublin Core and Metadata Applications(DC 2005), Nov 2005b.M. Junnila. Tietosisältöjen semanttinen yhdistäminentoimintakuvausten avulla (Event-based approachto semantic linking of data content). Master’s thesis,University of Helsinki, March 6 2006.24 http://www.seco.tkk.fi/projects/finnonto/35

M. Junnila, E. Hyvönen, and M. Salminen. Describingand linking cultural semantic content by usingsituations and actions. In Semantic Web at Work- Proceedings of the 12th Finnish Artificial IntelligenceConference STeP 2006, Volume 1., Nov2006.T. Kauppinen, R. Henriksson, J. Väätäinen, C. Deichstetter,and E. Hyvönen. Ontology-based modelingand visualization of cultural spatio-temporalknowledge. In Semantic Web at Work - Proceedingsof the 12th Finnish Artificial Intelligence ConferenceSTeP 2006, Volume 1., Nov 2006.T. Kauppinen and E. Hyvönen. Modeling and reasoningabout changes in ontology time series. InR. Kishore, R. Ramesh, and R. Sharman, editors,Ontologies: A Handbook of Principles, Conceptsand Applications in Information Systems.Springer-Verlag, Dec 2006. In press.A. Vehviläinen, E. Hyvönen, and O. Alm. A semiautomaticsemantic annotation and authoring toolfor a library help desk service. In Proceedings ofthe first Semantic Authoring and Annotation Workshop,ISWC-2006, Athens, GA, USA, November2006. To be published.K. Viljanen, T. Känsälä, E. Hyvönen, and E. Mäkelä.Ontodella - a projection and linking service forsemantic web applications. In Proceedings ofthe 17th International Conference on Databaseand Expert Systems Applications (DEXA 2006),Krakow, Poland. IEEE, September 4-8 2006.G. P. Zarri. Semantic annotations and semanticweb using nkrl (narrative knowledge representationlanguage). In Proceedings of the 5th InternationalConference on Enterprise Information Systems,Angers, France (ICEIS 2003), pages 387–394, 2003.E. Mäkelä, E. Hyvönen, and S. Saarela. Ontogator— a semantic view-based search engine service forweb applications. In Proceedings of the 5th InternationalSemantic Web Conference (ISWC 2006),Nov 2006.E. Mäkelä, E. Hyvönen, S. Saarela, and K. Viljanen.OntoViews – a tool for creating semantic web portals.In Proceedings of 3rd International SemanticWeb Conference (ISWC 2004), Hiroshima, Japan,November 2004.T. Ruotsalo and E. Hyvönen. Knowledge-basedrecommendation based on heterogenous metadataschemas, 2006. Paper under contruction.M. Salminen. Kuvien ja videoiden semanttinensisällönkuvailu (Semantic content description ofimages and videos. Master’s thesis, University ofHelsinki, May 2006.A. T. Schreiber, B. Dubbeldam, J. Wielemaker, andB. J. Wielinga. Ontology-based photo annotation.IEEE Intelligent Systems, 16:66–74, May/June2001.J. Sowa. Knowledge Representation. Logical,Philosophical, and Computational Foundations.Brooks/Cole, 2000.J. van den Berg. Subject retrieval in pictorialinformation systems. In Proceedings ofthe 18th international congress of historical sciences,Montreal, Canada, pages 21–29, 1995.http://www.iconclass.nl/texts/history05.html.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 36

Ontology-based Modeling and Visualization of CulturalSpatio-temporal KnowledgeTomi KauppinenUniversity of Helsinki andHelsinki University of Technology (TKK) Media TechnologySemantic Computing Research Group (SeCo)http://www.seco.tkk.fi/tomi.kauppinen@tkk.fiRiikka HenrikssonHelsinki University of Technology (TKK)Laboratory of Geoinformation and Positioning TechnologyFIN-02151 Espoo, Finlandriikka.henriksson@tkk.fiJari VäätäinenGeological Survey of FinlandP.O. Box 96FIN-02151 Espoo, Finlandjari.vaatainen@gtk.fiChristine DeichstetterJohannes Kepler Universität Linz andHelsinki University of Technology (TKK)c.deichstetter@gmx.atEero HyvönenHelsinki University of Technology (TKK) andUniversity of HelsinkiSemantic Computing Research Group (SeCo)http://www.seco.tkk.fi/eero.hyvonen@tkk.fiAbstractGeographic knowledge is essential in handling a large variety of resources, including cultural contentssuch as museum artifacts, maps, books, photographs, and videos. The metadata of such resourcesoften need to refer to a geographic entity or region, for example to the place where an artifact wasproduced, used, or found. In this paper, we examine how geographical knowledge may be representedontologically to enable different types of searches, visualization, and inference in cultural semanticportals and other semantic geo-applications. In particular, we show how change in time betweenhistorical regions can be explicated as an ontology and be used for reasoning. Regarding search andvisualization, we show how maps of different time periods can be visualized as transparent overlayson top of Google Maps, how cultural content can be visualized on them, how geo-based queries canbe formulated based on maps, and how additional services, such as a meta-search system, can beintegrated in the mash-up system. The work presented is being integrated at the practical level in thecultural semantic cross-domain portal CultureSampo.1 IntroductionA large proportion of cultural resources such as artifacts,collections, books, photographs, and videos aregeographically referenced and thus should be identifiedby search terms that refer to locations (Joneset al., 2001; Stuckenschmidt and Harmelen, 2004).This is because they are produced, found or usedin those locations, or they have some other relationshipto the location in question. By georeferencingthe resources (Schlieder et al., 2001), different spatialqueries can be enabled in order to find interestingconnections.The term georeference refers to the act of assigninglocations to geographic objects, which are entitiesrepresenting some part on or near the Earth’s surface.The simplest form of georeferencing is place naming.However, the problem of representing this geographicallyreferenced knowledge is complex due to manyreasons. Place names are usually unique only within37

an area or domain of the Earth’s surface (e.g. cityname is usually unique within the domain of a state)and they may become obsolete through the time. Forexample, different countries and other regions aremerged, split and renamed due to reorganizations atthe social and cultural levels thus causing new territorialshapes. Place names may also have differentmeaning to different people, and within the differentcontexts they are used. Well-known systems for georeferencinguniquely across domains are postal codesand geographic coordinate systems (the system of latitudeand longitude), from which the former is basedon a human activity and the latter one on physical features(Longley et al., 2001; Mark et al., 2001).Ontology-driven information systems (Guarino,1998) have been proposed to provide means to representand manage this kind of complex knowledge.The idea behind modern systems is to (Visser, 2004)“geo-enable” the Web and allow for complex spatialinformation and services accessible and useful withall kinds of applications e.g. from library systems tomuseum systems.In information systems, the term ontology shouldbe considered as a synonym to conceptual model ina way that it is independent of its philosophical roots(Welty and Guarino, 2001). The idea is that modeledontologies capture the properties, classes and theirmutual relationships (Guarino and Welty, 2002) —i.e. the essential semantics of the Universe of Discourse(UoD). For example, the spatial overlap of regions(Visser, 2004; Stuckenschmidt and Harmelen,2004) is important in a sense that it affects the wayontological annotation of resources should be made.This knowledge concerning geographic, spatialentities has to be represented in a machineunderstandable,reusable and shareable way so thatit can be used, for example, in the Semantic Web(Berners-Lee, 1998; Berners-Lee et al., 2001) as ontologies.In this paper we examine how ontological knowledgecan be used in order to visualize different kindsof georeferenced information and to enable differentprecise searches and meta searches. We are especiallyinterested in how the results could be used at a practicallevel in a cultural semantic cross-domain portalsuch as CULTURESAMPO (Hyvönen et al., 2006)and in creating a prototype of a national geo-ontologyserver.In essence, we propose that the following spatiotemporal,ontological issues should be handled in astate-of-the-art-system.1. Ontological representation of spatial entities andtheir mutual relationships.2. Inference based on explication of complexknowledge. We will show how knowledge abouthistorical changes in regions can be modeled andutilized in information retrieval.3. Visualization of semantic web content on maps.We show how cultural content can be visualizedon maps by using Google Maps as an externalservice.4. Multi-map visualization by using different kindsof maps simultaneously. It is shown how mapsof different time periods can be used by usingtransparent overlays, which gives the end-user akind magnifying glass to zoom into history.5. Polygon-based searches. We will illustrate howto express polygons, and map them into ontologicallocation individuals (instances) such ascities, in order to annotate and search semanticcontent.6. Combining map visualization with other services.Visualization on the map can be easilycombined with other services. As a demonstrationof how this can add value, a meta search serviceimplemented on top of the semantic portalMuseumFinland (Hyvönen et al., 2005) is presented.In the following we will overview how we have addressedthese issues by creating ontologies, reasoningengines and building working demonstrations usingexisting portals.2 Representation of OntologicalRelationshipsSpatial structures and their mutual relations areclearly the most essential form of geographic knowledge.The principles by which the geographic domain- and geo-ontologies - are primarily structuredfrom the theoretical viewpoint are topology (the theoryof boundaries, contact and separation), mereology(the theory of part/whole) and geometry (Market al., 2001). This is because geographic objects arenot merely located in space; they are tied intrinsicallyto space inheriting many of the structural propertiesfrom the Earth’s surface. Reasoning in termsof part-of relationships is powerful and appears to bewell-suited for geographic representation and inference.Part-of relationships are used for containmenthierarchies and are closely related to representationNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 38

of different types of administrative hierarchies. However,mereology alone cannot deal with some verybasic spatial relations, such as the relations within,contains, andoverlaps, and this is where topologicalreasoning is required (Casati et al., 1998; Mark et al.,2001; Jones et al., 2002).There has been an active philosophical discussionabout the essential spatial relations (e.g. in (Varzi,1996)) and different attempts to formalize the relations,such as RCC-8 (Randell et al., 1992) and(Egenhofer, 1989). Furthermore, methods to measurespatial relevance, such as topology (of neighboringregions), directions (of related regions), and distances(between regions) combined with partonomical relevance(cf. (Stuckenschmidt and Harmelen, 2004),chapter 8) have been developed.Semantic web ontologies represent geographicknowledge (among other types of knowledge) interms of the Resource Description Framework (RDF)(Brickley and Guha, 2004) and the Web OntologyLanguage (OWL) 1 . RDF and Uniform ResourceIdentifiers (URI) form the basis of the Semantic Web.OWL and RDF are used for defining concepts andmetadata, and URIs are used for identifying resourcesused in the descriptions uniquely.As a part of our research, we are developing 2 aFinnish geo-ontology SUO (Suomalainen paikkaontologia)using Semantic Web technologies RDF andOWL. According to the plans SUO-ontology willmodel different classes of geographical places - eitherman-made or natural - and their topological andmereological relations, as well as relations definingcoordinate-based geometry for points, curves andpolygons. Currently SUO includes approximately150 classes such as city, province, graveyard, lake orriver. These classes will be populated with thousandsof instanses from the Place Name Register 3 .3 Explicating Change of RegionsGeographic regions change over time. Regions, forexample, are split and merged due to various reorganizationsat the social and cultural levels. For example,in 1991 East Germany and West Germany weremerged to form Germany, as depicted in figure 1.WestGermany, East Germany and Germany are here individualsof an ontology. The x-axis depicts time andthe y-axis the relative areas of the countries. The his-1 http://www.w3.org/2001/sw/2 We have used The Protégé Ontology Editor available at(http://protege.stanford.edu/).3 Place Name Register was obtained from the National LandSurvey of Finlandtory is full of events like this (Smith and Brogaard,2003). Budapest was formed via the unification offormer towns Buda and Pest, the Czech Republic andSlovak Republic were formed through the separationof Czechoslovakia, and so on. This kind of spatialchanges of individuals at the geographical, social andcultural levels are problematic from the informationannotation and retrieval viewpoint. Content is annotatedby using historical and contemporary conceptsand names. However, when querying, concepts andnames from other time periods may be used, whichleads to low recall and precision.Figure 1: Germany, East Germany and West Germanyover time and space.To address the problem we developed an ontologicalmethod (Kauppinen and Hyvönen, 2006, 2005)for representing spatio-temporal changes in regionsthat essentially define a geo-ontology time series.The figure 2 illustrates the steps of using the method.In the initial data of ontological changes are maintainedas a spreadsheet table to ease the editing 4 .Each change, such as a merge or a split of two counties,is represented by a line in the table and is calleda "change bridge".This table is then transformed automatically intoRDF-form representing the changes and regions involved.A set of rules is used to construct temporalregions based on the change information. As a simpleexample, since we know that an administrationalregion of Viipuri has changed both in 1906 and in1921, the area of Viipuri has remained the same between(the beginning of) 1906 and (the end of) 1920and hence a temporal region Viipuri (1906-1920) iscreated. In a similar manner the rule set is used toconstruct all the other temporal regions in the ontologytime series. At the next phase, the essential propertiesare inferred for these temporal regions. We declaredthat the essential property (Guarino and Welty,4 We have used the freely available OpenOffi ce Calc(http://www.openoffi ce.org/) but there are other alternatives availableas well.39

Initial Ontology ConstructionRule−based Reasoning ofTemporal RegionsInference of Areal Sizesand Partonomic RelationsFigure 3: Annotation and indexing concepts matched.Inference of Coveragescal coverings are made explicit using the sizes ofgeospatial resources.Ontology time−series3. Global Coverings. Global overlaps are calculatedby chaining local coverings and by consideringdifferent change paths between concepts.Figure 2: Process of creating an ontology time-series.4. Visualization of global coverings.2002) for each region is the size of the area. For example,since we know that the area of Helsinki hasbeen measured during the year 2005, we can inferthat a temporal region Helsinki (1966-NOW) has thisareal value as it has remained the same between 1966and now. Each temporal region in the system has anown identifying URI. This means that if the area of aregion changes, there will be a new URI correspondingto a new region with the new value for the size ofthat area.An inference engine then reasons the coverages(i.e., how much a region X overlaps an other regionY, and vice versa) between all temporal regions ofthe ontology. The calculation is based on the initialsizes of the regions and their change history. The coveragescannot be calculated directly based the actualpolygons of the regions, because these are not known,only the sizes and the changes. In the below, the basicsteps of the method are shortly described.1. Local Bridges. Changes are modeled as individualsof basic change classes, such as split andmerged.2. Local Coverings. The bridges represented inRDF are transformed into a form where the lo-With the method, it has been calculated, e.g.,that the temporal region Lappeenranta (1989-) covers12% of the temporal region Viipuri (-1906). Theglobal coverage information can be used to match annotationsand queries, as depicted in figure 3. Forexample, the photo depicted in the figure is stored ina database of the Geological Survey of Finland GTK(Väätäinen, 2004). According to the attached metadata,the photo is taken within the region Viipuri (-1906). This means that there is a 12% change that thephoto is actually within the boarders of the currentcity of Lappeenranta in Finland. This may be a surpriseto many, since Viipuri was annexed to the SovietUnion after the World War II. The explanation is theannexed Viipuri constitutes a temporal region that isdifferent from Viipuri (-1906).The method is being applied to construct a completeFinnish time-location ontology (Suomen AjallinenPaikkaOntologia, SAPO) of counties and communessince the mid 1800’s, and to maintain the ontologyin the future. It has already been identified(Väätäinen, 2004) that from the beginning of 20thCentury there are already over 1100 changes (such ascreation, merge, split, and name change) in Finnishcommunes.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 40

Figure 4: Using multiple maps simultaneously. Ahistorical Karelian map depicting the park of Monreposin Viipuri is shown semi-transparently on topof a modern satellite image provided by the GoogleMaps service.4 Visualization using MultipleSimultaneous MapsIn order to visualize historical geo-content annotatedaccording to old temporal regions and place names,historical maps are needed. On the other hand, alsothe current view of places is usually needed at thesame time to bridge the conceptual cap between regionsof different era. To facilitate this, we created ascheme for using several overlaying maps simultaneouslyin visualizations. Creation of several layers isa common (de Berg et al., 2000) way to make mapsmore readable.The maps and satellite images of the Google Mapsservice were used as the contemporary view. To providethe historical view, we used a set Finnish mapsfrom the early 20 th century covering the area of theannexed Karelia before the World War II. The mapswere digitized and provided by the National LandSurvey of Finland 5 . In addition, a map of the Espooregion in 1909, provided by the Geological Survey ofFinland, was used.Figure 4 illustrates our scheme. On the left, a satelliteGoogle Maps image of the contemporary Viipuriregion is shown. In the middle, a smaller rectangulararea is shown with a semi-transparent 6 old Karelianmap that is positioned correctly and is of the samescale as the Google Maps image. This smaller view5 http://www.maanmittauslaitos.fi /default.asp?site=36 The level of transparency can be altered in the demonstration.shows the park of Monrepos in Viipuri, a famousFinnish pre-war cultural site that nowadays is a part ofRussia. The place cannot be found in current maps asit was, making it difficult for modern users to locatethe place geographically. Old maps and names on itcould be of substantial benefit when using the visualizationin annotating or searching content by mapsin systems such as CultureSampo (Hyvönen et al.,2006). The simultaneous views are useful also whencomparing geo-information from different eras (e.g.,how construction of cities has evolved) or from differentthematic perspectives (e.g., viewing a geologicalmap on top of a satellite image).In order to move around the user is able to use thezooming and navigation functions of Google Mapsand the Karelian view is automatically scaled and positionedaccordingly. In order to facilitate this, theKarelian maps were processed according to the followingsteps:1. Cropping and conceptualization of the map images.The digitized maps had a lot of useful informationin addition to the actual maps in theimage files. For example, the place name, scale,old coordinates etc. This knowledge was extractedand represented as an ontology. Additionalinformation in the margins of the map imageswere cropped away.2. Projection of the images. Projection was donemanually by adjusting the images by draggingand stretching them in an image editing programindividually using reference points. One couldalso use a specialized GIS software to do theprojection conversion. However, when this wastested, no clear benefits were obtained. For example,relative positional accuracy between thesatellite images and the projected map imagesdid not increase remarkably and the process wasrather time consuming.3. Renaming the image files according the GoogleMaps API specifications.4. Publishing of the image files on a public webserver.5 Using Points to Explicate aSearch AreaGeographic objects are traditionally modeled aspoints, lines, polygons or fields in spatial databases.Depending on the information a point carries, it canbe either an entity point, a label point or an area point.41

An entity point represents the coordinate position ofa point entity such as a tower, a label point carriestextual information of the object in question and anarea point represents the center point coordinates ofa polygon (Zhang and Goodchild, 2002). There hasalso been discussion (Schlieder et al., 2001) about extendingontologies (gazetteers) with more exact positionslike a point or polygon in a given coordinatereference system. For representing areas, polygonswould clearly be most satisfying.However, polygon data is (Schlieder et al., 2001)often 1) proprietary or 2) may be unavailable for e.g.historic regions. Furthermore, 3) detailed polygondata is computationally expensive to process and 4)the exactness of the polygon data should be set toa needed abstraction level. Area points are morewidely available. In Finland, names and coordinatesare available in the Place Name Register produced bythe National Land Survey of Finland.We have created a method, n-point search, forsearching this kind of coordinate data. A search queryin this method is done by pointing out n points on amap. The user clicks on the map and a polygon isformed, accordingly.The idea is that the n points define either a simplepolygon (without an inner ring) or a complex polygon(with 1...n inner rings) that bounds the search area inquestion. In other words, the defined polygon can beeither complex, concave or convex. If an area pointof a certain place is found inside the user-given polygon,it is retrieved and added to the results. Matchingplaces were found using the Jordan Curve Theorem(Haines, 1994).N-point search would also support the polygonoverlays if the regions would be modeled as polygons.This would lead even more precise search results,but in our case such polygon data is not availablecurrently. However, in the future we plan to createpolygons and use them as well.We have also created a special handling for twospecial cases, namely, for those cases where n =1or n = 2. If n = 1 a circle is drawn around thepoint 1 and the places that are inside the circle areretrieved. An alternative treatment for the n =1situationwould be to find the nearest places. Naturallyboth of these treatments could be offered for a user.And if n =2, we create a bounding box, where point1 and point 2 are the opposite corners, e.g. South-West and North-East corners, of the search area.We have implemented the n-point search as aGoogle Maps application where Scalable VectorGraphics (SVG) (Ferraiolo et al., 2003) is used fordrawing the polygon as an other transparent layer ontop of a map. An example of using the n-point searchis depicted in figure 5. The n polygon corners aredepicted as small crosses. The system has found outthree historical communes of the annexed Karelia, Viipuri,Makslahti, and Koivisto, whose center pointsare situated within the user-specified search polygon.In the future, we plan to use the idea of userdefinedpolygons also in ontology population and inannotating contents.Figure 5: Search results using the n-point search: Viipuri,Koivisto and Makslahti are matched.6 Combining Visualization withOther ServicesVisualizations on the map can be combined interactivelywith services on the web. In this section it isshown how cultural content can be projected on a mapservice as interactive elements that can be used to invokerelated services, such as search engines.To test this idea, we have built an applicationthat uses Google Maps API 7 to visualize hundredsof Finnish locations on a zoomable map. User cansearch information from different portals with locationnames just by clicking the location on a map.Figure 6 depicts the idea. On the left pane, aGoogle Maps view is shown where the red buttonsrepresent Finnish communes and other places of interest.The user has clicked on a button on Helsinki.As a result, the system has queried MuseumFinland(Hyvönen et al., 2005) with the the concept of“Helsinki” as either the place of manufacture or place7 http://www.google.com/apis/maps/New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 42

Figure 6: A searchable map interlinked with the semantic portal MuseumFinland.of usage of the artifacts included in the semantic portal.The result is shown on the right pane. In addition,a bubble with several links is opened on the left. Thelinks are search requests to different search engineswhere the location selected is used as a query parameter.The search engines engaged in the application include,in addition to the default search engine MuseumFinland,for example Wikipedia 8 , Google Images9 , Muisti-project cultural portal 10 and the Old photosservice 11 provided by the Geological Survey of Finland.The search links are created automatically by constructinga search query to the search engines. For example,since the URL’s of Wikipedia are simple andpermanent, and include the keyword at the end of theURL, the corresponding place name could be used asthe keyword. For the capital of Finland, Helsinki, forexample, the system guessed that it has a Wiki pagehttp://en.wikipedia.org/wiki/Helsinki which happensto be the case. By selecting the link, this page isshown on the right pane instead of the MuseumFinlandpage. Also an automatic linking to the wwwpagesof the municipalities in Finland is provided.However, there are two problems with this approach:First, the automatically constructed link or8 http://www.wikipedia.org/9 http://images.google.com10 http://www.lib.helsinki.fi /memory/etusivue.html11 http://www.gtk.fi /palvelut/info/geokuvat/index.phpquery may lead to a non-existing page or it mightreturn empty results. For example, some Wikipediapages may be missing. Nevertheless, this way peoplecan be guided to the corresponding Wikipedia pageto contribute their knowledge about places. Second,place names have several meanings that cannot necessarilybe disambiguated. For example, “Nokia” is acity in Finland, a telecom company, a person, and ananimal in Finnish. Semantic disambiguation could bemade, if the services were supporting ontology-basedquerying based on URIs. For example, when usingMuseumFinland this would be possible because thesystem supports search based on concepts (URIs) inaddition to literal keywords.In the “Geo-MuseumFinland” application of figure6 places are visualized on the map and used for constructingqueries to search engines. We also made arelated application for the semantic portal CULTURE-SAMPO (Hyvönen et al., 2006) where search hits arevisualized in the same vein. CULTURESAMPO uses aplace ontology that is based on the one used in MuseumFinlandand defined in RDF. Language (OWL)(Smith et al., 2004). However, the original place ontologydid not have any coordinates. The place ontologywas therefore enriched by introducing a hasCoordinatePoint-propertyand by extracting values forthese properties from the Place Name Registry obtainedfrom the National Land Survey of Finland.The result is a visualization of the search results onthe map. The implementation was done as a Google43

Maps application where Ajax 12 is used for data interchangeof search results between the browser and theserver. The figure 7 shows how the search results arevisualized as interactive buttons. The user has clickedon a button depicting the hits related to the city ofOutokumpu, and a bubble is opened with links to theactual artifacts, in this case two basins (“vati”). Byselecting a link, the user will get a more detailed explanationabout the item, and semantic recommendationsto related materials customary.ReferencesTim Berners-Lee. Semantic web road map. Technicalreport, World Wide Web Consortium, September1998.Tim Berners-Lee, Jim Hendler, and Ora Lassila. Thesemantic web. Scientific American, 284(5):34–43,May 2001.D. Brickley and R. V. Guha. RDF Vocabulary DescriptionLanguage 1.0: RDF Schema W3C Recommendation10 February 2004. Recommendation,World Wide Web Consortium, February 2004.R. Casati, Barry Smith, and A. C. Varzi. Ontologicaltools for geographic representation. In N. Guarino,editor, Formal Ontology in Information Systems,pages 77–85. IOS Press, Amsterdam, 1998.Figure 7: Visualization of search results in semanticportal CULTURESAMPO.7 ConclusionsIn this paper we examined how ontological knowledgecan be used in order to represent and visualizedifferent kinds of georeferenced data and to enablesearches. We proposed and demonstrated explicationof complex spatio-temporal relations between geographicalobjects, search based on spatio-temporalontologies, query formulation based on maps, and visualizationof historical contents on old and contemporarymaps.The systems overviewed will be used when creatinggeo-based applications and tools within thethe National Semantic Web Ontology Project in Finland13 , especially in CULTURESAMPO and in creatinga prototype of a national geo-ontology server.AcknowledgmentsOur research is a part of the National Semantic WebOntology Project in Finland 14 (FinnONTO) 2003-2007 funded mainly by the Finnish Funding Agencyfor Technology and Innovation (Tekes).12 http://en.wikipedia.org/wiki/AJAX13 http://www.seco.tkk.fi /projects/fi nnonto/14 http://www.seco.tkk.fi /projects/fi nnonto/Mark de Berg, Marc van Kreveld, Mark Overmars,and Otfried Schwarzkopf. Computational geometry:algorithms and applications. Springer-VerlagNew York, Inc., Secaucus, NJ, USA, second edition,2000.M. Egenhofer. A formal definition of binary topologicalrelationships. In Foundations of Data Organizationand Algorithms, 3rd International Conference,FODO 1989, volume 367 of Lecture Notesin Computer Science, pages 457–472, 1989.Jon Ferraiolo, Dean Jackson, and Jun Fujisawa. ScalableVector Graphics (SVG) 1.1 specification W3Crecommendation. Technical report, World WideWeb Consortium W3C, January 14 2003.N. Guarino. Formal ontology and information systems.In Proceedings of the 1st International Conferenceon Formal Ontologies in Information Systems.IOS Press, 1998. Trento, Italy.Nicola Guarino and Christopher A. Welty. Evaluatingontological decisions with ontoclean. Communicationsof the ACM, 45(2):61–65, 2002.Eric Haines. Point in polygon strategies. In PaulHeckbert, editor, Graphics Gems IV, pages 24–46,Academic Press, 1994.Eero Hyvönen, Eetu Mäkelä, Mirva Salminen, ArttuValo, Kim Viljanen, Samppa Saarela, Miikka Junnila,and Suvi Kettula. MuseumFinland – finnishmuseums on the semantic web. Journal of Web Semantics,3(2):25, 2005.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 44

Eero Hyvönen, Tuukka Ruotsalo, Thomas Häggström,Mirva Salminen, Miikka Junnila, MikkoVirkkilä, Mikko Haaramo, Tomi Kauppinen, EetuMäkelä, and Kim Viljanen. CultureSampo—Finnish culture on the Semantic Web: The visionand first results. In Semantic Web at Work — Proceedingsof the 12th Finnish Artificial IntelligenceConference STeP 2006, volume 1, Helsinki, Finland,2006.C. Jones, R. Purves, A. Ruas, M. Sanderson, M. Sester,M. van Kreveld, and R. Weibel. Spatial informationretrieval and geographical ontologies – anoverview of the spirit project, 2002.Christopher B. Jones, Harith Alani, and DouglasTudhope. Geographical information retrieval withontologies of place. In In Proceedings of the InternationalConference on Spatial Information Theory,pages 322–355. Springer-Verlag, 2001.Tomi Kauppinen and Eero Hyvönen. Modeling coveragebetween geospatial resources. In Posters andDemos at the 2nd European Semantic Web ConferenceESWC2005, pages 49–50, Heraklion, Crete,2005.Tomi Kauppinen and Eero Hyvönen. Modeling andReasoning about Changes in Ontology Time Series,chapter 11 in book: Ontologies: A Handbookof Principles, Concepts and Applications in InformationSystems. Integrated Series in InformationSystems, Volume 14. Springer-Verlag, 2006.Barry Smith and Berit Brogaard. Sixteen days. TheJournal of Medicine and Philosophy, 28:45–78,2003.Michael K. Smith, Chris Welty, and Deborah L.McGuinness. W3C OWL Web Ontology LanguageGuide W3C Recommendation, February 2004.Heiner Stuckenschmidt and Frank Van Harmelen. InformationSharing on the Semantic Web. Springer-Verlag, Berlin Heidelberg, New York, 2004.A.C. Varzi. Parts, Wholes and Part-whole Relations:the Prospects of Mereotopology. Data and KnowledgeEngineering, 20:259–286, 1996.Ubbo Visser. Intelligent information integration forthe Semantic Web. Springer-Verlag, Berlin Heidelberg,New York, 2004.Jari Väätäinen. A database containing descriptionsof changes of counties in Finland. The GeologicalSurvey of Finland (GSF), Espoo, Finland, 2004.Christopher A. Welty and Nicola Guarino. Supportingontological analysis of taxonomic relationships.Data Knowledge Engineering, 39(1):51–74,2001.J. Zhang and M. Goodchild. Uncertainty in GeographicalInformation. Taylor & Francis, London,2002.Paul A. Longley, Michael F. Goodchild, David J.Maguire, and David W. Rhind. Geographic InformationSystems and Science. John Wiley & Sons,England, August 2001.David M. Mark, André Skupin, and Barry Smith.Features, objects, and other things: Ontologicaldistinctions in the geographic domain. In SpatialInformation Theory: Foundations of GeographicInformation Science (Lecture Notes in ComputerScience 2205), pages 488–502, 2001.D.A. Randell, Z.Cui, and A.G. Cohn. A SpatialLogic based on Regions and Connection. In Int.Conf. on Knowledge Representation and Reasoning,Boston, October 1992.C. Schlieder, T. Vögele, and U. Visser. Qualitativespatial representation for information retrieval bygazetteers. In Proceedings of Conference of SpatialInformation Theory (COSIT), volume 2205, pages336–351, Morrow Bay, CA, 2001.45

Improving the Quality of Medication by Semantic WebTechnologiesJuha PuustjärviLappeenranta University of TechnologyP.O.Box 20 FIN-53851juha.puustjarvi@lut.fiLeena PuustjärviThe Pharmacy of KäpyläKäpyläntie 8 Helsinki Finlandleena.puustjarvi@kolumbus.fiAbstractDuring the past few years several organizations in the healthcare sector have produced standardsand representation forms using XML. For example, patient records, blood analysis and electronicprescriptions are typically represented as XML-documents. This generalization of XMLtechnologiessets a promising starting point for the interoperability of the various organizations inthe healthcare sector. However, the introduction of XML is not enough but many other XML-basedtechnologies have to be introduced in order to achieve a seamless interoperability between the organizationswithin the healthcare sector. The main goal of this article is to show the gains the interoperabilityof the health care systems and the deployment of the Semantic Web technologies canprovide for electronic prescription systems. In particular, we present an e-prescription ontology andthe querying facilities that the deployed ontology provides. We also illustrate how the coordinationof the interoperability required by electronic prescription systems can be automated by utilizingXML-based process languages.1 IntroductionElectronic prescription is the electronic transmissionof prescriptions of pharmaceutical products fromlegally professionally qualified healthcare practitionersto registered pharmacies. The scope of the prescribedproducts varies from country to country aspermitted by government authorities or health insurancecarriers. For electronic prescription to be acceptedby the physicians, pharmacies and patients itmust provide added benefits to all participants.The problems related to prescribing medicationare discussed in many practitioner reports and publicnational plans, e.g., in (Bobbie, et al., 2005) and(Chadwick and Mundy, 2004). These plans shareseveral similar motivations and reasons for the implementationof electronic prescription systems(EPSs). These include: reduction of medication errors,speeding up the prescription ordering process,better statistical data for research purposes, and financialsavings.A salient trend in medication is that the numberof new medications increases every year. As eachdrug has its unique indications, cross-reactivity,complications and costs also the prescribing medicationbecomes still more complex every year. However,applying computing technology for prescribingmedication this complexity can be alleviated inmany ways.Today there exists a great diversity of competingtechnological solutions for electronic prescriptionsystems. For example, a citizen may have a memorycard, or electronic prescriptions may be transferredvia the internet or EDI. There is also diversity inused distribution, e.g., drugs may be transferred tohome or they may be picked from pharmacies.In addition, many prescription writer applicationstake advantage of internet and other applicationssuch as expert databases, and systems that includeinformation about patients’ demographic dataand medication history. So, modern prescriptionwriters are internet-based applications that interoperatewith many other health care information systems.During the past few years several organizationsin the healthcare sector have produced standards andrepresentation forms using XML. For example,patient records, blood analysis and electronic pre-New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 46

scriptions are typically represented as XMLdocuments(Jung, 2006; Liu et al, 2001; Mattocks,2005; Stalidis et al 2001; Woolman, 2001). Thisgeneralization of XML-technologies sets a promisingstarting point for the interoperability of the variousorganizations in the healthcare sector. However,the introduction of XML itself is not enough butalso many other XML-based technologies have to beintroduced in order to achieve a seamless interoperabilitybetween the organizations within thehealthcare sector.In this article we illustrate the interoperabilitywithin the healthcare sector from electronic prescriptionspoint of view. In particular, we illustrate:How XML-based technologies can be utilizedin modelling the concepts that are relatedto prescription writing.How Web service technology can be usedin implementing the interoperability ofelectronic prescription system and otherhealthcare systems.How the coordination of the interoperabilityrequired by electronic prescription systemscan be automated by utilizing XMLbasedprocess modelling languages. In particularwe show how the Business ProcessModeling Notation (BPMN) (BPMN,2006) and Business Process ExecutionLanguage for Web Services (BMEL4WS)(BPEL4WS, 2006) can be used for automatingthe coordination of electronic prescriptionprocesses.The rest of the paper is organized as follows. First inSection 2, we give a motivation by illustrating apaper based prescribing system and electronic prescribingsystem. Especially we will illustrate theway the physician can utilize the new querying facilitiesthe EPS (Electronic Prescription System).Then, in Section 3, we first give a short introductionto ontologies and then we present a simple e-prescription ontology. We also illustrate how theontology can be utilized in prescribing medication.Then, in Section 4, we consider the architecture ofan electronic prescription system based on serviceoriented architecture. In Section 5, we illustrate howBusiness Process Modelling Notation can be used inautomating the coordination of electronic prescriptionprocesses. Finally, Section 6 concludes the paperby discussing the advantages and disadvantagesof our proposed approach.2 Prescription processesWe now illustrate a scenario of an electronic prescriptionprocess that interoperates with other healthcare systems. The process goes as follows: first apatient visits a physician for diagnosis. In prescribingmedication the physician uses a prescriptionwriter. The electronic prescription writer (EPW)used by the physician may interact with many otherhealth care systems in constructing the prescription.For example, the EPW may query previous prescriptionsof the patient from the prescription holdingstore. The EPW may also query patient’s recordsfrom other health care systems.Once the physician has constructed the prescriptionthe EPW may send the prescription to the medicalexpert system which checks (in the case of multidrug treatment) whether the prescribed drugs havemutual negative effects, and whether they havenegative effects with other ongoing medical treatmentof the patient. Then the EPW sends the prescriptionto a medical database system, whichchecks whether the dose is appropriate.The medical database may also provide drugspecificpatient education in multiple languages. Itmay include information about proper usage of thedrug, warnings and precautions, and it can beprinted to the patient. Then the EPW sends the prescriptionto a pricing system, which checks whethersome of the drugs can be changed to a cheaper drug.(This activity is called generic substitution and itaims to promote cost effective drug therapies and toclarify the responsibilities of the prescriber and thepharmacy as well as to enhance both patient autonomyand the efficacy of the pharmaceutical market.)Once the checks and possible changes have beendone the physician signs the prescription electronically.Then the prescription is encrypted and sent toan electronic prescription holding store. Basicallythe holding store may be centralized or distributedstore. The patient will also receive the prescriptionin the paper form, which includes two barcodes. Thefirst identifies the address of the prescription in theholding store, and the second is the encryption keywhich allows the pharmacist to decrypt the prescription.The patient is usually allowed to take the prescriptionto any pharmacy in the country. At thepharmacy the patient gives the prescription to thepharmacist. The pharmacist will then scan both barcodesby the dispensing application, which thenrequests the electronic prescription from the electronicprescription holding store. After this thepharmacist will dispense the drugs to the patient andgenerates an electronic dispensation note. Finallythey electronically sign the dispensation note and47

send it back to the electronic prescription holdingstore.3 Deploying ontologies in prescribingmedicationThe term ontology originates from philosophywhere it is used as the name of the study of the natureof existence (Gryber, 1993). In the context ofcomputer science, the commonly used definition is“An ontology is an explicit and formal specificationof a conceptualization” (Antoniou and Harmelen,2004). So it is a general vocabulary of a certaindomain. Essentially the used ontology must beshared and consensual terminology as it is used forinformation sharing and exchange. On the otherhand, ontology tries to capture the meaning of aparticular subject domain that corresponds to what ahuman being knows about that domain. It also triesto characterize that meaning in terms of conceptsand their relationships.Ontology is typically represented as classes,properties attributes and values. So they also providea systematic way to standardize the used metadataitems. Metadata items describe certain importantcharacteristics of their target in a compact form.The metadata describing the content of a document(e.g., an electronic prescription) is commonly calledsemantic metadata. For example, the keywords attachedto many scientific articles represent semanticmetadataEach ontology describes a domain of discourse.It consists of a finite set of concepts and the relationshipbetween the concepts. For example, withinelectronic prescription systems patient, drug, and e-prescription are typical concepts. These conceptsand their relationships are graphically presented inFigure 3.medicinal productliteralprecedescorrespondspricedrugincludese-prescriptionproduct groupincludessubstitutabledrugIn Figure 1, ellipses are classes and boxes are properties.The ontology includes for example the followinginformation:E-prescription is prescribed by a physician,and it is targeted to a patient.An e-prescription of a patient may precedeother e-prescription of the patient, i.e., thee-prescriptions of the same patient arechained.Each e-prescription includes a drugEach drug has a price, and it may have oneor more substitutable drugs.Each drug corresponds a medicinal product,e.g., acetylsalicylic acid is a drug andits correspondence medicinal product isAspirinEach drug belongs to a product group, e.g.,aspirin belongs to painkillers.Each patient record is associated to a patientand it is written by a physicianThe information of the ontology of Figure 1 canbe utilized in many ways. For example it can be inautomating the generic substitution, i.e., in changingmeciucinal products cheaper medicinal productswithin substitutable products. It has turned out thatin Finland the average price reduction of all substitutableproducts has been 10-15 %. In addition theautomation of the generic substitution decreases theworkload of the pharmacies.In Figure 2, the ontology of Figure 1 is extendedby instances, i.e., it includes the instances of theclasses drug, medicinal product, physician, patientand e-prescription. So allows a wide variety of queriesincluding:Give me all medicinal products thatcorresponds the drug asetylsalicylicacid.What drugs is included in the medicinalproduct Aspirin.isPrescribedByphysicianisTargetedAtpatientGive me all prescriptions prescribed toJack TaylorisWrittenBye-patient recordisAssociatedGive me all prescriptions prescribed byphysician John SmithFigure 1. An e-prescription ontology.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 48

acetylsalicylic acidtypetypeprecedesJack SmithGive me all prescriptions including medicinalproduct named Panadol.drugincludesisPrescribedByphysiciantypetypeisPrescribedBye-prescriptionId 123correspondse-prescriptiontypetypecorrespondstypeisPrescribedByprecedesisTargetedAtpatienttypeAspirinIs TargetedAte-prescriptionId 678John TaylorFigure 2. An extension of the e-prescription ontology.The graphical ontologies of Figure 1 and 2 canbe presented by ontology language in a machineprocessable form. The most commonly used ontologylanguages are XML (Harold. and Scott Means,2002), XML Schema XML (Harold. and ScottMeans, 2002), RDF (Daconta et al. 2003), RDFSchema (Daconta et al., 2003) and OWL (Singhand Huhns, 2005).XML (Extensible Markup Language) is ametamarkup language for text documents. It is thesyntactic foundation layer of the Se-mantic Web.All other technologies providing features for theSemantic Web will be built on top of XML. ParticularlyXML defines a generic syntax used to mark updata with simple human readable tags. An importantfeature of XML is that it does not have a fixed set oftags but it allows user to define tags of their own.For example, various communities have definedtheir specialized vocabularies (set of tags) for variousdomains such as MathMl for mathematics,BSML for bioinformatics and GovML (GovernmentalMarkup Language) for government.typemedicinal producttype4 The architecture of the serviceoriented EPSWe now describe the architecture that can beused for providing the services described in Section2 and 3. The architecture is based on the serviceoriented computing paradigm. Basically, servicesare a means for building distributed applicationsmore efficiently than with previous software approaches.The main idea behind services is that theyare used for multiple purposes. Services are alsoused by putting them together or composing them.Therefore every aspect of services is designed tohelp them to be composed.In the health care sector service oriented computingparadigm provides flexible methods for connectingelectronic prescription system to the otherrelevant health care systems. For example, electronicprescription writer can interact through a Webservice with the health care system that supportspatient records. There may also be components thatare used by different healthcare systems. For example,medical database may provide services formedical information systems as well as for electronicprescription system.The communication is based on Web servicesand SOAP-protocol. Originally they provided away for executing business transactions in the Internet.Technically web services are self-describingmodular applications that can be published, locatedand invoked across the Web. Once a service is deployed,other applications can invoke the deployedservice. In general, a web service can be anythingfrom a simple request to complicated business orehealth processes.The components of the electronic prescriptionsystem are presented in Figure 1.Prescriptionholding storeWeb ServiceinterfaceWeb ServicenterfaceMedicaldatabasesystemPrescription writerWeb ServiceinterfaceExpertdatabasesystemHealth caresystemWeb ServiceinterfaceWeb ServiceinterfacePrescriptionpricingauthorityFigure 3. The interaction of a prescription writer.49

Each component in the architecture communicatesthrough a Web service interface. Further eachmessage is presented as an XML-document andeach XML-document is carried by the SOAP protocol.5 Using BPMN in modelling e-prescription processesThe Business Process Modeling Notation(BPMN) is a standard for modeling business processflows and web services. The BPMN 1.0 and theUML 2.0 Activity Diagram from the OMG arerather similar in their presentation. However, theActivity diagram has not adequate graphical presentationof parallel and interleaved processes, whichare typical in workflow specifications.We now give an overview of the BPMN. First wedescribe the types of graphical objects that comprisethe notation, and then we show how they work togetheras part of a Business Process Diagram (BPD).After it we give a simple electronic prescriptionprocess description using BPD.In BPD there are tree Flow Objects: Event, Activityand Gateway:• An Event is represented by a circle and itrepresents something that happens during thebusiness process, and usually has a cause orimpact.• An Activity is represented by a rounded cornerrectangle and it is a generic term for a workthat is performed in companies. The types oftasks are Task and Sub-Process. So, activitiescan be presented as hierarchical structures.• A Gateway is represented by a diamond shape,and it is used for controlling the divergence andconvergence of sequence flow.In BPD there are also three kind of connectingobjects: Sequence Flow, Message Flow and Association.• A Sequence Flow is represented by a solid linewith a solid arrowhead.• A Message Flow is represented by a dashedline with an open arrowhead and it is used toshow the flow of messages between two separateprocess participants.• An Association is represented by a dotted linewith a line arrowhead, and it used to associatedata and text with flow objects.In Figure 4 we have presented how the process ofproducing electronic prescription (described in Section2) can be represented by a BPD.Send the prescriptionto expert databasesystemSend the prescriptionto medical databaseSend the prescriptionto pricing authorityFigure 4. A BPD-description of the prescription process.6 ConclusionsConstructa prescriptionYesYesYesNonegativeeffectsCheckthe doseCheckthe pricesSign the prescriptionSend the prescription toprescription holding storeIn this article we have illustrated the interoperabilitywithin the healthcare sector from electronicprescriptions point of view. In particular, we illustratedhow XML-based technologies can be utilizedin modelling the concepts that are related to prescriptionwriting, and how web-service technologycan be used in implementing the interoperability ofelectronic prescription system and other healthcaresystems.In addition, we have illustrated how the coordinationof the interoperability required by electronicprescription systems can be automated by utilizingXML-based languages BPMN and BPEL4WS. Thereason for using BPMN is that the BPMN notationis readily understandable for the employees of thehealth care sector. It is also readily understandablefor the business analyst that create the drafts ofhealth care processes as well as for the technicaldevelopers responsible for implementing the technologythat will perform those processes. Also aNoNoNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 50

notable gain of BPMN specification is that it can beused for generating executable BMEL4WS code.A consequence of introducing Semantic Webtechnologies in health care sector is that it significantlychanges the daily duties of the employees ofthe health care sector. Therefore the most challengingaspect will not be the technology but ratherchanging the mind-set of the employees and thetraining of the new technology.ReferencesAntoniou, G. & Harmelen, F., A semantic webprimer. The MIT Press, 2004Bobbie, P., Ramisetty, S., Yussiff, A-L. and Pujari,S. Desgning an Embedded Electronic_prescriptionApplication forHome_Based Telemedicine using OSGiFramework.http://cse.spsu.edu/pbobbie/SharedFile/NewPdfs/eRx-Final2Col.pdf, 2005BPMN- Business Process Modeling Notation(BPMN), http://www.bpmn.org/ , 2006BPEL4WS – Business Process Language for WebSevices.http://www.w.ibm.com/developersworks/ httpwebservices/library/ws-bpel/Chadwick, D., and Mundy, D. 2004. The SecureElectronic Transfer of Prescriptions.http://www.healthinformatics.org/hc2004/Chadwick%20Invited%20Paper.pdf, 2004.Daconta, M., Obrst, L. & Smith, K., The semanticweb. Indianapolis: John Wiley & Sons. 2003.Gruber, Thomas R. Toward principles for the designof ontologies used for knowl-edgesharing. Padua workshop on Formal Ontology.,1993Harold, E. and Scott Means W., XML in a Nutshell.O’Reilly & Associates, 2002.Jung, F., XML-based prescription drug databasehelps pharmacists advise their customers.http://www.softwareag.com(xml/aplications/sanacorp.htm, 2006Liu, C., Long, A., Li, Y., Tsai, K., and Kuo, H.:Sharing patient care records over the WorldWide Web. International journal of MedicalInformatics, 61, p. 189-205. 2001.Mattocks E. 2005. Managing Medical Ontologiesusing OWL and an e-business Registry / Repository.http://www.idealliance.org/proceedings/xml04/papers/85/MMOOEBRR.htmlSingh, M., and Huhns, M., Service OrientedCXDomputing: Semantics Proceses Agents.John Wiley & Sons, 2005.Stalidis, G., Prenza, A. Vlachos, N., Maglavera S.,Koutsouris, D. Medical support system forcontinuation of care based on XML web technology:International journal of Medical Informatics,64, p. 385-400, 2001.Woolman, P. S.: XML for electronic clinical communicationin Scotland. International journalof Medical Informatics, 64, p. 379-383. 2001.51

RosettaNet and Semantic Web ServicesPaavo Kotinurmi⋆ Helsinki University of TechnologyFinlandpaavo.kotinurmi@tkk.fiArmin Haller† DERINational University of Ireland, Galwayarmin.haller@deri.orgAbstractIn this paper we present a semantic B2B gateway based on the WSMX semantic Service OrientedArchitecture to tackle heterogeneities in RosettaNet messages. We develop a rich RosettaNet ontologyand use the axiomatised knowledge and rules to resolve data heterogeneities and to unify unit conversions.We define adaptive executable choreographies, which allow a more flexible integration ofsuppliers using RosettaNet and make it easy to introduce more competition to the supply chain. Thesolution is justified with a scenario-based evaluation and by analysing RosettaNet specifications andtheir implementations.1 IntroductionB2B integrations offer increased speed, less errorsand reduced cost of information exchange betweenbusiness partners. However, integration efforts stillsuffer from long implementation times and high-costs[Preist et al. (2005)]. This has lead to business modelswith simple processes, in which long term rigidpartnerships are established [Kotinurmi et al. (2006)].RosettaNet is a prominent standard for B2B integrationthat represents an agreement on the message exchangepatterns, the message content and a securetransportation mechanism between companies operatingin the IT and electronics industries. The messagecontent of structurally valid RosettaNet PartnerInterface Processes (PIP) is defined by either DTD forthe older PIPs or XML Schema for the recently updatedones. However, the interoperability challengesare only partly solved. DTDs and XML Schemaslack expressive power to capture all necessary constraintsand do not make all document semantics explicit.Being able to express constraints in machineinterpretableformat is expected by RosettaNet expertsas well [Damodaran (2004)].Companies can use the same PIP messages differentlyas the messages contain many optional elements.Some companies may support only partsof the enumerated values for packaging-units, unitof measurements and currencies that are in theRosettaNet dictionaries. When the number of partnersincreases, handling these heterogeneities comesincreasingly important and point-to-point syntactictransformations using scripts are not the most lucrativeoption. This can lead to the situation that buyersuse only one supplier as the information systemsdo not easily support more competitive arrangements.The example scenario that we use in this paper highlightsthis from the buyer’s (requesters) role in a collaboration.Quotes are asked from multiple suppliers(service providers) that use the messages differently.Rules handling the data heterogeneity of theirresponse messages are required to decide on the bestsupplier for a given order. In the interactions the useof RosettaNet PIPs 1 3A1 for quotes and 3A4 for purchaseorders are handled.We propose a semantic B2B gateway, which buildsupon the Web Service Modelling eXecution environment(WSMX) [Haller et al. (2005)]. This solutionapplies semantic Web Service (SWS) technologies toRosettaNet collaborations. However, we do not implythe use of SWS technologies to the business partners,as the current knowledge of those technologies is low.In our example scenario only the requester uses SWStechnologies, the partners simply use current RosettaNetXML interfaces.The main contributions of this paper are as follows:• We encode the information exchanged in RosettaNetPIP3A1 messages in a formal ontology.The ontology includes constraints that cannotbe represented in current PIP message schemes,such as dependency between fields.• We define axioms in the ontology, which constrainthe interpretations of certain elements inRosettaNet messages (e.g. units of measurement,packaging size)• We further add domain specific rules to theknowledge base that are applied at run time to1 http://www.rosettanet.org/pipdirectoryNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 52

esolve data heterogeneities in the RosettaNetmessages.• We define and execute a choreography, which allowsto easily integrate new partners to our solutionand to handle the data heterogeneities introduced.The paper is structured as follows: first we give abrief introduction to semantic Web Services in section2. Section 3 presents the architecture of our semanticB2B gateway and describes its components.Section 4 outlines our use case scenario, its designtime modelling tasks and presents evaluation results.In section 5 we position our solution to related work.Finally the paper is concluded in section 6.2 Semantic Web ServicesThe Web Service Modeling Ontology (WSMO) [Romanet al. (2005)] provides a conceptual model anda language for semantic markup describing all relevantaspects of general services, which are commonlyaccessible through a web service interface. However,the semantic mark up provided in WSMO canbe grounded to any service (e.g. CORBA, GRIDetc.) The ultimate goal of such a markup is to enablethe (total or partial) automation of tasks (e.g.discovery, selection, composition, mediation, executionand monitoring) involved in both intra- and interenterpriseintegration. The WSMO conceptual modelis formalised by the Web Service Modeling Language(WSML) family of ontology languages, used to describeWSMO elements (goals, ontologies, services,mediators). WSML consists of a number of variantsbased on different logical formalisms. The differentvariants of the WSML correspond with differentlevels of logical expressiveness and are bothsyntactically and semantically layered. In addition,WSMO defines the underlying model for the WebService Execution Environment (WSMX). WSMXis an integration platform conforming to the principlesof a Service Oriented Architecture (SOA) whichfacilitates the integration between different systems.The integration process is defined by adaptive operationalsemantics, defining the interactions of middlewareservices including discovery, mediation, invocation,choreography, repository services, etc. Thus,WSMO, WSML and WSMX form a coherent frameworkcovering all aspects of semantic Web Services(SWS).Although WSMX aims to allow a dynamic discoveryof partners in the collaboration, our focus lies onhow SWS technology tackles the data heterogeneitiesin e-business collaborations. This has two reasons,first current business practice does not consider anintegrated discovery and invocation of services. The“discovery” of business partners is conducted whenthe infrastructure is set up and are commonly basedon well-established and long-running business relations.Second, the technology required for a dynamicdiscovery of rich semantic descriptions is not matureenough in business settings. The reasoners to performmatchmaking only scale on light semantic descriptionsof services with very limited expressivity.Thus we omit the functional description of a service(its capability description which is mainly used duringthe dynamic discovery) and the formalisation of arequester’s goal. Instead we avail of the WSMO serviceinterface - a description of the communicationpatterns according to which a service requester consumesthe functionality of the service. The WSMOservice choreography [Roman and Scicluna (2006)]is based on the formalism of Abstract State Machines(ASMs) [Gurevich (1994)] and allows to model dataexchanged in every state of a service execution. Itconsists of three core elements (1) a signature, (2)a grounding, and (3) transition rules. The signaturedescribes static parts of state descriptions definedin terms of imported ontologies. Each statehas a grounding associated defining how concepts inthe ontology are linked with messages. Transitionrules express changes of states in the choreographyby changing the set of ontology instances (adding, removingand updating instances to the signature ontology).An example of such a choreography descriptionis described in section 4.1.3 Semantic B2B GatewayIn this section we introduce the architectural considerationtaken into account for a semantically enhancedRosettaNet B2B integration. We detail theimplementation of a generic semantic B2B gateway,being truly agnostic about whether the partners useWSML messages or any ordinary schema language.Our solution is based upon the WSMX platform.WSMX follows a component based design with adaptiveoperational semantics. Every component providessome service and can be orchestrated withinWSMX as required. A semantic RosettaNet B2BGateway as defined in this paper relies on the followingfour components:• The Adapter Framework consists of B2Badapters to lift and lower XML instances in themessages sent from the partners to WSML class53

hierarchies and a middleware adapter connectingto the back-end applications of the requester.• The Choreography Engine executing externalservice by evaluating a WSMO choreographydescription. The actual invocation of the serviceand the choice of the right transport level is performedby the Communication Manager.• The Resource Manager constitutes the knowledgebase and its interface and provides accessto the local component repositories that storedefinitions of WSMO entities. In our case,these are the domain ontology, the domain specificrules stored in our knowledge base and thechoreography specifications stored in the repository.• The Reasoner allows to perform queries on theknowledge base of facts and rules to determinethe answer by logical deduction.WSMX is shipped with these standard components.However, the adapter instances have to be builtbased on the requirements from RosettaNet interfaceswith service providers. The complete architecture ofour solution is depicted in figure 1.Knowledge BaseReasonerChoreography EngineWeb Service InterfaceWeb Service InterfaceMiddlewareWSMLWeb Service InterfaceMiddlewareAdapterWeb Service InterfaceXMLService RequesterWSMLWS InterfaceB2B AdapterWS InterfaceSemantic B2B gatewayXMLXMLService Provider AWS InterfaceService Provider BWS InterfaceB2B ServerB2B ServerFigure 1: Overview of B2B gateway architecture3.1 Resource ManagerThe resource manager coordinates the access to therepositories of the different components in the architectureand as such gives the illusion of a centralrepository. This repository acts as a knowledge baseand has to be populated with the ontology artifactsused in the different components in our B2B gateway.Most importantly, at least one domain ontologyto formally capture the message content exchangedin any of our RosettaNet collaborations is required.Ideally existing domain ontologies should be reused.Since an industry wide recognised ontology for businessmessages has not been established yet, we builtthe ontology ourselves. It is based on the RosettaNetspecification, which itself can be regarded as a lightweightontology.However, the task is to not only simply translatethe DTDs or XML Schemas to a richer and formallanguage (i.e. WSML), but to model all the constraintson the semantics of the business documents.We include constraints which are not expressible inDTD or XML Schema and capture implicit knowledgeprovided in the RosettaNet message guidelinesand accompanying documentation to facilitate thepromised benefits of automatically resolving data heterogeneities.An example of the first, a cardinality constraint notexpressible in current RosettaNet messages schemasare the constraints imposed on the “BusinessDescription”element used in all RosettaNet PIPs. The “BusinessDescription”element includes business propertiesthat describe a business identity and its location.At least one of the possible three subelements “businessName”,“GlobalBusinessIdentifier” or “Partner-BusinessIdentification” has to be provided in order tobe a valid PIP. It is easy to include such constraints inour WSML ontology as shown in listing 1:✞☎concept businessDescriptionbusinessname ofType (0 1) businessNameglobalbusinessidentifier ofType (0 1)globalBusinessIdentifierpartnerbusinessidentification ofTypepartnerBusinessIdentificationnfpdc#relation hasValue validBusinessDescriptionendnfpaxiom validBusinessDescriptiondefinedByforall ?x ( ?x memberOf businessDescription implies?y memberOf businessName or?y memberOf globalbusinessidentifier or?y memberOf partnerbusinessidentification).✝Listing 1: Cardinality Constraints spanning multipleelementsListing 2 shows examples of implicit knowledgewe have captured in our RosettaNet ontology. For example,RosettaNet defines a list of 335 possible valuesfor unit of measurements, with the logical relationshipsbetween values unspecified. We made theselogical relations explicit and included these axiomatisationsin our ontology. The first axiom “resolveMeasurementUnitType”in listing 2 shows how the measurementunits defined with natural language text inthe RosettaNet PIPs can be resolved to its correspondingnumerical value. The numerical value can subsequentlybe used for further calculations (c.f. section4.1). The second part of the listing defines a function✆New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 54

used to relate a kilogram value to its equivalent poundvalue.✞☎277 axiom resolveMeasurementUnitType278 definedBy279 forall ?x(?x[globalProductUnitOfMeasurementCodehasValue ”dozen”] memberOf quoteLineItem implies?x[globalProductUnitOfMeasurementCode hasValue”12”]).280 forall ?y(?y[globalProductUnitOfMeasurementCodehasValue ”10−pack”] memberOf quoteLineItemimplies ?y[globalProductUnitOfMeasurementCodehasValue ”10”]).281282 relation poundKilo (ofType productQuantity, ofTypeproductQuantity)283 nfp284 dc#relation hasValue poundKiloDependency285 endnfp286287 axiom poundKiloDependency288 definedBy289 forall ?x,?y (290 poundKilo(?x,?y) equivalent291 ?x memberOf productQuantity and292 ?x[globalProductUnitOfMeasureCode hasValue293 ”Kilogram”]294 memberOf quoteLineItem and295 ?y memberOf productQuantity and296 ?y[globalProductUnitOfMeasureCode hasValue297 ”Pound”]298 memberOf quoteLineItem and299 ?x = wsml#numericDivide(?y,?x,0.45359237)).✝Listing 2: Definitional facts in the domain ontology3.2 Adapter FrameworkThe adapter framework provides transformationfunctionality for every non-WSML message sent tothe B2B gateway. In our scenario, two adapter instancesare required. One for every RosettaNet collaborationpartner, who still use XML Schema orDTD based RosettaNet messages and one to connectto the middleware system, providing access to theback-end applications.The first adapter receives every RosettaNet messageand applies the lifting or lowering rules definedin the adapter to map every message instance basedon its source schema (in our case XML-Schema) toa WSML instance based on its ontological schema.Listing 3 shows one incoming XML instance andlisting 4 its corresponding WSML instance after thetransformation rules were applied 2 .It has to be noted that the adapters act for WSMXas the actual service provider. The adapter functionality(i.e. the Web Service endpoint of the adapter)is registered as a service with WSMX and not theRosettaNet service of the partner itself. Thus theadapter is further responsible to execute the correctendpoint of the partner service. However, adapters2 More details on the lifting and lowering of XML Schema toWSML can be found in Kotinurmi et al. (2006).✆only perform data manipulation, their interface behaviourreplicates the behaviour of the underlyingpartner service.The second adapter instance receives every internalmessage sent from the middleware bus, representinga gateway to every non Web Service compliant backendapplication.3.3 ReasonerThe Reasoner is required to perform query answeringoperations on the knowledge base, which itself ispopulated with domain ontologies and domain specificrules at design time and ontology instances atrun time. Our reasoner has to handle the WSML-Rule variant and should have built-in predicates forhandling basic data-types, basic arithmetic functionsas well as basic comparison operators.Dedicated reasoners for the different WSML variantsare still under development. However, there area number of implementations based on existing reasonersfor WSML-Rule available: one is based onthe FLORA-2 reasoner, which is also used in themediation component of WSMX. FLORA-2 is arule-based knowledge representation formalism anda reasoner for this formalism. It is based on F-Logic[Kifer et al. (1995)], a frame-based logic, WSML isalso based upon. Further, there are translations fromWSML-Rule to F-Logic available. The reasoner includesa Web Service interface which is used by theservice requester to perform queries on the knowledgebase. Messages received from the back-end systemsof the requester are sent through the middleware.Thus, the middleware adapter has to encoderules how to translate requests from these back endsystems to queries on the knowledge base and returnthe results. This is out of the scope of this paper andwill be addressed in future work.3.4 Choreography EngineThe Choreography Engine as part of WSMX is responsibleto control the sending and receiving ofmessages and update the state of the choreographyaccording to the message content. In contrast tothe common use of choreography languages as nonexecutabledescriptions, the engine operates and executesWSMO choreography description such as theone defined in section 4.1.As mentioned earlier WSMO choreographiesare modelled as Abstract State Machines and areprocessed using standard algorithms during runtime.The state in the machine is represented by ontology55

44 47 20448 52 dozen53 92 93 Better product94 102 114 115 USD116 117 198118 119 Listing 3: XML message instance62 instance QuoteLineItem1 memberOf rfq#quoteLineItem65 rfq#globalProductUnitOfMeasurementCode hasValue ”dozen”82 instance quantitySchedule1 memberOf83 core#quantitySchedule84 core#productQuantity hasValue ”204”106 instance substituteProductReference1 memberOf107 core#substituteProductReference108 core#GlobalProductSubstitutionReasonCode109 hasValue ”Better product”129 instance totalPrice1 memberOf core#totalPrice130 core#financialAmount hasValue FinancialAmountTot132 instance FinancialAmountTot memberOf133 core#FinancialAmount134 core#globalCurrencyCode hasValue USD135 core#monetaryAmount hasValue ”198”Listing 4: and its lifted WSML instanceinstances. According to the instance data, a transitionrule is selected from the rule base within a choreography.The consequent of the rule is interpreted and theontology instance is modified accordingly. It is theresponsibility of the Choreography Engine to maintainthe state of a conversation and to take the correctaction when that state is updated. For example, theupdate of the state of a choreography instance maybe the result of a message received from a serviceprovider.Since WSMO choreography descriptions are expressedin a constraint-based fashion, they are verysuitable to be changed and extended once new partnersare introduced into a collaboration. It is easy toinclude transition rules triggering on certain parts of amessage sent only by one partner to introduce partnerspecific data handling (c.f. section 4.1.4 EvaluationIn our example scenario, we discuss the handlingof heterogeneous partners engaged in quoting for aproduct with a PIP 3A1 Request for Quote (RFQ).We describe the collaboration set up and orchestrationthrough examples from the requester’s (buyer’s)point-of-view. Then we discuss evaluation resultsthrough analysing existing PIP message specificationsand example instances and the general benefitsfrom our example scenario.4.1 Collaboration Set UpIn order to retrieve RFQ messages from differentpartners at run-time and still being able to processtheir response messages, the requester has to set upmapping rules and define a choreography descriptionin the B2B gateway at design time.Mapping rules developed in the Collaboration Set-Up phase capture any data heterogeneity that is notresolved by the definitional facts in the domain ontology(c.f. section 3.1).These domain specific rules (conversion relationsin our case) define how attribute values in the differentWSML instances can be transformed. One suchexample is given in listing 5. It defines a function tocalculate the unit price by taking the “financialAmount”and “productQuantity” given in the RosettaNetontology instance. This rule can be used by the requesterto compare the prices of two or more partners.The “financialAmount” in a PIP3A1 RFQ can referto different quantities of the product. We made thedifferent packaging sizes and its corresponding valueexplicit in the ontology as described earlier in section3.1. Now the requester can query the knowledgebase to automatically compare the prices provided as“financialAmount” by the partners transparently on aunit basis.✞☎295 relation unitPrice (ofType financialAmount, ofTypeproductQuantity, ofType decimal)296 nfp297 dc#relation hasValue unitPriceDependency298 endnfp299300 axiom unitPriceDependency301 definedBy302 forall ?x,?y,?z (303 unitPrice(?x,?y,?z) equivalent304 ?x memberOf financialAmount and305 ?y memberOf productQuantity and306 ?z = wsml#numericDivide(?z,?x,?y)).✝Listing 5: Definition of a conversion relationNext the choreography description has to be defined.In order to allow a collaboration with its suppliers,the choreography interface of the requester inour scenario has to be compliant with the interfacebehaviours of the partners providing a RFQ response.Since all suppliers in our Supply Chain use RosettaNet,there is already agreement on the message ex-✆New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 56

change patterns. However, there are still mismatcheson the messages sent and received in the collaboration.Thus, we introduce rules in the choreography(c.f. listing 6) of the requester handling data mismatchesin the RosettaNet messages.It has to be noted that the model defined in listing 6can actually be regarded as an orchestration in termsof the WSMO conceptual model. Orchestrations inWSMO are modelled in the same way as choreographies.In fact the only difference is on the conceptuallevel. Whereas WSMO choreographies describe theinterface behaviour of one service, orchestrations describehow a service makes use of other WSMO servicesor goals in order to achieve its capability. In ourcase, we alter the interface behaviour of the requesterby introducing rules, which are for example calling acurrency conversion service of an external party. Thisinformation in fact would not be part of the choreographydescription. However, in the WSMX system itis executed in the same engine. Only when the requesterwants to publish its interface behaviour to apartner, it is relevant to decide on the abstraction levelof what parts of the orchestration he wants to publish.An extract 3 of such a choreography is shown inlisting 6. Please note that the “//...” symbol denotesparts omitted in the listing. The namespace declarationsin the first lines of a WSMO choreography definitionare also omitted in the listing.✞☎31 choreography32 stateSignature33 importsOntology {34 ”http://www.wsmx.org/ontologies/rosetta/coreelements”,35 ”http://www.m3pe.org/ontologies/rosetta/CTRLASM”36 }37 out rfq#Pip3A1RFQRequest withGrounding ”http://example.org/webServices#wsdl.interface(ServicePortType/RFQ/Out)”38 out curr#currConvRequest withGrounding ”http://www.webcontinuum.net/webservices/ccydemo.wsdl#”39 in rfq#Pip3A1RFQResponse withGrounding ”http://example.org/webServices#wsdl.interface(ServicePortType/RFQ/In)”40 transitionRules41 if (?Pip3A1RequestForQuoteRequest[42 fromRole hasValue ?fromRole,43 globalDocumentFunctionCode hasValue44 ?globalDocumentFunctionCode,45 quote hasValue ?quote,46 thisDocumentGenerationDate hasValue47 ?thisDocumentGenerationDate,48 thisDocumentIdentifier hasValue ?thisDocumentIdentifier,49 toRole hasValue ?toRole50 ] memberOf rfq#Pip3A1RFQRequest) and51 //...171 update(?controlledState[currentState hasValue 1]memberOf ctrlasm#controlledState)172 endIf173174 if (?controlledState[175 currentState hasValue 1176 ] memberOf ctrlasm#controlledState) and3 The full listing can be found at:http://www.m3pe.org/ontologies/rosettaNet/177 exists ?Pip3A1RequestForQuoteRespond178 (?Pip3A1RequestForQuoteRespond memberOf rfq#Pip3A1RFQResponse) and179 //...357 update(?controlledState[currentState hasValue 2]memberOf ctrlasm#controlledState)358 endIf359360 if (?controlledState[361 currentState hasValue 2362 ] memberOf ctrlasm#controlledState) and363 exists ?globalProductSubstitutionReasonCode,364 ?productIdentification365 (?substituteProductReference[366 globalProductSubstitutionReasonCode hasValue367 ?globalProductSubstitutionReasonCode,368 productIdentification hasValue ?productIdentification369 ] memberOf core#substituteProductReference) then370 add( # memberOf rfq#Pip3A1RFQRequest)371 endIf372373 if (?controlledState[374 currentState hasValue 2375 ] memberOf ctrlasm#controlledState) and376 exists ?globalCurrencyCode, ?monetaryAmount377 (?totalPrice[378 globalCurrencyCode hasValue ?globalCurrencyCode,379 monetaryAmount hasValue ”USD”380 ] memberOf rfq#totalPrice) then381 add( # memberOf curr#currConvRequest)382 endIf✝Listing 6: Choreography in WSMLAs described in section 2, a WSMO choreographydefinition consists of a state signature (c.f. line number32-39 in listing 6), which imports one or possiblymany ontologies and transition rules (c.f. line number40-382), which are basic operations, such as adding,removing and updating on the instance data of thesignature ontology. In our example we import twoontologies, the message ontology capturing conceptsin RosettaNet messages and a control State ASM ontology,which allows us to define termination and toimpose an explicitly modelled control flow for certainparts of the choreography.The transition rules in listing 6 starting at line 39capture only a small number of heterogeneities possiblyoccurring in a RFQ collaboration. New rules canbe added when new partners are introduced into thescenario with different data heterogeneities.The first transition rule (lines 41-172) defines theontology schema of the message sent by the buyerA (the mode “out” in the “stateSignature” states thepassing direction, thus that the message is sent) andthat the “currentState” of the machine is updated tothe value “1” after a successful transmission of themessage. The grounding only defines one WSDL interfaces.However, since the requesters wants to getquote response from multiple providers, the actualgrounding list would include more endpoints.The second rule (line 174-358) checks the controlstate, to ensure that the “Pip3A1RFQRequest”message was successfully sent and that the✆57

“Pip3A1RequestForQuoteRespond” is received.The constraints in the antecedent of the rule act asa schema validation in the sense that the rule onlytriggers if the message returned by a partner includesall required fields. For space reasons these parts ofthe choreography (between lines 179-357) are notshown in listing 6.The last two transition rules only trigger on a partof the message instance which differs between suppliers.These are rules the requester includes to anticipatepossible data heterogeneities. If the requesterknows that some partners will provide the amount inUSD, the fourth transition rule (lines 373-382) ensuresthat a currency conversion service is used tocalculate the value in Euro. The third transition rule(lines 360-371) fires if a partner provides a substituteproduct. It results in sending out a new RFQ request.4.2 Evaluation ResultsThe evaluation is accomplished by analysing RosettaNetPIPs, real instances obtained from companiesand by using a descriptive scenario to demonstrate theutility of our solutions.We first took a sample 56 PIP message specificationsrepresenting 29,5 % of the total 190 messages.The PIPs cover product information, ordermanagement, inventory management and manufacturingclusters in RosettaNet classification. Measuringunits were used in 24 (43%) and currency informationin 28 (50 %) of the PIP messages in the sample.Moreover, we analysed two production messageinstances and their processing instructions. The PIPswere PIP 3A4 Request Purchase Order and 2A13 DistributeMaterial Composition Information. The instancesuse only a part of the whole PIP data modeland most of the optional elements are not used. Bothmessages used unit of measure information, but onlyone contained currency information. Both supportedonly kilograms as mass units and Euros as currencies.In both cases, the PIP had just been taken intouse with a couple of partners.The axiomatised message semantics and mappingrules introduced in this paper would have both wideusage across different PIPs and that current real-lifeXML-based integrations lack this kind of functionalitymotivates the need for this kinds of solutions.This also provides practical examples of the benefitsof SWS technologies to current implementations.Although axiomatised knowledge can also be representedin XSLT scripts or custom codings, the mainadvantage of expressing these kinds of definitionalfacts in an ontology is its reusability. The “pound-Kilo” relation captures a conversion that is generallyapplicable across a wide range of PIP interactions.This reuse is a major advantage over scripts, whichare typically rewritten for every schema conversioncausing point-to-point integrations that are hard tomaintain and test that validation covers the necessaryconstraints.The example scenario discussed throughout thepaper on quoting and purchasing highlights the requestersneed to save on purchasing 4 . Having suppliersfrom different countries brings heterogeneitiesas the partners are likely to use different currenciesor other measuring or packaging units. Benefits forthe buyer result from decreased costs of purchasingas the best value deals can be selected based on bestquotes. The suppliers benefit from being able to easierintegrate to the buyer without having to make potentiallycostly changes to their current integration interfaces.In addition, the heterogeneities of using differentstandards, such as UBL, are easier handled bylifting them to ontologies where mappings can easilybe performed.5 Related WorkThere are a number of papers discussing the use ofSWS to enhance current B2B standards. Some concentrateon ontologising B2B standards. Foxvog andBussler (2005) describe how EDI X12 can be presentedusing WSML, OWL and CycL ontology languages.The paper focuses on the issues encounteredwhen building a general purpose B2B ontology,but does not provide an architecture or implementation.Anicic et al. (2006) present how two XMLSchema-based automotive B2B standards are liftedusing XSLT to OWL-based ontology. They use a twophasedesign and run-time approach similar to our’s.The paper is based on different B2B standards and focusesonly on the lifting and lowering to the ontologylevel.Others apply SWS technologies to B2B integrations.Preist et al. (2005) presented a solution coveringall phases of a B2B integration life-cycle.The paperaddresses the lifting and lowering of RosettaNetXML messages to ontologies, but no richer knowledgeis formalised or used on the ontological level.Trastour et al. (2003b,a) augment RosettaNet PIPswith partner-specific DAML+OIL constraints and useagent technologies to automatically propose modificationsif partners use messages differently. Their4 See http://www.m3pe.org/ontologies/rosettaNet/ for completeexamples of XML instances from which the heterogeneities can beautomatically solved.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 58

approach of accepting RosettaNet in its current formand lifting to semantic languages is similar to ours,but we go further by axiomatising implicit knowledgeand by providing mappings to resolve data heterogeneities.6 ConclusionOur semantic B2B gateway allows a buyer to tackleheterogeneities in RosettaNet interactions. The solutionapplies axiomatised knowledge and rules to resolvedata heterogeneities and to unify various unitconversions using functions to capture definitionalfacts, such as the relation between pounds and kilograms.Such relations between different enumerationsare not specified by RosettaNet. The conversionshave potential use in significant portion of the190 RosettaNet PIP messages as shown in the evaluationof the PIP schemas and the current observed integrationslack the support for such heterogeneity. Wefurther defined adaptive executable choreographies,which allow a more flexible integration of suppliersand make it easy to introduce more competition tothe supply chain.Our future work includes further extending andtesting our solution and developing an adapter to thereasoning component, which translates the requestssent from the back-end systems of the requester toqueries on the knowledge base and returns the resultsto the applications.AcknowledgementThe authors thank Eyal Oren and Tomas Vitvarfor fruitful discussions leading to the ideas of thispaper. This material is based upon works supportedby the Finnish Funding Agency for Technologyand Innovation (Tekes) and the GraduateSchool for Electronic Business and Software Industryand the Science Foundation Ireland under GrantNo. SFI/04/BR/CS0694.ReferencesNenad Anicic, Nenad Ivezic, and Albert Jones. AnArchitecture for Semantic Enterprise ApplicationIntegration Standards. In Interoperability of EnterpriseSoftware and Applications, pages 25–34.Springer, 2006. ISBN 1-84628-151-2.Suresh Damodaran. B2b integration over the internetwith xml: Rosettanet successes and challenges.In Proceedings of the 13th internationalWorld Wide Web conference on Alternate track papers& posters, pages 188–195, 2004.Douglas Foxvog and Christoph Bussler. OntologizingEDI: First Steps and Initial Experience. InInt. Workshop on Data Engineering Issues in E-Commerce and Services, pages 49–58, 2005.Yuri Gurevich. Evolving algebras 1993: LipariGuide. In Egon Börger, editor, Specification andValidation Methods, pages 9–37. Oxford UniversityPress, 1994.Armin Haller, Emilia Cimpian, Adrian Mocan, EyalOren, and Christoph Bussler. WSMX – A SemanticService-Oriented Architecture. In Proc. of the3rd Int. Conf. on Web Services, pages 321 – 328.IEEE Computer Society, 2005.Michael Kifer, Georg Lausen, and James Wu. Logicalfoundations of object-oriented and frame-basedlanguages. Journal of the ACM, 42(4):741–843,1995.Paavo Kotinurmi, Tomas Vitvar, Armin Haller, RayBoran, and Aidan Richardson. Semantic web servicesenabled b2b integration. In Int. Workshop onData Engineering Issues in E-Commerce and Services,June 2006.Chris Preist, Javier Esplugas Cuadrado, Steve Battle,Stuart Williams, and Stephan Grimm. AutomatedBusiness-to-Business Integration of a LogisticsSupply Chain using Semantic Web ServicesTechnology. In Proc. of 4th Int. Semantic Web Conference,2005.Dumitru Roman and James Scicluna. Ontologybasedchoreography of wsmo services. Wsmo finaldraft v0.3, DERI, 2006. http://www.wsmo.org/TR/d14/v0.3/.Dumitru Roman, Uwe Keller, Holger Lausen, Josde Bruijn, Rubn Lara, Michael Stollberg, AxelPolleres, Cristina Feier, Christoph Bussler, and DieterFensel. Web Service Modeling Ontology. AppliedOntologies, 1(1):77 – 106, 2005.David Trastour, Claudio Bartolini, and Chris Preist.Semantic Web support for the business-to-businesse-commerce pre-contractual lifecycle. ComputerNetworks, 42(5):661–673, 2003a.David Trastour, Chris Preist, and Derek Coleman.Using Semantic Web Technology to Enhance CurrentBusiness-to-Business Integration Approaches.In Proc. of the Int. Enterprise Distributed ObjectComputing Conference, pages 222–231, 2003b.59

CASCOM: Context-Aware Service Coordination in MobileComputing EnvironmentsHeikki HelinTeliaSonera Finland OyjP.O.Box 970, FIN-00051 Sonera, FINLANDheikki.j.helin@teliasonera.comAhti SyreeniTeliaSonera Finland OyjP.O.Box 970, FIN-00051 Sonera, FINLANDahti.syreeni@teliasonera.comAbstractThe research project CASCOM will implement, validate, and trial value-added support for businessservices for mobile workers and users across mobile and fixed networks. The vision of the CAS-COM approach is that ubiquitous application services are flexibly co-ordinated and pervasivelyprovided to the mobile users by intelligent agents in dynamically changing contexts of open, largescale,pervasive environments. The essential approach of CASCOM is the innovative combinationof intelligent agent technology, semantic Web services, peer-to-peer, and mobile computing for intelligentpeer-to-peer (IP2P) service environments. The services are provided by software agentsexploiting the co-ordination infrastructure to efficiently operate in highly dynamic environments.1 Introductionproviding better customer satisfaction. For serviceproviders, CASCOM provides an innovative platformfor business application services.The essential approach of CASCOM 1 is the innovativecombination of agent technology, semantic Web The project will carry out highly innovative researchaimed at providing a framework for agent-services, peer-to-peer, and mobile computing forintelligent peer-to-peer mobile service environments.The services of CASCOM environment are ronments. CASCOM will integrate and extend existbaseddata and service co-ordination in IP2P envi-provided by agents exploiting the CASCOM coordinationinfrastructure to efficiently operate in computing, service co-ordination, and P2P computingtechnologies in areas such as agent-based mobilehighly dynamic environments. The CASCOM intelligentpeer-to-peer (IP2P) infrastructure includes service environment with its agents and coingin mobile environments. A generic, open IP2Pefficient communication means, support for contextawareadaptation techniques, as well as dynamic plemented and deployed in CASCOM mostly asordination mechanisms will be prototypically im-service discovery and composition planning.open-source software enabling instant take-up andCASCOM will implement and trial value-added use within European and world community.support for business services for mobile workers and In general, it is expected that the outcomes ofusers across mobile and fixed networks. The vision CASCOM will have significant impact on the creationof a next-generation global, large-scale intelli-of the CASCOM approach is that ubiquitous applicationservices are flexibly co-ordinated and pervasivelyprovided to the mobile users by agents in methods for service provision, discovery, composigentservice environment. Both, research results ondynamically changing contexts of open, pervasive tion and monitoring, and the deployed prototype ofenvironments.an open IP2P service environment in the context ofFor end users, the CASCOM system provides nomadic computing will advance the state of the artseamless access to semantic Web services anytime, of European and world knowledge in areas relatedanywhere, and using any device. This gives freedom to the deployment of services in open systems.to mobile workers to do their business whenever andwherever needed. For network operators, CASCOM 2 Technical Approachaims towards vision of seamless service experienceFigure 1 depicts the technologies that we use in theCASCOM project. Software agents will be a key1 http://www.ist-cascom.orgNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 60

Figure 1: CASCOM Technologiestechnology to address the challenges of the CAS-COM architecture (see Figure 2). IP2P networksprovide an environment for agents to collaborate aspeers sharing information, tasks, and responsibilitieswith each other. Agents help to manage the P2Pnetwork complexity, and they will improve thefunctionality of conventional P2P systems. Our innovationsin this domain will concern the developmentof context-aware agent-based semantic Webservices, and flexible resource-efficient coordinationof such services in the nomadic computingfield. Further, context-awareness is investigatedin the context of IP2P environment and we will developcontext-aware agents which provide variousbusiness application services.Using agents in wireless environments has beenstudied extensively. We will build on the previouswork by using existing agent platforms as a basis ofour architecture. However, the P2P aspects are insufficientlytaken into account in these platformsand therefore our research represents advancementsin this direction. CASCOM will provide solutionsfor agent communication between agents withoutassumption of any fixed infrastructure.Service co-ordination mechanisms of P2P systemscan be applied to multi-agent systems to improvetheir efficiency. Although this may be acceptedon a conceptual level, the combination ofagents and P2P environments certainly deservesmore innovative research, especially regarding nomadicenvironments. As an example, many P2Poverlay network algorithms lacks support for rapidnode movements. The dynamic topology of IP2Pnetworks, characteristics of wireless network connections,and the limited capacity or mobile devicespose several challenges that have been addressedinadequately in service discovery architectures. InCASCOM, we will investigate mechanisms for servicediscovery algorithms for dynamic IP2P environments.The problem of service co-ordination can besplit into several sub problems: discovery, compositionplanning, execution monitoring, and failurerecovery. CASCOM will advance the state of the artby carrying out innovative research on how theseproblems can be solved in IP2P environments. EspeciallyCASCOM will provide flexible and efficientmatching algorithms to be performed in largescale and resource limited IP2P environments.Using AI planning formalisms in service compositionand planning are developed for problemswhere the number of operators is relatively smallbut where plans can be complex. In Web servicecomposition for open, large-scale IP2P environmentsplanning methods dealing with huge numberof possible service are required. However, plans arenot necessarily very complex, and therefore planningmethods must follow more closely the structureof the service directories. CASCOM will developplanning mechanisms that establish plan fragmentsdirectly on top of the service directory to solve thisproblem.AcknowledgementsThis work has been supported in part by the EuropeanCommission under the project grant FP6-IST-511632-CASCOM. The authors would also like tothank all CASCOM consortium members (DFKI(Germany), TeliaSonera (Sweden/Finland),ADETTI (Portugal), EPFL (Switzerland), URJC(Spain), FrameTech (Italy), UniBAS (Switzerland)and EMA (Finland)).Figure 2: The CASCOM IP2P architecture61

Towards reflective information managementJari Yli-Hietanen * Samuli Niiranen *†* Digital Media Institute* Digital Media InstituteTampere University of Technologyjari.yli-hietanen@tut.fiAbstractTampere University of Technology† Visiting at Decision Systems GroupBrigham and Women’s HospitalHarvard Medical Schoolsamuli.niiranen@tut.fiWe discuss the evolution of information management and argue that the current paradigm is reachingits limits in terms in ability to provide more utility. Our proposition is that this is due to the natureof current information management tools being built around the paradigm of a document, in theabstract sense of the word, sharing many of their limitations. We discuss reflective informationmanagement breaking free from this through a native ability to directly reflect on information.1 IntroductionProponents of evolutionary psychology, StevenPinker being perhaps the most famous example,argue that the mental evolution of primates towardsthe sentient man was pushed forward by the environmentsfaced by early hunter-gatherers (e.g., see(Pinker, 1999)). The result, Homo Sapiens, is theultimate hunter-gatherer, a social animal who’smental and physical skills are tuned at co-operationand collaboration at the small scale of hunting parties.Considering the role of information management inthis context, humans are adept at processing andstoring information natively when it comes to mattersclose to the hunter-gatherer life style. Naturalspoken language as a tool of collaboration, especiallywhen the collaborating parties share a commonground and history, is an evolutionary adaptationproviding an extremely efficient informationmanagement tool for the hunter-gatherer (see(Pinker, 2000)). The emergence of complexity, aswe currently understand the term, in human societiesbeyond the ad-hoc organizations of the huntergatherercoincided with the growing populations andrelated urban settings facilitated by the surplus offood created by advancing agriculture. The resultinglarge, complex societies needed new informationmanagement tools to handle the large amounts informationrelated to their management. Written languageand its practical manifestation, the writtendocument, emerged as the tool to handle the informationoverflow created by advancing societies.The use of documents to convey and store the largemass of information created by advancing agricultural,and later on industrial, societies represented afundamental break from innate information management.Although documents are written in expressivenatural language, they fail to reflect the innatelinked nature of information management native tothe human mind. When we document something, wealmost always lose a part of the original piece ofinformation. Typically some part of the originalcontext, of which we are not necessarily even consciousof at the time of the documentation, is lost inthe process. With documentation, the worst case isthat relevant information is lost into a sea of unrelateddata without meaning and context. The keyquestion arising from these considerations is if continuingon the documentation-centric path is theoptimal choice in the contemporary world of thedigital computer. Also, although highly expressive,the interpretation of natural language compositionsis highly context-sensitive and currently relies onour native disambiguation facilities and a sharedcommon ground with the composer.What comes after digital documentation in informationmanagement? Looking at the contemporarydevelopments of information technology, one keytrend is the emergence of linkage. For example, akey power of the World Wide Web is that it pro-New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 62

vides a way to dynamically link documents together.The benefits of linkage in the case of digital documentsare obvious, especially when the goal is tosearch and sort them. Linkage is to documents whatroads are to cities. Still, linked documents do notprovide a panacea for the complex informationmanagement needs sprouting around us in businessand elsewhere. This is true even if we expand theconcept of a digital document to include rigidlycoded and structured information in order to solvethe problem of ambiguity inherent to natural languageas we will discuss later on. It can argued thatinformation management systems built around theconcept of a digital document, even a linked andconventionally structured one, have failed to fulfillthe promises made in the early age of computing.Could the role of linkage in information be broughtto a new level and could this change bring aboutadvantages for the management of information?Looking at this question from the perspective ofbusiness or other organizations provides a numberof interesting insights. Organizations targeting complexactivities have traditionally been hierarchicalwith formalized responsibilities and roles. Still,modern management science has raised the questionwhether this organizational approach is the optimalone in all situations (Wilkinson, 1998). So what isholding back the trend towards decentralization inorganizations targeting complex activities? Complexactivities involve many pieces of information whichcurrent information management tools, based primarilyon electronic documents, cannot reflect in anatural way. The result is that a hierarchy is requiredto gradually transform management decisions downto practical work orders and, on the other hand, toprovide reporting of factory floor activities back tothe decision makers.As a practical example, a single work order document,even if available in the context of an enterpriseresource planning system, cannot natively reflecton the activity it relates to. Not all the linkages,relevant to the purpose and meaning of the workorder, are available. Native means here the ability toobtain the relation between this piece of informationand the goals of the organization. Comparing largeorganizations to single-person ventures it can beargued, that a single-person venture has a relativeadvantage in information management. Namely, theentrepreneur can natively understand the entirevalue chain.2 Reflective information managementA viewpoint we denote ‘reflective information management’approaches the presented problem by introducinga way to manage information with manydependencies always relevant to complex activities.How can we briefly characterize the approach?There are two key aspects to it. First of all, informationis natively handled as an unrestricted linkage ofatomic reflections, collected ideally directly fromtransactions resulting from financial or other concretedecisions made by people and including otherinformation relevant to the activity at hand. Thedescriptive power of a diverse and large set of directobservations cannot be stressed enough. We arguethat the resulting associative power of the linkagecan create a foundation for reducing the need forlayered, manual information management. Secondly,looking at the operational side of reflectiveinformation management, the emphasis is put onhuman abstraction level, iterative collaboration betweenhumans and information management tools.Simply put, reflective information management triesto replace the middle men reading and writingdocuments produced in a layered way with the abilityto directly obtain the relation between each pieceof information and the goals of the activity in question.How does reflective information management relateto the numerous attempts at intelligent computingand artificial intelligence? In contrast to attempts ata generic intelligent machine, using approaches suchas neural networks or expert systems (Russell andNorvig 2003), reflective information managementattempts to construct a mechanism supporting peoplein their information management activities. Thefocus is moved from standalone, unsupervised intelligentsystems towards incrementally and cumulativebuilt tools, which continuously collaborate withusers through the efficient and flexible use of abstractionlevel traversal. Within this collaboration,the key tool is support for reflective communicationamong collaborating people instead of support forformal documentation.As stated, the capability of the approach is to enableseamless traversal within different levels of abstractionand different dimensions of an available informationmass. The goal is not to automate decisionmaking but to provide people making decisions withcontext-optimized access to information. The keypoint is that access to information is provided athuman abstraction level in a way that people canconcentrate on decision making instead of formattingand interpreting information to and from machine-readableformats. Even more important goal is63

the capability to provide the people with the information(e.g., in the form of seemingly simple buthighly contextualized messages) currently relevantto their decision making.Going back to the presented historical review towardsthe use of documents, in the abstract sense ofthe word, from natural collaboration, we note thatultimately reflective information management takesthe way of organizing human collaboration, also incomplex activities, back to its natural form, which ismore close to communication among tribesmen thanto sterile documentation in hierarchical settings.3 Unrestricted linkageConsidering the described characteristics of reflectiveinformation management, the goal of the approachis human abstraction level information managementin settings where the innate human abilityis insufficient due to the variety or volume of possiblyrelevant pieces of information.Hampton (2003) has discussed how humans useabstraction and context in concept representation.Picking up some key points from his discussion,native human information management involves aflexible abstraction mechanism. Characteristic ofthis flexibility is that context heavily influenceshow, for example, a specific heard word is interpreted.Another characteristic he lists is the ability toflexibly add new dimensions to previously learnedconcepts without hindering the ability to pick thecurrently most relevant dimensions during a cognitiveprocess. Also, human learning is characterizedby the ability to fluidly leverage previously learnedconcepts and their relations while facing new phenomena.Considering implications of these characteristics forreflective information management, the key observationis the unbounded nature of human abstractionlevel information management. This leads into theconclusion that design-while-use is the only possiblebasic paradigm for reflective information managementtools. However, we also limit the scope of thetools by not setting the goal at autonomous or unsupervisedoperation but rather at the capability of thetools to serve as collaborative agents. The key capabilityof the tools is thus the ability for abstractiontraversal and dimension surfing in an unboundedconcept space. This is in contrast to the approachused, for example, in Semantic Web research whereconcepts are modeled using pre-constructed ontologies.Expanding on this, conventionally information hashad to be rigidly and strictly coded and structured tobe machine-processed. This formalization sets ahindrance for information processing as typicallythe structure for the information has to be designedbefore processing. This leads at minimum to increasedcomplexity in the case where the operationenvironment evolves during use. In practice, suchinformation processing systems work satisfactorilyonly in bounded cases where the boundaries of thecase space can be predicted reliably. For these reasons,reflective information management does notrely on structures fixed before use but rather on anevolving, unrestricted linkage of direct observationsand other information. Thus reflective informationmanagement is suited also for unbounded cases.We will next describe one possible physical manifestationfor the described unrestricted linkagewhere a set of concept relations with practicallyunbounded expressive power using natural languagevocabulary for concept and relation labels providesthe basis for concept modeling. The following is anexample of concept relations from health care:12892 “low-density lipoprotein”(is) “cholesterol”12893 “LDL” (is) “low-density lipoprotein”...44137 “laboratory result” (for)“patient” 114604539368744138 *44137c (is) “290245-1234”44139 *44137a (is) “LDL value”44140 *44139c (is) “89 mg/dL”As seen from this example a concept relation has asyntax similar, for example, to the one used in theW3C RDF standards (see http://www.w3.org/RDF/).Specifically, each relation is a 5-tuple. Basically, themember B relates the member A and C to each otherwith an optional timestamp indicating the absolutemarkup time.The couplings between the concept relations areeither implicit or explicit. For example, an implicitcoupling exists between all relations containing a“LDL value” labeled concept through the existenceof the similar string. Explicit couplings are definedthrough labeling A, B and C members with a referencenotation. This notation uses relation sequencenumbers and A, B or C membership as points ofreference. For example, when the member C is la-New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 64

eled “*441437c”, it indicates a reference to the Cmember of the relation with identification 441437. Itis important to note that the representation does notmake any difference between data and metadata.One mechanism for operation in an unbounded casespace is the implicit coupling between natural languagelabeled concepts. This enables coupling of anew concept relation to already existing conceptrelations without the need to modify them or to evenverify their existence. It should also be noted thatsince pre-defined ontologies cannot be used withunbounded case spaces, natural language is the onlypractical source for concept labeling. The unavoidableambiguity of natural language has conventionallyhindered its use in applications involving machine-processing.We propose that instead of formalizingthe used language, ambiguity should besolved by firstly utilizing context as availablethrough the associative power of the linkage and therelated descriptive power of the direct observationscontained in it and secondarily by making a clarifyingquery to appropriate user. Naturally, the abilityto learn from these interactions is a critical facilityhere.4 Case: health careHealth care work illustrates especially well the characteristicsof a field involved in complex activitieswhere documents have had an important role in informationmanagement. Attempts at fully digitalizingthe patient record, where the goal has been toimprove health work efficiency, have run into majorproblems, especially if one considers the originalgoal of increased efficiency (Dorenfest, 2000),(Heeks, 2005), (Wears and Berg, 2005), (Ash et al,2004), (Kuhn and Giuse, 2001). The introduction ofan electronic patient record has perhaps removed theconventional archivist and document courier fromthe picture but something has been lost at the sametime. It can be argued that the electronic record hasincreased formalization of health care work, due tothe use of strictly structured and inflexible electronicpatient documents (Berg and Toussaint, 2003). Thisincreased formalization has broken up natural pathsof information sharing and work organization. Thevalue of tacit knowledge, which is by definitionimpossible to formalize (Nonaka and Takeuchi,1995), and a shared common ground between collaboratinghealth care workers have an essential rolein successful and efficient clinical work (Stefanelli,2001). Of course, it cannot be denied that early informationprocessing systems in health care, as wellas in other industries, have been successful at theautomation of actuarial work exemplified by repetitivecomputation of tabular numerical data. To clarifythis, processing of documents can be enhancedby digitalization but not all work can be made moreefficient by digitalizing documents.Expanding on the set of concept relations presentedin the section 3, we will next present a simple exampleon how information management operationscan be carried out in this context. Specifically, welook at how presenting a simple goal via naturallanguage fragments can provide for contextual informationaccess to information and semantic sensorand action interfacing.Let us assume that for a patient group laboratory testresults for LDL cholesterol are made available in thedescribed representation through a semantic sensorinterface mapping a laboratory system’s structureand semantics for the LDL values directly to conceptrelations. Now, a medical professional searchingfor a specific patient’s LDL values initiates asearch with the query “cholesterol 290245”.Search and traversal algorithms as well as an iterativequery process can be applied to provide theprofessional with the value “89 mg/dL”.Assuming that we describe action interfaces with thesame relation mechanism, we can furthermore directlycouple the found LDL value and the recipient’saddress information, for example, to a physicalsoftware component capable of sending an e-mailmessage again with the help of an iterative queryprocess. This way to relate the pieces of informationand the actions operating on them enables, at least inprinciple, a native ability to directly reflect on information.5 DiscussionThe current discussion represents merely a startingpoint and a vision for more detailed work in the areaof human abstraction level information management.Important open research questions include,but are not limited to, performance issues related tothe utilization an unrestricted linkage and the repeatabilityof operations carried out in the context ofthe described paradigm.ReferencesSteven Pinker. How the Mind Works. W.W. Norton& Company, New York, 1999.65

Steven Pinker. The Language Instinct: How theMind Creates Language. Harper PerennialModern Classics, New York, 2000.Adrian Wilkinson. Empowerment: theory and practice.Personnel Review. 27(1): 40-56, 1998.Stuart Russell, Peter Norvig. Artifical Intelligence:A Modern Approach. Prentice Hall, 2nd edition,Upper Saddle River, 2003.James A. Hampton. Abstraction and context in conceptrepresentation. Philosophical Transactionsof the Royal Society of London B. 358: 1251-1259, 2003.Sheldon Dorenfest. The decade of the ‘90s. Poor useof IT investment contributes to the growinghealthcare crisis. Health Care Informatics.17(8):64–7,2000.Richard Heeks. Health information systems: Failure,success and improvisation. International Journalof Medical Informatics. Aug 19, 2005.Robert L. Wears, Marc Berg. Computer technologyand clinical work: still waiting for Godot.Journal of American Medical Association.293(10): 1261-3, 2005.Joan S. Ash, Marc Berg, Enrico Coiera. Some unintendedconsequences of information technologyin health care: the nature of patient care informationsystem-related errors. Journal ofAmerican Medical Informatics Association.Mar-Apr;11(2):104-12, 2004.Klaus A. Kuhn, Dario A. Giuse. From hospital informationsystems to health information systems.Problems, challenges, perspectives.Methods of Information Medicine. 40(4):275-87, 2001.Marc Berg, Pieter Toussaint. The mantra of modelingand the forgotten powers of paper: A sociotechnicalview on the development of processorientedICT in health care. International Journalof Medical Informatics. Mar;69(2-3):223-34, 2003.Ikujiro Nonaka, Hirotaka Takeuchi. The Knowledge-Creating Company. How Japanese CompaniesCreate the Dynamics of Innovation. OxfordUniversity Press, New York, 1995.Mario Stefanelli. The socio-organizational age ofartificial intelligence in medicine. Artificial Intelligencein Medicine. 23(1): 25-47, 2001.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 66

Elastic Systems: Role of Models and ControlHeikki Hyötyniemi ⋆⋆ Helsinki University of TechnologyControl Engineering LaboratoryP.O. Box 5500, FIN-02015 TKK, FinlandAbstractNeocybernetics, and specially the framework of elastic systems, gives concrete tools for formulatingintuitions concerning complex systems. It can be claimed that a cybernetic system constructs a modelof its environment, and it applies model-based control to eliminate variation in its environment. Thereexist various valuable control intuitions that are available for understanding cybernetic behaviors, includingexperiences with adaptive control. It turns out that even the evolutionary processes making thesystems more and more complex are thermodynamically consistent when seen in the control perspective.1 IntroductionIt seems that artificial intelligence is returning to itsroots: It used to be cybernetics, as proposed by NorbertWiener that was one of the cornerstones of earlyAI, and, still, the same ideas hold. The original reasonfor intelligence is not to do fancy reasoning, butto survive, and to constitute a functional whole in itsenvironment.Behaviors in cybernetic systems are based on interactionsamong low-level system components, constitutingstructures of feedback. In another perspective,feedbacks can be seen as control. This understandinghas been exploited, and, indeed, cybernetics hasalso been defined as being study of communicationand control in an abstract sense. The claim here isthat the framework of neocybernetics makes it possibleto make the abstract intuitions about communicationand control concrete and analyzable.Since the introduction of cybernetics back in1950’s, control theory, or, more generally system theoryhas matured. Whereas there was plenty to explorein traditional control structures during the lastdecades, it seems that by now the easy intuitions havebeen exhausted. The limits of the control paradigmare now visible, and it is evident that there are problems.It needs to be recognized that control theory is notin all respects an appropriate framework to understandnatural complex systems: The main counterargumentis that control practices are based on extremelyreductionistic approaches and centrally operatingcontrol structures. The real cybernetic populations— economies, ecologies, etc. — are based ondistributed operations, and it is difficult to see howthe centralized structures could be relaxed and reorganized.Getting distributed by definition means looseningthe controls, and major shifts in thinking are neededin engineering and AI. Indeed, traditional centralizedcontrol is a prototype of Western ways of structuringand mastering the world — but when applied in modelingof complex systems, there is the unpleasant feelof finalism and purpose-orientedness in the models 1 .There is a two-way contribution here: Control theorygives powerful tools to understand complex systems,but, at the same time, neocybernetic modelsmay give fresh perspectives to the control community.Today, one cannot even imagine the approachesthat someday perhaps become possible what comesto distributed agents and complex networks.What is more, even the most fundamental notionsof system theory are challenged by cybernetic considerations.Before having some function, a complexsystem intuitively does not deserve to be called a system.Whereas in system theory there are some crucialconcepts, like the distinction between the insideof the system and the outside environment, in cyberneticsystems such boundaries become blurred, theenvironment equally reacting to the system. A systemcould be defined not as being characterized by someformal structures, etc. — a system is a functionallycomplete sustainable entity.1 Perhaps because of the monoteistic view of natural order, onesearches for the underlying primum movens — Jean-Paul Sartre hassaid that even the most radical irreligiousness is Christian Atheism67

2 Cybernetics as controlNew useful intuitions can be reached when the contentsof familiar concepts are “refilled” with fresh semantics.When studying complex systems, controlengineering seems to offer just the right connotations.2.1 Information vs. noiseRoss Ashby coined the Law of Requisite Variety in1952:The amount of appropriate selection thatcan be performed is limited by the amountof information available.This is a deep observation. The concept of informationis, however, left somewhat vague here, and theconsequences remain obscure. To make it possible toefficiently apply mathematical tools, such basic conceptsnecessarily have to be defined in an accuratemanner. How is information manifested in data?When looking at the neocybernetic models in(Hyötyniemi, 2006a), one can see how the modelssee the data: When studied closer, it turns out thatthe weighting matrix in the pattern matching cost criterionbecomesW = E{ΔuΔu T }. (1)This means that data is weighted by the correlationmatrix when evaluating matches among patterns: Theneocybernetic system must see information in variation.Traditionally, when doing parameter fittingapplying maximum likelihood criteria for Gaussiandata, the approach is opposite — variation is interpretedas something top be avoided — and theweighting matrix is the inverse of (1).When applying Shannons information theory, orKolmogorov / Chaitin (algorithmic) information theory,the definition of information is strictly syntactical.There is no domain area semantics involved,and thus extreme universality is reached. However,some paradoxes remain: What you expect, containsno information, and noise has the highest informationcontent. When applying the neocybernetic viewof information, semantics (in a narrow, formalizedway) is included in manipulations, making the analysesnon-universal — but there is universality amongall cybernetic systems. This semantics is based notonly on correlations, but on balanced tensions amongvariables. What is expected, is the most characteristicto the system; uncorrelated noise has no relevancewhatsoever.The neocybernetic models are fundamentallybased on correlation matrices — principal subspaceanalysis is just a way of formally rewriting and redistributingthis correlation information. The correlationmatrices contain atoms of information, entriesE{¯x i ū j } revealing cumulated (co)variations amongvariables. Covariances and variances — such measuresfor information are easily expressed and exploited,and they are also the basis of modern identificationand minimum-variance approaches in controlengineering.2.2 Model-based controlIt turns out that a cybernetic system is a “mirror” ofits environment, optimally capturing the informationthere is available. This is not merely a metaphor —note that the formulas in the neocybernetic model (seeHyötyniemi (2006a)) can be given very concrete interpretations:• Model. It turns out that the neocybernetic strategyconstructs the best possible (in the quadraticsense) description of the environment by capturingthe information (covariation) in the environmentaldata in the mathematically optimal principalsubspace based latent variables¯x = ( E {¯x¯x T }) −1E{¯xΔuT } Δu. (2)• Estimate. It turns out that the neocyberneticstrategy constructs the best possible (in thequadratic sense) estimate of the environmentstate by mapping the lower-dimensional latentvariable vector onto environment applying themathematically optimal least-squares regressionfittingû = E {¯xΔu T } T (E{¯x¯x T } ) −1¯x. (3)• Control. It turns out that the neocyberneticstrategy integrates modeling and estimation tomaximally eliminate variation in the environment:ũ = u − û (4)The issue of modeling Δu rather than u directly isstudied in Sec. 3.1 (when q increases, u and Δu approacheach other what comes to the n most significanteigenvalues). Note that in the case of “intelligentagents” that are capable of explicitly taking thecompetition into account, so that explicit feedback isconstructed, original u rather than Δu can directly bemodeled.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 68

Cumulatedinformation( structure)Cumulatedmatter( biomass)Unmodeled noiseFlow of information/emergyFlow of matter/ energyDissipative wasteSources ofinformation( variations)Sources ofmatter/energy( levels)plex system dynamics (Prigogine, 1997) — now theseare just a side effect. It is the information in the environment(or variations in the data) that dictates thestructures within the higher-level system, whereas itis the matter (or actual levels in the data) that cumulateas some kind of biomass within this predestinatedstructure of some kind of niches. One could even saythat the cybernetic model to some extent captures thePlatonian idea beyond the changing world.Figure 1: Abstract flows in a cybernetic systemThis all means that a cybernetic system implementsmodel-based control of its environment. In terms ofinformation as defined above, this control is the bestpossible. The implemented control is far from trivial:It constitutes a multivariate controller where then most significant variation directions are equalized(or nullified). It needs to be emphasized that thepresented control scheme is just an emergent phenomenon,as there are no centralized control units or“master minds”, but everything is based on the local,mindless agents that know nothing about the “bigpicture”. The symmetric structure of the modeling /estimation loop reminds of Heraclitus’ words: “Theway up and the way down is the same”.2.3 Flows of information and matterThe feedback part in the closed-loop structure aboveis only an abstraction: It does not correspond to someseparate real processes because it only represents thenon-ideality of the information transfer. It is interestingto note that for the closed loop structure toemerge, two different kinds of processes need to cooperate— first there is the information flow into themodel, and then there is the material flow dictated bythe model. Without the other flow the other couldnot exist either. One could say that a cybernetic systemconstitutes a marriage mind and matter, combiningthese two incompatible dualistic viewpoints (seeFig. 1).In the figure, there are the two flows shown separately:On top, there is the flow of information (oremergy), and on bottom, there is the flow of matter(and energy). Most of the flows are wasted — in informationflow, the uncorrelated noise becomes filtered,whereas in material flow, it is the dissipativelosses that do not get through into the higher-levelsystem. Note that it is often assumed that it is thesedissipative flows that are the manifestation of com-2.4 Hunger for informationA cybernetic system sees information (emergy) as resourcesavailable in the environment. In other words,variation is the “nourishment” for systems, and beingcapable of exploiting these resources is a prerequisiteof surviving in an environment. That is why, it seemsthat evolutionary surviving systems are “hungry” formore and more information. Again, this sounds teleological— but if some system applies this strategyby accident, it immediately has evolutionary benefitin terms of increasing resources. There is no guidinghand needed — but it is like with Gaia: Even thoughall behaviors can be reduced to lower levels, simplestmodels are found if stronger emergent-level assumptionsare applied.It turns out that this eternal hunger has resulted invery ingenious-looking solutions for reaching moreinformation, and, to achieve such sophistication, thesystems have typically become ever more complicated.First, the variable values visible in the environmentcan be actively changed by the system: Whenan organism develops the ability to move, changingones environment also changes the variable values.Second, there exist an infinity of environmental variablesto choose from, and with enough ingenuity, theresource vector can be augmented in different ways,or the environment can be seen in new ways. To illustratethis, see Fig. 2 — there it is shown how theinformation content of a signal can reside in differentfrequency regions. The mathematically compactdefinition of information as being interpreted as variancemakes it possible to exploit frequency-domainmethods for analysis. Assume that the system concentrateson band-limited signals, so that the signalsare filtered asdu sdt = −μ su s + μ s u in , (5)and, similarly, there is an exponential “forgettinghorizon” what comes to the covariance estimates:dÊ{¯x su T s }dt= −γ s Ê{¯x s u T s } + γ s¯x s u T s . (6)69

The parameters μ s and λ s are filtering coefficients.Then it turns out that only variation in the darkestarea in the figure becomes cumulated in the model (orin the covariance matrix), whereas higher-frequencysignals are only filtered by the system. Too high frequencesare invisible altogether to the current system,leaving there room for other systems to flourish. Asthe world gets older, even slower-scale behaviors becomestatistically modellable — meaning that thereis room for ever increasing number of coexisting systems.The systems are hungry, but they are not greedy.Whereas a system exhausts variation in its environment,there is the same variation inherited in the systemitself (remember that PCA model maximally relaysvariation to latent variables). This gives rise toa cascade of trophic layers: Another system can startexploiting the variation that is now visible in the priorsystem. When the next trophic layer has been established,there is room for a yet higher trophic layer, etc.This kind of succession of systems can be representedas a sequence of “ideal mixers” of information. Whennew layers are introduced, the ecosystem becomesmore and more continuous and smooth – becominga partial differential equation (parabolic PDE) diffusionmodel filtering the incoming variation. All looseinformation seems to give rise to new systems.Heraclitus said that the underlying principle in natureis fire — in the cybernetic perspective, it is thisfire that is the driving force, but it seems that the goalsof nature could best be explained in terms of a fire extinguisher.There are the physical (chaotic) processes (planetsorbiting and rotating, followed by climatologicalphenomena, etc.) that shuffle the originally “noninformative”flow of solar energy, originally generatingthe information for other systems to exploit. Theinput variables on the lowest level are temperatures,nutrients in the soil, diseases, rainfall, etc., and on thelevel of herbivores, it is then the spectrum of plants toforage on.The higher-level systems can appear in very differentphenospheres, starting from very concrete systemsand ending in very abstract memetic ones (seeFig. 3). The interprtetation of signals changes fromconcrete resources to available/required functionalities,and explicit formulations become impossible.For example, take politics — also there exist interestingpossibilities for applying the cybernetic thinking.Why democracy seems to prosper eventhough it is less efficient than a dictatorship?Assuming that there is complete informationavailable in the society, democ-1EnvironmentSystem itselfForgottenVeiled s’ s sFigure 2: Different systems concentrate on different timescales, seeing the environment in different waysracy represents the most cybernetic politicalsystem, information on the bottom(needs of citizens) being maximally transferredto the top (decision makers). Partiesdetermine profiles of opinions; party popularity(number of votes x i ) reflects needsu j in the society, and this is reflected in itspossibilities of implementing party visions.2.5 Towards optimum?In cybernetic systems the critical resource is information.In human-made “constructivistic” systems(technical, scientific, ...), the same principle seemsto apply, but the variables are more difficult to quantify;the critical resource can be said to be knowledge,and one is always “at the edge of understanding”. Assoon as there is some new understanding about relationshipsamong variables, it is exploited — this becomesmanifested in industrial plants, for example,where new controls are introduced to make the systemremain better in balance. These developments areimplemented by humans, but, after all, the system followsits own evolution where individual human signalcarriers have little to say.Complex systems seem to develop towards becomingmore and more cybernetic. Regardless of the domain,the limiting factor in this evolutionary processseems to be related to extracting and exploiting information(or knowledge). Typical examples are foundin working life. The other prerequisite for “cybernetization”— better understanding of the system andgaining more information — is implemented throughsupervision, questionnaires, and more paper work ingeneral, and the other — applying more efficient controlsbased on the acquired information — is implementedthrough increasing administration, organizationalchanges, missions and visions, and even “developmentaldiscussions”. This is all good, isn’t it?Unfortunately, the same efficiency pursuit has alsocome to universities. The role of science is to questionand find alternative explanations, avoiding fixedNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 70

economicalsystemssocialsystemsmemeticsystemstechnicalsystemsecologicalsystemsbiologicalsystemsphysicalsystemsFigure 3: Systems in different phenospheres try to extinguish the Heraclitus’ fireframeworks and controls. This is, of course, farfrom being efficient, and cybernetization is going on.Whereas the scientific problems being studied are —by definition — missing compact representation, differentkinds of variables are needed to make researchwork better quantifiable. This means that deep problemshave to be trivialized, and new measures for“good research” are defined. Earlier the scientificwork was evaluated in terms of how well the observedphenomena can be described, but nowadays it is theconcrete practical value that is assessed. The traditionalideal in scientific work was self-organizationand low hierarchies, but now one is getting back towardshierarchies.3 Control intuitionsWhen the control notions are employed, there aremany intuitions directly available concerning the behaviorsin cybernetic systems. For example, the controlsystem can become too good.3.1 Adaptive controlAdaptation is the key property in truly cybernetic systems,meaning that they are adaptive control systems,trying to implement more efficient controls based onsimultaneous observations of their environments. Ifone has control engineering background, one can understandwhat happens in a truly cybernetic system:Adaptive controllers are notorious in control engineering,as they can behave in pathological ways.The reason for the “explosions” is loss of excitation.Good control eliminates variation in data — and afterthis there is no information where the model tuningcan be based on, and gradually the model becomescorrupted. After that, when the model is nomore accurate, the variation cannot all be eliminated,and there will exist information in observations onceagain. The model starts getting better, and after thatthe control gets better, and the cycle of good andbad closed-loop behavior starts again; the collapses incontrol performance can be quite catastrophic. Thiskind of behavior is typical in loops of simultaneousmodel identification and model-based control. Thisresult is paradoxical: Good balance on the lower levelresults in high-level instability.In complex cybernetic systems, the model adaptationscan be more complex than in typical adaptivecontrols. For example, the sampling rate can becomefast as compared to the system dynamics (compare to“quartal capitalism” in economy), but increase in sensitivityin any case follows. The evolutionarily survivingsystems are on the edge of chaos where variationis no more information but only noise.Extreme optimization results in “stiffness” of thesystem, and worsened fitness in changing conditions.It is again easy to see connections — compare to ancientempires: It seems to be so that there is a lifespanfor all cultures, after which even the strongestcivilization collapses. For example, during Pax Romana,there were no enemies, and the army was graduallyruined – and there was a collapse after a severedisturbance 2 . And this does not only apply to humansocieties: For some reason, massive extinctions seemto take place in 62 million year cycles (Rohde andMuller, 2005). Do you need some meteors to explainextinctions — or is this simply because of evolutiondynamics?However, as it turns out, the cybernetic strategywhere the feedback is implemented implicitlythrough the environment, results in “gentle” adaptivecontrol, form of buffering, where the variation is not2 But explicit emphasis on the army results in the Soviet-typecollapse: If there is no real need at some time, such investments arecybernetically non-optimal, meaning that the system cannot outperformits competitors in other fields in the evolutionary struggle71

fully eliminated, and the closed loop behavior doesnot become pathological. This is because it is Δurather than the estimate u itself that is being eliminatedfrom the input data, and some level of excitationremains, making the overall system evolutionarilystable and sustainable.However, being too ambitious, implementing extremeoptimization, and full exploiting the informationcompletely wiping out excitation, is also a possiblescenario in a cybernetic system. This happensif the agents are “too smart”, implementing the feedbacksexplicitly. An agent can not only see the resourcesbut also the competitors and take their actionsinto account — implementation of such explicitfeedback results in combined Hebbian/anti-Hebbianlearning (see Hyötyniemi (2006b)).3.2 Inverting the arrow of entropyThe second law of thermodynamics states that in aclosed system entropy always increases (or remainsthe same if the system is reversible). This means thatthe the system goes finally towards “heat death”, thestate of maximum probability, where all variations areeliminated. All physical systems fulfill this principle.However, it can be claimed that cybernetic systemsare thermodynamically inconsistent as it seems thatthey operate against the arrow of entropy: As newstructures emerge, the probability decreases. Thisdifference between “normal” physical systems and“abnormal” cybernetic ones causes an uneasy feeling:Even though there are no outright contradiction here(the cybernetic domain is not closed), different lawsapply, and it seems that there must exist different sciencesfor evolutionary systems.In the neocybernetic framework, this paradoxseems to vanish altogether. Remember that the targetof entropy is heat death where everything is in balance— but it was balances that were asumed to bethe goal of the neocybernetic systems, too. When theboundaries between the system and its environmentare set appropriately, it turns out that all processes gotowards entropy increase, including the evolutionaryones. There is a minor decrease in the overall entropywhen the model of the environment is constructedoff-line, once-for-all, but thanks to this model thereis huge ever-lasting increase in entropy caused bythe on-line variation suppression due to the modelbasedcontrol. When the environment is seen as residinginside the higher-level control system, it turnsout that at all levels maximization of entropy takesplace. Such observations perhaps offer tools for modelingprinciples: If it is assumed that all systems goPhilosophy(logic)Engineering(control)Mathematics(linear theory)Mathematics(syntax)Engineering(semantics)Philosophy(metaphysics)Figure 4: Hierarchy of disciplines. Traditional view, onthe left, sees engineering as simple application of the finetheories; the new view, on the right, sees control engineeringas delivering substance to philosophiestowards entropy as fast as possible, an extremely efficientconceptual framework is available (this principlecould be called “maximum entropy pursuit”).The inversion of the arrow of entropy makes it possibleto attack many mysterious phenomena where itseems that improbability cumulates beyond all limits— in the new perspective, such cumulation is nomore a paradox. For example, the challenge of originof life can be attacked. The essence of understandingthe life processes is, again, in the functions; semanticsof life is buried in the notorious concept of elanvital. In the neocybernetic setting, such finalistic argumentsbecome issues of entropy production.3.3 Rehabilitation of engineeringSince 1960’s, after the great discoveries of moderncontrol theory, there have been no real breakthroughsin the branch of control engineering. It seems thatthis stagnation does not need to last long: There isa Golden Age of control engineering ahead. Controltheory and tools can be applied not only in technicalapplications, but also in understanding reallycomplex system — biological, social, economical,etc. There do not necessarily exist explicit controls insuch systems, but understanding the natural dynamicsin such systems is still based on control intuitions.It is traditionally thought that philosophy is the basisof all science: Logic is part of philosophy determiningthe rules of sound thinking. Mathematics if“applied logic”, implementing the logical structuresand manipulating them according to the logical rules.Natural sciences, on the other hand, can be seen as“applied mathematics”, where the ready-to-use mathematicalformulas are exploited to construct models.Finally, the engineering disciplines are “applied science”.Engineering is inferior to the more fundamentalways of structuring the world.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 72

This is a formal view of how deductive scienceis done, how new truths are derived. However, alsothese viewpoints need to be reconsidered: If the presentedneocybernetic modeling can cast some lightonto the mysteries of what is the essence of complexsystems, the deepest of the philosophical branches,metaphysics, is addressed. It is mathematics that offersthe syntax for discussing the issues of what isthere beyond the observed reality, and it is controlengineering that offers the semantics into such discussions.It can be claimed that control knowledge isnecessary for understanding complex systems, naturalor artificial (see Fig. 4), involving intelligent ones.4 Mastering the environmentThe whole cybernetic machinery can be studied in theframework of control and entropy maximization. Thefeedback needs not be implemented in such an implicitway as in the prototypical cases of “static control”,where time axis is abstracted away, but the controlintuition extends to dynamic, transitory cases.4.1 Artificial reflexesIt is reflexes that can be seen as atomary manifestationsof intelligence, representing reasonable behaviorwith no brains involved, and automated sensor/motorreactions (conditioned reflexes) can be seenas extensions of that, being learned rather than hardwired.When variations in the environment are interpretedas resources (or threats), low-level intelligencemeans immediate benefit: Reaching towardsresources (or away from them) can be seen as controltowards zero activation, or “heat death” of the localenvironment.To study “artificial reflexes” the originally staticmodel framework needs to be extended to dynamiccases. As the system was previously seen as a mirrorof the environment, now it is the controller that implementscurrent state as a mirror between the pastand the future, and, what is more, an adapted controlshould implement some kind of balance betweenthe past and the future. One needs to have a modelnot only of the current environment but also of how itcan be changed; one has to be capable of simulation,or estimation of the future. In the control engineeringperspective, one could speak of model-predictivecontrol combined with dead-beat control (see Åströmand Wittenmark (1997)).Assume that the observation vector u is coded as a“perception” or “mental view” ¯x. These variables denotedeviations from the nominal reference values, soCurrent T=xk ()=ck () TFuture=xk+ ( 1)Figure 5: Control as a (antisymmetric) mirror between thepast and the futurethat in the goal state there should hold ¯x ≡ 0, meaningextreme loss of variation in the environment. Theperceptions are transformed into control signals c thatare connected to actuators (muscles). The resultingresponses in the environment are then observed andtransformed to another set of perceptions. Based onsuch assumptions, the control scheme in Fig. 5 canbe presented. The right-hand side in the figure representsthe process of model construction, where thedependencies between the control signal and resultingchanges in the observations are recorded, and theleft-hand side represents model usage, or on-line constructionof the control signals. The model constructionis carried out applying the neocybernetic principlesfor the “input” ¯x(k +1)and “latent variable”¯c(k), now assuming that the control and its effectsare in balance. The signals with “double bars” are thebalance values after the previous-level signals ¯x arefurther exploited as inputs in the control constructionprocess.The intuitive idea of this control is to make thechange in the state inverse of the current state, meaningthat, when the new state is reconstructed as a sumof the old state and the change, the outcome will bezero state — meaning successful control. The pastand the future perceptions are “folded” on top of eachother. Whereas adaptation of the actuation model canonly take place after the effects are visible, the on-linecontrol construction is strictly causal. Coupling thesuccessive perceptions together implements the trickof collapsing the time structure back to singularity.The above control scheme is based on a very simpleview of the environmental dynamics: The assumptionnow is that if nothing is done, nothingchanges in the observed environment. This makesprediction simple. Yet, the causal structure is no moreself-evident, as it is no more physical exhaustion ofthe variation taking place as a side-effect, feedbacksbeing explicitly implemented, and control structureshave to be explicitly initialized. The minor extensionof the assumed environment presupposes that differentkinds of extensions are implemented in the basic73

neocybernetic model structure. In general, better controlmeans that new structures need to be employed,and there is more need for explicit control of this control.4.2 From reactivity to proactivityWhen looking at truly complex systems, it turns outthat the environment itself consists of other systems.When one has a coevolving system of systems, therole of external disturbations becomes smaller andsmaller, and it is interaction among subsystems thatmainly takes place. When one knows in advancehow the environment will react, controls can be designedbeforehand. When the subsystems share thesame design instructions, changing the environmentcan turn from reactive to proactive. Indeed, the genetic(memetic) codes are used for bootstrapping thebiological (cognitive) systems, activating dynamicattractors one by one, until the final phenotype isreached.It seems that no matter how sophisticated structureshave been developed during evolution, naturehas not found the way to put up a running systemwithout repeating the whole sequence. As observedalready by Ernst Haeckel, “ontogeny recapitulatesphylogeny”.When a biological system is to be put up in a neworganism (or memetic system in another mind), theinformation can only be stored and transmitted in theform of sequential genetic code (memetic scripturesor spoken tradition). When the environment is favorable,the codes can be interpreted, and the highdimensionaldynamics are started in a hierarchic manner,more complex structures being based on simplerones, making the system functional and “alive”.5 ConclusionIn today’s AI, specially in modern robotics, the approachesare typically based on input/output orientation,“intelligent” systems being black boxes, actuallyin the spirit of the age-old Turing test. Followingthe developments in cognitive science, perhapsthis Brooksian “artificial behaviorism” will somedaychange to “artificial cognitivism” or constructivismwith emphasis on the internal models and modelbasedcontrol. Neocybernetics is a candidate offeringsuch model structures.Good control is based on a good model — it is thismodel that is the key issue assumedly in all cyberneticsystems, however abstract they happen to be. The basicidea does not change if the intuitions cannot easilybe turned into concrete numbers. Even all human activitiescan be interpreted in this framework of findingmodels and exploiting them for control. For example,scientific activities try to find models for the world —and technology is thereafter employed to exploit thisunderstanding and exhaust the new resources, thusbringing the resource variations to zero.Arthur Schopenhauer once said that art is the onlyfield of human activity that is free of struggle, aestheticexperience making it possible to temporarilyescape the “rat-race”. In the cybernetic setting, thisactivity, too, is in line with other goal-directed activities:If art helps to see the world in new perspectives,it makes it possible to construct alternative models.In all, the purpose of life is to understand the world— to find a model, and then exploit the resources.Indeed, neocybernetic considerations have closeconnection to philosophies. Along the lines of Easternwisdom, neocybernetics emphasizes the role ofbalances in all kinds of living systems. However,in some sense neocybernetics goes deeper than that:Good life is not only about finding the balance. Tofind the model, to be capable of identifying the environment,there has to exist enough excitation aroundthat balance. It can be assumed that autoimmunediseases, for example, are caused by an incompletemodel caused by absence of natural microbial attacks.Full life is not characterized by loss of challenges;rather, happiness is one’s knowledge of being capableof coping with any challenge one can face.According to Eastern philosophers, the reason forsuffering is missing knowledge and understanding— with appropriate models the world and its constituentsbecome a meaningful whole:Before Zen, men are men and mountains aremountains, but during Zen, the two are confused.After Zen, men are men and mountainsare mountains again.ReferencesK.J. Åström and B. Wittenmark. Computer-Controlled Systems.Prentice Hall, New Jersey, 1997.H. Hyötyniemi. Elastic Systems: Another View at Complexity.SCAI’06, Helsinki, Finland, 2006a.H. Hyötyniemi. Neocybernetics in Biological Systems.Helsinki University of Technology, Control EngineeringLaboratory, 2006b.I. Prigogine. End of Certainty. The Free Press, 1997.R. Rohde and R. Muller. Cycles in fossil diversity. Nature434, 2005.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 74

≤ ≤ 75

New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 76

≤ ≤ ≤ ≤ ≤ ≤ ≤ ≤ 77

New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 78

≤ ≤ New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 80

Object-oriented declarative model specification in C++Risto Lahdelma **University of TurkuDepartment of Information TechnologyJoukahaisenkatu 3, FI-20520 TURKU, FinlandRisto.lahdelma@cs.utu.fiAbstractMathematical optimization models are commonly implemented by specifying them in a high-leveldeclarative modelling language. These languages allow specifying the overall model structure separatelyfrom the actual numerical values that are needed to form a specific instance of the model.The existing modelling languages are typically interpreted when the model is generated. In largescalelinear and mixed integer programming the interpretation and generation time may be substantialcompared to the time to solve the model. This paper presents MPLC++, which is an object-orientedmathematical modelling language based on C++ classes and objects. Because the modelclasses are standard C++, they can be compiled and executed efficiently when the model is generated.This approach is particularly suitable in embedded time-critical on-line applications of mathematicaloptimization models.Keywords: declarative programming, object-oriented mathematical modelling, linear programming,mixed integer programming, C++1 IntroductionMathematical optimization models are commonlyimplemented by specifying them in a high-level declarativemodelling language. Examples of such languagesor environments are GAMS /BKM88/,UIMP /ElMi82/, GXMP /Dol86/, AMPL /FGK90/,MPL /Kri91/, PDM /Kri91a/, LPL /Hür93/ andMME /Lah94/. These languages allow specifyingthe overall model structure separately from the actualnumerical values that are needed to form a specificinstance of the model. Surveys of modelling languagescan be found for example in /GrMu92/,/Kui93/ and /BCLS92/.Most existing modelling languages require a separateinterpreter environment or a pre-processor.Sometimes the modelling language is integratedwith a solver, but often the solver is a separate programaccepting some intermediate format. Thus,when models are embedded in various design, planningand control applications, the resulting systemconsists of several programs, which exchange datathrough large model files. Alternatively it may bepossible to link the modelling language interpreterdirectly into the application. The problem with theabove approaches is the overhead caused by• inter-process communication through files,• interpreting the modelling language,• re-generating the internal model format forthe solver,• post-processing and transferring the resultsback to the application and• re-initialising the solver in successive optimisationruns with only slightly differentparameters.In large-scale linear and mixed integer programmingthis overhead may be substantial compared to thetime to actually solve the model.This paper presents an implementation of an objectorientedmathematical modelling languageMPLC++ in C++ using classes and instances. Becausethe model classes are standard C++, they canbe compiled and executed efficiently to generatemodel instances. This approach is particularly suitablein embedded time-critical on-line applicationsof mathematical optimization models.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 82

2 Object-oriented modellinglanguage in C++MLC++ is a high-level, hierarchical, object-orientedmodelling language, which can be efficiently implementedin C++ using objects and classes. MPLC++supports modelling with generic, re-usable modelcomponents in a way similar to MME (MathematicalModelling Environment) /Lah94/. Model parameterscan be inline-coded in C++ or retrieved fromauxiliary sources, e.g. external databases. Modelscan be generated directly into the internal storageformat of an integrated optimizer, optimised and theresults can be directly retrieved by the application.The basic elements of MPLC++ are implemented asC++ classes according to the class hierarchy shownin Table 1.}Table 1. The basic elements of MPLC++Class Abstract syntaxModels can be instantiated from model classes withModel Name Set(Var) Set(Param) Set(Model) Set(Constr) different arguments. Models instantiated from insideConstr Name Cexp

directly in C++ programs as instances of the Varclass or the derived classes Real, Integer, Binary.For example:Var x("x", 0, 100, "kg");Min=0, Max=100, Unit="kg"Real y("DB001"), z;Unit= ""// Name="x",// Min=0, Max=INF,Here x becomes a continuous decision variable inthe range[0,100], measured in kilograms. The namex occurs twice in the previous definition. The C++variable x is defined as a reference to a decisionvariable with run-time name "x". The C++ name xcan be used in the succeeding C++ code where constraintsand submodels are defined. The run-timename "x" is stored in a symbol table. Without a runtimesymbol table it would be impossible to interpreta character string as a variable or to attachmeaningful names to a listing of variable values.The C++ name and run-time name may differ aswith y in the previous example. If the run-time nameis omitted, the system will automatically generate aunique run-time name.All arguments are optional. The unit defaults to anempty string signifying a dimensionless quantity.Real and Integer variables are by default non-negative,which is common in mathematical programming.Binary variables are by default 0/1 variables.2.3 ParametersParameters are named symbolic constants, whichcan be defined directly in C++ programs as instancesof the Param class or possible subclassesthereof. Example:Param a("a",10,"g");// Name="a", Val=10,// Unit="g"When parameters are used instead of ordinary C++constants or variables, the system is able to keeptrack of how the model depends on the parameters.This makes it possible to support various parametricanalysis techniques. It is also possible to implementspreadsheet-like incremental model re-generationtechniques when some parameters are modified.2.4 ConstraintsWe consider here only linear constraints. Constraintsmay have a C++ name and a run-time name.Linear constraints can be double inequalities withconstant left and right hand sides or a single inequalityor equality between two linear expressions. Unitcompatibility is automatically checked in expressionsand constraints. Many standard units are builtinto the system, but new units and conversion rulescan also be defined. Incompatible units result in runtimeerrors. Unit conversions can be used for scalingexpressions. For exampleConstr b("b");// name “b” for0

GrMu92/ Greenberg H.J., Murphy F.H.: A Comparisonof Mathematical Programming ModelingSystems; Annals of Operations Research, vol.38, December 1992/Hür93/ Hürlimann T.: LPL: A mathematicalprogramming language; OR Spektrum; Band 15,Heft 1, Springer-Verlag 1993/Kri91/ Kristjansson B.: MPL Modelling SystemUser Manual, Maximal Software Inc., Iceland1991/Kri91a/ Krishnan R.: PDM: A knowledge-basedtool for model construction; Decision SupportSystems; Vol. 7, No. 4, 1991/Kui93/ Kuip C.A.C: Algebraic Languages forMathematical Programming; European Journalof Operational Research, Vol. 67, No. 1, May1993/Lah94/ Lahdelma R.: An Object-OrientedMathematical Modelling System; Acta PolytechnicaScandinavica, Mathematics and Computingin Engineering Series, No. 66, Helsinki 1994, 77p.85

Solving and Rating Sudoku Puzzles withGenetic AlgorithmsTimo Mantere and Janne KoljonenDepartment of Electrical Engineering and AutomationUniversity of VaasaFIN-65101 Vaasafirstname.lastname@uwasa.fiAbstractThis paper discusses solving and generating Sudoku puzzles with evolutionary algorithms. Sudoku isa Japanese number puzzle game that has become a worldwide phenomenon. As an optimizationproblem Sudoku belongs to the group of combinatorial problems, but it is also a constraintsatisfaction problem. The objective of this paper is to test if genetic algorithm optimization is anefficient method for solving Sudoku puzzles and to generate new puzzles. Another goal is to find outif the puzzles, that are difficult for human solver, are also difficult for the genetic algorithms. In thatcase it would offer an opportunity to use genetic algorithm solver to test the difficulty levels of newSudoku puzzles, i.e. to use it as a rating machine.1 IntroductionThis paper studies if the Sudoku puzzles can beoptimized effectively with genetic algorithms.Genetic algorithm (GA) is an optimization methodthat mimes the Darwinian evolution that happens innature.According to Wikipedia (2006) Sudoku is aJapanese logical game that has recently becomehugely popular in Europe and North-America.However, the first puzzle seems to be created inUSA 1979, but it circled through Japan andreappeared to west recently. The huge popularityand addictivity of Sudoku have been claimed to bebecause it is challenging, but have very simple rules(Semeniuk, 2005).Sudoku puzzle is composed of a 9×9 square thatare divided into nine 3×3 sub squares. The solutionof Sudoku puzzle is such that each row, column andsub square contains each integer number from [1, 9]once and only once.In the beginning there are some static numbers(givens) in the puzzle that are given according to thedifficulty rating. Figure 1 shows the originalsituation of the Sudoku puzzle where 30 numbers for81 possible positions are given. The number ofgivens does not determine the difficulty of thepuzzle (Semeniuk, 2005). Grating puzzles is one ofthe most difficult things in Sudoku puzzle creation,and there are about 15 to 20 factors that have aneffect on difficulty rating (Wikipedia, 2006).6 8 9 4 77 67 6 24 8 5 97 19 7 6 28 2 54 13 2 4 5 9Figure 1. A starting point of the Sudokupuzzle, where 30 locations contains a staticnumber that are given.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 86

Figure 2 shows the solution for the puzzle in fig.1. Note that each column, row and sub square of thesolution contains each integer from 1 to 9 once. Thestatic numbers given in the beginning (fig. 1) are intheir original positions.2 3 6 5 8 9 4 1 78 5 4 2 7 1 3 6 99 7 1 6 4 3 8 2 54 6 8 1 2 5 7 9 37 2 3 9 6 4 5 8 11 9 5 7 3 8 6 4 26 8 7 3 9 2 1 5 45 4 9 8 1 7 2 3 63 1 2 4 5 6 9 7 8Figure 2. A solution for the Sudoku puzzlegiven in figure 1.The objective of this study is to test if geneticalgorithm is an efficient method for solving Sudokupuzzles, and also if it can be used to create generateSudoku puzzles and test their difficulty levels. If thedifficulty rating of the Sudoku puzzles in thenewspapers is consistent with their difficulty for GAoptimization, the GA solver can also be used as arating machine for Sudokus.1.1 Genetic AlgorithmsAll Genetic algorithms (Holland, 1982) arecomputer based optimization methods that use theDarwinian evolution (Darwin, 1859) of nature as amodel and inspiration. The solution base of ourproblem is encoded as individuals that arechromosomes consisting of several genes. On thecontrary to the nature, in GAs the individual(phenotype) is usually deterministically derived fromthe chromosome (genotype). These individuals aretested against our problem represented as a fitnessfunction. The better the fitness value an individualgets, the better the chance to be selected to a parentfor new individuals. The worst individuals are killedfrom the population in order to make room for thenew generation. Using crossover and mutationoperations GA creates new individuals. In crossovergenes for a new chromosome are selected from twoparents using some preselected practice, e.g. onepoint,two-point or uniform crossover. In mutation,random genes of the chromosome are mutated eitherrandomly or using some predefined strategy. TheGA strategy is elitist and follows the “survival of thefittest” principles of Darwinian evolution.1.2 Related workSudoku is a combinatorial optimization problem(Lawler et al, 1985), where each row, column, and3×3 sub squares of the problem must have eachinteger from 1 to 9 once and only once. This meansthat the sum of the numbers in each column and rowof the solution are equal. Therefore, as a problemSudoku is obviously related to the ancient magicsquare problem (Latin square), where different sizeof squares must be filled so, that the sum of eachcolumn and row are equal. The magic squareproblem has been solved by GAs (Alander et al,1999, Ardel, 1994) and also the Evonet Flyingcircus (Evonet, 1996) has a demonstration of amagic square solver.Another related problem is a generatingthreshold matrix for halftoning (Kang, 1999)grayscale images. In the halftoning, the gray valuesof an image are converted into two values, black orwhite, by comparing the value of each pixel to thevalue of the corresponding position at the thresholdmatrix. The values in the threshold matrix should beevenly distributed. Therefore the sums of thethreshold values in each row and column of thethreshold matrix should be nearly equal.Furthermore, the demand of homogeneity holds alsolocally, i.e. any fixed size sub area should have anearly evenly distributed threshold value sum. Thisguarantees that the resulting image does not containlarge clusters of black or white pixels. Thresholdmatrices have been previously optimized by GAse.g. in (Alander et al, 1998 and 1999; Kobayashiand Saito, 1993; Newbern and Bowe, 1997; Wiley,1998).There seems to be no scientific papers onSudoku in the research indices, but there are somewhite papers on the Internet. In (Gold, 2005) a GAis used to generate new Sudoku puzzles, but themethod seems inefficient, since in their example theGA needed 35700 generations to come up with anew puzzle. In our results, we created a new Sudoku,in average, in 1390 generations.There is also a ‘Sudoku Maker’ (2006) softwareavailable that is said to use genetic algorithminternally. It is claimed that the generated Sudokusare usually very hard to solve. Unfortunately, thereare no details, how GA is used and how quickly anew Sudoku puzzle is generated.Sudoku can also be seen as constrain satisfactionproblem, where all the row and column sums must87

e equal to 45. Constrained optimization problemshave been efficiently optimized with evolutionaryalgorithms e.g. in (Li et al, 2005; Mantere, 2005;Runarsson and Yao, 2000).2 The proposed methodSudoku is a combinatorial optimization problem,where each row, column and also each nine 3×3 subsquare must have a number from {1, 2, …, 9}exactly once. Therefore we need to use a GA that isdesignated for combinatorial optimization problems.That means that it will not use direct mutations orcrossovers that could generate illegal situations:rows, columns, and sub squares would contain someinteger from [1, 9] more than once, or some integerswould not be present at all. In addition, the geneticoperators are not allowed to move the static numbersthat are given in the beginning of the problem(givens).Consequently, we need to represent the Sudokupuzzles in GA program so that the givens will bestatic and cannot be moved or changed in geneticoperations. For this purpose we have an auxiliaryarray containing the givens. We decided to presentSudoku puzzle solution trials as an integer array of81 numbers. The array is divided to nine sub blocks(building blocks) of nine numbers corresponding tothe 3×3 sub squares from left to right and from topto bottom.The crossover operation is applied so that itexchanges whole sub blocks of nine numbersbetween individuals. Thus the crossover pointcannot be inside a building block (fig. 2).The mutations are applied only inside a subblock. We are using three mutation strategies thatare commonly used in combinatorial optimization:Individual 1:swap mutation, 3-swap mutation, and insertionmutation (fig. 3).Each time mutation is applied inside the subblock, the array of givens is referred. If it is illegal tochange that position, we randomly reselect thepositions and recheck until legal positions are found.In swap mutation, the values in two positions areexchanged. In 3-wap mutation the values of threepositions are rotated, either clockwise orcounterclockwise. In insertion mutation, one or morenumbers are inserted before some number and all thefreely changeable numbers are then rotatedclockwise or counterclockwise.To design such a fitness function that would aida GA search is often difficult in combinatorialproblems (Koljonen and Alander, 2004). In this casewe decided to use to a simple fitness function thatpenalizes different constraint violations differently.Sub block:The help array:2 3 6 8 5 4 9 7 1 0 0 6 0 0 0 0 7 0Swap mutation:Illegal attempt of swapmutation:2 9 6 8 5 4 3 7 1 2 3 6 8 5 4 9 7 13-swap mutation:2 4 6 3 5 8 9 7 1 2 3 6 8 5 4 9 7 1Insertion mutation:Illegal attempt of insertion1 2 6 8 3 5 4 7 9 mutation: 2 3 6 8 5 4 9 7 1Figure 4. The types of mutation used inSudoku optimization. Up left is one sub blockand up right the static values of that sub block(6 and 7 are givens). The mutation is appliedso that we randomly select positions insidethe sub block, and then compare the selectedpositions to the help array if the positions arefree to change.2 3 6 8 5 4 9 7 1 5 8 9 2 7 1 6 4 3 4 1 7 3 6 9 8 2 5 4 6 8 7 2 3 1 9 5 1 2 5 9 6 4 7 3 8 7 9 3 5 8 1 6 4 2 6 8 7 5 4 9 3 1 2 3 9 2 8 1 7 4 5 6 1 5 4 2 3 6 9 7 8Individual 2:3 1 6 8 5 2 4 7 9 4 8 9 1 7 2 6 3 5 4 8 7 3 6 5 9 2 1 4 3 8 7 2 1 5 9 6 1 6 5 8 4 9 7 3 2 8 9 7 4 5 1 6 3 2 5 8 1 6 4 7 3 9 2 3 9 2 7 1 8 4 5 6 7 5 2 3 6 4 9 8 1Individual n:x x 6 x x x x 7 x x 8 9 x 7 x 6 x x 4 x 7 x 6 x x 2 x 4 x 8 7 x x x 9 x x x 5 x x x 7 x x x 9 x x x 1 6 x 2 x 8 x x 4 x 3 x 2 x x 2 x 1 x 4 5 x x 5 x x x x 9 x xThe help array:0 0 6 0 0 0 0 7 0 0 8 9 0 7 0 6 0 0 4 0 7 0 6 0 0 2 0 4 0 8 7 0 0 0 9 0 0 0 5 0 0 0 7 0 0 0 9 0 0 0 1 6 0 2 0 8 0 0 4 0 3 0 2 0 0 2 0 1 0 4 5 0 0 5 0 0 0 0 9 0 0The possible crossover pointsFigure 3. The representation of Sudoku puzzles in GA. One possible solution (individual) is an array of 81numbers that is divided into nine sub blocks of nine numbers. The crossovers points can only appearbetween sub blocks (marked as vertical lines). The auxiliary array is used for checking static positions, ifthere is a number that is not equal to zero that number cannot be changed during the optimization, only thepositions that have zero are free to change.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 88

The condition that every 3×3 sub square containsa number from 1 to 9 is guaranteed intrinsically.Penalty functions are used to force the otherconditions.Each row and column of the Sudoku solutionmust contain each number between [1, 9] once andonly once. This can be transformed to a setinequality constraints. The first two equations (1)require that row and column sums should be equal to45 and the other two equations (2) require that eachrow and column product should be equal to 9!.ggggi1j1i2( x)= 45 −j2( x)= 45 −( x)= 9! −( x)= 9! −9j=19i=19∏j=19∏i=1xxi,jxi,ji,jxi,j(1)(2)Another requirement is derived from the settheory. It is required that the each row x i and columnx j , must be equal to set A = {1, 2, …, 9}. Thefunctions (3) calculate the number of missingnumbers in each row (x i ) and column (x j ) set.A = {1,2,3,4,5,6,7,8,9}ggi3 i , (3)j3( x)=( x)=A − xA − xwhere | ⋅ | denotes for the cardinality of a set.In the optimal situation all constraints (1), (2),and (3) should be equal to zero. The overall fitnessfunction is a linear combination of the partial costfunctions (1–3):f ( x)= 10*(jgij2g ( x)+i1( x)+ 50*(jjgij1( x))+gi3( x)+ijgi2(x)+(4)g ( x))The fitness function (4) was chosen after a seriesof tests of different ideas for constraints andparameter values. It penalizes most heavily if somenumbers of set A are absent in the row or columnset. It also penalizes if sums and products of a rowor column are not equal to the optimal situation.This fitness function setting worked satisfactorily,j3but in the future we need more considerationwhether or not this is a proper fitness function forthe Sudoku problem.3 Experimental resultsIn order to test the proposed method we decidedto use an integer coded GA. The size of the GAchromosome is 81 integer numbers, divided intonine sub blocks of nine numbers. The uniformcrossovers were applied only between sub blocks,and the swap, 3-swap and insertion mutation withrelative probabilities 50:30:20, respectively, insidesub blocks, with an overall mutation probability0.12. The population size was 100, and elitism 40.We tested five Sudoku puzzles taken from thenewspaper Helsingin Sanomat (2006) marked withtheir difficulty rating 1-5 stars. These had 28 to33 symmetric givens. We also tested four Sudokustaken from newspaper Aamulehti (2006), they weremarked with difficulty ratings: Easy,Challenging, Difficult, and Superdifficult. They contained 23 to 36nonsymmetrical givens.We also generated new Sudoku puzzles (nogiven numbers in the beginning), marked withdifficulty rating: New Sudoku. We tried to solveeach of the ten Sudoku puzzles 100 times. Thestopping condition was the optimal solution found,or max. 100000 generation, i.e. max. six milliontrials (fitness function calls).The results are given in table 1. The columns(left to right) stand for: difficulty rating, the count ofhow many of the 100 optimization runs foundoptimal solution, and minimum, maximum, average,median as well as standard deviation calculated onlyfrom those test runs that found the solution (showedin count section). The statistics represent the amountof generations required to find the solution.Table 1 shows that the difficulty ratings ofSudoku’s correlate with their GA hardness.Therefore, the more difficult Sudokus for a humansolver seem to be also the more difficult for the GA.The GA found the solution for Sudoku’s withdifficulty rating 1 star and Easy every time. TheEasy from Aamulehti was the most easiest to solvein general, even easier than to generate a NewSudoku.With other ratings the difficulty increasedsomewhat monotonically with the rating, and thecount of those optimization runs that found asolution decreased. In addition, the average andmedian GA generations needed to find solutionincreased.89

Table 1. The comparison of how effectively GA finds solutions for the Sudoku puzzles with differentdifficulty ratings. The rating New means no givens (new Sudoku) and numbers 1-5 are Sudoku puzzles withsymmetric givens and their difficulty ratings taken from the newspaper Helsingin Sanomat (2006). TheSudokus with ratings Easy to Super difficult are taken from newspaper Aamulehti (2006) and theyhad nonsymmetrical givens. Count shows how many of the 100 GA optimization runs found the solution andother columns are the descriptive statistics of how many GA generations was needed to find the solution.Difficulty rating Givens Count Min Max Average Median StdevNew 0 100 206 3824 1390.4 1089 944.671 star 33 100 184 23993 2466.6 917 3500.982 stars 30 69 733 56484 11226.8 7034 11834.683 stars 28 46 678 94792 22346.4 14827 24846.464 stars 28 26 381 68253 22611.3 22297 22429.125 stars 30 23 756 68991 23288.0 17365 22732.25Easy 36 100 101 6035 768.6 417 942.23Challenging 25 30 1771 89070 25333.3 17755 23058.94Difficult 23 4 18999 46814 20534.3 26162 12506.72Super difficult 22 6 3022 47352 14392 6722 17053.33100000Generations100001000EasyNew1 star2 stars3 stars4 stars5 starsChalleng.Super d.Difficult1000 20000 40000 60000 80000 100000AverageFigure 5. The difficulty order of tested Sudokus. The minimum and maximum generations needed to solveeach Sudoku from 100 test runs as a function of average generations needed.However, the most difficult puzzle (5 stars)from Helsingin sanomat was solved 23 times out of100 test runs, which was almost as often as 4 starspuzzle (26 times). With the exception of Easy theother Sudokus in Aamulehti were found to be moredifficult than Sudokus in Helsingin sanomat. TheSudoku with rating Challenging was almost asdifficult as 4 and 5 stars Sudokus in Helsinginsanomat.The puzzles Difficult and SuperDifficult of Aamulehti were much more difficultthan any of the puzzles in Helsingin sanomat. Thedifficulty rating of these two also seemed to be inwrong mutual order (fig. 5), since Difficult washarder for GA than Super Difficult. However,both were so difficult that GA found the solutiononly a few times, so the difference is not statisticallysignificant.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 90

Figure 5 shows the difficulty order of the testedSudokus. The {Easy, New Sudoku, 1 star}seems to form a group of the most easiest Sudokus,while 2 and 3 stars Sudokus lie by themselvesbetween the previous and the group containing {4stars, 5 stars, Challenging}. The two mostdifficult Sudokus {Super difficult,Difficult } are in the final group.The New Sudoku (no givens) is the secondeasiest for GA in average. This means that GA canbe used for generating new puzzles. It takes for GAbetween [406, 3817] generations ([24400, 229060]trials) to create a new open Sudoku solution that canfurther be used to create a new puzzle.If GA is used as puzzle generator, one firstsearches any possible open solution and thenremoves some numbers and leave the givens. Thedifficulty ratings of new Sudoku can also be testedwith GA by trying to solve it after the givens areselected.4 Conclusions and futureIn this paper, we studied if Sudoku puzzles canbe solved with a combinatorial genetic algorithm.The results show that GA can solve Sudoku puzzles,but not very effectively. There exist more efficientalgorithms to solve Sudoku puzzles (Wikipedia,2006). However, in this study the aim was to testhow efficient pure GA is, without problem specificrules, for solving Sudokus. The GA results can ofcourse be enhanced by adding problem related rules.The other goal was to study if difficulty ratingsgiven for Sudoku puzzles in newspapers areconsistent with their difficulty for GA optimization.The answer to that question seems to be positive:Those Sudokus that have a higher difficulty ratingproved to be more difficult also for geneticalgorithms. This also means that GA can be used fortesting the difficulty of a new Sudoku puzzle.Grading puzzles is said to be one of the mostdifficult tasks in Sudoku puzzle creation (Semeniak,2005), so GA can be a helpful tool for that purpose.The new puzzles are usually generated by findingone possible Sudoku solution and then removingnumbers as long as only one unique solution exists.The GA can also be used to select the removednumbers and also testing if there is still only oneunique solution. This can be tested e.g. by solvingthe Sudoku 10 or more times with other GA, and ifthe result is always identical, we can be relativelysure (but not completely) that a unique solutionexists. However, it may be wiser to use moreeffective algorithms to test the uniqueness of asolution.The future research may consist of generating amore efficient hybrid GA and algorithms createdspecifically to solve Sudoku. Another research areawould be to study if it is possible to generate a fitnessfunction based on an energy function (Alander,2004).It has been said that 17 given number is minimalto come up with a unique solution, but it has not beenproved mathematically (Semeniak, 2005). The GAcould also be used for minimizing the number ofgivens and still leading to a unique solution.AcknowledgementsThe research work of the author T.M. was funded bythe research grant from the TietoEnator fund of theFinnish Cultural Foundation.ReferencesAamulehti. Sudoku online. Available via WWW:http://www.aamulehti.fi/sudoku/ (cited11.1.2006)Alander, J.T. Potential function approach in search:analyzing a board game. In J. Alander, P. Ala-Siuru, and H. Hyötyniemi (eds.), Step 2004 – The11th Finnish Artificial Intelligence Conference,Heureka, Vantaa, 1-3 September 2004, Vol. 3,Origin of Life and Genetic Algorithms, 2004, 61-75.Alander, J.T., Mantere T., and Pyylampi, T.Threshold matrix generation for digital halftoningby genetic algorithm optimization. In D. P.Casasent (ed.), Intelligent Systems and AdvancedManufacturing: Intelligent Robots and ComputerVision XVII: Algorithms, Techniques, and ActiveVision, volume SPIE-3522, Boston, MA, 1-6November 1998. SPIE, Bellingham, Washington,USA, 1998, 204-212.Alander, J.T., Mantere T., and Pyylampi, T. Digitalhalftoning optimization via genetic algorithms forink jet machine. In B.H.V. Topping (ed.)Developments in Computational mechanics withhigh performance computing, CIVIL-COMPPress, Edinburg, UK, 1999, 211-216.Ardel, D.H.. TOPE and magic squares, a simple GAapproach to combinatorial optimization. In J.R.Koza (ed.) Genetic Algorithms in Stanford,Stanford bookstore, Stanford, CA, 1994.Darwin, C. The Origin of Species: By Means ofNatural Selection or The Preservation ofFavoured Races in the Struggle for Life. Oxford91

University Press, London, 1859, A reprint of the6th edition, 1968.Evonet Flying Circus. Magic square solver.Available via WWW:http://evonet.lri.fr/CIRCUS2/node.php?node=65(1996, cited 11.9.2006)Gold, M.. Using Genetic Algorithms to Come upwith Sudoku Puzzles. Sep 23, 2005. Available viaWWW:http://www.csharpcorner.com/UploadFile/mgold/Sudoku09232005003323AM/Sudoku.aspx?ArticleID=fba36449-ccf3-444f-a435-a812535c45e5 (cited 11.9.2006)Helsingin Sanomat. Sudoku. Available via WWW:http://www2.hs.fi/extrat/sudoku/sudoku.html(cited 11.1.2006)Holland, J. Adaptation in Natural and ArtificialSystems. The MIT Press, 1992.Kang, H.R.. Digital Color Halftoning. SPIE OpticalEngineering Press, Bellingham, Washington, &IEEE Press, New York, 1999.Kobayashi, N., and Saito. H. Halftone algorithmusing genetic algorithm. In Proc. of 4th Int.Conference on Signal Processing Applicationsand Technology, vol. 1, Newton, MA 28. Sept. –1. Oct. 1993, DSP Associates, Santa Clara, CA,1993, 727-731Koljonen, J. and Alander, J.T. Solving the “urbanhorse” problem by backtracking and geneticalgorithm – a comparison. In J. Alander, P. Ala-Siuru, and H. Hyötyniemi (eds.), Step 2004 – The11th Finnish Artificial Intelligence Conference,Heureka, Vantaa, 1-3 September 2004, Vol. 3,Origin of Life and Genetic Algorithms, 2004,127-13.Lawler, E.L., Lentra, J.K., Rinnooy, A.H.G., andShmoys, D.B. (eds.). The Traveling Salesmanproblem – A Guided Tour of CombinatorialOptimization. John Wiley & Sons, New York,1985.Li, H., Jiao, Y.-C., and Wang, Y.: Integrating thesimplified interpolation into the genetic algorithmfor constrained optimization problems. In Hao etal. (eds) CIS 2005, Part I, Lecture Notes onArtificial Intelligence 3801, Springer-Verlag,Berlin Heidelberg, 2005, 247-254.Mantere, T. A min-max genetic algorithm for minmaxproblems. In Wang, Cheung, Liu (eds.)Advances in Computational Intelligence andSecurity - The Workshop of 2005 Int. Conferenceon Computational Intelligence and Security - CIS2005, Xi'an, China, December 15-19, 2005.Xidian university press, Xi’an, China, 2005, pp.52-57.Newbern, J., and Bowe Jr., M. Generation of BlueNoise Arrays by Genetic Algorithm. InB.E.Rogowitz and T.N. Pappas (eds.) HumanVision and Electronic Imaging II, San Jose, CA,10.-13. Feb. 1997, Vol SPIE-3016, SPIE - Int.society of Optical Engineering, Bellingham, WA,1997, 441-450.Runarsson, T.P., and Yao, X.: Stochastic ranking forconstrained evolutionary optimization. IEEETransactions on Evolutionary Computation 4(3),2000, pp. 284-294.Semeniuk, I. Stuck on you. In NewScientist 24/31December, 2005, 45-47.Sudoku Maker. Available via WWW:http://sourceforge.net/projects/sudokumaker/(cited 27.1.2006)Wikipedia. Sudoku. Available via WWW:http://en.wikipedia.org/wiki/Sudoku (cited11.9.2006)Wiley, K.: Pattern Evolver, an evolutionaryalgorithm that solves the nonintuitive problem ofblack and white pixel distribution to produce tiledpatterns that appear grey. In The Handbook ofGenetic Algorithms. CRC Press, 1998.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 92

Using SOM based resampling techniques to boost MLPtraining performanceJussi Ahola * Mikko Hiirsalmi †Technical Research Centre of Finland * Technical Research Centre of Finland †FI­02044 VTT, Finland FI­02044 VTT, FinlandJussi.Ahola@vtt.fi Mikko.Hiirsalmi@vtt.fiAbstractWe describe SOM based modeling work that has been conducted in a research project that aims atdeveloping a flight parameter based fatigue life analysis system for the Finnish Air Force F-18fighters. The idea of using SOM based clustering as a basis for iteratively selecting poorly modeledtest samples to be included into the training set has been investigated. We call these schemes SOMresampling. Our tests indicate that clear gains could be achieved in the test cases when measured interms of the mean squared error criterion as typical in MLP network training.1. IntroductionThis work constitutes a part of a larger project todevelop a flight parameter based data analysissystem to estimate flight specific fatigue lifeexpenditure for the Finnish Air Force F-18 fighters.In Figure 1, the general analysis chain of the systemis shown. In the analysis chain MLP 1 neuralnetworks have been used to learn regression modelsto estimate strain in a selected position near eachdesired critical structural detail. The strain valuesare adjusted to the desired position through finiteelement models and fatigue life analysis software isused to estimate the fatigue life expenditure ofwhole flights. Therefore there is no direct linkbetween the neurally estimated strain values and thefatigue life expenditure estimates. During theoperational phase of the system the measurementdata is based on currently available instrumentationonboard the aircraft. These data points arepreprocessed by filtering and interpolationtechniques to produce refined time series data. Forthe training phase the onboard data has beensupplemented with additional strain gaugemeasurement producing the real strain values in theselected positions. These values are targets for theneural network models. One neural network modelhas been created for each structural detail. Early1 MLP means Multi Layered Perceptron being one form of socalled feed forward neural network architectureswork with the system design and the neural networktraining have been documented in (Salonen, 2005).Figure 1: The analysis chain used in proof ofconcept study to predict strain gauge response(Tikka, Ahola and Hiirsalmi, 2006)Currently the amount of training data is ratherlimited covering 25 flights where each flightconsists of a temporal sequence of flightmanoeuvres. In the coming years hundreds of newflights are going to be carried out with two airplanesequipped with the strain gauge measurementsallowing one to gather new data for neural networktraining during their normal operation. As the93

amount of the data grows the neural network modelsare going to be relearned. The input variables havebeen selected by aeronautical engineers to reflectmeaningful dependencies. The selection of training,validation and test data sets is another critical area inneural network modelling. Data selection issues andmodel complexity have been discussed in (Bishop,1995) and in (Hastie, Tibshirani and Friedman,2001). The fundamental idea in the selection of thedatasets is that they represent well the phenomenathat the neural network is being trained to predict.The following points should be considered:• The data sets should all come from the samedistribution of data, i.e. they should be sampledfrom the same distribution. Bias between thedata sets is harmful for gaining good predictionresults.• There must be enough data samples to be ableto learn statistically accurate model parameters.The complexity of the network has an impacton the number of training samples needed forlearning. On the other hand, if the complexityof the underlying model is too small theachievable modelling accuracy will bediminished.• In MLP network training the distribution of thetraining data samples should be adequatelybalanced so that there are approximately equalamounts of samples in each value segment ofthe target variable because typical trainingsession tries to minimize the mean squarederror and areas with lower amounts of sampleswill not be optimized as well. On the otherhand, it is also possible to stress importanttarget value ranges by over-representing thosesamples.Training data is used by the MLP algorithm tolearn the weight and bias parameters. Testing data isused during the training session to evaluate howwell the learned neural networks perform onpreviously unseen data. The best performingnetwork is selected for production usage. Validationdata is used to evaluate the performance of theselected network on previously unseen data. Itshould give a measure of the average behaviour ofthe network on previously unseen data. The dataselection methods used in the project have beenbetter described in (Salonen, 2005) and in (Tikka,Ahola and Hiirsalmi, 2006). One of the problemspresent is that the overall life cycle performance hasnot been evaluated with totally unseen test data – inthe testing phase the whole time series data has beenused to evaluate the performances. This mayproduce optimistic performance results in caseswhere extreme strain samples are overly representedin the training data set which generally raises theprobability of over-fitting. A better evaluationmethod could be achieved by leaving a few flightsfor testing purposes or by averaging the results ofseveral training sessions where a small amount ofdifferent flights are left for testing. These tests mustwait until we get new training data from furtheroperational flights.Our role in this project has originally dealt withvalidation of the modelling results by using SOM 2modelling in a qualitative sense in order to findinput data clusters reflecting typical flight states andlooking for systematic errors in the predictionresults. Cluster specific prediction error histogramsand time series plots which have been coloured withthe cluster colours have been used to search forproblematic areas. As an extension it was consideredworthwhile to investigate whether the learning dataset could be adjusted iteratively during the trainingcycle to automatically improve the balance of thelearning data samples. The basic idea has been thatif a SOM map unit has a large portion of predictionerrors on the test data samples these samples may bepoorly represented in the learning data set andfurther samples should be selected from this area. Insome areas the distribution of prediction errors mayalso be rather flat around the zero level meaning thatthe data set is inconsistent or that they do not form acompact input parameter cluster.2. Resampling methodstestedIn this work our research hypothesis has been thatthe modelling efficiency of feed-forward neuralnetworks can be improved iteratively by selecting asmall portion of the less well predicted test samplesas additional training samples. We have tested a fewapproaches to accomplish this task and haveevaluated their performance with real measurementdata.In order to set a reference line for evaluationswe have at first constructed a simple approachwhere we compute the squared modelling errors (ei2= (measuredi-modeledi)2) for test data samples andselect the desired amount of the worst predicted testsamples as new training samples. This is oursampling method 1.All the other tested methods are based on SOMclustering information (Kohonen, 2001). At first we2 Self-Organizing Map (SOM) is an unsupervised learningmethod for vector quantizing a high-dimensional data set into alower dimensional output space while at the same time preservingneighbourhood topologyNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 94

have computed a 2-dimensional SOM map based onnine explanatory variables of the feature vector ofthe training data samples (the predicted value hasbeen masked off from this training). We havetherefore attempted to find the typical inputparameter combinations that reflect so called ‘flightstates’. A relatively sparse SOM map has been usedfor this task in order to get a reasonably small set ofcodebook vectors for further processing. We havethen computed the best matching units (BMUs 3 ) forall the training samples and using these samples wehave computed the average modelling error for eachmap unit. We have then computed the BMUs for allthe test data samples and we have identified the listof test data samples for each map unit. We use theselists for random selection purposes with the nextthree sampling methods.In sampling method 2, we scale the samplinglikelihoods of each SOM map unit based on theaverage modelling error of those training datasamples that are closest to that map unit. Withineach map unit the sampling likelihood is uniformbetween the data samples. Therefore, it is morelikely to select new samples from the test datasamples that have been projected to the map unitswith the highest average modeling error.In sampling method 3, we extend the previousmethod by additionally scaling the selectionlikelihood of the test data samples by their relativemodelling error within their BMU map unit.Therefore, it is more likely to select new samplesfrom the test data samples that have been projectedto the map units with the highest modeling dataerror and among these test data samples we preferthe ones that have the largest modeling error.In sampling method 4, we extend the previousmethod by additionally scaling the selectionlikelihood of the test samples by their quantizationerror 4 within their BMU map unit. Therefore, it ismore likely to select new samples from the test datasamples that have been projected to the map unitswith the highest average modeling error and amongthese test data samples we prefer the ones that havethe largest modeling error and at the same time fitless well to their BMU map unit (those that havelarge quantization errors).3. Evaluation test resultsIn the evaluation tests the initial training data setcontained 14995 training samples that were selectedfrom a data set of more than 550000 data samples.The samples contained nine explanatory variablesand one target variable to be predicted. The3 BMU means the best-matching unit and it is computed for adata sample as the minimum distance to any of the SOM mapunit vectors.4Quantization error means the distance between a data sampleand a SOM map unit vector.variables were already scaled in a proper way in therange of [-0.8, 0.8]. The proper network complexityhad been selected while initially training the originalnetworks and in order to get comparable results weused the same network architecture with 9 inputvariables, 25 hidden nodes in hidden layer 1 and 1output variable. Therefore, in this case the networkcomplexity was 276 weight parameters to beoptimized. The transfer function used in layer1 wastansig and in the output layer linear. The trainingalgorithm was Levenberg-Marquardtbackpropagation algorithm with the performancefunction being the minimization of the MSE errorand with the early stopping threshold value being 5epochs.During each resampling iteration we taught 5networks with random initial states and selected theminimum results on a previously unseen test data setas the reference value for each iteration for eachsampling scheme. In the first iteration we computedthe starting values with the initial training data setand after that we added 1000 randomly sampled testdata samples to the training set and recomputed theMLP networks.min MSE65.554.543.5x 10 ­3Minimum MSE of different resampling approachessample1sample2sample3sample42 4 6 8 10 12 14iterationFigure 2: Minimum Mean squared error (MSE) ofthe tested sampling methods on each iteration stepIn Figure 2, we have summarized theperformances achieved with the different iterativeresampling schemes tested. We have used the MSEerror (mean squared error) as the quality criterion asit is the criterion that is being minimized whentraining our MLP networks. In the figure iterationstep 1 marks the initial starting state without theiterative resampling steps. We can see that with thesimple sampling scheme, sample1, only moderategains can be achieved during the first steps but withmore steps the results start to oscillate and theaccuracy becomes somewhat unstable. The SOMbasedschemes (samples 2-4) behave fairly similarlyto each other. All of them show clear gains in themodelling accuracy. The achieved accuracies tend tovary between successive iterations but the trend is95

towards more accurate results. We can see thatsample2 has the biggest oscillations but producesslightly more accurate results than sample3.Sample4 produces fairly consistent, non-oscillatoryresults and the best accuracy with iteration step 10.This achieved best accuracy produced a 31% gain inaccuracy as compared to the starting situation withthe initial training data set.4. DiscussionOur test results indicate that MLP modellingaccuracy may be boosted by our SOM basediterative resampling schemes when we measuresuccess with the same performance criterion aswhen teaching the MLP networks. This indicatesthat in typical MLP learning our SOM basedresampling schemes may be used to boostperformance. The techniques are however onlypreliminary methods and we believe that they couldbe enhanced by better considering therepresentativeness of the map units, possiblyclustering the map units and by better consideringthe target variable distribution within each map unitwhile determining the sampling likelihoods.In (Tikka, Ahola and Hiirsalmi, 2006) it hashowever been indicated that no clear gains can beachieved when the performance gains are measuredin terms of consumed life time. We believe that thisis caused by the fact there is no direct link from thestrain values that have been predicted by the MLPneural networks but the effect has been transferredby FEM transfer models and by the life cycleestimation software which inevitably cause noise inthe predictions.SOM based clustering might also be useful in theoriginal data selection phase as an alternative to theK-means clustering previously used in (Tikka,Ahola and Hiirsalmi, 2006).ReferencesChristopher M. Bishop. Neural Networks for PatternRecognition. Oxford University Press, Oxford,1995.T. Hastie, R. Tibshirani and J. Friedman. Theelements of statistical learning : data mining,inference and prediction, 2001.Teuvo Kohonen. Self-Organizing Maps. Springer,3rd, extended edition, 2001.Risto Laakso & al., On the Neural NetworkApplications as Tools to Improve AircraftStructural Integrity Management Efforts, ICAF2005 conference 6. – 10.6.2005, Hamburg,Germany.Tuomo Salonen. An Assessment of the Feasibility ofa Parameter Based Aircraft Strain MonitoringMethod. M.Sc Thesis at the TechnicalUniversity of Helsinki, 12.7.2005,www.aeronautics.hut.fi/edu/theses/full_thesis/Salonen_Tuomo_2005.pdf.Jarkko Tikka, Jussi Ahola, Mikko Hiirsalmi.Parameter based fatigue life analysis, influenceof training data selection to analysisperformance. American Institute ofAeronautics and Astronautics Modeling andSimulation Technologies Conference, 2006.5. ConclusionsThe idea of using SOM based clustering as a basisfor iteratively selecting poorly modeled test samplesto be included into the training set has beeninvestigated. We call these schemes SOMresampling. Our tests indicate that clear gains couldbe achieved in the test cases when measured interms of the mean squared error criterion as typicalin MLP network training.AcknowledgementsThis work has been funded by the Finnish Air Forceand the Technical Research Centre of Finland. Wewant to especially thank Jarkko Tikka for setting upresearch challenges and Risto Laakso for letting usshare his SOM modelling work in this domain.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 96

Data-Based Modeling of Electroless Nickel PlatingHans-Christian Pfisterer ⋆⋆ Control Engineering LaboratoryPL 550002015 TKKFinlandh.pfisterer@gmx.deHeikki Hyötyniemi †† Control Engineering LaboratoryPL 550002015 TKKFinlandheikki.hyotyniemi@tkk.fiAbstractThis work connects the neocybernetic theory of elastic systems to modeling of a complex industrialprocess. It turns out that using simple mathematics combined with multilinear tools leads to linearmodels that can be used for monitoring and control. As an application example, electroless nickelplating is studied.1 IntroductionWhat has artificial intelligence to do with modeling ofchemical processes? It seems that such a very technicalapplication domain would have little in commonwith the AI approaches. Still, in all complex systemsthe challenges are similar. One would need to understandemergence to escape from the individual lowlevelbehaviors to the level of relevant functionalities.In the work at hand the industrial process of electrolessnickel plating is studied. This chemical systemis very complex since many chemicals are involvedand the process has a lot of variables being observedand controlled in order to reach good plating results.What is more, for commercial reasons not all compoundsand reactions have been disclosed. It is verydifficult also for a domain expert to understand thebehavior of the system.In this paper, a very simple, data-based model forcontrol purposes of the nickel alloy thickness and itsphosphorus content is presented. This model wasachieved using basic ideas and mathematical toolsbased on the theory of neocybernetics.2 NeocyberneticsSince models of industrial systems and observed naturalsystems get more and more complex and verycomplicated to understand the traditional way ofmathematical modeling with dynamic nonlinear differentialequations and constraints leads to an unmanageablemess of functions and equations. Furthermore,one has to be a domain expert to understand theinterconnections between variables in order to writethe mathematical equations for the description of thebehaviors in the first place.Neocybernetics, introduced by H. Hyötyniemi(2006), offers a new approach to get reasonable systemmodels and helpful system understanding whenlooking at a system from a different angle. Since industrialsystems are monitored extensively the datacan be used to find connections among the variablesand possibly associate some variables with others.The system structure is hidden in this data and it willemerge when manipulated in an appropriate way. Usingthe right mathematical tools and a holistic view ofthe system this modeling machinery is also availablefor non-experts of the particular domain. Here informationabout the behavior of the system is retrieveddirectly from the measurement data.2.1 Key pointsTo see a system through neocybernetic eyeglassessome assumptions about the system have to be made.There are some basic principles (see H. Hyötyniemi(2006) and H.-C. Pfisterer (2006)):• Emergence: Some unanticipated functionalitycan appear after the cumulation of some simpleoperations.• Dynamic Balance: The emphasis is on systemsin some kind of balance rather than on the processitself.• Linearity pursuit: The system behavior is consideredto be linear as long as nonlinearity is notabsolutely necessary.97

• Environment: Neoybernetic systems are assumedto be oriented towards their environment,they reflect and mirror their environment.All these principles and assumptions are reasonableand in many natural and industrial systems they canbe fulfilled. The linearity assumption needs a furtherexplanation (see Section 5), especially when modelingchemical systems.2.2 ElasticityThe system is assumed to be in thermodynamic balance.The changes in the environment are seen as disturbancescausing tensions in the system, pushing thesystem away from the balance. According to the LeChatelier principle (H. Hyötyniemi, 2006), the systemyields, but after the pressure vanishes, the originalbalance is restored. In a sense, the neocyberneticideas are a functionalization of the intuitions concerningcomplex homeostatic systems.2.3 Degrees of freedomAs mentioned above the hidden structure of the systemwill emerge when the modeling concentrates onthe remaining degrees of freedom in behaviors ratherthan on constraints and restrictions. For simple systemsthis is not reasonable — for very complex systemswhere many variables are strongly connectedand many constraints have to be considered this approachhelps to avoid an unmanageable mess of equations.When concentrating on the non-constraineddegrees of freedom, the emphasis is on phenomenathat are not explicitly seen in any formulas; one couldspeak of emergent models.Now the system is kept in dynamic balance andone searches for the structure of covariation that isstill left in the variable space. A model based on thedegrees of freedom approach is as useful as a traditionalone but has the advantage of being simple andunderstandable. With this “inverse thinking” not allconstraints and chemicals and reactions need to beknown, as long as it can be assumed that the internalinteractions and feedbacks always manage to keep thesystem in balance. More information about degreeof freedom modeling can be found in H. Hyötyniemi(2006) and H.-C. Pfisterer (2006).3 Multivariate toolsUsing the above mentioned key points and utilizingstrong mathematical tools a practical modeling machinerycan be set up. After applying the modelingtools a black box between known variables and unknownones to be estimated can be filled with life.For these purposes the known data (column vectorsof data samples) is collected in a matrix X and theunknown variables in a target space formed by thematrix Y . Here, it is assumed the k sample vectors oflength n and m are stored as rows in the matrices Xand Y , respectively.Principal Component Analysis (PCA)The information in terms of covariations in the datais captured in the (unscaled) correlation matrixR = X T X.A lot of hidden information can be revealed by eigenvaluesλ i of this matrix, and by their correspondingeigenvectors θ i as can be read in H. Hyötyniemi(2001). The eigenvectors are orthogonal and pointin the direction of maximum variation in the originaldataset. The amount of variety of each particulardirection is given by the corresponding eigenvalue.Directions of most variety are assumed to carry mostinformation about the system and it is reasonable totake these into account in the model; hence the modelis built on the degrees of freedoms found in the correlationstructure of the data.Large eigenvalues stand for directions of essentialinformation, small eigenvalues stand for directionswhich most probably contain redundant informationor only measurement noise. When the data is projectedonto the subspace Z of dimension N ≤ nby the mapping matrix F 1, redundancy and measurementnoise in the data can be reduced.Multilinear Regression (MLR)To find the connection to the target space Y a regressionis applied. In order to achieve good results thebase of the starting space should be orthogonal andno redundancy should be there. Using the dataspaceZ these prerequisites are fulfilled and the MLR algorithmcan find a mapping F 2 from Z to Y . Hencea combination of both procedures, now called PrincipalComponent Regression (PCR), can explain thebehavior in Y by the information already given in X.This is illustrated in Figure 1.F1X Z YFigure 1: Data spaces and projections with PCRF2New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 98

PSfrag replacementsPartial Least Squares (PLS)This algorithm includes a view into the target space Yalready while building the mapping F 1. Not only thedataspace X is scanned for the essential information(collected in Z1) but also in Y the key information isextracted (to Z2) and only then an overall mapping isobtained which now bridges input and output. Thissteps and transformations are illustrated in Figure 2.F1X Z1 Z2Figure 2: Data spaces and projections with PLSAfter introducing the industrial process and somespecific variables these tools will be applied and theirresults shown in Section 6.4 Electroless nickel platingThe process of electroless nickel plating is a step ofsurface finishing. Printed wiring boards (PWBs) consistof a epoxy laminate base and a copper layer inwhich electric circuits are etched. Witout protectionthe copper would oxidize very fast, especially whencoming in contact with humidity. In order to enhancethe lifetime of the PWB and to improve its mechanicalproperties the PWB is coated with a gold layer ofabout 0.01µm. But since the copper would diffuse inthe gold and form again an oxidizeable compound thetwo layers are separated by a nickel layer of around4µm. Figure 3 shows a cut through a plated PWB andexplains the layers in detail; since the gold layer is sothin it can not be seen here.F2a b c d eF3YThe use of the word “electroless” is misleading; itemphasizes the difference between the method wherean external power supply is connected to the substratewhich is to be plated and the approach of an activesubstrate that deposits nickel atoms from ions on itssurface. Here nickel is provided in form of ions in anaqueous solution, beside the reducer hypophosphitethat provides the necessary electrons for the transformationfrom nickel ions to real atoms. Due to theused reducer the alloy has a phosphorus content ofup to 15 weight percent. The catalytically activatedsubstrate is immersed into the bath and an alloy ofnickel atoms and some phosphorus can be built, becausenickel itself is also catalytically active itself.There are electric currents in the bath caused by theconversion of ions to atoms, so the process is not reallyelectroless.The bath consists of many more chemicals whichhelp to keep the bath stable and in a desired status.The composition of these chemicals is either notexactly known or a well kept industry secret whichmakes the traditional modeling of the bath behaviorvery difficult or even impossible. Furthermore onehas to be a chemical domain expert to understand theconnection between the chemicals and their behaviorespecially when connected to the substrate and its expandingnickel alloy.The characteristics of the bath are observed continuouslyand as many variables as possible are measured.Hence online measurements of the currentbath nickel concentration, bath temperature and pHvalue are available and with good controllers keptalong desired values to keep the bath constitutionconstant. The substrate and its characteristics aremeasured in a laboratory when the plating is finished.Summarizing the dataspaces X and Y (Section 3) basicallycontain the following information:X: Nickel concentration, pH value, ammonia concentration,ammonia and nickel sulfate pumping,plating area and temperature.Y : Alloy thickness, phosphorus content, hypophosphiteconcentration and orthophosphite concentration.Figure 3: Cut through a plated substrate. Layers: (a) ametallic fastener and (b) a piece of conductive plasticto fix the sample, (c) Ni-P layer, (d) copper layer, (e)the base of PWB (epoxy laminate) (K. Kantola, 2004)The original dataset X is further prepared and expandedas explained in the following section. Theinformation about hypophosphite and orthophosphiteis not really a plate characteristic and for that reasonnot further studied here.99

5 LinearityThe system is in a state of thermodynamic balance.This state is kept constant by a very strong and accuratecontrol mechanism provided by the bath surrounding.Balance also is one of the neocybernetickey points and so is very important to this approach.The thermodynamic equilibrium can be described bythe constantK = Cb1 B 1. . . C b βB βC a1A 1. . . C aαA α.This constant is highly nonlinear dependent on theconcentrations of the different chemicals in the solution.But applying a logarithm on both sides anddifferentiating the expression leads to0 = b 1∆C B1¯C B1−a 1∆C A1¯C A1∆C Bβ+ · · · + b β ¯C Bβ∆C+ · · · − aAαα ¯C Aα,where ∆C i / ¯C i are deviations from nominal values,divided by those nominal values, meaning that it isrelative changes that are of interest. For more informationabout this equations see H.-C. Pfisterer (2006)and H. Hyötyniemi (2006).Even more the whole system can be representedin a linear way when using variables in a properway. Nonlinearity can by avoided by appropriatepreprocessing of the variables (by employing relativechanges among the data). What is more, the dynamicnature can be put in the static form when the variablesare selected in a clever way. The following points explainhow to use information in order to stay in thelinear domain even if the real system is nonlinear andcan also be found in H. Hyötyniemi (2006):• Temperature: According to the Arrhenius formula,reaction rates k are functions of the temperature,so that k ∝ exp(c/T ). When this issubstituted, the model remains linear if one augmentsthe data vector and defines an additionalvariable z T = ∆T/ ¯T 2 .• Acidity: The pH value of a solution is definedas pH = − log c H +. Because this is a logarithmof a concentration variable, one can directly includethe changes in the pH value among thevariables as z pH = ∆pH.• Voltage: In electrochemical reactions one maycharacterize the “concentration of electrons”. Itturns out that according to the Butler-Volmertheory the amount of free electrons is exponentiallyproportional to the voltage. Hence, aftertaking logarithms, the “electron pressure” can becharacterized by z e − = ∆U.• Physical phenomena: It is evident that phenomenathat are originally linear like diffusioncan directly be integrated in the model, assumingthat appropriate variables (deviations from anominal state) are included among the variables.In practice, some reactions are not in balance;rather, there can be a constant flux of some chemicalexiting from the reacting solution. The above balanceexpressions can be written also for such “dissipative”reactions, and linear models are again applicablearound the equilibrium flux. This means thatalso the rates of change need to be included amongthe variables that characterize the dynamic state ofthe process.Some of the variables to be included in the regressionmodel are integrals over time — for example, inthe coating process, the layer thickness is the integralof the nickel reduction rate. Because of the modellinearity, such integrals can be transferred from themodel output to the input — meaning that the layerthickness can be modeled as the integrals of the variablesare included among the variable values themselvesamong the data.Hence apart from the plain data also some featurescan be included in the dataset X. Since the measurementsample is only taken at the precise time whenthe plate is taken out of the bath it makes sense toinclude more information about the whole time theplate is immersed. A weighted integral over the platingtime solves this problem, hence the dataset X isextended with integrals of the plain variables whichdoubles the dimension of X. The mentioned connectionbetween layer thickness and nickel reduction rateis another reason for using integrals in the input structure.Furthermore an old set of data can be includedby weighting the old data differently in comparison tothe recent information. This smoothing of data againadds featured data to X.A schematic view and a summary of all the mentionedvariables in the input structure can be seen inFigure 4.6 Modeling the plating processWhen using the described setup the first step is PCR(Section 3). Figure 5 shows the eigenvalues λ i of thecorrelation matrix of X in descending order. Sincethe dimension of the used dataspace is n = 18 thereare as many eigenvalues, each one representing theNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 100

TimelineBath variablesNickel concentrationTemperaturepH value...Integrals ofNickel concentrationTemperaturepH value...Smoothing ofNickel concentrationTemperaturepH value...Plate variablesThicknessPhosphor contentOrthophosphiteHypophosphitePlating timePSfrag replacementsNumeric value5432191% of maximum00 2 4 6 8 10 12 14 16 18Eigenvalue indexTimelineFigure 5: Latent variables θ i (PCA) and the numericalvalues of the corresponding eigenvalues (equalsimportance of information among data)Plain variablesCalculated featuresLaboratory measurementsOnline informationSeldom informationFigure 4: Schematic view of input data set along thetimeline of the plating processThickness [µm]54.543.518 input vectors, 2 latent vectors capture 76.5465%measured dataestimate, SSE = 0.028895electrochemical model, SSE = 0.03708830 5 10 15 20 25 30 35 40 45Time [h]amount of information kept in the corresponding directionθ i . It can be seen clearly that the last directionsdo not carry any information; that shows theredundancy in the data. Since the noise is assumedto be uncorrelated it is evenly distributed in all directions,hence some directions carry mostly noise andno real information. In the end it is reasonable to useonly the first 7 directions for a new dataspace Z fromwhich a final mapping to Y should be obtained. Thisfirst N = 7 vectors carry already 91% of the full information.The alloy thickness and its phosphorus content canbe accurately estimated with these 7 vectors. UsingPLS as an data preparing algorithm the dataspace caneven be reduced to a dimension of N = 2 and obtainan even better estimate. The results of the modelgained by PLS analysis are presented here. For moreresults and discussions see H.-C. Pfisterer (2006).Figure 6 shows the process data of the alloy thicknessdivided in three parts. The first one was used formodel estimation along the above described way. Theother two parts are validation sets. Blue informationis the real data, measured in the laboratory and storedby a data acquisition system. Green data is a modelestimation, obtained with a traditional complex electrochemicalmodel by K. Kantola (2004) and used asa comparison to validate the linear model at hand. InThickness [µm]Thickness [µm]54.543.54.53.5Validation 1measured dataestimate, SSE = 0.082567electrochemical model, SSE = 0.1459530 5 10 15 20 2554Validation 2measured dataestimate, SSE = 0.05134electrochemical model, SSE = 0.0593930 5 10 15 20 25Overall Validation SSEs equals 0.064752Figure 6: Alloy thickness: measured data and twocompared estimatesred color the data estimation for the alloy thicknessby the linear model can be seen and compared to realdata and the other model estimation.The same setup of data was used to estimate thephosphorus content of the nickel alloy (Figure 7).In red color again the result of the linear model at101

Phosphor content [w%]Phosphor content [w%]Phosphor content [w%]9.598.5818 input vectors, 2 latent vectors capture 76.5465%7.5measured data7estimate, SSE = 0.14906electrochemical model, SSE = 0.210446.50 5 10 15 20 25 30 35 40 45109.598.58Time [h]Validation 1measured dataestimate, SSE = 0.77085electrochemical model, SSE = 0.40147.50 5 10 15 20 2598.587.5Validation 2measured dataestimate, SSE = 0.030103electrochemical model, SSE = 0.03374170 5 10 15 20 25Overall Validation SSEs equals 0.34825Figure 7: Alloy phosphorus content: measured dataand two compared estimateshand along with real data (blue) and results of anothermodel (green).The linear model provides a very accurate estimationof the alloy thickness and an accurate estimationof its phosphorus content. This is no surprise in thefirst parts of either dataset, since they were used tobuild the model and the model is tailored by the algorithmespecially for them. But the correct estimates inthe other parts confirm and validate the good performanceof the linear model. The power and the correctcombination of strong mathematical tools produceda very accurate result which can keep up and evenbeat complicated and inscrutable models. Since thismodel is a linear combination of measured data it isalso possible to see the importance of each measuredvariable for the characteristic of each output variable.It is interesting to see that if the estimate of the linearmodel is wrong also the estimate of the complexmodel is wrong. This is a hint for missing informationthat is not available for the modeling machineryin either case or some mistakes in the measurements,hence no fault of the linear model. Further it shouldbe pointed out that there is an improvement when usingPLS instead of PCR and also when including integralsin the input data vector. However, smoothedold data can not improve the results.The overall result combined with the simplicity ofits design leads a way to a good control machineryfor nickel plating processes. The model can easilybe used and adjusted by process workers and it canprovide the necessary information for a controller inform of an observer for the plate characteristics. Theinformation about these characteristics is already hiddenin the anyway measured bath variables and it wasjust necessary to reveal it and make it emerge.AcknowledgementsMany thanks to Prof. Dr.-Ing. Dr.h.c. Michael Zeitzfrom the Institute for System Dynamics (ISYS) of theUniversity of Stuttgart who made the paper at handpossible and gave his support and advice.ReferencesH.-C. Pfisterer. Data-Based Modeling of ElectrolessNickel Plating. Master’s thesis, Helsinki Universityof Technology, Control Engineering Laboratory,2006.H. Hyötyniemi. Multivariate Regression. Technicalreport, Helsinki University of Technology, ControlEngineering Laboratory, 2001.H. Hyötyniemi. Neocybernetics in Biological Systems.Technical report, Helsinki Universityof Technology, Control Engineering Laboratory,2006.J. E. Jackson. A User’s Guide to Principal Components.Wiley-Interscience, 2003.J. Harju. Characterization of Electroless Nickel PlatingProcess. Master’s thesis, Helsinki Universityof Technology, Control Engineering Laboratory,2002.K. Kantola. Modelling of Electroless Nickel Platingfor Control Purposes. Master’s thesis, HelsinkiUniversity of Technology, Control EngineeringLaboratory, 2004.R. Tenno, K. Kantola, H. Koivo, A. Tenno.Model Identification for Electroless Nickel PlatingThrough Holes Board Process. Technical report,Helsinki University of Technology, Control EngineeringLaboratory, 1996.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 102

Lähdekoodin symbolinen analysointi tekoälyn näkökulmastaErkki LaitilaDepartment of Mathematical Information Technology, University of JyväskyläOy P.O. Box 35, FIN-40014 Jyväskylä, FinlandSwMaster OySääksmäentie 14, 40520 Jyväskyläerkki.laitila@swmaster.fiTiivistelmäTämä artikkeli käsittelee lähdekoodin automaattisen käsittelyn menetelmiä ja tuo esille uutena konstruktiona symbolisenanalyysin, joka on tunnettuja menetelmiä korkeatasoisempi, koska siinä tulee esille semioottinen, tulkitseva näkökulma.Esitämme sen tärkeimmät toiminnot ja piirteet tekoälyn näkökulmasta. Artikkelissa kuvataan analysointiprosessi ja senhyödyntämiseen soveltuva symbolinen tiedonhakumenetelmä, jonka avulla käyttäjä voi saada symboliseen malliinperustuen tarvitsemaansa pragmaattista informaatiota tarkoituksena helpottaa nykyisen Java-kielisen ohjelmaversionymmärtämistä haluttaessa kehittää uusi versio. Ratkaisu on ohjelmoitu ja koeajettu Visual Prolog-kehittimellä, jokahybridikielenä ja symbolista ohjelmointia tukien tarjoaa erinomaiset menetelmät vastaavan työkalun toteutukseen.1 JohdantoTuskin mikään ohjelmistotoimitus valmistuu joensimmäisellä kerralla lopulliseen muotoonsa.Pikemminkin voidaan sanoa, että ohjelmistojenkehittäminen on lähes poikkeuksetta jatkuvaa tieto- jainformaatiovirtaa, siis prosessi, joka sisältää lukuisiavaiheita ja toimituksia. Kaikissa tilanteissauudelleenkäytettävän koodin määrä tulisi, luotettavuus–ja kustannussyistä, olla mahdollisimman suuri.Lyhimmillään toimitussykli on päivittäistä koostamistasovellettaessa päivän luokkaa. Tietoliikennealallatoimitusten väli saattaa olla kireimmillään vain muutamiaviikkoja, missä ajassa kaikki mahdollinen hyöty entisestäversioista olisi pystyttävä siirtämään uuteen versioon.Olemassa olevan koodin hyödyntäminen on sitenmerkittävä haaste. Sitä kuvataan seuraavassametodologian näkökulmasta.1.1 TakaisinmallintaminenTakaisinmallintaminen (software reverse engineering)tarkoittaa olemassaolevan koodin hyödyntämistä.Erityisesti uudelleenmallintaminen (reengineering) tähtääsaadun informaation käyttöön uudessa ohjelmaversiossa.Selvästi todettu ongelma ohjelmistotyössä on, ettäohjelmistoprosessi on luonteeltaan vain yksisuuntainen,sillä sen tuotoksesta, kuten työn laadusta, määrästä,monimutkaisuudesta ja lopullisen koodin käyttäytymisestäon ollut vaikeaa saada palautetta. Koskapalaute on puuttunut, ei prosessia ole voitu optimoidasamalla tapaa kuin miten teollisen tuotannon prosessit onaikanaan saatu hallintaan. Juuri ohjelmistoprosessienoptimoitavuus on todettu korkeimmaksi alan kehityksentasoksi, mutta vain aniharvat yritykset ovat päässeetkorkeimmalle CMMI-tasolle (CMMI). Ilmeistä kuitenkinon, että jos ohjelmistoprosessista saataisiin yhtä hyvä jaajanmukainen palaute kuin parhaimmissatuotantoprosesseissa asianmukaisella mittaustekniikallasaadaan, myös ohjelmisto-organisaatiot saisivatmerkittävän parannuksen toiminnan laatuunsa, millä olisipositiivisia vaikutuksia luotettavuuteen,läpimenoaikoihin, töiden delegoitavuuteen jaennenkaikkea mahdollisuuteen saada uuttaydintietämystä jatkokehitykseen.Tässä artikkelissa kuvattu symbolinen menetelmätähtää juuri takaisinmallintamispalautteen parantamiseen.1.2 Semioottinen näkökulmaohjelmistotyöhönSemiotiikka tarkoittaa merkitysten ja kommunikaationtutkimusta. Se on tämä artikkelin kannalta kiinnostavaaihe, koska ohjelmoijahan on aikanaan jättänyt koodiinjälkensä, joiden vaikutuksia ja tarkoituksia ylläpitäjähaluaa tutkia jatkokehityksen kannalta.Peircen tulkintojen mukaan (1998) semiotiikkajakaantuu kolmeen pääkäsitteeseen, joita ovat ovatmerkki (sign), kohde eli objekti ja tulkinta (interpretant).Merkki viittaa vastaavaan objektiin ja tulkinta yhdistääniitä toisiinsa. Esimerkiksi nähdessään jäniksen jäljet,ihminen tekee tulkinnan, joka yhdistää jäljet oletettuujänikseen, jolloin jänis on objekti ja jäljet ovat vastaava103

merkki. Toisin kuin merkit, tulkinnat ovat aina sisäisiäesityksiä. Merkit voidaan jakaa osoittimiin (indice,index), ikoneihin ja symboleihin riippuen siitäminkälaisesta objektista on kysymys. Esimerkiksi kirjaon tietämyksen osoitus (tunnusluku, indeksi), samoinsavu on osoitus tulesta.Vieläpä vaikka kausaalista yhteyttä tai mitäänattribuutteja olioon ei olisikaan, silti voi löytyä jokinindikaatio (merkki) luomaan yhteyksiä objektiinsa.Symbolin käyttö soveltuu sellaiseen tarkoitukseen, jajuuri tällaista epäsuoraa kytkentää käyttää hyväkseentässä artikkelissa kuvattu symbolinen menetelmä. Koskakaikki ohjelman sisäiset artifaktat ovat näkymättömiä,tulee analysointiohjelmiston muodostaa riittävän tarkkaja yleinen virtuaalimalli viittauksineen ja symboleineenkäyttäjän näkösälle kuvaruudulle. Näytön koko onkuitenkin rajattu ja työkalun on siten mahdotonta tuottaakerralla riittävää informaatiota yhtenä kokonaisuutena.Siksi analysoinnin tulee olla interaktiivista navigointia,joka mahdollistaa uusia tiedonhakuja eri näkökohdista.1.2.1 Tietojärjestelmien semiotiikkaMorris (1971) jakaa semiotiikan kolmeen dimensioon,joita ovat syntaksi, semantiikka ja pragmaattinentarkastelu, jolloin pragmaattinen osuus viittaa Peircensemioottiseen tulkintaan. Seuraavassa kuvataan, kuinkaFrisco (1998) on määritellyt nämä kolme dimensiotatietojärjestelmien kehittämisen suhteen.Frisco luokittelee informaatiojärjestelmänsemioottiset tasot seuraavasti. Järjestelmän rakentaminenaloitetaan fyysisen maailman tasolta, joka sisältäävaltaisan määrän signaaleja, merkityksiä jaluonnonlakeja. Järjestelmän suunnittelu tehdäänempiirisen kokemuksen avulla luoden kokonaisuudestamalleja ja uusia käsityksiä. Tulos ohjelmoidaan, jolloinsaadaan syntaksia, käytännössä kiinteä ohjelmistotiedostoineen, kääntäjineen ja toimintaympäristöineen.Kun syntaksia ja koodia tulkitaan semantiikan kautta,saadaan tietoa koodin rakenteiden merkityksistä,väittämistä ja viittauksista ulkomaailmaan. Silloin kunjärjestelmää tutkitaan nimenomaan eri roolien tarpeistalähtien, tavoitetellaan pragmaattista tietoa. Sillä onFriscon mukaan selvä yhteys järjestelmän tuottamaanhyötyyn. Kokonaisvaikutuksen tulee näkyä tietojärjestelmääympäröivässä sosiaalisessa maailmassa,jonka tarpeisiin tietojärjestelmän tulisi tuottaa lisäarvoa.Friscon määrittelykonseptia on kritisoitu esimerkiksisiitä, että se pyrkii formalisoimaan lähes epäformaalejaasioita, mutta takaisinmallintamisen kannalta tätä edelläkuvattua ketjua voidaan pitää kelvollisena lähtökohtana,koska takaisinmallintamisessa keskitytään nimenomaanformaalin koodin hyödyntämiseen.1.3 Koodin analysoinnin teknologiatLähdekoodin analysoiminen liittyy ohjelmistojentakaisinmallintamisen (software reverse engineering)alueeseen, jonka tarkoituksena on juuri helpottaa uusienohjelmaversioiden tuottamista. Koodin analysoinnissaperiaatevaihtoehtoina on tutkia koodia käsin perinteisilläeditoreilla ja kehitystyökaluilla tai käyttää parhaitenkoodimassan käsittelyyn soveltuvia erikoismenetelmiä,joita on varsin vähän.Kielioppien taustateoriat on tunnettu 50-luvulta asti jakoodin jäsentämisen metodologiat kehitettiin 70-luvullaUnixin mukana. Ylläpidon kannalta haitallista on, ettäanalysointimenetelmät ovat kehittyneetkääntäjäteknologian ja koodin testauksen menetelmiensivutuotteina siten, että suurtakaan huomioita ei olekiinnitetty juuri analysointituloksen hyödynnettävyyteenkoko prosessin kannalta, informaatiojärjestelmänkokonaishyödystä puhumattakaan.1.3.1 Abstract Syntax Tree (AST)Vallitseva käytäntö koodin käsittelystä analysointiohjelmistoissaperustuu Abstract Syntax Treeteknologiaan(AST). Siinä työkaluun tallentuu kieliopinmukaisia abstrakteja rakenteita, jotka muodostavatkeskenään tiiviin hierarkian. Vaikka AST on hyödyllinenkääntäjiä suunniteltaessa, se ei kuitenkaan annamahdollisuuksia syvälliseen tulkintaan, koska kielensemantiikka ei kokonaisuudessaan sisälly kielioppiin.Esimerkiksi muuttujaviittausten ja metodiviittaustenlogiikka ja oliosidosten toiminta sekä rakenteiden välinensemantiikka puuttuvat AST-määrittelystä.1.3.2 UML ja round-trip-engineeringKäytännön tasolla oliopohjaisen koodin analysoinnissaon viime aikoina selvästi yleistynyt round tripengineering - niminen menetelmä, joka auttaaluokkarakenteiden poimimista koodista. Valitettavastinäin syntyvien UML-notaatioiden tarkkuus ei riitäpitkälle. Lisäksi eri työkalut saattavat tuottaa erilaisiakaavioita samalle lähdekoodille.Toisaalta UML:n eri esitystavat eivät skaalauduhyvin riittävän isojen näyttöjen luomiseen, minkä takiakäyttäjä joutuu monesti tutkimaan lukuisia erilaisiakaavioita voidakseen tehdä johtopäätöksiä nykyisenkoodin toiminnasta. Toistuva erillisten kaavioidenlukeminen on työlästä ja se kuormittaa käyttäjänlyhytkestoista muistia merkittävästi, jonka takiavarsinainen ongelma saattaa jäädä ratkaisematta.Olio-ohjelmoinnin kehittäjäjärjestö OMG keskittyypääasiallisesti uusien sovellusten kehittämiseen ja sitäkautta uuden koodin luomiseen. Ehkä sen takia heidänNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 104

mallinsa eivät tarjoa tarpeeksi tarkkoja esityksiä, jotkamahdollistaisivat koodin tarkimpien lauseidentallentamisen tai suorittamisen, mikä olisi tarkananalyysin perusedellytys. Siten käsitteellä executableUML ei tarkoiteta vastaavan koodin suorittamista vaanlähinnä vastaavan mallin tietojen evaluointianimenomaisen mallin kannalta, ei koodin kannalta.1.3.3 Koodin ymmärtämisen peruskäsitteetKuten edellä todettiin, takaisinmallintamisen luomapalaute on arvokasta kehittäjäorganisaatiolleen, ja ettäsyntaksi tarjoaa erinomaisen pohjan uudelleautomaatiolle, koska käsiteltävä koodin informaatio onluonteeltaan formaalia. Esille tulee kuitenkin kysymys,mikä osa koodista on merkityksellistä ja arvokasta? Vasonko koodi vain tasapaksua massaa, jonka lukeminensiltikin onnistuu parhaiten editorilla?Ohjelman ymmärtämisen teorioita on kehitetty 80-luvulta lähtien, jolloin Pennigton (1987) kehittitarkastelun alhaaltapäin alkavan tarkastelunviisikkomallinsa (function, control flow, data flow, state& operation) ja Brooks (1983) kehitti yleiset teoriattarkastelun hypoteeseille. 90-luvulla von Mayrhauser(1992) loi integroidun metamallin, johon liitettiin osanaylhäältäpäin alkava osittava tarkastelutapa sekätilannemalli, joka on kumulatiivista ohjelmasta saatuainformaatiota toimialansa tietoon yhdistettynä.Wiedenbeck (2002) on edelleen laajentanut aiempiateorioita 2000-luvulla kattamaan olio-ohjelmienymmärtämisen tarpeet. On esitetty myös abstraktimpiakäsitteitä kuten suunnannäyttäjä (beacon) ja lohko(chunk) kuvaamaan koodin ymmärtämistä esimerkiksilajittelualgoritmien osalta, mutta niiden merkitysoliomaisen koodin tarkastelussa ei ole enää kovinmerkittävä, koska oliot ja metodit rajaavat tehokkaastitoimintoja sisäänsä muodostaen uuden helpottavanabstraktiotason.Sen sijaan uutena asiana olio-ohjelmointi on tuonutsuunnittelumallit ja kerrosarkkitehtuurit uusiksitarkastelutavoiksi. Niiden jäljittäminen koodista onpaljon tutkittu alue ja koodin rakenteen uudelleensuunnittelu (refactoring) on siihen soveltuvakonkreettinen käyttökohde.1.4 Tekoälyn ja koodin analysoinnin suhdeTekoälyyn liittyvät keskeisesti tietämyksen hankinta,palautus ja käsittely, ihmisen ja koneen välinenvuorovaikutus, päättelymekanismit ja älykkäätkäyttöliittymät. Lähdekoodin tutkimuksen alueeltavoidaan siihen liittyen määritellä, että ohjelman ja sentoiminnan ymmärtäminen on oppimisprosessi, jossalukija pyrkii kehittämään tietämystään tietojärjestelmännykyarvon kasvattamiseksi. Ohjelma sisältää paljonsumeaa tietoa, jota ohjelmoijat ovat tallentaneet koodiinomalta kokemustaustaltaan optimoiden. Sen sisältö onusein osittain virheellistä. Tietoa on jopa miljoonia rivejä,joten sitä on mahdotonta esittää näytössä kerrallaan,tarvitaan siis erilaisia haku- ja suodatusmekanismeja.Parhaille formaalien kielten analysoinninsovelluksille on ominaista, että koodin tieto saadaanpoimittua tarkkaan semantiikan mukaan. Penningtoninviisikkomalli muodostaa tämän tiedon perustan. Semäärittää mikä on kiinnostavaa dataa. Käyttäjä onliikkeellä yleensä joko perehtymistarkoituksessa taipaikantamistarkoituksessa. Molemmissa tapauksissa hänhaluaa evaluoida kriittisimpiä koodiosuuksiaapukysymysten ja hypoteesien kautta. Yhteinen nimittäjäkaikille näille tarpeille on koodin ymmärtämisen tehostaminentyökalun avulla (program comprehension, PC).1.5 Tutkimuksen rajausErilaisia ohjelmointikieliä tunnetaan suuri joukko.Painopiste teollisuudessa on siirtynyt oliopohjaisiinkieliin, mutta silti alan tutkimus huomattavan suureltaosin käsittelee proseduraalisia kieliä kuten C, Pascal taiCobol. Oliopohjaisista pääkielistä C++ on laajin,monimutkaisin ja ehkä jopa tärkein, mutta senanalysointiin on varsin rajatusti menetelmiä. Sen sijaanJava-kieltä ja sen käsittelyä on tutkittu paljon, koska seon kielenä selväpiirteinen.Tässä artikkelissa käsittelemme etupäässä Javaa,koska sen selväpiirteisen kieliopin ansiosta onmahdollista päästä nopeasti hyvin tarkkaankäsittelytulokseen. Lisäksi edistysaskeleidenosoittaminen paljon tutkitulle Javalle voi olla selväosoitus uuden tutkimuslähestymisen paremmuudestaperinteisempiin tutkimuslähestymisiin nähden.Symbolinen analyysi on SwMaster Oyn toimestakäynnistynyt innovaatio, jota Erkki Laitila tutkiiväitöstutkimuksessaan Jyväskylän Yliopistossa. Sentekninen osuus keskittyy koodin symbolien ja niihinliittyvän logiikan kuvaamiseen mahdollisimman hyvinVisual Prolog- kehittimellä, joka on hybridikieli sisältäenerinomaisen oliojärjestelmän mallintamistarkoituksiin jalogiikkaohjelmoinnin perustan koodin logiikan,assosiaatioiden ja riippuuvuksien käsittelyyn.Symbolinen koodin malli on uusi atomistinenkonstruktio, joka mahdollistaa koodin tutkimisen kaikistalähtökohdista, jotka liittyvät Java-koodin semantiikkaanja simuloidun koodin käyttäytymiseen ja ajovaikutuksiin.Kuva 1 (seuraava sivu) esittää molekylääristä rakennettasidoksineen. Se muistuttaa ohjelmakoodia, koskamolemmissa on sidoksia. Tärkein asia ohjelmistojahahmotettaessa on ymmärtää kuinka eri lähestymistavatkuten käyttötapaukset, ohjausvuo ja ohjelmasuunnitelmatliittyvät toisiinsa. Olio-ohjelmointi aiheuttaa omiaperiaatteellisia ongelmiaan, koska vastaavat kutsutsijoittuvat näkymättömiin eri metodien sisälle koodiin.Syntyviä olioita ei voida myöskään tarkkailla millään105

analysoinnin perusmenetelmillä. Siten näin havainnollistakuvaa ei tänä päivänä ole saatavissa, muttamahdollisuuksia on, mikäli tässä tutkimuksessa kuvattuteknologia pääsee aikanaan käytäntöön asti.Kuva 1. Symbolinen, atomistinen malli.Voimakkaasti yksinkertaistaen voidaan sanoa, ettäkoodin ymmärtäminen on siinä olevien symbolien jarakenteiden välisten suhteiden ymmärtämistä. Nämäsuhteet muodostavat keskenään symbolivuon, jokatuottaa hyödyllisintä informaatiota silloin kun se onjäsentyneenä ohjelman suoritusjärjestykseen. Sitenkuvassa 1 ylinnä voisi olla main-metodi, joka haarautuisialaspäin suorituslogiikan mukaisesti.2 AnalysointimenetelmätPerinteisiä koodin analysointimenetelmiä ovat staattinenja dynaaminen analyysi. Staattinen analyysi ei sovelluhyvin oliomaisille kielille kuten Javalle, koska vain osatoiminnoista on tulkittavissa suoraan koodista. Toisaaltadynaaminen analyysi vaatii lähes aina monimutkaisiaajojärjestelyjä.Staattisella analyysillä voidaan tuottaariippuvuuskaavioita ja kutsupuita, jotka toimivatkinhyvin proseduraalisille kielille jopa niin, että niidenavulla voidaan saada täsmällinen tieto mitä koodinosiamikäkin proseduuri käyttää ja missä järjestyksessäohjelmavuo etenee, mikä on arvokasta tietoa mm.vianpaikannustilanteissa. Vaikeita haasteita staattiselleanalyysille ovat osoittimet (pointer) japysähtymättömyyden analysointi.Dynaaminen analyysi perustuu pitkälle ohjelmanajamiseen debuggerilla ja sitä kautta koodin toimintojenjäljittämiseen valittujen pysäytysehtojen, testitapausten jajäljityskomentojen mukaisesti. Näin saadaan talteen jälki,trace, joka voi sisältää erittäin paljon aivan turhaakintietoa. Jopa 98 % jäljestä voi olla epäkuranttia.Edellä mainittujen menetelmien rajoitteet tuntien onaiheellista kysyä, voidaanko muodostaa sellainen uusianalysointitapa, jonka painopisteenä olisi koodiinformaationjalostaminen ja muokkaaminen sellaiseenmuotoon, että syntyvästä ratkaisusta voitaisiinselektiivisesti kyselyin poimia haluttu informaatio?Voisiko syntynyt menetelmä esimerkiksi toimia kutentietokanta, jolloin käyttäjä saisi vapaasti valita minkäasioiden välisiä suhteita hän haluaisi tarkastellasuunnitellessaan muutoksiaan?Mietittäessä vastauksia edellisiin kysymyksiinpäädytään ideaalisen analyysin määritelmään: Millainenolisi ideaalinen analysointimenetelmä, jos sen voisivapaasti määritellä puhtaalta pöydältä?Vertauskohtia voimme tietysti löytää muidentietojärjestelmien ja teknologioiden parhaistatoteutuksista, joita ovat mm. relaatiomalli,informaatioteoria, diagnostiikan perustietous,asiantuntijajärjestelmien perustyypit. Näiden kauttavoimme päätyä synteesinomaisesti seuraaviinvaatimuksiin: Analyysi voidaan tehdä milloin vain mistä tahansakoodin osasta. Analyyysi poimii käsittelyyn kaikki tarvittavat asiatkuten Penningtonin ja Wiedenbeckin määrittelemättiedot. Tuotoksella tulee olla ideaalinen tiedonsiirtosuhdeilman turhaa tietoa. Vastaava analysointiprosessi tukee tiedonhankintaa,mahdollistaen uuden tietämyksen hankinnan. Vastaus staattiseen kysymykseen saadaan nopeasti.Silti ratkaisu vaativaan ongelmaan saisi kestääkauemminkin, jos vertailukohtana käytetään käsintehtävää perushakua, mikäli haku vaatii runsaastisyvällistä laskentaa.. Menetelmän tulee tukea käyttäjän pohdintaasyntaksin ja semantiikan suhteista, mieluumminmyös pragmaattiselta kannalta katsottuna. Menetelmän tulisi voida tunnistaa arkkitehtuurinkaikki keskeiset piirteet.Ohjelmakoodi sisältää erilaisia tasoja, kerroksia jamenetelmiä, joten käytännön tutkimuksessa on tehtävärajauksia. Tässä tutkimuksessa keskitytääna) ”kovoteoriaan”, joka tuottaa mahdollisimmanytimekkään, mutta ilmaisuvoimaisen mallinb) ”pehmoteoriaan”, joka määrittelee ihmisentyypillisen käyttäytymismallin ylläpitoongelmiaratkottaessa.Siten abstraktimmat käsitteet, jotka ovat etäällä kieliopinmäärittelyistä, jätetään huomioimatta. Esimerkiksikerrosrakenteen poimimista koodista ei tutkita eikämyöskään suunnittelumallien tunnistamista.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 106

3 Koodin käsitehierarkiaKoodin tutkiminen muistuttaa monin tavoin tutkimusta,koska sen lukeminen vailla tarkoitusta ei ole mielekästä.Aina tarvitaan jokin tavoite, jolloin lukija joutuumiettimään tavoitteen ja koodin välistä suhdetta (vonMayrhauser 1992). Se onnistuu vain olettamuksia elihypoteesejä tehden. Näin koodin tarkasteluun voidaansoveltaa tieteen peruskäsitteistöjä alla olevaan tapaan.Ontologia kuvaa koodin olemuksen ja sen keskeisenrakenteen. Epistemologia tunnistaa koodista olennaistatietoa pyrkien määrittelemään sen tietämystä.Metodologia kuvaa tavan, jolla selektiivinen tiedonkeruuja tietämyksen johtaminen siitä voi tapahtua. Lisäksitarvitaan vielä menetelmä, joka soveltuu arkielämäntarpeisiin ja erityisesti työkalu, joka helpottaa käyttäjäntoimia.3.1 PeruskäsitteetSeuraavassa esitetään lyhyesti tutkimuksen keskeisettermit, jotka selostetaan myöhemmin tarkemmin.3.1.1 Sanasto ja lyhenteetG kieliopin symboliX koodin syntaksin rakenne kieliopissa GN rakenteen X tyyppi (non-terminal)Y X:n vaikutus koodissa, semantiikkaZ X:n merkitys lukijalle, pragmatiikkaB X:n käyttäytymismalli (input/output-tarkastelu)S koodista saatu symboliO symbolista S luotu olioL logiikka, joka olioon O sisältyy (interpretant)M malli, joka koostuu olioistaA analyysi, joka hyödyntää mallia MQ kysely, jolla analyysi A käynnistyyH hypoteesi, jonka perusteella kysely Q voidaan tehdäP päättelyprosessi, josta käsin luodaan hypoteeseja HT ylläpitotehtävä päättelyprosessin P aktivoimiseenI pragmaattinen informaatio ylläpitoa tukemaanK tietämys (knowledge), jota prosessissa P kehitetäänSymbolinen analyysimenetelmä toimii ideaalisesti, jos sepystyy yhdistämään edellä mainitut käsitteet ideaalisesti.3.1.2 MääritelmiäInformaatio on uutta, käyttäjäkohtaista merkittäväätietoa, jonka määrä on minimoitu tarpeeseensa.Tietämys on sellaista tietoa, jota voidaan käyttääratkaistaessa muita samantyyppisiä ongelmia.Tietämyksen hankintaan liittyy siten oppimista jasisäistämistä. Arvokkain tieto, jota metodologia voituottaa, on juuri tietämystä, taitoa ratkaista tuleviaongelmia yhä järjestyneemmin verrattunatapauskohtaiseen käsittelyyn, jossa ratkaiseminentapahtuu yhä uudelleen entiseen tapaan. Uudentietämyksen avulla nykyistä menetelmää voidaan kehittääedelleen, jolloin päästään positiiviseen kierteeseen jaoptimoimaan menetelmän parametreja sekäeliminoimaan virhemahdollisuuksia. Walenstein (1998)esittää, että näin tietämys muuttuu säännöiksi ja taidoiksi.3.1.3 OhjelmaesimerkkiSeuraavassa esitettyä lyhyttä esimerkkiä käytetäänjatkossa havainnollistamaan symbolisen analyysin ideaa.Siinä oleva luokka Server käynnistää muutamaa lausettakäyttäen (4-10) Server-olion.// Start the server up, listening on an// optionally specified port1 class Server extends Thread {2 {3 protected int port;4a public static void main(String[]4b args) {5 int port = 0;6 if (args.length == 1) {7a try { port =7b Integer.parseInt(args[0]); }8acatch (NumberFormatException8b e) { port = 0; }9 }10 new Server(port);11 }12 }Javan koodi on selkeä. Se sisältää vain varattuja sanoja jajoko viittauksia Javan kirjastoon tai käyttäjän antamiasymboleita viittauksineen. Varatut sanat rajaavatrakenteita (nonterminal), esimerkkeinä luokka Server jasen sisältämä metodi main.Javaa tuntematon lukija ei voi päätelläesimerkkikoodin (X) aiheuttamasta toiminnasta (B)paljoakaan, koska hän ei tunne kielen semantiikkaa (Y),riippuvuussääntöjä (N) eikä käsitteitä (G). Se sijaankokenut Java-ohjelmoija ymmärtää yllä olevan toiminnantäydellisesti (Z) luettuaan jokaisen rivin huolellisesti.3.2 Koodin ontologiaFormaalien kielten jäsentäminen ei ole ongelmallista,koska kielen määrittely on alkuvaiheissa suunniteltusäännönmukaiseksi. Seuraavassa kuvataan kieltentulkinnan mahdollisuuksia tietosisällöistä lähtien.107

3.3 Kieliopin määritelmäKieliopin määritelmät ovat peräisin jo puolen vuosisadantakaa (Chomsky 1956). Seuraavassa formaalin kielenkieliopin määritelmä:G = Symboli N edellä tarkoittaa jakokelpoista termiä(non-terminal), jolla on vähintään kaksitasoinenhierarkinen rakenne. on kielen kaikkien varattujensanojen luettelo sisältäen välimerkit ja avainsanat(terminal). R on joukko sääntöjä (production rule), jotkakertovat kaikki sallitut rakenteet, joita kukin N voi saada.S0 on kielen käännöksen aloitussymboli, jolla kuitenkaanei ole analysoinnissa juuri merkitystä.Tärkeimpiä rakenteita Javan osalta ovat luokanmäärittely ClassDeclaration, metodi MethodDeclaration,lause Statement, lauseke Expression ja muuttujaIdentifier.Ontologian ja kapseloinnin takia luokan määrittely onensiarvoisen tärkeä. Se on erittäin selkeä. Luokka sisältäämäärittelynsä mukaan luokan jäseniä (metodi jaattribuutti), ja sillä on perintämäärittely. Yhdessä nämäulkoiset määrittelyt muodostavat luokan allekirjoituksen,luokkasopimuksen.Javan sanastossa erityisesti jokainen viittauskäsitteeseen Identifier viittaa joko JDK-kirjastoon taikäyttäjän antamaan symboliin, joka voi olla joko paketin,luokan, metodin, attribuutin tai muuttujan nimi. Kaikkimuut kielen rakenteet tulee tunnistaa suhteessa Identifierviittauksiin.3.3.2 AST:n rajoitukset analysoinnissaYleinen tapa käsitellä kielen rakenteita on tallentaa neAST-muotoon. Siihen liittyy kuitenkin paljonkinongelmia analysointitarkoituksessa. AST-rakenteilla einimittäin ole yksiselitteistä viittaussemantiikkaa, jonkamukaan voitaisiin poimia itsenäisesti evaluoitavaksiyksilöllisiä koodin rakenteita esiin kuten if-lausemetodista main. Siten sisempiä AST-rakenteita ei olemahdollista käsitellä ja evaluoida itsenäisesti vaan vainosana laajempaa kokonaisuutta. Siksi tarvitaankehittyneempi menetelmä, jota kuvaamme seuraavassa.3.3.2 Kielioppien saatavuus ja ongelmatKielioppeja eri kielille ja myös Javalle on saatavissainternetistä. Ongelmana on kuitenkin kielioppitiedostojenlaatu, erilaiset kirjoitusvariaatiot ja niidentaltiointitarkkuus yleisesti, koska osa niistä on pelkästäänperiaatteellisia ja vain osa riittävän tarkkoja, että nemahdollistaisivat sellaisenaan parserin rakentamisen.Esimerkiksi Sunin julkaisema version 3 (Java1.5)kielioppi, joka vastaa uusimman kääntäjän tilannetta jajulkaistaan myös Java 3rd Edition-kirjan mukana, onongelmallinen. Siinä on useita virheitä ja lisäksimoniosaisten termien semantiikkaa ei ole kuvattutarkoituksenmukaisesti. Hankalinta on, että Suninilmoittama kielioppi ei esitä tarkkaan lausekerakenteita,koska siinä kaikilla laskentatoiminnoilla yhteenlasku,kertolasku, jakolasku ja loogiset vertailut olisivirheellisesti sama prioriteetti ja siten samalaskentajärjestys.Siksi jotta riittävään tarkkuuteen päästäisiin, tuleeSunin toimittama kielioppi unohtaa kokonaan jaorganisoida kielen rakenteet itse uudelleen ja muuntaa neristiriidattomaan, hierarkisempaan muotoon.Prolog-kielelle on kehitetty edistyneitä kieliopinkäsittelyn menetelmiä kuten definite clause grammar(DCG), jossa yksittäisen lausekkeen prioriteetti voidaanilmaista lukuna. Tämä menetelmä soveltuu tulkkaavalleja ISO-standardin mukaiselle Prologille. Tutkimuksenmukainen Visual Prolog-kehitin sisältää omanmenetelmän, jota kutsutaan semanttiseksikielioppirakenteeksi, minkä on alkujaan kehittänyt LeoJensen, Prolog Development Center. Alla if-lauseenkuvaus tällä menetelmällä:1 Statement =2 “if” Condition “then” Statement ->3 iff(Condition, Statement).Termin nimenä on Statement, jonka yksi mahdollinenmuoto olisi edellä mainittu if-lause, jonka syntaksisisältää hierakisesti alemmalla tasolla sijaitsevat termitCondition ja Statement. Varattuja sanoja siinä ovatlainausmerkeissä olevat sanat if ja then. Syntaksin oikeajärjestys ilmenee ”->” -symbolia edeltävästä osuudesta.Sen oikealla puolella oleva rakenne iff argumentteineenon muotoa, joka syntyy jäsennyksen tuloksena vastaavantyökalun muistiin. Iff-rakenne kuvaa siis semantiikkaa,mistä käytämme tässä artikkelissa nimitystä Y. Näinsaatiin ytimekäs kokonaisuus: rivi 1 kertoo määrittelynnimen (N), rivi kaksi kertoo syntaksin (X) ja rivi kolmesemantiikan (Y).Jos käytännön tilanteessa asianomainen if-lause eijohda halutulla tavalla sen sisältämän osuudenkäynnistymiseen, vika voi olla vain sen Conditionosassa.Tällöin tämä ehto olisi vianpaikantajallepragmaattista tietoa (Z).3.3 Koodin mallien epistemologiaEpistemologia pyrkii erottamaan todellisen ja riittäväntietämyksen paikkansa pitämättömästä tietämyksestä. Setutkii mm. tiedon luonnetta, tietojen välisiä yhteyksiä.OMG ei erikseen puhu epistemologiasta tutkiessaanmetodologioita ohjelmistokehityksen näkökulmasta,sehän ei edusta tiedettä vaan käytännön päämääriä.Kuitenkin koodin ja mallien synkronointi on OMG:nNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 108

päätavoitteita. OMG:n painopisteenä onmallintamiskokonaisuus nimeltään Model DrivenArchitecture, MDA (MDA 2006), jossa keskeisimpänätietosisältönä on MOF-määrittely (MOF 2006). MOF:ssakoodintarkan informaation käsittely osoittautuumahdottomaksi, koska Javan lause-tason informaatiopuuttuu siitä (tosin siinä on OCL-rajoitekieli).MOF tarjoaa koodista poimitulle informaatiolleseuraavat mallit ja tasot. Taso M0 eli systeemitaso kuvaakoodia sellaisenaan. Taso M1 eli luokkataso viittaakoodista poimittuun luokkamäärittelyjen rajapintaan.Taso M2 viittaa metamalliin, jolla määritellään luokkienyleiset rakenteet. Ylinnä on taso M3 metametamalli, jokamäärittelee yleisen transformaatiomallin lähinnämuunnostarkoituksiin.Koska MOF ei tue lausetarkkaa käsittelyä, sentarjoama ymmärtämistuki jää suppeaksi. Niinpäalimmalle tarkastelutasolle tarjotaan vaihtoehtona ASTrakenteidenavulla muodostettua mallia, mutta siinähänon muutamia perusongelmia, joita on kuvattu tämänartikkelin alkuosassa.Seuraavassa jaksossa koodista poimittavissa olevaatietoa käsitellään erillisten teorioiden valossatarkoituksena muodostaa looginen ketju sellaisen eheänmallin luomiseksi, joka sisältäisi sekä MOF:n kaltaisenmetariippuvuusinformaation että koodin tarkanlauserakenneinformaation, jotta tulosta voitaisiin käyttääyhtenäiseen tarkasteluun.3.3.1 T1 Koodin jäsentämisen teoriaKoodista saadaan jäsennyksen tuloksena rakenteet (N),jotka parserin jäljiltä ovat yhdessä hierarkisessarakenteessa, jonka juurena on aloitussymboli (S0),esimerkiksi TranslationUnit, joka jakaantuu paketin jaluokan määrittelyihin.Prologilla jäsennetty parsintapuu (hierarkinenkokonaisuus) on predikaattilogiikan (1. kertaluvunlogiikka) mukaisesti täydellisesti aksiomatisoitavissa.Parsinta on täydellinen ja todistettavissa, jos kukinrakenne säilyttää alkuperäisen muodon ja riittävän tarkaninformaation (koherenttius ja konsistenssius).3.3.2 T2 Abstraktiotason nosto, symbolinen kieliLaskentasääntöineen ja sisäkkäisine lauseineen Javastasaadut rakenteet ovat tarpeettoman monimutkaisiasellaisenaan analysoitaviksi. Kun rakenne on kerranjäsennetty oikein, sitä voidaan merkittävästi (noin 90 %)yksinkertaistaa pudottamalla sisäkkäisiä itsestään selviävälitasoja ilman, että uusi sisältö vääristyy tai muuttuuepäloogiseksi.Jatkoanalysoinnin kannalta tärkeää on ryhmitellärakenteet mahdollisimman hyvin. Analysointitarpeitaovat mm. seuraavien rakenteiden tunnistaminen, suluissajatkossa käytetty lyhenne: määrittelylauseet (def),konstruktorikomennot (creator), dataa muuttavatrakenteet (set), dataan viittaavat rakenteet (ref),metodiviittaukset (invoke, get), operaatiot (op),silmukkalauseet (loop), ehdolliset lauseet (path,condition), vakiot (value) ja muut lauseet. Näin saadaan11 lauseryhmää.Tällä ryhmittelyllä päästään seuraavaan tavoitteeseen.Eri analyysit voidaan määritellä abstraktien rakenteidenvälisinä yhteyksinä, esimerkiksi data flow – analyysi onsaman datan set- ja ref-rakenteiden välinenkeruutoiminto. Ohjausvuo on valitusta koodinosastakoottu get- ja creator-lauseiden kokonaisuus, joka käyläpi muiden lauseiden tarvittavat alarakenteet.Ryhmittelylogiikasta voidaan muodostaa uusisymbolinen kieli, jolle tutkimuksessa on annettu nimiSymbolic. Kuten jokaisella formaalilla kielellä, myösSymbolicilla voi olla oma syntaksi ja semantiikka. Näinollen sille voidaan luoda jäsennin ja koodigeneraattori,joista on hyötyä muunnettaessa abstrahoituja rakenteitalukijan paremmin ymmärtämään muotoon tai tarvittaessaluoda sujuva testausympäristö analysointikokoonpanollealkuarvojen syöttömahdollisuuksineen.Jos Java-kielen keskeinen lause on nimeltäänStatement, voisii sitä vastaava korkeamman tasonrakenne Symbolic-kielessä olla Clause. Näin saadaantranslaatiotarve Statement2Clause, jota kuvataankinseuraavassa.3.3.3 T3 TranslaatioFormaalien kielten translaatioon on kehitetty erillisiämonimutkaisia menetelmiä, jotka perustuvat useinminulkoiseen translaatiomekanismiin (translation engine).Mielenkiintoista on, että translaatio voidaan tehdä myös”suoraan” ilman erillistä sääntölogiikkaa rekursiivisensijoittamisen avulla. Prolog tukee tällaista vaihtoehtoatunnistaessaan automaattisesti kielen rakenteet. Tällöinaivan tavallinen sijoituslause käy translaatioksi.Seuraava rekursiivinen xslate-komento muuttaa Javakielenif-lauseen Symbolisen kielen path-lauseeksi:xlate(iff(Condition,Statement)) =path(xlate(Condition),xlate(Statement)).Olennaista ylläolevassa komennossa on se, ettävasemman puolen suluissa on muunnettava Java-termi jayhtäsuuruus-merkin jälkeen Symbolic-kieleen syntyvärakenne. Koska jokaisen alarakenteen yhteydessäkutsutaan alas laskeutuen uudelleen translaatiokomentoa,kaikki alarakenteet muuntuvat automaattisesti uuteenmuotoon. Translaation suorittamiseksi jokaiselleerilaiselle Javan termille tarvitaan yleensä vain yksiedellämainitun kaltainen sijoituslause. On vain muutamapoikkeus.Siirto Javasta symboliseen kieleen(Statement2Clause) voidaan validoida analyyttisesti109

translaatiolausekkeiden kautta sekä testitulosteilla.Lisäksi Visual Prolog sisältää tyypintarkistustoiminnon,joka tarvittaessa varmistaa, että kukin alalauseke onkattavasti määritelty. Se poistaa oikein käytettynäpuutteet muunnoksen kattavuudessa.3.3.4 T4 Semantiikan talteenottoOhjelmointikielten semantiikan ilmaisemiseen on useitaesitystapoja eli notaatioita. Merkittävimpiä ovattoiminnallinen semantiikka (operational semantics). Erässen muoto, luonnollinen semantiikka (natural semantics),on Prologin kannalta merkittävä, koska se on Prologyhteensopiva.Siten jokainen luonnollisen semantiikanlause voidaan periaatteessa ohjelmoida Prolog-lauseiksi.Myös Prolog-lauseita voidaan tietyin edellytyksinmuuntaa luonnollisen semantiikan esitystapaan, jolloinsitä voidaan käyttää kuvaamaan Prolog-koodinkäyttäytymistä alkuperäistä korkeammalla tasolla.Luonnollinen semantiikka tarjoaa seuraavia etujaperinteisempiin semantiikkoihin nähden:Kaikki sen oliot ovat äärellisiä termejä, jotenrakenteen käsittelyyn ei tarvita monimutkaistatyyppiteoriaa.Koska jokaiselle rakenteelle voidaan määritelläuseita päättelysääntöjä, sillä voidaan mallintaa myösepädeterminististä hakua, joka on Prologin parhaitaominaisuuksia.Epädeterminisitinen määrittely mahdollistaapolymorphististen rakenteiden mallintamisen.Luonnolisen kielen semantiikan yleinen lausemuoto onseuraava:Kuva 2. Luonnollisen semantiikan periaate.Kuvassa 2 alarivillä on todistettava lauseke, joka sisältäähypoteesin H, johon liittyy parametrikokonaisuus T jatavoitetulos R. Se ratkeaa, jos ylemmällä rivillä olevahypoteesi H1 argumentein T1 ratkeaa tuottaen tuloksenR1, joka edelleen mahdollistaa muiden hypoteesien H2… Hn toteutumisen. Lopullinen tulosjoukko R onkombinaatio ehdollisten hypoteesien tuottamastatulosjoukosta. Tämä lauseke toteutuu vain jos ehto condtoteutuu myös.Prolog sallii predikaatteineen tällaisen lauseenkirjoittamisen. Visual Prolog erityisesti sisältääoliojärjestelmän, joka myös soveltuu sekä ylläolevienhypoteesien ohjelmointiin että koodin muuttamiseenvastaavanlaisiksi hypoteesimäärittelyiksi. Alla pieniesimerkki, jossa hypoteesit on muutettu predikaateiksi:h(T, Cond) = R:-ifTrue(Cond),R1 = h1(T1), … Rn=hn(Tn),Result = [R1,R2,… Rn].Predikaatti ifTrue tarkistaa aluksi onko ehto Condvoimassa. Jos on, siirrytään h-predikaattien evaluointiin.Seuraavassa esimerkissä hypoteesit ovat oliota (tässätutkimuksessa symbolisen mallin elementtejä):evaluate(T, Cond) = R:-ifTrue(Cond),R1 = H1:evaluate(T1),…Rn = Hn:evaluate(Tn),R = [R1, … Rn].Esimerkin selostus: Visual Prologissa suoratdataviittaukset voivat kohdistua vain ko. olion sisälle. Seestää haitalliset sivuvaikutukset. Jokaisella oliolla tuleeolla metodi evaluate, joka laskee oman tuloksensa (R i )rekursiivisesti.Ohjelmointikielten semantiikassa keskeisiä asioitaovat seuraavat asiat: Dataviittausten huomiointi Metodikutsujen huomiointi parametreineen Polymorphististen metodien huomiointi Perintähierarkian huomiointiKirjallisuutta semantiikan rajoitteista ja erityisesti Javakoodinsemantiikasta on runsaasti. Javan spesifikaatiosisältää täsmälliset määrittelyt kielensä piirteistä edellämainitut asiat huomioiden.Symboliseen malliin nämä yllämainitut asiat saadaantoteutettua siten, että alkuperäinen Javan rakenne javiittaus siihen korvataan joko mallia luotaessa tai mallialäpikäytäessä alkuperäistä laajemmalla rakenteella(tyypillisesti oma haku- metodi lookUp), joka suorittaatäsmällisemmän ja yleisemmän identifioinnin rakenteelle.Esimerkiksi symboli port saattaa olla sekä metodinargumenttina, muuttujana ja myös luokan attribuuttina.Siten eri metodeissa sama muuttujanimi viittaa eri asiaan.Tätä kutsutaan alias-ilmiöksi. Sen vuoksi metodikutsuissatuloargumentit ja parametrit on saatavavastaamaan toisiaan.Semantiikkateorian T4 tarkoituksena on varmistaa,että mallinläpikäyntialgoritmi voi toimia eri tilanteissasamaan tapaan kuin alkuperäinen Java-koodi tunnistenvastaavat rakenteet. Se tulkitsee siten koodinkäyttäytymistä ohjelmoijan eduksi. Ohjelmoijan kiusanaon ns. jojo-ilmiö, jossa kutsuttua oliorakennettajoudutaan etsimään laajalti olion perintähierarkiasta.Semantiikkateorian tuloksena saadaan kerättyä UML:nsekvenssikaavion mukaiset tiedot, mutta tarkempanasisältäen kaikki lausekkeet ja eri suoritusvariaatiot.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 110

3.3.5 T5 Symbolinen malli ja mallin kutojaErilaisia malleja ja mallienluontimekanismeja (modelweaver) on kehitetty runsaasti mm. OMG:n toimesta(Bezevin 2005). OMG on verrannut erilaistenmalliratkaisujen ominaisuuksia seuraavasti. Tärkeitävertailtavia piirteitä ovat modulaarisuus, muunnettavuus,jäljitettävyys, formalisoitavuus, koodin suoritettavuus,aspektien poiminnan mahdollisuus ja ratkaisunerikoistamisen mahdollisuudet. MOF-mallien keskeinenheikkous on assosiaatioiden käsittelyssä. Koskaoliomallit voivat itsessään käyttää vain olioita, joidenheikkoutena on suljettu sisäinen rakenne, MOF-malleissajoudutaan jokainen assosiaatio pilkkomaan useiksiolioiksi (kuten association, associationEnd, link jalinkEnd). Pilkkomisen takia mallien läpikäynti vaatiipaljon ohjelmointia ja monimutkaisia algoritmeja.Takaisinmallintamisen kannalta tärkeimpiä mallienpiirteitä ovat modulaarisuus, muunnettavuus,jäljitettävyys, formalisoitavuus ja suoritettavuus. OMGtoteaa, että MDA:lla ei voida tuottaa suoritettavaa mallia,vaikkakin ”executable uml” on paljon käytetty käsite.Sen sijaan äskeisten teorioiden T1-T4 mukaan kehitettymalli on simuloitavissa eli mallinnettavissa niin pitkälle,että myös koodin sivuvaikutukset voidaan analysoida.Samalla simuloinnista saadaan tuloksena jälki (trace),joka vastaa dynaamisen analyysin tulosjoukkoa.Symbolinen malli on formalisoitavissa tarkkaan jamuutettavissa myös MDA-malleiksi ns. xmi-esitystavanavulla (XMI 2005).Symbolisen mallin keskeisin ja lähes ainoa rakenneon symbolinen elementti (SE), jolla on optimoituperintähierarkia. Se periytyy Symbolic-nimisestäluokasta, joka sisältää teorian T2 mukaiset määritelmät,parserin ja koodigeneraattorin. Eri tyyppiset elementit,jotka kuvattiin teorian T2 yhteydessä, toteutetaanperuselementin SE erikoistuksina. Kunkin elementinmuistiin tallennetaan siihen kuuluva koodi, joka onluontivaiheessa muuntamalla teorian T3 mukaiset T2:nrakenteet omiksi elementeikseen ja prosessoimalla saatukoodi vielä teorian T4 mukaisesti. Näin saadaan aikaanatomistinen malli.3.3.6 T6 Selektiivinen kyselyTunnetuin koodin tarkastelun menetelmä on nimeltäänviipalointi (slicing). Se tuottaa tietystä koodin osastasiihen suoranaisesti liittyvät muut ohjelmalauseetsuoritusjärjestyksessä joko taakse- tai eteenpäin.Viipalointiin liittyy kuitenkin paljon rajoituksia ja sesoveltuu vain koodin matalan tason tarkasteluun. Siksiolemme valinneet paremman perusmenetelmän, joka onnimeltään leikkely eli chopping.Chopping on tärkeä analysointikeino, joka tarkoittaahaluttujen kohteiden saavutettavuuden tutkimista. Se käyerinomaisesti syy-seuraussuhteiden käsittelyyn, joka onolennaista vianpaikannustehtävissä. Chopping pyrkiianalysoimaan kuvan 3 mukaan perättäistenfunktiokutsujen joukkoja, jotka alkavat tietystä Startkohdastaja etenevät aikajärjestyksessä Target-kohtaansaakka:Start = fk • fk-1 • ... •f • Target.Chopping-yhtälö voidaan ilmaista monella eri tavallasymbolista notaatiota käyttäen. Ilmaisu lopusta alkuun onmuotoa:Target = fk • fk-1 • ... •f (Start).Sisäkkäisessä muodossa se voidaan kirjoittaa:Target = f (f (f ( ... f( Start ) ) ) )Tai peräkkäisessä muodossa listana:Target = [f(XK), f(XK-1).. f(“Start”)].Chopping-analyysin ohjelmointi Prologilla onselväpiirteistä. Seuraava koodi sisältää rekursiivisensäännön, joka kerää informaatiota seuraavilta perättäisiltäsolmuilta, jotka on määritelty f-rakenteina sisältäenkunkin funktion nimen ja vastaavat argumentit. Tuloksetsaadaan kumulatiivisesti kerättyä predikaatin toiseenargumenttiin Sequence-muuttujan välityksellä:chopping (Target, [f(This)|Sequence]):-f(NextId,Arguments),NextId:chopping(Target,Sequence).Cstartf 1 f 2call q f k 1 f kstart qf 4f 3ret( )f 5f k 2exit qf k 3targetKuva 3. Chopping ohjausvuon tulkinnassa.3.3.7 T7 Tyhjentävä haku ongelman ratkaisijanaPerinteinen koodin analysointi tarkastelee tyypillisestiohjelman lauseiden välisiä suhteita sellaisenaan [Slicing,Reps] ilman korkeamman abstraktion malleja. Tarkkakäsittely johtaa usein laskennallisesti liian vaativiinalgoritmeihin, jos aluetta ei voida rajata tarpeeksi.Äsken kuvaamamme selektiivinen kysely, chopping, sensijaan tuottaa suppean tietoaineiston, joukon mallinelementtejä, jota voidaan käsitellä monin eri tavointyhjentävästi mm. tarkastelemalla eri suorituspolkuvaihtoehtoja,datan kulkua ja sivuvaikutuksia jamahdollisia aktivointeja eri tilanteissa.Yhden metodin osalta kaikki Penningtonintietotarpeet voidaan aina poikkeuksetta poimiatäydellisesti siten, että tuntemattomat parametritkäsitellään viittauksina. Mikä tahansa suorituspolkuvoidaan myös evaluoida yhtenä kokonaisuutena, jolloinvoidaan tutkia pienimpiä ilmiöitä koodista.111

Ohjelmasilmukoiden käsittely on silti ongelmallista, jossuorituskertoja tulee lukemattomia. Siinä tapauksessatoistokertoja on voitava rajoittaa. Samoin ulkoistenrajapintojen tulkitseminen ehtolausekkeissa vaatiialkujärjestelyjä ennen testausta.3.3.8 T8 Javan täydellinen relaatiomalliJavaa ja sen koodista kehitettyä symbolista malliatarkastellaan järjestelmällisimmin sitä vartensuunnitellun, yhtenäisen relaatiorajapinnan kautta. Näinvoidaan tunnistaa tärkeimmät käsitteet (entity) ja niidenväliset suhteet (relation), mikä tuottaa kelvollista tietoavianpaikannukseen. Relaatiomalli sisältää seuraavatperusmääritykset, joiden välisiä relaatioita on mahdollistatutkia: Luokan nimi, class Luokan jäsenet Luokan super-luokka Metodin sisä ltämät elementit Metodin sisältämät staattiset kutsut Metodin suorittamat dynaamiset kutsut Kuhunkin elementtiin liittyvät navigointitiedot. Kuhunkin elementtiin liittyvät sivuvaikutukset JäljityshistoriaKun koodin yksittäiset rakenteet saadaan näinyksittäisillä kyselyillä hallintaan, tarkastelua voidaanlaajentaa käyttämällä kyselyjen välillä loogisiaoperaatioita ja alakyselyjä.3.3.9 T9 SelitysmalliTietämysteknisesti tarkasteltuna koodin informaatio onmielenkiintoista. Halutessamme kysyä syyseuraussuhteita,koodista saadaan läpikäyntialgoritmillaloogiset ehdot ja rakenteet, jotka vaikuttavat kyseiseentarkasteluväliin. Saatujen symbolisten rakenteiden joukkovoidaan muuntaa luonnolliselle kielelle, esimerkiksienglanniksi, jolloin saadaan selväkielinen perusteluketjutarvittavine parametreineen.Yhteistä kaikelle käsittelylle on se, että saaduntulosinformaation muoto on aina sama kyselystäriippumatta. Se helpottaa ohjelmointia ja toteutusta jatulosten integrointia. Tulosinformaatiolle voidaan valitaerilaisia tulosteita ja tarkastelutapoja kuten what, why taihow, jolloin näytölle syntyvä aineisto saadaanvastaamaan parhaiten käyttäjän tekemää kyselyä.Tulostuksessa voidaan toki käyttää myös luonnollistasemantiikkaa ja sen esitysasua, mutta se on ohjelmoijillevieraampi.3.3.10 T10 Pragmaattisen tulkinnan teoriaYlläpito sisältää paljon erilaisia tehtäviä ja alatehtäviä,joissa koodin tarkastelu on tarpeen. Aiheen laajuudenjohdosta sitä käsitellään tässä kohdassa vain muutaminesimerkein Penningtonin tietotarvemäärittelyn pohjalta: Jokainen työkalun tuotos, joka johdattaa käyttäjäävianpaikannustehtävässä ja supistaa alkuperäistäinformaatiojoukkoa, on pragmaattista informaatiota. Jokainen perehtymistilanteessa käyttäjälle tuotettuvastaus kysymyksiin what, why ja how, onpragmaattista informaatiota. Jokainen syy-seuraussuhdetta erittelevä haku, jokaon kohdistettu vian määritykseen, tuottaapragmaattista vian rajauksen informaatiota. Jokainen metodin suorituspolkujen hahmottamiseenliittyvä kokonaisuus palvelee vianhaun tarpeitatutkittaessa ohjelman pysähtyvyysongelmia. Jokainen kriittisen kuuntelijaolion tiloihin liittyväkysely tuottaa pragmaattista informaatiota.Teknologiamielessä pragmaattisen informaationtarkastelu vaatii käyttäjän tarpeiden tunnistamista jametodologian sovittamista niihin. Saatu tietämys onkuitenkin arvokasta, sillä se palvelee kiireistä käyttäjäähänen jokapäiväisessä työssään sellaisenaan.4 MetodologiaEdellä kuvattu ontologia ja tietämyksen määrittely eriteoria-alueineen toimivat lähtötietoina metodologiantarkastelulle.4.1 Metodologian toteutusSeuraavassa esityksessä Peircen semioottista käsitteistöä,jonka muodostavat merkki (sign), kohde (object) jalogiikalla toteutettava tulkinta (interpretant, logic L),lähestytään symbolista alkaen. Yhdistämällä nämäkäsitteet saadaan hahmoteltua kokonaisuus SOL. Malli(M) määritellään yksinkertaisesti säiliöksi, joka sisältäämalliin luodut oliot. Mallin analysointi (A) tapahtuukyselyiden (Q, Query) avulla oliorajapinnan kautta.4.1.1 Symbolin toteutus (S)Kieliopista saadaan suoraan kaikki symbolit, jotkaviittaavat lähdekoodin muuttujiin. Niitä ovat luokan nimi,metodi, attribuutti ja muuttujanimet. Koodin tarkasteluuntämä metoditarkkuus ei vielä riitä, koska kun saatuasymbolista mallia halutaan tutkia esimerkiksivianpaikannuksen tarkoituksessa, tarvitaan suorituspoluntarkkuus ja sen toteuttamiseen suorituspolkuelementti.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 112

Jos erityisesti käyttäytymismalli ja elementtienvaikutukset kiinnostavat, tarvitaan lisäksisivuvaikutuselementti, joka tallentaa kunkin elementinsuorittamat ulkoiset muutokset.Symboli tarkoittaa siis symbolisessa mallissakäsitettä, joka voi olla Käyttäjän antama alkuperäinen symboli Suorituspolkuun viittava symboli Sivuvaikutukseen viittaava symboliSymboli on käytännössä vastaavaan työkaluun luotuviittaus vastaavaan olioon, joka voidaan tulostaa javisualisoida eri tavoin sen sisällön ja tulkinnan mukaan.Kun käyttäjä napauttaa symbolin kuvaketta näytöllä hänpääsee kaikkiin sitä vastaavan olion ominaisuuksiinkäsiksi, jolloin saadaan aikaan semanttinenkäyttöliittymä. Symboli on siis esittämisen,kommunikaation ja ohjauksen väline.4.1.2 Objektin toteutus (O)Määrittelynsä mukaan oliojärjestelmän olio ja luokkasisältävät abstrahointia, yleistämistä ja erikoistamistatukevia piirteitä. Ne kaikki ovat hyödyllisiä lähdekoodinanalysoinnissa ja soveltuvat Peiren semiotiikkaanseuraavasti: Abstrahoinnilla voimme luoda käsiterakenteita, jotkaovat kaikkien analyysien kannalta yhteisiä ja kehittäänäin mahdollisimman teoreettista yhteistä mallia. Yleistämisellä voimme määritellä yhdenmukaisenolion työkalun luokkana. Tässä sitä kutsutaansymboliseksi elementiksi, symbolicElement. Sepystyy mallintamaan kaikkia ohjelman rakenteita,koska kaikilla rakenteilla on merkittävä määräyhteisiä piirteitä.Voimme luoda riittävän tarkkoja erikoistettujaolioita symbolicn alaluokkina siten, että nejakaantuvat teorian T2 ryhmittelyn mukaan. Voimme periyttää kaikki oliot superluokastasymbolic , joka sisältää symbolisen kielenmäärittelyt. Näin kaikkien olioiden välille saadaanaikaan symbolinen kieli ja sen täydellinen tuki.Ohjelmistokehitystasolla keskeisin olio, symbolinenelementti, toteuttaa kaikki tarvittavat piirteet rajapintojenansiosta. Se periytyy symbolic-luokasta kuten edellämainittiin ja sitä käyttävät super-luokkana kaikkisymbolisen kielen tyyppirakenteet.Puhdas oliomalli ei kuitenkaan sellaisenaan riitäolioiden esittämiseen, koska kapselointi rajoittaisitiedonsaantia olioiden välillä ja tekisi mallinassosiaatioiden läpikäyntialgortmien rakentamisen hyvinvaikeaksi, koettaisiin samat ongelmat kuin xmi-malleissa.Siksi tarvitaan täydentävä ohjelmointiparadigma,logiikka, jota kuvataan seuraavassa.4.1.3 Tulkinnan toteutus, logiikka (L)Olion sisäiset tiedon tallennukset suoritetaan logiikankautta predikaatteina ja faktoina. Oliot muodostavatkeskenään navigointia tukevan kaksisuuntaisenverkoston, joka saadaan aikaan child- ja parent- sekänext- ja previous-faktoilla. Elementin sisältämä kooditallentuu contents-nimiseen faktaan, joka on käytännössäsymbolisen kielen rakenne, Clause.Mallin läpikäyntialgoritmit käyttävät hyväkseen jokonavigointifaktoja tai contents-faktan sisältöä.Prologin päättelykone tarjoaa tukea läpikäynninohjelmoimiseen. Näin saadaan aikaan traverse-toiminto.Mallin suorittamiseen tarvitaan run-toiminto, jokapalauttaa oletuksena symbolisen kielen lauseita.4.1.4 Mallin toteutus (M)Yleistä symbolista mallia voidaan luonnehtia seuraavallamäärittelyllä:model(L, B, T, F),jossa L (layers) on tarvittavien mallin kerrosten määrä, B(bases) on tarvittavien erilaisten superluokkien määrä, T(types) on ratkaisun erilaisten tietotyyppien määrä ja F(features) on sellaisten funktioiden (piirteiden) määrä,jotka voidaan toteuttaa symbolisella elementillä.Ylläolevasta tarkastelusta voimme päätellä seuraavaa: L on 1, koska tarvitaan vain 1 kerros, joka sisältääkaiken informaation. B on 1, koska kaikki oliot periytyvät symbolisestaelementistä, joka edelleen periytyy symbolisenkielen määrittelystä.T on 11, koska symbolisen kielen rakenteeseenliittyy 11 perustyyppiä. Mallilla toteuttavien piirteiden määrä, F, onperiaatteessa ääretön, koska symboliselle mallillevoidaan ohjelmoida lukemattomia erilaisiakäsittelijöitä eri tarkoituksiin, mm. vastaamaankyselyihin.Edelliseen verrattuna MDA-mallit ovat huomattavastimonimutkaisempia, mutta silti heikompiaominaisuuksiltaan ja laajennettavuudeltaan.Tyypillisimmät mallin toiminnot ovat seuraavat: Build rakentaa mallin ja sen oliot Traverse käy mallin sisältöä läpi tietyin oletuksin Chop hakee kahden metodin väliin kutsuhierarkiassasijoittuvan metodien joukon. Run ajaa (simuloi) koodia halutusta kohdasta lähtien. Reset poistaa oliot muistista.4.1. 5 Analyysin toteutus (A)Edellä mainittiin mallin ja oliorajapinnan keskeisimpiäpiirteitä. Ohjelmistoteknisesti analyysi on kuuntelija113

(observer), joka mallin käsittelyn käynnistyttyä jääodottamaan dataa. Analyysin parametrit määrittelevätmistä piirteestä kulloinkin on kysymys. Tulokset saadaansymbolisen kielen muodossa.4.1.6 Kysely eli Query (Q)Kuten edellä kerrottiin, tarkastelun tarpeilla onkysymysten kautta yhteys hypoteeseihin, joista edelleenon yhteys kyselyihin. Vastaus kysymyksiin what, why jahow saadaan joukkona symbolisen kielen rakenteita,jotka voidaan edelleen muuntaa kaavioiksi, näytöksi,tekstiksi tai xmi-muotoon siirrettäväksi muihinkehitysvälineisiin.Tyypillinen kysely liittyy ongelman rajaukseen jaantaa tuloksena joukon elementtejä, joita epäillään. Näitäkutsumme ehdokkaiksi, vikaehdokkaiksi. Kun käyttäjäsaa näkyville listan vikaehdokkaista, hän voi valitasopivan etenemistavan navigointiin ongelman ratkaisuntäsmentämiseksi.4.2 Yhteenveto metodologiastaSymbolisen mallin rakentamiseen ja hyödyntämiseentarvitaan seuraava teoriakokonaisuus:T1 Java - koodin lukemiseen parsintapuuksiT2 Uusi abstraktiotaso parsintapuulleT3 Translaatio symboliseen kieleenT4 Javan semantiikan siirto malliinT5 Mallin kutoja luomaan yhtenäisen aineistonT6 Suppea kyselyjärjestelmä alkurajaukseenT7 Laajat kyselyt suorituspolkujen analysointiinT8 Kohdekielen ja mallin relaatiomalliT9 Selittämisen teoriat muuntamaan tulokset logiikaksiT10 Teoriat pragmaattisen informaation tuottamiseen5 Ylläpitotehtävän suorittaminenTutkimusrajauksessa tulotietona toimii ylläpitoon luotutehtävä, jota kutsutaan muutosvaatimukseksi (ChangeRequest, CR). Tällöin kyseessä on ongelma taiparannusehdotus tai perehtymistarve.Seuraava proseduuri pyrkii luomaan toimivanyhteyden ylläpitotehtävän ja symbolisen mallin kyselynvälille.5.1 YlläpitometodiYlläpitotehtävän kulkua kuvataan seuraavassa lyhyenkirjainyhdistelmän TPHQ avulla.5.1.1 Task (T)Ylläpitotehtävän käsittelyä hajautetussa ohjelmistotyössäon kuvannut Walenstein (2002) RODS-käsitteistöllään,joka esittää kuinka tehtävän suoritus pyritäänminimoimaan, kuinka suoritusalgoritmi halutaanoptimoida, kuinka tehtävän eri osat jaetaan useilletekijöille ja kuinka erikoistuneita asiantuntijoita pyritäänkäyttämään tehtävän osien suorittamiseen. Jos tehtävä eiole triviaali, se alkuvalmistelujen jälkeen muuttuuprosessiksi, josta kerrotaan seuraavaksi.5.1.2 Prosessi (P)Prosessin tarkoituksena on ratkaista monivaiheinenongelma, joka vaatii useita erillisiä toimenpiteitä. Koodintarkastelun välivaiheita on tutkinut mm. von Mayrhauser(1992). Ihminen ajattelee tällaisessa tilanteessakysymysten ja vastausten kautta. Letovsky (1983) onkuvannut ylläpitoon liittyviä kysymyksiä luokittelulla :what, why ja how. Käyttäjä haluaa selvittää mitä jokinkoodin osa tekee (what), miksi se suorittaa tiettyjätoimintoja (why) ja kuinka se toimii (how). Koska nämätiedot eivät selviä lukematta koodia, hän joutuu arvamaanja tekemään olettamuksia. Näin hän tulee luoneeksihypoteeseja.5.1.3 Hypoteesi (H)Hypoteeseilla käyttäjä todistaa toimiiko esimerkiksiServer-luokka niin kuin hän oletti. Jos se toimii, hän voikehittää hypoteesin pyrkien todistamaan sen oikeaksi taivääräksi. Lopullinen päätelmä voidaan tehdä kunhypoteeseista on saatu riittäviä tuloksia. Vaikka tällainenhypoteesien käsittelyn prosessi kuulostaa vaivallolliselta,se on monessa tilanteessa esimerkiksi vianhaussa ainoavaihtoehto. Jos käyttäjällä on käytössään vain editori, työon hyvin hankalaa, mutta kehittyneet takaisinmallintamisentyökalut voivat tässä häntä auttaa.Peirce esittää ongelman lähestymiseen kolme erillistätapaa: abduktion, induktion ja deduktion. Seuraavassamuutama esimerkki niistä: Abduktion käyttö ohjausvuon käsittelyssä: Kaikkiohjelman sekvenssit, jotka alustavat olion Server,ovat palvelimeen liittyviä olioita. Esimerkiksi koskametodi main alustaa serverin, metodi main onkriittinen palvelimen käynnistymisen kannalta. Induktio: Muuttuja port saa komponentin sisälläserverin aloituslogiikassa arvoja 80,81 ja 82, jotkakaikki ovat oikeita. Toinen vaihtoehto on se, ettäkomponenttia kutsutaan ulkoa, jolloin portille eivoida asettaa käynnistyarvoa, mikä on riski. Siten onvarmistettava nämä haarat ja tarkistuksena lisättäväoletusarvoksi 80. Näin voidaan yleistäen(induktiivisesti) päätellä, että kaikki tapaukset ovatkunnossa. Deduktio: Olion Server käynnistymiseen vaikuttavatkomentoriviltä saatu arvo ja if-lausekkeessa saatuarvo. Käynnistyminen estyy, jos poikkeustilannelaukeaa main-metodissa. Nämä lauseet muodostavatNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 114

täydellisen logiikan (and) serverin käynnistymiseen.Jos olio ei käynnisty oikein, vian täytyy olla jossakinnäistä kohdista.Letovskyn kysymyksillä on seuraava yhteys koodinsisältöön: What – kysymys viittaa elementin tarkoitukseen jatoiminnallisuuteen (function). Why – kysymys viittaa syy-seuraussuhteeseen, jokaon vianpaikannuksessa hyvin hyödyllinen piirre. How-kysymys viittaa elementin sisäiseen logiikkaan,sen suorituspolkuihin.5.1.4 Kysely, Query (Q)Kun kysymys on tunnistettu, se voidaan muuntaakyselyksi. Jos kysymykseen liittyy kaksi rajausta, se voiolla esimerkiksi muotoa:Query = why(Start,Target,Options)Jos rajauksia on vain yksi, se on muotoaQuery = what(Object)taiQuery = how(Object)5.1.5 Systemaattinen vai opportunistinen oppiminenKäyttäjä voi tehdä kyselyjä systemaattisesti taiopportunistisesti optimalla hakujen määrää.Systemaattinen haku vie usein liian paljon aikaa, jotenkäyttäjän ammattitaito tulee esille, kun hänen pyrkiiselviytymään ratkaisuun mahdollisimman joustavasti,mutta kuitenkin riittävän kattavasti.Työkalu voi tukea tiedonhakua mm. tarjoamallajoustavan navigoinnin ja minimoimalla kyselyynkohdistuvan informaation. Näistä piirteistä kerrotaanseuraavassa.6 Työkalu JavaMasterEdellä selostettiin karkealla tasolla kohdealueenmetodologia ja työskentelymetodi. Seuraavaksi käydäänläpi työkalu nimeltä JavaMaster ja sen peruspiirteitä.Työkalun sisältämän Visual Prolog- kielisenlähdekoodin koko on noin 40.000 riviä. Se sisältää noin450 luokkaa ja toimii Windows-ympäristössä.6.1 Työkalun arkkitehtuuriTakaisinmallintamistyökalun arkkitehtuuriksi muodostuuvarsin luontevasti Model View Control (MVC) –rakenne, koska lähdekoodista saatu malli muodostaatärkeimmän osan (Model). Käyttäjä ohjaa työkaluaControl-osan kautta saaden aikaan näkymiä (View).Navigointi tapahtuu näyttöjen kautta tai työkalunpäämenuista.6.2 Työkalun piirteetJavaMasterin pääpiirteitä ovat: 1) lähdekoodin latausmuistiin ja malliksi, 2) ohjelmamallin tietojen selaus erivälilehtiä hyödyntäen, ja 3) kyselyjen suorittaminentarkastelun hypoteesien mukaisesti.Työkalu tarjoaa lukuisia erilaisia tarkastelutapoja.Latausvaiheessa se luo koodista työlistan, joka palveleelähinnä koodiin perehtyjää, joka sen mukaisesti voikäydä läpi vaativimman aineiston, jota hän voi katsellapuiden, taulukkonäyttöjen, riippuvuuskaavioiden, umlkaavioidenja xml-dokumenttien muodossa.Kuva 4 esittää JavaMasterin käyttötilannetta. Koodiladataan siinä tiedostopuun (kohta 1.) avulla muistiin.Lataus tuottaa puuhun (2.) ohjelmamallin mukaisenhierarkian. Sitten käyttäjä napauttaa Worklist-painiketta,jolloin työkalu näyttää vasteenaan sarjannavigointikysymyksiä (4. Q/A). Navigointitilannettavastaava informaatio muodostuu eri näyttöihin (3.).1.4Q/A2.3.Kuva 4. JavaMasterin käyttöliittymä.6.3 Työkalun evaluointiJavaMasterin evaluoimiseksi on useita mahdollisuuksia: Koodi ja algoritmit voidaan verifioida analyyttisestilause lauseelta. Työkalun tulokset voidaan evaluoida kokeellisestitutkien mm. kutsupuiden kattavuus. Koodin sisäisiä rakenteita voidaan verrata ASTtyökaluihin,joita löytyy mm. Eclipse-työkalusta. Käyttötapauksina voidaan tutkia tyypillisäongelmanpaikannustilanteita, esimerkiksi tilanteita,joissa tietokantapalvelin toimii väärin. Tälloinselvitetään miten kuvattu ongelmatilanne onpurettavissa hypoteeseiksi ja kuinka niitä vastaavatkyselyt on suoritettavissa työkalun avulla.115

Työkalun evaluointiprosessi on vielä kesken, mutta senabsoluuttista kapasiteettia on jo evaluoitu syöttämällätyökalun symboliseen malliin miljoonan elementinaineistoja, jotka määrältään vastaavat MicrosoftinWord97-ohjelmiston AST-solmujen määrää. Tällöin ontodettu, että myös suurten ohjelmistojen analysointi onsymbolisella mallilla mahdollista, vaikkakin menetelmänsuurimmat edut löytynevät tiettyjen rajattujenongelmalliseten koodialueiden tarkastelusta monin eritavoin.7 YhteenvetoEsitetty symbolinen malli kuvaa sellaisenaan eräänlaisenvirtuaalitoteutuksen, joka abstrahoiduilla rakenteillaanmuistuttaa alkuperäistä koodia. Siten se poikkeaamerkittävästi konkreettisista työkaluista, jotka pyrkivättarjoamaan käyttäjälle aina täydellisen ratkaisun.Abstrahoinnin tueksi Walenstein (2002) on esittänyt, ettätäydellisen ratkaisun tarjoaminen johtaa käyttäjänylikuormittumiseen, siksi abstrahointi olisi suositeltavaa.Mutta mikä olisi abstraktin toteutuksen esikuva, onkosellaista esitetty aiemmin?Kovin monenlaisia erilaisia teorioita abstrakteistasimuloitavista koneista ei ole tehty, jos virtaalikoneita eitässä huomioida. Ehkä tunnetuin niistä on Turingin kone(Kokkarinen 2003), joka käy läpi tulotietona olevaanauhaa ja ohjausyksiköllään pyrkii reagoimaan nauhantietoon määrittelyn mukaisesti.Symbolinen analyysi muistuttaa monin tavoinTuringin konetta ja sen ohjausyksikköä. Symbolinenmallihan sisältää tilakonemaisen ohjausyksikönpoimittuaan alkuperäisestä koodista logiikkapolut jalauseiden semantiikan. Mutta toisin kuin Turingin kone,symbolinen malli sisältää myös formaalin tietoaineiston,joka jäljittelee alkuperäistä koodia.Suorituksen aikana symbolinen malli tallentaavälitulokset ja lopputulokset perusolioihinsa, joten niidensisältö palautuu mallin kyselyn kautta käyttäjälle. Sitensaatu informaatio kuormittaa ”ohjausnauhaa” eli ihmisenja koneen välistä kommunikointiväylää mahdollisimmanvähän. Sehän oli abstrahoinnin tavoitekin.Toinen mielenkiintoinen asia on tarkastella, kuinkahyvin saatu konstruktio vastaa semioottista käsitteistöä.Symbolisen mallin pääkäsitteistä voi havaita, että esitettymetodologia sopii suhteellisen hyvin yhteen Peircenkäsitteistöön, koska molempiin liittyy kolme pääsuuretta:merkki (sign /symboli), kohde (object) ja tulkinta(interpretator) - sillä tarkennuksella, että tulkintaasymbolisessa mallissa vastaa logiikka, joka on toteutettumallin elementin sisältämänä predikaattina.Edellä kuvattu kolmijako muodostui metodologiaantuntematta Peircen teoriaa vielä suunnitteluvaiheessa,mikä on mielenkiintoinen havainto. Siitä voidaan vetääjohtopäätös, että konstruktio pääsi kehittymään oikeallatavalla oikeaan suuntaan jo alusta alkaen.KiitoksetSwMaster Oy on mahdollistanut alustavan tutkimuksenja työkalun kehittämisen vuodesta 2000 alkaen. Senlisäksi vuoden 2005 alusta Jyväskylän yliopistonCOMAS-ohjelma on rahoittanut Erkki Laitilanväitöskirjahankkeeseen liittyvää tutkimusta.ReferencesPaul Anderson, Thomas Reps, and Tim Teitelbaum."Design and Implementation of a Fine-GrainedSoftware Inspection Tool", IEEE TSE 29 (8), 2003,pp. 721-733.Richard Brooks. Towards a Theory of UnderstandingComputer Programs, IJMS 18 (6), 1983.Jean-Marie Burkhardt, Francoise Detienne and SusanWiedenbeck. "Object-Oriented ProgramComprehension: Effect of Expertise, Task andPhase", Empirical Softw. Eng. 7 (2), 2002, pp. 115-156.Gerardo Canfora, A. Cimitile, and U. de Carlini. "ALogic-Based Approach to Reverse EngineeringTools Production", IEEE TSE 18 (12), 1992, pp.1053-1064.Thomas Cheatham, Glen Holloway and Judy Townley."Symbolic Evaluation and the Analysis ofPrograms", IEEE Trans. Softw. Eng. 5 (4), 1979,pp. 402-417.Noam Chomsky. Three models for the description oflanguage. IRE Transactions on Information Theory,IT-2:3:113–124, 1956.CMMI. Capability Maturity Model Integration (CMMI),http://www.sei.cmu.edu/cmmi.Eclipse. Eclipse Research Community: Eclipse, OpenSource Project. http://www.eclipse.org [2003].David Flanagan. Java Examples in a Nutshell: Example7-5, ftp://ftp.ora.com/examples/nutshell/java/examples, O'Reil-ly, 2000.Frisco. A Framework of information system concepts.http://www.mathematik.unimarburg.de/~hesse/papers/fri-full.pdf,1998.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 116

Susan Horwitz, Thomas Reps, and David Binkley.Interprocedural slicing using dependence graphs.ACM Trans. Program. Lang. Syst., 12(1):26–60,1990.Ilkka Kokkarinen. Tekoäly, laskettavuus ja logiikka,2003.Erkki Laitila. Visual Prolog: Teollisuuden Sovellukset,Docendo/Teknolit, Jyväskylä, Finland, 1996, 228 p.Erkki Laitila. Method for Developing a Translator and aCorresponding System. Patent: W02093371,http://www. espacenet.com, 2001.Erkki Laitila. "Program Comprehension Theories andProlog Based Methodologies", Visual PrologApplic. & Lang. Conf. (VIP-ALC 2006), PrologDevelopment Center, 2006.Stan Letovsky. "Cognitive Process in ProgramComprehension", Empirical Studies ofProgrammers: First Workshop, Ablex, 1986, pp. 80-98.Esko Marjomaa. Theories of Representation,http://cs.joensuu.fi/^marjomaa/tr/tr/docAnne-Liese von Mayrhauser and Ann Marie Vans."Hypothesis-Driven Understanding ProcessesDuring Corrective Maintenance of Large ScaleSoftware", Proc. Int. Conf. Softw. Maint. (ICSM1997), pp. 12-20.Charles W. Morris. Signs, Language and Behaviour,Prentice-Hall, New York, 1946.Charles W. Morris. Writings on the general theory ofsigns. The Hague: Mouton (1971).OMG. UML, Model Driven Architectures,http://omg.org [2005].Prolog Development Center. Visual Prolog 6.3,http://www.visual-prolog.com, 2005.Prolog Development Center. Visual Prolog Applic. &Lang. Conf. (VIP-ALC 2006), http://www.visualprolog.com/conference2006, 2006.Thomas Reps. "Program Analysis via GraphReachability", ACM SIGPLAN 2000 Conferenceon Programming Language Design andImplementation (PLDI'2000).Tamar Richner and Stand Ducasse. "Recovering High-Level Views of Object-Oriented Applications fromStatic and Dynamic Information", Int. Conf. Softw.Maint. (ICSM 1999), IEEE, pp. 13-22.Spice. Software Process Improvement and CapabilityDetermination. http://www.sqi.gu.edu.au/spice.Sun. The Grammar of Java Programming Language,http://java.sun.com/docs/books/jls/secondedition/html/syntax.doc.html, 2003.Turing Machines. http://plato.stanford.edu/entries/turingmachine.Andrew Walenstein. Cognitive Support in SoftwareSupport in Software Engineering tools: ADistributed Cognition Framework,ftp://fas.sfu.ca/pub/cs/theses/2002/AWalensteinPhD.pdf.Norman Wilde,and Ross Huitt. "Maintenance Support forObject-Oriented Programs", IEEE Trans. SoftwareEng. 18 (12), 1992, pp. 1038-1044.Stefano Zacchiroli. Understanding Abstract Syntax Tree,http://www.connettivo.net/article.php3?id_article=49, 2003.Charles S. Peirce. Kotisivut, C.S. Peirce,http://peirce.org, 2006.Nancy Pennington. "Stimulus Structures and MentalRepresentations in Expert Comprehension ofComputer Programs", Cognitive Psychology 19 (3),1987, pp. 295-341.117

Risto.lahdelma@cs.utu.fi aiyron@utu.fiKeywords: Fuzzy sets, linear programming, chance constraint, scrap charge optimization, steel production.A general trend during the past decades is thatscrap-based steelmaking has increased its share,reaching around 40% of global crude steel productionin 2001 (Rautaruukki 2001). Steel is also theworld’s most important recycled material. The useof the steel scrap as a raw material for steelmakingresults in a saving of 600 million tonnes of iron oreand 200 million tonnes of coke each year (EURO-FER 2001). With the growing concern on environmentalissue, the popularity of using scrap could furtherincrease because scrap-based steelmaking emitssignificantly less CO 2 as compared with integratedsteelmaking using metallurgical coke as reductantfor iron-making. Undoubtedly, the use of the scrapoffers the opportunity to produce high qualityproducts most economically. In the meantime, italso poses challenge in charge optimization causedby the uncertainty of chemical composition of scrap.The uncertainty mainly comes from two sources.First, the constituents in the scrap and steel productare diverse. With the development of steel productsfor higher grades and higher performance, use ofsteel materials in combination with nonferrousmetals or non-metallic materials has increased. Dependingon the melting process and the requirementsof the particular product, some of the constituents inscrap are considered impurities while others arevaluable additives.Second, the diverse scrap materials are generally dividedinto scrap types for handling and the materialsincluded in each scrap type are heterogeneous becausethe classification varies based on different criteria.That means there are large deviations in materialproperties inside class, sometimes larger thanbetween classes. Therefore, it is difficult to come upwith accurate chemical composition (element concentration)analysis for the scrap.Scrap-based steelmaking process starts with thecharge of the predetermined scrap mix in an electricarc furnace (EAF). One furnace of refined steel(with required chemical composition) is called aheat. When the scrap mix is melted, there is a considerablerisk for the outcome failing to satisfy thecomposition requirements for the product because ofthe uncertainty in the chemical composition for thescrap. The objective of the scrap charge optimizationis to select the most economical scrap mix foreach produced steel grade while minimizing the failurerisk caused by uncertainties in raw material consistence.There are three kinds of charge optimization modelsbased on different planning horizon: single-heatmodel (AFS 1986) for short-term planning (e.g.daily operations), multi-heat model (Kim & Lewis1987) for medium-term planning (e.g. weekly ormonthly) and product recipe model (Lahdelma et al.1999) for long-term planning (annually). Anycharge plan is implemented on single-heat basis andon-line analysis is applied to identify whether theNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 118

outcome match the predicted characteristics and tocorrect the possible bias in future predictions(Wilson et al. 2001). However, if the charge planningis designed using the single-heat model, thismay result in non-optimal use of raw materials.Minimizing raw material costs in one heat can eatup the cost-efficient raw materials and therefore increasethe costs in future heats. The multi-heat modelcan allocate the available raw materials optimallyamong the different heats based on the current rawmaterial stock and predicted deliveries. For longtermplanning, it is more convenient to use productrecipe model which can viewed as a model where allheats of the same product are grouped. Therefore,the product recipe model is much smaller than themulti-heat model. The long-term scrap charge planis designed based on forecast customer orders andpossible available raw materials. The product recipemodel can also be applicable to medium-term planning.In terms of handling uncertainty, there are severalmethods of constraining failure risks. Bliss (1997)added safety margins to the product standard by usinga tighter product standard in optimization.Lahdelma et al. (1999) added safety margins to thechemical composition for scrap in optimization. Theapproach of adding safety margin can be viewed asthe extension of the deterministic model to accommodatethe uncertainty. Turunen (2000) constrainedthe failure risk based on stochastic chance constraintsto guarantee that the failure rate is less thana predetermined level (or to allow small violationsin some constraints). However, stochastic chanceconstrained programming models are difficult tosolve since the deterministic version of the model isnon-linear (Kall & Wallace 1994, Watanabe & Ellis1994). The solution can be even more complicated ifthe stochastic model includes simultaneously chanceconstraints and ordinary constraints with stochasticparameters.In this paper, we formulate the scrap charge optimizationproblem based on the product recipe model.We represent the uncertainty for the scrap compositionand the composition specification for productstandard based on fuzzy set theory (possibility theory,Zadeh 1978). Then we constrain the failure riskbased on a possibility measure. Consequently, thescrap charge optimization problem is modeled as afuzzy linear chance constrained programming problem.This approach can be viewed as explicitly integratingmethods of adding safety margin forproduct standard and for scrap composition and allowingsmall violations (chance constraints) in someconstraints under a single framework. Since theconstraints in the scrap charge problem mainly addressthe specification of the product, the crisp equivalenceof the fuzzy constraints should be less relaxedthan that purely based on the concept of softconstraints. For general discussions on the interpretationof the ordinary fuzzy constraints based on theconcept of soft (flexible, approximate) constraints,we refer to Canz (1996), Dubois & Prade (1980),Slowinski (1986), Werners (1987), Zimmermann(1978). For interpretation of general chance constraintsdirectly based on the possibility measure, werefer to Liu & Iwamura (1998). Here we interpretchance constraints based on the likelihood measureand the resulting crisp constraints are much stricterthan those based directly on the possibility measure.The property of the likelihood measure introducedin this paper is similar to that of the likelihood profilediscussed in the fuzzy machine scheduling context(Wang et al. 2002). There are two ways to interpretordinary fuzzy constraints: soft constraints andtolerance constraints. The strict tolerance constraint(Dubois & Prade 1980) means that that fuzziness ofthe right-hand side of the constraint is interpreted asa maximum tolerance for that of the left-hand sideof the constraint. However, the system with thestrict tolerance constraints may be inconsistent.Therefore, we interpret the ordinary fuzzy constraintsbased on the framework of relaxed toleranceconstraints where we try to eliminate the possibleconflicts of the strict tolerance constraints by droppingthe relatively less critical constraints or byweakening some constraints. Finally a crisp modelwith consistent and compact constraints is formed.The resultant crisp model is a standard linear programming(LP) model, which can be solved bystandard LP software.The paper organizes as follows. In Section 2, we reviewa generic fuzzy linear programming model anddescribe the relationship between the fuzzy numberand the stochastic parameter with given distribution.In Section 3, we describe the scrap-based steelmakingprocess first, and then we formulate the scrapcharge optimization problem as a fuzzy chance-constrainedlinear programming model. In Section 4, wetransform the fuzzy model into a crisp model withcompact and consistent constraints based on the applicationcontext. In Section 5, we report the simulationresults.We define fuzzy linear programming (FLP) as theextension of the classical linear programming (LP)in operations research (Taha 1992) in the presenceof uncertainty in the optimization process wheresome or all of the coefficients are represented asfuzzy quantities (numbers) based on fuzzy set theory(possibility theory). A non-fuzzy quantity can beviewed as a special case of fuzzy quantity as discussedlater. A generic FLP model is given below.s.t. 119

where x j are decision variables, are cost coefficients, are technical coefficients, and are right-hand side coefficients. Some orall of these coefficients can be fuzzy numbers. Formula(1) is the objective function to be optimized.Constraints (2) are called chance-constraints. Pos{.}denotes the possibility of events {.}. The predeterminedaspiration level is usually interpreted asconstraint reliability. The constraints (2) mean thatthe possibility for violating shouldbe less than 1-α i. Constraints (3) are ordinary linearconstraints.Fuzzy numbers play a central role in the fuzzy programmingmodel. First, we review the concept offuzzy numbers briefly. Then we represent the statisticaluncertainty based on fuzzy set theory and establishthe relationship between the fuzzy number andthe stochastic parameter with given distribution.A fuzzy number is a convex normalized fuzzy set Aon the real line R with membership function f A(x) (x∈R). We represent the fuzzy number based on themodified L-R fuzzy number. The examples of L-Rfuzzy numbers can be referred to Dubois & Prade(1980) and Slowinski (1986). The representation forA is the 3-tuple of parameters ,where a, and are mean, left andright spreads of the fuzzy number. The memberfunction f A(x) is given below. where L and R are continuous decreasing functionssuch that L(0) = R(0) = 1, L(1) = R(1) = 0. If bothL and R functions are linear, then the resultant fuzzynumber is called triangular fuzzy number as shown in Figure 1, where and are L and R functionsand Anon-fuzzy number can be treated as a special case ofthe fuzzy number where both the left and rightspreads are zero. Figure 1 Relationship between a fuzzy number and a stochastic parameter withdensity function p(x) (a is the mean of the stochasticparameter).Figure 1 also illustrates the relationship between afuzzy number and a stochasticparameter with density function p(x). The uncertaintyinvolved in many applications can be viewedas statistical uncertainty. Canz (1996) discussed thebasic approach for representing statistical uncertaintyusing trapezoidal fuzzy numbers. Here, wefollow the similar logic and represent the statisticaluncertainty using modified L-R fuzzy numbers. Firstwe assume a probability density function p(x) fromthe statistical data, and then we transform it into apossibility distribution (fuzzy set) based on theshape of p(x) and the mean and standard deviationof the statistical data. If the mean and standard deviationof the density function p(x) isβ and σ respectively,then for a fuzzy number , and ,where ε is related to the skewness (the third moment,Banks & Carson 1984) of the probabilitydensity function p(x), n 1 and n 2 are confidencefactors. The choice of n 1 and n 2 depends on shape ofp(x) and the application context. If the density functionp(x) is skewed (e.g. log normal distribution),then n 1 and n 2 should be different. If p(x) is symmetric(e.g. normal distribution), then a = β, n 1 = n 2.However, if the parameter has non-negativity restrictionthen . If , this is equivalent to case that n 1 andn 2 take different values. The value setting of n 1 andn 2 depends on the requirement of the application.The choice of L and R functions is flexible andsimple linear functions can satisfy the needs formost applications.To understand the scrap charge optimization problem,we start by introducing the scrap-based steelmakingprocess, e.g. stainless steel-making process.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 120

Scrap-based stainless steel-making pro-Figure 2.cessFigure 2 shows a sketch of the scrap-based stainlesssteel-making process. The scrap metal is chargedinto an electric arc furnace (EAF) and melted. Thenthe melt is moved into argon-oxygen decarburization(AOD) converter where the carbon is removedby argon and oxygen injection in order to meet thesteel quality requirement. After AOD, the moltensteel is moved into ladle furnace (LF) for secondarymetallurgical operations such as adding alloying elements,degassing and temperature homogenization.The scrap consists of materials from differentsources such as old cars, structural steel, cans, electricappliances and industrial by-products. Scrapscontain also waste from the manufacturing processof steel products such as cut edges of steel platesand ends of coils. Generally the diverse materialsare divided into several standard types for trade.After the scrap materials are delivered to the meltshop,the steel producers have their own internalclassification system that further divides the standardtypes into different subtypes based on origin,supplier, chemical contents and size distribution andrequirements. The total number of scrap types isabout 20 for a normal carbon-steel meltshop. Forstainless and speciality production, the classificationcan be even finer and the number of scrap types canbe more than 100 (Sandberg 2005). For the stainlesssteel production, it is particularly important to sortgrades with high content of valuable alloying metalssuch as nickel or molybdenum in their own classesso that the scrap can substitute for the expensivepure alloying materials to provide alloying elementsin the new product. It would be good for the productionif all different alloys could be separated to differentclasses but it seems that there are large deviationsin properties (e.g. element concentrations) insideclasses, sometimes larger than between classes.Some scrap classes are much more heterogeneousthan others. That is, there is uncertainty for elementconcentration for the scrap. The uncertainty cancause errors for element concentrations in the finalproducts.The scrap also contains oxides, organic materialsthat burn in the process, and the metal itself formsoxides in the process. The yield of the raw material(material yield) is the share of the raw material thatbecomes part of the product, typically 70-95 %. Wealso use a yield parameter for each element (elementyield coefficient). The element yield coefficients dependon the conditions in the furnace, while the materialyield depends mainly on the physical andchemical properties of the material. The two yieldparameters multiplied together give the net yield ofan element for a raw material. The yield and elementconcentrations of the scrap types can be estimatedbased on spectrographic analysis in conjunctionwith analysis of the process and heat data (Ikäheimo1999, Sandberg 2005).The process conditions may also vary and produceerrors not related to scrap. Inaccuracies in the chargingprocess constitute a third source of error.However, the element concentrations for the scrapare the most significant source of uncertainties inthe process. We can treat this uncertainty as thesources for all the errors.The objective of the charge optimization is to fulfilla set of customer orders at the lowest cost usingavailable raw materials. The costs include the rawmaterial cost and an approximation of the costs insuccessive process stages (the AOD converter andLF). Charge optimization models usually have constraintsfor the chemical balance of the steel, productionamounts and material resources (Kim &Lewis 1987). Here we also consider the uncertaintyin element concentrations for different scrap types.The uncertainty of the element concentrations forthe scrap will cause the element concentration in thefinal product to deviate from the product standard.The product standard is specified by lower and upperlimits for each alloying element concentration.A failure happens when the concentrations of the alloyingelements of the molten steel in EAF exceedsthe upper limits of the product standard because it isusually difficult to remove any element from the liquidsteel. Falling below lower limits is not as criticalbecause the element contents can be increased byadding pure alloying elements. However, it causescost increase. On the one hand, the raw material costwill increase because the pure alloying materials aremore expensive than the scrap. On the other hand,additional processing cost will be incurred in thesubsequent stages in LF because the processing timewill be increased. Therefore, the ideal scrap mixwould yield the element concentrations that areclose to the lower bounds of product standard.The orders are divided into |K| batches and eachbatch consists of products of the same standard.The element concentrations of raw materials and theelement concentration requirements of final productare represented as modified L-R fuzzy numbers. Weconstrain the failure risk based on the possibilitymeasure to control the failure rate within the predeterminedlevel. The following notations are introduced.Index SetI Set of all useful elements in the product includingelement iron.I 1 Set of alloying elements.J Set of all raw materials, J = J 0∪J 1.J 0 Set of scrap materials.121

J 1 Set of pure alloying materials, each ofwhich is made up of one alloying element.K Set of batches.Parametersa j Yield of raw material j∈J.b i Yield coefficient of element i∈I.c j Cost of raw material j∈J. For the alloyingmaterials j∈J 1, additional processing cost isincluded in material cost. Mass of products in batch k∈K. , Lower and upper bounds for chargeof raw material j∈J in batch k∈K. , Lower and upper bounds for usage (consumption)of raw material j∈J. Aspiration levels for controlling the upperlimit of the concentration of alloying elementi∈I 1 in batch k∈K. Concentration of elementi∈I in raw material j∈J (fuzzy number). Concentration of alloyingelement i∈I 1 in batch k∈K (fuzzy number).Decision variables Charge (consumption) of material j∈J inbatch k∈K.Then the scrap charge optimization problem forminimizing the raw material costs including additionalprocessing costs is represented as follows. ∈∈ ∈∈ ∈∈ ∈ ∈ ∈∈ Constraints (7)--(9) together are used to control theconcentrations of alloying elements for raw materials.Constraints (7) are general requirements for theconcentrations of the alloying elements for the scrapmaterials. The concentrations of the alloying elementsfor the scrap in EAF cannot exceed theproduct standard. Constraints (8) are chance constraintsparticularly constraining the failure riskbased on a possibility measure. The failure possibilitythat the concentrations of the alloying elementsfor the scrap exceed the upper limits of the productstandard must be less than the predetermined levels for each alloying element i∈I 1 in eachbatch k∈K. The effect of the chance constraints istwo-fold. On the one hand, they emphasize that controllingthe upper limits of the alloying element concentrationsis critical. On the other hand, they giveflexibility to control the limits by choosing differentaspiration levels, allowing explicitly small violationsof the upper limit of the product standard.Constraints (9) state that the concentrations of thealloying elements in the final product must meet theproduct standard. Constraints (10) state the massbalances between raw materials and final products.Here the constraints are crisp. We should know thatthe summation of concentrations of all elementsmust be one in the original raw materials. Some elementsmay be burnt off in the process and the summationof concentrations of all elements that becomepart of product is fixed for a given material.The mass is the aggregate of the share of all elementsthat become part of products for the selectedmaterials. Therefore, the uncertainty in element concentrationshas only slight effect on mass becausethe increase in concentration for one element impliesthe decrease in concentration for other elements.The variations in mass mainly come from thepossible different element yield coefficient b i fordifferent element i ∈ I. Constraints (11) and (12)give the bounds for raw material consumption.In the fuzzy environment, there are two ways totreat the ordinary fuzzy constraints: tolerance constraintsand soft (approximate) constraints (Dubois& Prade 1980). Tolerance constraints mean that thefuzziness of the right-hand side of the constraint isinterpreted as a maximum tolerance for that of theleft-hand side of the constraints. The soft constraintsmean that the satisfaction of the constraints is determinedby the membership function and there iscompensation between the satisfaction of the constraintsand fulfillment of the objective. That is, theinterpretation of the fuzzy constraints based on theconcept of soft constraints results in the crisp constraintsthat are more relaxed than the correspondingconstraints assuming that there is no fuzziness in thecoefficients, while the crisp constraints based on theconcept of tolerance constraints are much stricter.Based on this logic, we can find later the interpretationof the chance constraints directly based on thepossibility measure follows the line of soft constraints.In case of multi-criteria decision environment wherecriteria are described by objective functions andpossible alternatives are implicitly determined byconstraints, it is reasonable that the fuzzy constraintsare interpreted as soft constraints which can providea certain degree of freedom in determining the set offeasible solutions. However, for the scrap chargeproblem, the interpretation of the fuzzy constraintsshould be stricter than that based on the concept ofsoft constraints because the constraints mainly imposephysical restrictions on the product and mustNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 122

e somewhat strictly satisfied. Therefore, we interpretthe chance constraints based on the likelihoodmeasure to obtain a stricter interpretation than thatbased directly on possibility measure. We interpretthe ordinary fuzzy constraints based on the frameworkof the relaxed tolerance constraints where wetry to eliminate the possible conflicts from the stricttolerance constraints by dropping the less criticalconstraints or by weakening some constraints. Forsimplicity, we assume that L-R fuzzy numbers aretriangular. Different L and R function types mainlyaffect the setting of the aspiration level and complexityfor the realization of chance constraints.Based on the operation of fuzzy numbers where is also a triangular fuzzy number.Then constraints (8) becomeLet (x) be the membership function of thefuzzy number (Figure 3). ,u∈[0,1] are L and Rfunctions. Then based on (5) (x) canberepresentedas follows. Figure 3. The membership function of a fuzzy elementconcentration .Then the left hand side of (14) can be calculated asfollows. If the realization of (14) is directly based on (16),we can see that any aspiration level can be satisfiedby the value (element concentration) less than . This implies the concept of soft constraints.However, this realization is too relaxed for the scrapcharge problem because it cannot control failure rateeffectively.To enforce the chance constraints more strictly, weconstruct the likelihood of the valid element concentrationwith fuzzy possibility distribution. Wedefine the likelihood of the valid element concentrationfor a fuzzy element concentration in the similarway as Wang et al. (2002) defined the job completionlikelihood profile for the fuzzy processing timein the machine scheduling context.Let denote the likelihood of the valid elementconcentration for a fuzzy element concentration . For a given , represents the likelihoodof its element concentration to be valid withina certain allocated upper limit y (variable) for theproduct standard. If the element concentrationis invalid and =0.If thenthe element concentration is completely valid = 1. When should vary123

from 0 to 1 based on an increasing function. Wewish that is a continuousbijective mapping, i.e. there is only element concentrationin corresponding to a given aspirationlevel . can be constructedbased on as follows (Figure 4). For a generic ordinary inequality fuzzy constraint where and are L-R fuzzy numbers, and . If afuzzy constraint is interpreted as a tolerance constraint,then crisp equivalence is given below(Dubois & Prade 1980). μ Figure 4. The likelihood of the valid element concentrationfor a fuzzy element concentration .Our application background requires that for the chance constraints. This meansthat Then the crisp equivalence of chance constraints (8)based on (13) and (18) is given below. ∈∈ where the first, second, and third constraints imposethe constraints in terms of mean, right and leftspread respectively. This means that one fuzzy constraintin principle should be transformed into threecrisp constraints.To enforce the right spread constraint in (21), wecan combine the first and second constraints in (21)and obtain To enforce the left spread constraint in (21), we cancombine the first and third constraints in (21) andobtainwhere β i is a scaling factor. However, simultaneously enforcing the three constraintsin (21) may cause inconsistence; especiallythe third constraint may conflict with the first twobecause the third constraint imposes the restrictionin the opposite direction of the first two (22). For thecurrent charge problem, we can choose not to enforcethe spread constraints, which we will discuss alittle bit later.If the constraint (20) is an equality constraint, it canbe weakened to in the sense of Zadeh(Dubois & Prade 1980). The crisp equivalenceis similar to (21) with mean constraint (the first constraint)enforced as an equality constraint and theother two constraints remaining unchanged. Now wediscuss the transformations of constraints (7) and (9)for the scrap charge problem.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 124

In terms of constraints (7), when the right spreadconstraint (22) applies, it is in fact the chance constraint(19) with the aspiration level . Thisis the strictest case of the constraint (19), implyingthat enforcing right spread constraints make thechance constraint lose the flexibility to control theupper limit of the element concentration based ondifferent aspiration levels. Therefore, we choose notto enforce the right spread constraint. The leftspread constraint for constraints (7) is used to enforcethe lower limit of the element concentration.However, the lower limits are not critical.Moreover, enforcing the left spread constraints canonly results in shrinking the feasible region and thusincreasing the cost of raw materials. Therefore, theleft spread constraints are not enforced either. Thenconstraints (7) can be transformed based on themean constraints only. ∈ ∈ In terms of equality constraints (9), it is unnecessaryto enforce the spread constraints because addingpure alloying elements does not introduce new uncertainty.That is, we only need to enforce the meanconstraints For thebatch k, the product is up to the standard as long asthe concentration for element i falls between thelower limit and upper limit . Therefore,we can weaken the mean constraints to ∈ ∈ In practice, it is unnecessary to worry about the upperlimit because the high cost of pure alloyingmaterials forces the consumption of the purealloying materials to be minimized.Finally, the crisp equivalence of the original fuzzyprogramming model (6)-(12) becomes the objectivefunction (6) in conjunction with constraints (19),(24), (25) and (10)-(12). We can see that the numberof the constraints in crisp model does not increase ascompared with that in the original fuzzy model becausewe drop the unnecessary constraints for interpretationof strict tolerance based on the applicationcontext. The crisp equivalence bears some similaritywith the model resulting from adding the safetymargins for element concentrations for the raw materials(Lahdelma et al. 1999). For example, the interpretationof the chance constraints (19) is similarto the effect of adding safety margins for elementconcentrations for raw materials. However, differencesexist between these two approaches. Addingsafety margins is an approach to extending the deterministicmodel to accommodate uncertainty andthe complicated penalty cost structure to punish thepossible deviation of an intermediate target in EAFis kept in the extended model. The fuzzy model directlyaddresses the uncertainty and each constraintin the crisp equivalence has a clear link with thefuzzy constraint. Consequently, the crisp equivalenceof the fuzzy model is more compact than theextended deterministic model based on addingsafety margins because no intermediate target andpenalty cost is introduced in the interpretation offuzzy constraints. That is, fuzzy approach is morestraightforward. The crisp equivalence of the fuzzymodel is a standard LP model which can be solvedby the standard LP solver.We have tested our model with modified processdata from a Finnish steel plant for stainless steelproduction. The aim of the experiments is to reproducethe products based on the crisp equivalentmodel using the available materials and evaluate theuncertainty of element concentration on failure risk(failure rate) and material cost. The stainlessproducts contain five main elements: Iron (Fe),Manganese (Mn), Chromium (Cr), NickelNiandMolybdenum (Mo). The candidate raw materialscontain at least one of above five elements. To guaranteethe existence of a feasible solution, we assumethat the supply of pure materials that consist of onlyone of above five elements is unlimited. The concentrationsof alloying elements for raw materialsare represented by the mean and standard deviation.The concentrations of alloying elements for finalproducts are specified by lower and upper limits.Table 1 gives the concentrations of alloying elementsfor some raw materials and Table 2 gives theconcentrations of alloying elements for some finalproducts.Table 1 Concentrations of alloying elements forsome raw materialsMaterial Element concentrations (%)Mn Cr Ni MoFeCr-1 ± ± ± ±FeNi ± ± ± ±Ferrite -scrap ± ± ± ±LC-scrap ± ± ± ±Acid-proofscrapScandust scrap±±±±±±± ±Table 2 Concentrations of alloying elements forsome final productsProduct Element concentrations (%)Mn Cr Ni Mo710-2 [1,1.4] [16.5,16.9] [6.35,6.55] [0.65,0.8]720-1 [1.6,1.8] [18,18.4] [8.5,8.65] [0,0.4]720-4 [1.4,1.6] [18, 18.4] [10,10.2] [0,0.4]731-1 [1.6,1.8] [17,17.4] [9,9.15] [0,0.4]750-1 [1.4,1.6] [16.7,17.1] [11,11.15] [2,2.2]757-2 [1.5,1.8] [16.6,17] [10.5,10.7] [2.5,2.7]Next, we investigate failure risk and material costbased on stochastic simulation. The specific proceduresare given below.125

First, the fuzzy parameters such as the element concentrationsfor products and raw materials are realized.The realization of the concentration of elementi for product k , , isstraightforward. ,and correspond to thelower and upper limits for element concentrationspecified in product standard respectively, and . For the concentration of elementi for raw material j, ,we assume that element concentrations follow normaldistribution and choose suitable confidencefactors n 1 and n 2. Then can be realized basedon the mean and standard deviation of element concentrationsfor raw materials as discussed in Section2.Second, for a fixed aspiration level forelement i in product k, we solve the crisp equivalenceof the fuzzy model presented in Section 4 andobtain the selection of raw materials and relatedamount for producing a set of products.Finally, stochastic simulations are performed. Randomelement concentrations for scrap are generatedbased on normal distribution. Then based on the materialselection and related amount in the secondstep, the contribution of the scrap to the alloyingelements in the product is computed. If the scrapcontribution has already exceeded the upper limit ofthe product standard, then the failure is recorded.Otherwise, then amount of pure alloying materials isdetermined and the cost of raw materials is computed.We reproduce 11 products using 17 raw materials(13 scraps + 4 pure alloying materials). We test twoscenarios with different uncertainty degrees. The uncertaintydegree is represented by the ratio of meanand standard deviation of the element concentrations.In the first scenario (S1), the maximum uncertaintydegree among all the element concentrationsin all materials is moderate and about 30%. Inthe second scenario (S2), the uncertainty degree forall the element concentration in all materials isdoubled as compared with that of the first one. Thatis, the mean of each element concentration is sameas that of the first scenario while the standard deviationof each element concentration is doubled ascompared with that of the first scenario. We choose6 aspiration levels and set confidence factors to 3.For each aspiration level in each scenario, ten turnsof simulations are run. In each turn, 10 6 sets of randomelement concentrations are generated for theselected materials and then the failure rate and materialcost is computed based on 10 6 outcomes. Thenwe aggregate the results from ten turns of simulationto obtain average failure rate and material costs.Table 3 gives the average failure rate (FR) and normalizedcost (NC) for all of raw materials. The normalizedcost uses the cost of the total raw materialsfor aspiration level λ=1 in the first scenario as reference.Table 3 Average failure rate (FR, percentage) andnormalized cost (NC, percentage) for two scenariosat different aspiration levels.λ S1 S2FR NC FR NC0.75 21.96 98.49 24.45 100.610.8 11.81 98.63 10.53 101.080.85 5.41 98.84 4.71 101.520.9 2.20 99.17 1.80 101.920.95 0.76 99.58 0.61 102.291 0.10 100 0.25 102.63Based on Table 3, the failure rate can be controlledby the proper combination of confidence factors (n 2)and aspiration levels (λ) regardless of the uncertaintydegree. Here we can induce that the failurerate is unacceptable when λ

0.8 29.02 9 31.59 50.85 29.40 8 32.12 50.9 29.68 6 32.57 50.95 30.02 5 32.96 51 30.55 5 33.30 5Finally we discuss the mechanism of controlling thefailure rate. The model reallocates the materialsbased on the aspiration level. The reallocation includesthe increase/decrease of scrap types and adjustmentof the amount used for each material. Table4 shows cost share (CS) of pure alloying materialsand the number of different types of scraps selectedfor different scenarios at different aspiration levels.On average we can see that cost share for the purealloying materials increases by about 1.15% fromfailure rate about 5% (aspiration level 0.85) to lessthan 0.3% (aspiration level 1) for both scenarios.When the uncertainty degree doubles (from S1 toS2) the cost share increases by about 2.5% for thesame order of failure rate. Here the cost share variationfor the pure alloying materials is coincidentwith the overall material cost variation shown inTable 3. This implies that the material cost increasefor reducing the failure rate mainly attributes to theshare increase of the pure alloying materials. Basedon the number of scrap types selected for products,we can see the material selection can be widened asthe uncertainty degree decreases.The mechanism to control the failure rate is to increasethe share of pure alloying materials in a costefficientway. For the same scenario, when the aspirationlevel decreases, the decrease in cost share ofalloying materials means the increase in share of thescrap. This in turn means the overall uncertainty forthe selected material increases. This results in simultaneousincrease of the failure rate and decrease ofthe material cost because scrap is much cheaper thanpure alloying material. For different scenarios, asthe uncertainty degree increases, the share of thepure alloying materials must increase to realize thesame the order of failure rate.Above simulation results can provide severalguidelines for the practical production. First, for therough estimates of the element concentrations, alittle bit larger uncertainty degree estimation can beused to produce the initial selection of scrap mixwith less failure risk. Second, the scrap mix can bereselected with less material cost based on more preciseestimation for element concentration as moreprocess data are available. Finally, the scrap selectionbased on more reliable concentration estimationfrom large quantities of process data can be used toguide the material purchase.Handling the scrap is one of the most importantfunctions for scrap-based steelmaking. Using thewrong grades of scrap will result in higher raw materialcosts and difficulty in meeting product standard.The objective of the scrap charge optimizationis to select the most economical scrap mix for eachproduced steel grade while minimizing the failurerisk caused by uncertainties in raw material consistence.In this paper, we first presented the fuzzychance constrained model for the scrap charge optimizationbased on product recipe and then transformedit into a crisp model based on the applicationcontext for solution. Simulation shows that the failurerisk can be hedged by the proper combination ofthe aspiration levels and confidence factors for representingthe fuzzy number. The mechanism of controllingthe failure risk against uncertainty is to increasethe share of the expensive pure alloying materialsin the controlled manner. Therefore, there is atradeoff between controlling failure risk and increaseof the material cost. The model can be usedfor both determining the scrap mix for short-termoperation of the melting shops based on availablematerials and for guiding the material purchase forlong-term planning. The model can also be appliedin other scrap-based production processes such asaluminum and copper (Lahdelma 1998). 127

New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 128

Simulating processes of language emergence, communicationand agent modelingTimo Honkela, Ville Könönen, Tiina Lindh-Knuutila and Mari-Sanna PaukkeriAdaptive Informatics Research CentreHelsinki University of Technology, Finlandfirstname.lastname@tkk.fiAbstractWe discuss two different approaches for modeling other agents in multiagent systems. One approach isbased on language between agents and modeling their cognitive processes. Another approach utilizesgame theory and is based on modeling utilities of other agents. In both cases, we discuss how differentmachine learning paradigms can be utilized for acquiring experience from the environment and otheragents.1 IntroductionIn this paper, we discuss how the emergence of subjectivemodels of the world can be simulated usingdifferent approaches in learning and what is the roleof communication and language. We consider, inparticular, the role of unsupervised learning in theformation of agents’ conceptual models, the originalsubjectivity of these models, the communication andlearning processes that lead into intersubjective sharingof concepts, and a game theoretical approach tomultiagent systems including reinforcement learningprocesses.An intelligent agent usually has a purpose or agoal, probably set by the designer of the agent, andthe agent tries to act rationally for satisfying the goal.For making rational decisions, the agent has to modelits environment, perhaps including other agents. Inthe following, we consider some specific issues relatedto the modeling.1.1 Complex and non-stationary environmentsIn the easiest case, the environment in which theagent is located is static, i.e. all properties of the environmentremain constant for the whole life-time ofthe agent. However, often the situation is not so simple.The properties of the environment may vary withtime, i.e. the environment is called non-stationary.The non-stationarity may be due to the environmentitself or the limited resources of the agent. For example,in many real problem instances, the learningagent is not capable to sense the real state of the environmentand thus some relevant properties of theenvironment remain hidden.Another example of non-stationarity are multiagentsystems. In these systems, properties of the environmentfor each individual agent depend on theactions of all agents located in the real environment.Thus for acting rationally, agents should model alsoother agents in the system.Pfeifer and Scheier (1999) stress that the behaviorof an agent is always the result of systemenvironmentinteraction. It cannot be explained onthe basis of internal mechanisms only. They illustratethat the complexity that we as observers attribute toa particular behavior does not always indicate accuratelythe complexity of the underlying mechanisms.Experiments with very simple robots that merely reactto stimuli in their environment have shown thatrather complex behavior can emerge.1.2 Subjective versus intersubjectiveMoore and Carling (1988) state that “[l]anguages arein some respect like maps. If each of us sees theworld from our particular perspective, then an individual’slanguage is, in a sense, like a map of theirworld. Trying to understand another person is liketrying to read a map, their map, a map of the worldfrom their perspective.” In many computational approachesto semantics and conceptual modeling, anobjective point of view has been used: It is assumedthat all the agents have a shared understanding andrepresentation of the domain of discourse. However,Moore and Carling’s statement emphasizes the needfor explicit modeling of the other’s point of view.129

This requires modeling of the subjective use of languagebased on examples, and, furthermore, to modelintersubjectivity, i.e. to have a model of the contentsof other subjective models.As an example related to the vocabulary problem,two persons may have different conceptual or terminological“density” of the topic under consideration.A layman, for instance, is likely to describe a phenomenonin general terms whereas an expert usesmore specific terms.1.3 Learning agentsDifferent machine learning techniques can be utilizedfor adding adaptivity to agent-based systems. Thereare basically three major learning paradigms in machinelearning: supervised learning, unsupervisedlearning and reinforcement learning. In supervisedlearning, there exists a teacher having knowledge ofthe environment, in the form of input-output pairs,and the learning system for which the environmentis unknown. The teacher provides samples from theenvironment by giving correct outputs to inputs andthe goal of the learning system is to learn to emulatethe teacher and to generalize the samples to unseendata. In unsupervised learning, contrary to supervisedlearning, there exists no external teacher andtherefore no correct outputs are provided. Reinforcementlearning lies between supervised and unsupervisedlearning: Correct answers are not provided directlyto the learning system but the features of theenvironment are learned by continuously interactingwith it. The learning system takes actions in the environmentand receives reward signals from the environmentcorresponding to these action selections.2 Unsupervised learning of conceptualsystemsIn the following, we consider the unsupervised learningof conceptual systems. We first study the modelingan individual agent that learns to create a conceptualspace of its own and learns to associate wordsand expressions with conceptual space. The notion ofconceptual space is taken from Gärdenfors who alsopresents the basic motivation and framework for dealingwith conceptual spaces (Gärdenfors, 2000). Afterconsidering one individual agent, we consider a multiagentsystem in which a shared conceptual systemis formed in a self-organized manner.2.1 One agentThe Self-Organizing Map (SOM) (Kohonen, 1982,2001) is an unsupervised learning model whichis often considered as an artificial neural networkmodel, especially of the experimentally found ordered“maps” in the cortex. The SOM can be used asa central component of a simulation model in whichan agent learns a conceptual space, e.g. based on datain which words are “experienced” in their due contexts(Ritter and Kohonen, 1989). This approach hasbeen used, for instance, to analyze the collection ofGrimm fairy tales. In the resulting map, there are areasof categories such as verbs and nouns. Within thearea of nouns a distinction between animate and inanimatenouns emerges (Honkela et al., 1995). The basicidea is that an agent is able to form autonomouslya conceptual mapping of the input based on the inputitself.2.2 Community of agentsThe basic approach how autonomous agents couldlearn to communicate and form an internal model ofthe environment applying self-organizing map algorithmwas introduced, in a simple form, in (Honkela,1993). Later we developed further the framework thatwould enable modeling the degree of conceptual autonomyof natural and artificial agents (Honkela et al.,2003). The basic claim was that the aspects relatedto learning and communication necessitate adaptiveagents that are partially autonomous. We demonstratedhow the partial conceptual autonomy can beobtained through a self-organization process. The inputfor the agents consists of perceptions of the environment,expressions communicated by other agentsas well as the recognized identities of other agents(Honkela et al., 2003). A preliminary implementationof a simulated community of communicating agentsbased on these ideas did not succeed to fully demonstratethe emergence of a shared language (Honkelaand Winter, 2003).When language games (Wittgenstein, 1953) wasincluded in the simulation model, it resulted in a simplelanguage emerging in a population of communicatingautonomous agents (Lindh-Knuutila, 2005).In this population, each agent was able to createtheir own associations between the conceptual leveland the emerged words, although each agent hada slightly different conceptual representation of theworld. The learning paradigm for the conceptuallearning was fundamentally unsupervised, but thelanguage learning tested has been so far supervised,i.e. the communicating agents are provided feedbackNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 130

of the outcome of the game as well as the “right answer”.The reinforcement learning and the unsupervisedlearning models for language games remain tobe implemented.3 Game theoretical approach formultiagent systemsAs discussed in Section 1, an intelligent agent usuallyhas a goal. For studying agent based systemsformally, e.g. for developing learning algorithms forintelligent agents, it is useful that the goal can be expressedmathematically. Traditionally approach is todefine an utility function for the agent, i.e. there is ascalar value connected to each possible action measuringthe fitness of the action choice for satisfyingthe goal of the agent. The utility function is ofteninitially unknown and must be learned by interactingwith the environment. Machine learning techniquescan be utilized in this learning process, consider e.g.(Könönen, 2004b; Könönen and Oja, 2004; Könönen,2004a).3.1 One agentIn single-agent systems, achieving the rational behavioris simple: the agent always selects an action withthe highest utility value. There exists a vast number oflearning methods that utilize, in one way or another,the rational decision making in agent-based systems.3.2 Community of agentsThe branch of science studying decision making insingle-agent systems is called decision theory. However,when there exists multiple active decision makers(agents) in the same environment, decision theoryis not suitable for achieving rational behavior anymore. Game theory is an extension of decision theoryto multiagent systems. In game theory, agents explicitlymodel the dependency of their utility functionson the actions of all agents in the system. The goal ofthe agents is to find equilibrium actions, i.e. the actionsthat maximize their utility values assuming theall agents will use the same equilibrium. Jäger (2006)applies game theory to examine the shape formationin the conceptual spaces of the agents (Gärdenfors,2000).Theoretically, game theory provides a perfect toolfor modeling multiagent systems. In practice, thereare many problems with game theory. For examplethere can exist multiple equilibria and the agentsshould coordinate which one they will select. For calculatingan equilibrium, the agents should not onlymodel their own utility function but also the functionsof all other agents in the system. This is oftenintractable and therefore some other methods,e.g. Bayesian techniques, should be used for creatingmore coarser models of other agents. By usingthese “external” models, the game theoretical problemreduces to the simpler problem solvable by usingdecision theory.4 DiscussionLanguage does not need to be viewed plainly as ameans for labeling the world but as an instrument bywhich the society and the individuals within it constructa model of the world. The world is continuousand changing. Thus, the language is a medium of abstractionrather than a tool for creation and mediationof an accurate “picture” of the world. The point ofview chosen is always subject to some criteria of relevanceor usefulness. This is not only true for theindividual expressions that are used in communicationbut also concerns the creation or emergence ofconceptual systems. It makes sense to make such distinctionsin a language that are useful in one way another.In this paper, we have discussed the role of unsupervisedlearning in the formation of agents’ conceptualmodels in a non-stationary environment and agame theoretical approach for multiagent systems. Inthe future, we will study game theoretical models forlanguage games and how to apply machine learningapproaches for solving these games.ReferencesPeter Gärdenfors. Conceptual Spaces. MIT Press,2000.Timo Honkela. Neural nets that discuss: a generalmodel of communication based on self-organizingmaps. In S. Gielen and B. Kappen, editors, Proceedingsof ICANN’93, International Conferenceon Artificial Neural Networks, pages 408–411,Amsterdam, the Netherlands, September 1993.Springer-Verlag, London.Timo Honkela and Juha Winter. Simulating languagelearning in community of agents using selforganizingmaps. Technical Report A71, HelsinkiUniversity of Technology, Laboratory of Computerand Information Science, Espoo, Finland, 2003.131

Timo Honkela, Ville Pulkki, and Teuvo Kohonen.Contextual relations of words in Grimm tales analyzedby self-organizing map. In Proceedings ofICANN-95, International Conference on ArtificialNeural Networks, volume 2, pages 3–7. EC2 etCie, 1995.Timo Honkela, Kevin I. Hynnä, and Tarja Knuuttila.Framework for modeling partial conceptualautonomy of adaptive and communicating agents.In Proceedings of Cognitive Science Conference,2003.Gerhard Jäger. Convex meanings and evolutionarystability. In Proceedings of 6th Int. Conf. on theEvolution of Language, 2006.Teuvo Kohonen. Self-Organizing Maps. Springer,Berlin, Heidelberg, 2001.Teuvo Kohonen. Self-organizing formation of topologicallycorrect feature maps. Biological Cybernetics,43(1):59–69, 1982.Ville J. Könönen. Policy gradient method for teamMarkov games. In Proceedings of the Fifth InternationalConference on Intelligent Data Engineeringand Automated Learning (IDEAL-2004), pages733–739, Exeter, UK, 2004a.Ville J. Könönen. Asymmetric multiagent reinforcementlearning. Web Intelligence and Agent Systems:An International Journal (WIAS), 2(2):105–121, 2004b.Ville J. Könönen and Erkki Oja. Asymmetric multiagentreinforcement learning in pricing applications.In Proceedings of the International JointConference on Neural Networks (IJCNN-2004),pages 1097–1102, Budapest, Hungary, 2004.Tiina Lindh-Knuutila. Simulating the emergence ofa shared conceptual system in a multi-agent environment.Master’s thesis, Helsinki University ofTechnology, Laboratory of Computer and InformationScience, 2005.Terence Moore and Chris Carling. The Limitations ofLanguage. Macmillan Press, Houndmills, 1988.Rolf Pfeifer and Christian Scheier. UnderstandingIntelligence. MIT Press, Cambridge, MA, 1999.Helge Ritter and Teuvo Kohonen. Self-organizing semanticmaps. Biological Cybernetics, 61(4):241–254, 1989.Ludwig Wittgenstein. Philosophical Investigations.Blackwell Publishers, Oxford, 1953.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 132

Program Comprehension Theories andProlog based MethodologiesErkki LaitilaA Department of Mathematical Information Technology,University of Jyväskylä, P.O. Box 35,FIN-40014 Jyväskyläerkki.laitila@swmaster.fiSwMaster OySääksmäentie 14, 40520 JKLAbstractSoftware maintenance is said to account for more than 50 % of all software efforts. Of this the attemptsto understand the code can take 70 %. In spite of its importance, program comprehension isnot well understood. This paper tells how Prolog can be used in modeling source code. An essentiallynew method, symbolic analysis, is presented and compared to static and dynamic analyses,which form the bulk of the current practices . We will show how a multi-paradigm tool, VisualProlog, can serve as an excellent model formulation tool in describing complex source code to covertypical program understanding models1 IntroductionProgrammers are frequently struggling with programcomprehension (PC) tasks when planning andpreparing changes into code. A typical maintenancetask is a confirmed change request. Before thechange can safely be implemented, the code must atfirst be understood (von Mayrhauser 97).Although a number of new technologies havebeen created since the 70’s, writing object orientedsource code is not easy. Bezevin has described developmentdifficulties in his illustratively (05). Animportant theme in his document is the impedancemismatch between business and software. Problemswriting code the problems in understanding it becauseof the mismatch.A number of theories have been written for understandingprocedural programs (Pennington 87)and a few for understanding object-oriented programs(Burlington 02). The integrated mental modelis a well-established framework with three approaches:1) top-down model, 2) bottom-up modeland 3) situation model that refers to domain information,too (von Mayrhauser 97).Figure 1: An analogy to program bindings.Figure 1 illustrates molecular structures with bindings.It resembles source code in that both it also hasbindings. The most important point in software is tounderstand how to match different approaches likedomain information and program flows, and plansbehind them. Object-oriented code causes someproblems of its own, because invocations are hiddenin modules. Further, dynamic bindings of objects arenot visible, they must be traced step-by-step byreading the code.133

Maintenance Comprehensiontasks to beautomatedPostitioning incode analysis scienceStaticDynamicVerificationResults tomaintenanceContributionFormulating• Modularity: each model should be dividedinto smaller units.• Transformability: the model should allowtransformation for other purposes.• Executability: the model and possibly thecode should be executable for verificationpurposes.NewformulationNewsymbolicapproachFigure 2. Symbolic approach.We will build a unified concept framework startingfrom maintenance (Figure 2) to connect existingcomprehension requirements applying strong reductionismby using Visual Prolog in the formulation.This new symbolic approach, or rather a symbolicmodel, is comparable with static and dynamicanalyses.This paper is organized as follows. Section 2 describesrelated work while Section 3 considers theselected approach. Section 4 presents a layered architectureof a symbolic model for tools. Section 5contains the essential definitions. Section 6 presentsa small Java example and Section 7 the architectureshortly. Section 8 describes a tool named JavaMasterand Section 9 Visual Prolog understanding ingeneral. Finally, Section 10 concludes the paper.2 Source code modelsVisualPrologWidely used methods for PC include reverse engineering,UML–diagrams, document browsing, andcode exploration by editor. These methods providestatic information. Dynamic behavior information ismuch harder to capture, because capturing it requiresdebugging the complete installation.The focus of software science is on developingmodels. It has been argued that the paradigm of object-orientedprogramming is to be replaced by aparadigm of model engineering (Bezevin 2005). Themain idea of model driven architectures (MDA2005) is summarized as “everything is a model”.However, there are many modeling technologies andno consensus on how to use them. Some alternativesare grammarware, xml, and ontologies, includingsemantic web and MDA.OMG has, as a non-scientific organization, presentedthe main requirements for a modeling evaluationframework (OMG 05):• Traceability: data for the model should betraceable to its original source.The most detailed model, FAMIX (Famix 1999),was created in a large project. Unfortunately, it containsonly a static part. Dynamic models have notbeen implemented to cover all the source code. Oneessential problem in describing dynamic models inclass based architectures is how to express logicconnections between elements. The best way, aclass-oriented architecture, like Java or C++, canconnect things is by creating an association classcombined with two association end objects. Thisproduces two excess class types and the correspondentobjects. It makes modeling in MDA very complex.3 Research focusFor the research we set a goal to define a unifiedconcept to find a model between written code and itsbehavior in order to match these with user domaininformation (Figure 3). In this figure, the X-axisrepresents written code (what), the Y-axis its objectbasedbehavior (when), and the Z-axis user concepts(how).If we mark the syntax by the letter X, we can use thenotation Y = f (X) to describe a control flow startingfrom any starting symbol X. A control flow containsa group of methods calling each other with dependenciesincluded:Y = y(Start) = f1 • f2 • • Target.3.1 Control flow is essentialThe control flow y(Start) builds a semantic behaviormodel for a current situation (Section 5, later). Byusing this invocation vector as input, all interestinginformation relating to the referred methods can becollected from their statements transitively. Thisinformation, when filtered and grouped according touser options, builds a pragmatic high abstractionsituation model (von Mayrhauser 97). It will create anew projection Z that is in fact a group of candidatesthat are explaining the selected use case. A candidateis any symbolic name written by the programmer,like a class, a method or a domain name suchas bluetooth or a database symbol if these terms aremapped into a model dictionary.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 134

As a result we will get a useful mapping betweenall the dimensions because the tool can visualize theresults in many ways, for example resembling Figure1.ZYUser domainHow ?Dynamic behaviorWhen?Static codeWhat?XFigure 3. Reductionist research model.3.2 Reasons for selectionsWhy Java was selected as a “patient”, a language tobe analyzed? It is a popular and much studied programminglanguage but can suffer from comprehensionproblems. So showing what the research cancontribute to Java, in respect to the problem pointedat, has essential value for the research.Why Visual Prolog was selected as a “doctor”,a language to implement the tool? The reason wasthat Java models contain a great amount of heavylogic and object-based information, so a multiparadigmtool is needed. There are not many suchtools.4 Program ComprehensionarchitectureThere are numerous design patterns for creating datamodels and for interactive systems. Because of itscomplex structures, the most essential implementationrequirement for reverse engineering is layeredarchitecture. For our work we have selected a combinationof two existing architecture models: MDAand PCMEF.MDA defines the following levels:• M0 - source code files.• M1 - parsed structures as classes.• M2 - a model based on class definitions.• M3 - a meta meta model where the most essentialterm is a definition element. It is especiallysuitable for transformation.The second reference architecture, PCMEF, has thefollowing practical structure:• Foundation (F) is the base of the model, it definesthe notation for information• Entity (E) is the unit to make the model• Mediator (M) is a facade to the model classesthat refer to user interface.• Controller (C) is a feature to control the actionsand events of the tool.• Presentation (P) is the display side of the userinterface.We combined these two methods to connect codeand models into maintenance:M0M1M2M3M4M5M6M7Table 1. Resulting Architecture Layers.Java files and packages in diskParse trees as Java semanticsHigher level notation, a symbolic language asa foundation (F)Complete symbolic model for the code withsymbolic elements (E)Mediator (M) to master the modelController (C) to implement messagingUser interface for solving user hypotheses asmodel queries.Presentation (P) in visualizing maintenancetask information.5 Definitions for symbolic modelTraditional programming languages work numerically.Numeric evaluation means that the programcalculates terms that are directly compatible withlanguage types (Sun 03). Therefore it cannot handleunknown information, external symbols or interfaces.Contrary to this, using unknown external information,like a variable X, as a reference is not aproblem in symbolic evaluation.5.1 Symbolic analysisSymbolic evaluation of code was first described byCheatham et al (Cheatham 1979). We define symbolicanalysis as a process to simulate originalsource code to obtain symbolic results for programcomprehension purposes. The following list containsthe mapping from the analysis into VisualProlog:1. Symbolic term is an individual Vip term.2. Symbolic definition is a Vip clause to definerelations between terms.3. Symbolic evaluation is a task (goal or subgoal)to unify and solve the values for asymbolic definition.4. Simulation is a process to execute successivesymbolic evaluations for a query.135

5. Symbolic result set is a set of outputs.6. Symbolic presentation is a view to displaysymbolic results.5.2 Source codeUML has defined a large framework for objectorienteddevelopment (Arlow, Neustadt 2002). It istoo complex to be used for evaluation. So the smallestpossible definition for Java information isneeded for evaluation purposes in order to reachmaximal throughput.A simplified set for UML can be defined to be aJava information model. It contains the followingterms: element, message, influence, block and resultset. The main benefit of this simplified approach,compared with UML, is that symbolic informationcan be evaluated intensively. Furthermore, becausethe elements contain their type information, thesecan be translated into UML diagrams (class, stateand sequence diagrams). This is an excellent tool forporting and verification purposes.6 Java exampleFigure 4 gives is a very small excerpt from the beginningof a typical Java program.class Server {main(String[] args) {if (port == 0)port = Int.parseInt(args[0])new Server(port);}}The call tree is important for tracing other features,too, because it defines active code and causal orders.Some main benefits of symbolic analysis are partialevaluation, simulation of selected methods and indeterministic,exhaustive search of terms includingsimulation of dynamic bindings.7 ImplementationIn this Section a data flow from code into user interfaceis described.7.1 Layer M0: LanguageIn order to write a source code tool a grammar isneeded. For Java there is an excellent specificationmade by Sun (Sun 2003). Grammars define syntaxand any valid type of a parse tree for each program.The structure of a parse tree corresponds to the orderin which different parts of the program are executed.Thus, grammars contribute to the definition of semantics(Webber 2005).7.2 Grammar technologyPDC has created a grammar notation, GRM, and aparser generator for PDC Prolog described in theToolbox (PDC Toolbox 1990). It is based on differencelists and top-down parsing. GRM-notationmakes it possible to seamlessly combine syntax andwanted semantics of each term. Still, it is the responsibility/possibilityof the user to define the semanticsfor parsing orders, for example, correctly(Laitila 1996).In exploring code, programmers use a short set ofquestions (Letovksy 86) and alternative approaches(Pennington 1987). The most typical questions beingasked are: what, why and how. Let us have someexamples:• What does class Server do?• Why did constructor Server start?• How does it work?The most useful way to start investigating this is tofollow the main call tree (Fig 4).Figure 4. Beginning of call tree for class Server.Figure 5. SwToolFactory: Universal generator.SwMaster Oy has developed a Visual Prolog –based tool named SwToolFactory (Fig 5). It is completelyobject-oriented and contains the followingfeatures.• Class MetaGrammar contains common featuresfor a formal grammar.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 136

• Class MetaLanguage contains languagedependant tools and features.• Object MetaTerm instantiates grammarterms and features like a parser generator.• There is a collection of numerous codegenerators like a pretty printer generatorand generators for XML, AST etc.In various reengineering projects for C, Cobol, VBand C++, SwMaster has created a unified conceptframework (Figure 6) for formal languages as follows.For reading code the features needed are scanner(1), parser (2) and a saving element (3): either amodel weaver or a simple symbol table. The code isstored into Visual Prolog as hierarchical facts or assymbolic objects (8).The code information can be re-translated intosyntax by using three features: a model iterator (4),a code generator (5) and a pretty printer (6). Thereare a number of mathematical correspondences betweenthe parts of this “diamond”. A very interestingfeature created by this structure is the possibilityto make translations from one language into anotherby switching the terms in the input side (7) and theoutput side (8). By combining (7) and (8) from differentlanguages we can form a term translator.1 2 37 8 96 5 4Figure 6. Diamond, a unified concept frameworkand programming model for formal languages.This design concept is useful in many kinds of VisualProlog transformation tasks. As an example of acode generator for a pretty printer is presented. Theshortest Visual Prolog code for a pretty printer canbe defined as a concatenated list of syntax terms andrecursive invocations of lower level calls in the syntaxorder (below, gen-prefix is used). A recursiveterm generator gen_Statement for the for-statementof Java1.5 is shortly:gen_Statement(for(ForControl,Statement)) =concatList(["for", "(",gen_ForControl(ForControl), ")",gen_Statement(Statement)]).7.3 Defining semanticsFor defining semantics of source code numerousprinciples have been developed including denotationalsemantics and attribute grammars. The bestapproach for Prolog is argued to be natural semantics(Sethi 1996), which gives us an excellent tool toinvestigate the validity of Prolog implementations.An if-command is a good example. In the GRMgrammarnotation:Statement =“if” Condition then Statement ->iff(Condition, Statement).A Prolog based analyzer could provide a numberof different ways to analyze iff-statements: subtermsearch, deterministic evaluation, code simulation,indeterministic coverage test, traversal of evaluatingresults, information for displays, model making etc(Heering 2005). Overall, natural semantics is a goodway to document and to validate these sub-models.7.4 Layer M1: Parse tree for JavaWhen GRM-notation is used, each term has a definedparse tree that is defined in the right side of theterm. The start symbol term of Java3 is a compilationunit. The other main Java terms for analysis areclass declaration, type declaration, statement, expression,identifier and literal.7.5 Layer M2: Symbolic languageIn a normal practice, parse trees are converted intoAbstract Syntax Trees (AST) for analysis purposes(VMCAI - konferenssi 2005). However, ASTs sufferfrom many problems because of their strongconnection with the original syntax. This is why weuse another way to raise the abstraction level. Webuilt a completely new language, Symbolic, insteadof using Java itself. This simplifies programmingand it makes it possible to optimize code items accordingto analysis needs. In the symbolic notationthe nearest equivalent for Java statement is clause.Table 2 below describes categories, identification,purpose and principal use for the term clause.Table 2. Categories for symbolic clausesId Purpose Principal usedef Definition Identification, typescrea Creator Dynamic bindingsset Changes Data flowget Invocation Control flowref Reference Variable referencesop Operation Math & relat operatorsloop Loop Loop analysispath Condition State analysisval Constant Evaluating valuesother Non-core term Terms, not important137

We defined Symbolic to be an independent language,so that it has all the “diamond features” (Figure6). It is a class with its domains, parsers andgenerators. Symbolic is particularly aimed at evaluation,display and explanation purposes.For translation from Java into Symbolic eachJava statement is converted into one or more symbolicclauses depending on the statement. So a symbolicclause is not backwards compatible, but whenthe context of successive symbolic clauses maintainsa correct sequence, the clauses can be retranslatedinto Java with their original meaning.This is useful for program specialization purposes.7.6 Symbolic translationVip6 has an excellent function notation for translationpurposes. Below, a translation rule to translate aJava if-statement into a pathClause of Symbolic isshown. The predicate returns the term in anotherlanguage. Condit is an abbreviation for conditionand SL for statement list. Translation takes place atthe Prolog level, and not by replacing source structureslike traditional translators do when using complexexternal rules (TXL, Software Tech. Laboratory2004).statement2clause(iff(Condit,SL)) =pathClause(condition2cond(Condit),stmntList2clauseList(SL)).The translation principle is excellent for the use ofVip applications, especially in multi-layer architectures.It is best to keep the translation programmingapart from the syntax level, due to the complexitiesand peculiarities of the latter.7.7 Capturing control flowsIn Section 1 (Figure 1) it is argued that programsresemble molecular structures. In Section 6 a calltree was presented as a base for models. UML,which covers the vast majority of approaches for thesoftware developer, contains a 4+1 model to summarizeall diagrams (Arlow, Neustadt 2002). In orderto identify all behavior features of UML, a behaviormodel is defined to be a sum of all causalchains (Mellor, Balcer 2002). These are captured by“chopping” successive method invocations (f) forthe selected approach (Figure 7).startf 1 f 2call q f k −1f kstart qf 4f 3(ret) f k − 2targetf 5exit qf k − 3Figure 7. Control flow in chopping (Reps 2000).The most used principle of source code analysis iscalled slicing (Weiser 1984). It means detectingeffects of source code from a certain point. It is notvery useful for program comprehension, because itis only a low-level approach. Chopping is a moregeneral approach (Reps 2000). Chopping is consideredin terms of producing relevant information regardinga part of the program between Start andTarget as determined by the person attempting tounderstand the program. Chopping is essential becauseit can be used in cause-effect analysis which isparticularly relevant in trouble-shooting (vonMayrhauser 1997). The purpose of the followingrule is to solve the chopping equation (Figure 7):Target = fk • fk-1 •... •f (Start).In a symbolic notation it can be either nested:Target = f (f ( ... f( Start ) ) )or successive:Target= [f(XK) •f(XK-1)• f(“Start”)].In Vip, collecting chops (f) could be written:chopping(Target,[f(This)|Sequence]):-f(NextId, Arguments),NextId:chopping(Target, Sequence).7.8 Layer M3: Object-oriented modelInitially the symbolic code for each class was storedinside classes. However, tree based structures aredifficult to analyze, because 1) identifying elementsis not easy and 2) there is no easy way to point tospecified points to trees. Further, the user considersthe code elements as parallel, not as hierarchicalunits. That is why an object-oriented core is neededfor the symbolic model. In our case it is very naturalto use the categories of Symbolic language as thebase for model objects.To implement an object-oriented model, a modelweaver is needed. It divides the parse trees intosmaller element objects as shown in Table 2. Let usconsider a row X in Table 2. The purpose of theSymbolic Model Weaver is to create a Symbolic Element for each X and to store the host intothe parent element and the handle of the parent intothe child element. Each block in the code is replacedby a link element and the replacement is stored intothe parent as a link node. In accordance with theprinciple above, each element should have a minimumnumber of child elements. Because the elementsof each object are in the same order as in theoriginal code, it is guaranteed that elements can beevaluated correctly. This makes it possible to executemodels, which is the most challenging featureof modeling technologies (Section 2). The principleNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 138

uses, extensively, Prolog’s logic notation and VisualProlog’s object notation together. In Figure 8 there isa class diagram illustrating a program implementation.SymbolicElement-parentElement+new()+build()+run()Symbolic Element-contents+new()+build()+specialRun()Figure 8. Principles of symbolic model elements.The super class SymbolicElement inherited from theclass Symbolic, contains the methods to build eachsub-model, to run the sub-model and to traverse theresults. There is a method specialRun that runs thecontents of each element object.7.9 Layer M4: MediatorClass Mediator is a facade to combine the most essentialdata classes and activities. The Symbolicmodel is run via a class ModelEngine that has a runcommandwhich starts a new thread.It is said that hypothesis is the only driver for programunderstanding (von Mayrhauser 97).The userwants to investigate special parts of the software byquestions such as: what, why, when and how. Tospecify a run, a start and target combination is enteredby the user.The results of the query are a collection of Symbolicclauses combined with the query notation. Eachexplanation has the form: Explanation = query +answer.7.10 Layers M5, M6: Controller and UIController receives signals from the model and updatesdisplays according to those signals. UI is describedlater in Section 8.7.11: Layer M7: PresentationsAs stated earlier, each explanation contains thequestion and its handle and the results. By collectingexplanations systematically the user can updatehis/her situation model, which can contain informationlike:• What does class Server do?• Why did X start Y?• How does method Z work?Symbolic clause is the foundation for all displayeddata. In source code analysis theory there areabout 20 different visualization graphs and presentationtypes (Famix 1999). Most of them can be transformedfrom the result set of the symbolic model forJava.8. Tool JavaMasterAccording to the description above, a tool was developed,the development of which started in 2005.The current implementation parses Java 1.5 withoutproblems. The tool contains about 40.000 lines ofcode and 400 classes. Some statistics about thecode:1. Java 1.5 grammar file (300 lines)2. Java parser (2556 lines)3. Symbolic language (90 definition lines)4. Java to Symbolic translator (2200 lines)It is useful to note that the symbolic translator isshorter than Java parser. This shows that semantictranslation is a very expressive feature in code transformation.8.1 Tool featuresFigure 10 shows the principles of the tool. Thesource code is loaded from the tree (1). It provides aprogram model for another tree (2). Then the userselects the function Worklist, for which the tool'sresponse is to display a set of navigation questionsin (4 Q/A). Program information is displayed byabstract controls (3). Here a relatively large call treeis shown. The user sets hypotheses and formulatesqueries related to the suggestions (Q) which willhelp the user in focusing on the most essential control-flowsand objects in order to understand thecritical references and method invocations (Wiedenbecket al 2002).139

1.4Q/A2.3.9.1 Influence modelEach Prolog program can be thought as an activecontrol flow, which has inputs and outputs on severallevels. In Figure 12 inputs (callers) are up andoutputs (callees) down. Pure logic, including temporaryeffects, is in the base line. A vertical distancefor each influence type can be defined as follows:Com-components are the most distant lines as wellas external actions that cannot be repeated. The mostdistant lines are main influences, compared withside effects that are intermediate data and not amongthe wanted results.callerFigure 10. User interface of JavaMaster.9 Prolog comprehensionVisual Prolog resembles Java in most informationneeds (Sections 3 and 5). It contains program flows,classes, objects and dependency models. Some essentialdifferences are non-determinism, strong recursivenature, declarative syntax, and flow analysis.Further, public attributes are not allowed and predicatesare forming much more compact units than inmethods of Java. Most of these differences makeunderstanding easier. From the programmers pointof-viewit would be best not to use facts, becausefacts can cause side effects, but it is not always possibleto avoid this.Understanding side effects is essential in Prolog(Walker et al 1997). In Figure 11 there are twocases, A and B, using predicates p, q and r. Predicatep causes a side effect note(X) to case A but notto B. It has direct influence on predicate q but not onr. Still, if r is called after q then B is dependent ofnote indirectly.Summary: The side effects can be detected completelyby symbolic analysis because the phenomenin Figure 11 is a control flow... p(X) | q(Y) | r(X,Y)assert(note(X))note(X),s(X)s(X)A: Side effects B: No side effectsFigure 11. Prolog’s side effect model.Startcallee• • •Figure 12. Prolog’s influence model.TargetThis model can be useful in planning refactoring forVip classes.9.2 Visual Prolog applicationIn order to understand a complete Vip applicationwe can use the following main approaches (the relatedfigure in parenthesis):• Static: How has it been written? (F. 11)• Dynamic: How does it behave? (F. 12)• Symbolic: What are the explanations for candidatesbetween starts and targets referring to ourfocus? (F. 10)The symbolic approach is the most general one. Itis pragmatic by nature. It could give us useful snapshotsfrom the user’s approach combining the dimensionsof Figure 3!10 ConclusionsA seamless architecture for a symbolic programcomprehension methodology has been described inthis paper. Using Visual Prolog proved to be a rightchoice. When one of the most characteristic featuresof traditional languages is that they are leading intomore and more specific applications, Visual Prologhas, as shown in this paper in Sections 2..9, becauseof its declarative notation, another rather valuablefeature; it is leading into wider, more abstract thinking,thus creating more general approaches andtheories related to the software science.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 140

Symbolic analysis can, because of its theory ofgrammars, semantic translations, symbolic modelsand explanation features, be an important step infuture in getting closer to a grand unified theory ofprogramming language understanding.AcknowledgementsThis research is a part of a dissertation concerning asymbolic program comprehension methodology. Itis funded by COMAS of Jyväskylä University.SwMaster Ltd has enabled the initial research andproduct development for the tool since 2000.ReferencesJean Bezevin. Object to Model ParadigmChange, Jean-Marie Burkhardt, Francoise Detienneand Susan Wiedenbeck. "Object-Oriented Program Comprehension: Effectof Expertise, Task and Phase", EmpiricalSoftw. Eng. 7 (2), 2002, pp. 115-156.Thomas Cheatham, Glen Holloway and JudyTownley. "Symbolic Evaluation and theAnalysis of Programs", IEEE Trans.Softw. Eng. 5 (4), 1979, pp. 402-417.Serge Demeyer, Sander Tichelaar, PatrickSteyaert. FAMIX 2.0: The FAMOOSInformation Exchange Model (1999),Jan Heering, Paul Clint. Semantics of ProgrammingLanguages, Tool Approach,Erkki Laitila. Visual Prolog: Industrial Applications(only in Finnish). Jyväskylä,Finland: Teknolit, 1996. 228 p.Stan Letovsky. "Cognitive process in programcomprehension". In: E. Soloway& S. Iyengar (Ed.) Empirical Studies ofProgrammers: Norwood, NJ: Ablex,1986.MDA. The Architecture of Choice for aChanging World, Model Driven Architectures.Leszek Maciaszek, Bruc Lee Liong. PracticalSoftware Engineering, Addison-Wesley, 2005.Stephen J. Mellor, Marc J. Balcer. ExecutableUML, Addison-Wesley,2002.OMG. Object Management GroupNancy Pennington. "Stimulus Structures andMental Representations in Expert Comprehensionof Computer Programs",Cognitive Psychology, 1987.Thomas Reps. “Program Analysis via GraphReachability”, PLDI-Conference, 2000.Ravi Sethi. Programming Languages - Conceptsand Constructs. Addison-Wesley,1996.Sun. Grammar of Java, 2003.Software Technology Laboratory. The TXLProgramming Language: www.txl.ca. Technical report,2004.Prolog Development Center A/S. PDCProlog Toolbox,,1990Jim. Arlow, Ila Neustadt. UML and the UnifiedProcess, Addison-Wesley, 2002.Prolog Development Center A/S. VisualProlog, 2006,Anne-Liese von Mayrhauser, Marie Vans."Hypothesis-driven understanding processesduring corrective maintenance oflarge scale software". CSM 1997, 1997,pp. 12-20. IEEE Computer Soc.VMC05, Verification, Model Checking, andAbstract Interpretation Conference,Paris, 2005.Adrian Walker, Michel McCord, John Sowa,Walter Wilson. Knowledge Systemsand Prolog, 1987, Addison Wesley.Adam Webber. Modern Programming Languages,c/o Amazon, 2005.Marc Weiser. Program slicing. IEEE Transactionson Software Engineering,10(4):352–357, 1984.141

Norman Wilde,and Ross Huitt, "MaintenanceSupport for Object-Oriented Programs",IEEE Trans. Software Eng. 18 (12),1992, pp. 1038-1044.Norman Wilde, Allen Chapman, Paul Matthews,Ross Huitt, Describing ObjectOriented Software: What MaintainersNeed to Know, EDATS, In InternationalJournal on Software Engineering,pages 521–533, 1994.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 142

Describing Rich Content:Future Directions for the Semantic WebTimo Honkela and Matti PölläHelsinki University of TechnologyLaboratory of Computer and Information ScienceP.O. Box 5400 FI-02015 TKK, Finland{timo.honkela, matti.polla}@tkk.fiAbstractIn this position paper, we discuss some problems related to those semantic web methodologies that arestraightforwardly based on predicate logic and related formalisms. We also discuss complementaryand alternative approaches and provide some examples of such.1 IntroductionIt is clear that the use of standardized formats withincomputer science is beneficial. For instance, thewidespread use of the World Wide Web would nothave been possible without the adoption of HTML.Similarly, there are serious attempts to create standardsfor metadata, data about data, so that a piece ofart stored in electronic form would include informationabout, e.g., its creator, source, identification andpossible access restrictions. Moreover, metadata usuallyincludes also a textual summary of the contents, acontent description that provides information for theorganization and search of the data.1.1 Subjective and complex paths fromdata to metadataEspecially if pictorial or sound data is considered,an associated description, metadata, is useful. Thedescription may be based on a pre-defined classificationor framework, for instance, like an ontologywithin the current semantic web technologies. However,even if something like the identity of the authoror the place of publishing can rather easily be determinedunambiguously, the same is not true for thedescription of the contents. In the domain of informationretrieval and databases of text documents, Furnaset al. (1987) already found that in spontaneous wordchoice for objects in five domains, two people favoredthe same term with less than 20% probability. Bates(1986) has shown that different indexers, well trainedin an indexing scheme, might assign index terms fora given document differently. It has also been observedthat an indexer might use different terms forthe same document at different times. The meaningof an expression (queries, descriptions) in any domainis graded and changing, biased by the particularcontext. Fig. 1 aims to illustrate the challengingaspects of the overall situation. Human beings perceive,act and interact within complex environments.It has occasionally been assumed that much of theunderlying conceptual structure within human mindis readily given by some way, even already when weare born. There is a vast body of literature relatedto this question which we do not touch upon here.It suffices to note that the discussion in this paper isbased on the assumption that the conceptual systemsare mainly emergent: they are created, molded andshared by individuals in interaction with each otherand the rest of the accessible part of the world.Figure 1: An illustration of the basic problem of symbolgrounding: what is the process that iscapable of generating ontological structuresbased on complex raw perceptions and activities.Vygotsky (1986, originally published in 1934) hasstated that “... the world of experience must be greatlysimplified and generalized before it can be translatedinto symbols. Only in this way does communicationbecome possible, for the individual’s experience residesonly in his own consciousness and is, strictly143

speaking, not communicable.” Later, he continues:“The relation of thought to word is not a thing but aprocess, a continual movement back and forth fromthought to word and from word to thought. In thatprocess the relation of thought to word undergoeschanges which themselves may be regarded as developmentin the functional sense.” This means in practicethat conceptualization is a complex process thattakes place in a socio-cultural context, i.e., within acommunity of interacting individuals whose activitiesresult into various kinds of cultural artifacts such aswritten texts.It is a very basic problem in knowledge managementthat different words and phrases are usedfor expressing similar objects of interest. Naturallanguages are used for the communication betweenhuman beings, i.e., individuals with varying background,knowledge, and ways to express themselves.When rich contents are considered this phenomenonshould be more than evident. Therefore, if the contentdescription is based on a formalized and rigid frameworkof a classification system, problems are likelyto arise. Fig. 2 shows a simple example of two conflictingformalizations.Natural languages have evolved to have a certaindegree of compositionality to deal with such situations.Vogt (2006) has developed a simulation modelthat considers the transition of holistic languages versuscompositional languages. Holistic languages arelanguages in which parts of expressions have no functionalrelation to any parts of their meanings. It isto be noted, though, that the compositionality is amatter of degree in natural languages. One cannotassume that, for instance, each noun would refer toone concept and the conceptual structures would followisomorphically the composition of the syntacticalstructures. For instance, the existence of collocationscomplicates the situation.1.2 From two-valued logic to adaptivecontinuous-valued modelsIn addition to the different ways of creating conceptualhierarchies discussed above, the inherent continuousnature of many phenomena makes it impossibleto determine exactly, in a shared manner the borderlinesbetween some concepts or how some wordsare used. Actually, we prefer to consider concepts asareas in high-dimensional continuous spaces as suggestedby Gärdenfors (2000). Fig. 3 illustrated thebasic problem of dividing continuous spaces into discreterepresentations. Various approaches have beendeveloped to deal with this phenomenon, fuzzy settheory as a primary example Zadeh (1965).Figure 2: An illustration of two conflicting conceptualsystems on the surface level. Thesesystems prioritize the distinctive featuresdifferently.Figure 3: The obvious difference between continuousand discrete. The continuous can be discretizedbut there is a certain cost associated.The cost relates to the need to learnand deal with an increasing number of explicitsymbols.The basic semantic web formalisms are based onpredicate logic and other symbolic representationsand are subject to most of those problems that earlierAI formalisms have. In order to alleviate the problemsrelated to the traditional approach, there are alreadyexamples of research projects in which somesoft computing approaches, including fuzzy logic,probabilistic modeling and statistical machine learn-New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 144

ing, are applied. Even a collection named “Soft Computingin Ontologies and Semantic Web” has recentlybeen published Ma (2006). In the collection, a relatedtopic to the discussion above is presented by Holiand Hyvönen (2006). They consider modeling uncertaintyin semantic web taxonomies in particular inthe domain of geographical information. Nikravesh(2006) presents an approach which is based on theuse of, e.g., fuzzy logic, evolutionary computationand the self-organizing map.In this position paper, we outline some complementaryand sometimes even alternative approachesto the core methodologies currently applied withinthe semantic web research and development.2 Alternatives to highly formalizedmetadataIt may be useful not to define any artificial limitationsfor the descriptions. For instance, when the domaindevelops into directions which did not exist when theclassification system was developed, problems arise.Moreover, if the content it described using largeenough body of text the for better recall, i.e., higherlikelihood for finding the information is greater.However, tools for ensuring precision are needed.Precision refers to the number of relevant retrieveddocuments over the total number of retrieved documents.If a word or an expression is seen without the contextthere are more possibilities for misunderstanding.Thus, for human reader the contextual information isoften very beneficial. The same need for disambiguationcan also be relevant within information systems.As the development of ontologies and other similarformalizations are, in practice, grounded in the individualunderstanding and experience of the developersand their socio-cultural context, the status of individualitems in a symbolic description may be unclear.Fig. 4 illustrates the influence of varying socioculturalcontexts. Namely, if one considers the pragmaticmeaning (if not semantic meaning) of “summer’sday” in a Shakespeare sonnet, it is obvious thatthere is a great difference between the contexts thatare being referred to, for example, in Scandinavia andin Sahara.Similarly, the methods that are used to manage datashould be able to deal with contextual information,or even in some cases provide context. The Self-Organizing Map by Kohonen (2001) can be consideredas an example of such a method. Often it iseven possible to find relevant features from the dataFigure 4: Different contexts potentially related to theexpression given.itself Kohonen et al. (1997). However, a computerizedmethod – using a some kind of autonomousagent – does not provide an “objective” classificationof the data while any process of feature extraction,human or artificial, is based on some selections forwhich there most often are some well-grounded alternatives.Below, we describe some examples in which theideas and principles outlined above have been applied.3 Case studiesWe consider three cases which exemplify complementaryand alternative approaches to the use of (firstorder) logic-based formalisms in knowledge representation.3.1 Color labelingThe concept of color cannot be adequately studiedonly by considering the logico-semantic structure ofcolor words. One has to take into account the color asa physical phenomenon. Color naming also requiresconsideration of the qualities of the human color perceptionsystem. A thorough study of color naming,thus, would require consideration of at least linguistic,cognitive, biological, physical and philosophicalaspects Hardin (1988).Subjective naming of color shades has been studiedin a demonstrative web site where users can pickcolor shades and give name labels (or ’tags’) to thecolors. In this case color shades are represented as aRGB tuple indicating the intensities of red, green andblue color.As the database of tagged color shades grows, interestingproperties can be found by looking at thedistributions of individual tags. For example, the tag’red’ would result in a reasonably narrow distributioncentered around the point (1.0; 0.0; 0.0) in the RGBspace. Other tags, however, can have much more variationin the way people place them in the color space.145

For example, the tag ’skin’ or ’hair’ is an example ofa highly subjective name for a color shade due to variousskin and hair color conceptions. A related themeis the fact that the domain for which a color nameis an attribute has a clear influence on which part ofthe RGB space the color name refers to. Gärdenfors(2000) lists as an example the differences between thequality of redness of skin, book, hair, wine or soil.Fig. 5 below gives an example how the color spacecan be divided in a substantially different manner intwo languages.400’red’500 600 700’ao’Wavelength[nm]400 500 600 700 Wavelength[nm]Figure 5: Illustration of fuzzy color definitions. Theword ’red’ is usually referred to light wavelengthsclose to 700 nm while the Japaneseterm ’ao’ is associated with a wider scaleof wavelengths. In English the term ’ao’would cover both ’blue’ and ’green’.3.2 WEBSOM and PicSOMThe WEBSOM method was developed to facilitate anautomatic organization of text collections into visualand browsable document maps (Honkela et al., 1997;Lagus et al., 2004). Based on the Self-OrganizingMap (SOM) algorithm (Kohonen, 2001), the systemorganizes documents into a two-dimensional plane inwhich two documents tend to be close to each otherif their contents are similar. The similarity assessmentis based on the full-text contents of the documents.In the original WEBSOM method (Honkelaet al., 1996) the similarity assessment consisted oftwo phases. In the first phase, a word-category map(Ritter and Kohonen, 1989; Honkela et al., 1995) wasformed to detect similarities of words based on thecontexts in which they are used. The Latent SemanticIndexing (LSI) method (Deerwester et al., 1990) isnowadays often used for similar purpose. In the secondphase, the document contents were mapped onthe word-category map (WCM). The distribution ofthe words in a document over the WCM was used asthe feature vector used as an input for the documentSOM. Later, the WEBSOM method was streamlinedto facilitate processing of very large document collections(Kohonen et al., 1999) and the use of the WCMas a preprocessing step was abandoned.The PicSOM method (Laaksonen et al., 1999,2002; Koskela et al., 2005) was developed for similarpurposes than the WEBSOM method for contentbasedimage retrieval, rather than for text retrieval.Also the PicSOM method is based on the Self-Organizing Map (SOM) algorithm (Kohonen, 2001).The SOM is used to organize images into map unitsin a two-dimensional grid so that similar imagesare located near each other. The PicSOM methodbrings three advanced features in comparison with theWEBSOM method. First, the PicSOM uses a treestructuredversion of the SOM algorithm (Tree StructuredSelf-Organizing Map, TS-SOM) (Koikkalainenand Oja, 1990) to create a hierarchical representationof the image database. Second, the PicSOM systemuses a combination of several types of statistical features.For the image contents, separate feature vectorshave been formed for describing colors, textures,and shapes found in the images. A distinct TS-SOMis constructed for each feature vector set and thesemaps are used in parallel to select the returned images.Third, the retrieval process with the PicSOMsystem is an iterative process utilizing relevance feedbackfrom the user. A retrieval session begins withan initial set of different images uniformly selectedfrom the database. On subsequent rounds, the queryfocuses more accurately on the user’s needs based ontheir selections. This is achieved as the system learnsthe user’s preferences from the selections made onprevious rounds.WEBSOM and PicSOM methods are a means forcontent-driven emergence of conceptual structures.Fig. 6 illustrates how the clustering structure discernableon a self-organizing map can correspond to a hierarchicalstructure. This basic principle can be appliedin multiple ways to provide a bridge betweenthe raw data directly linked with some phenomenonand the linguistic and symbolic description of its conceptualstructure. Fig. 6 also shows why the SOM isgaining popularity as a user interface element replacing,e.g., traditional menus. With suitable labeling ofthe map, the user can easily browse the map zoominginto details when necessary. The structure of the mapNew Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 146

Figure 6: The emergent structure of an organizedself-organizing map reflects the structure ofthe underlying data. The clustering that isvisible on the left can be interpreted as thetree diagram on the right.depends on the data and its preprocessing. The preprocessingcan be used to bring up relevant structures.For instance, it may be useful to increase or decreasethe weight of some variables. If we consider a mapof an e-mail collection, one can decrease the weightof those variables that are commonly related to unsolicitede-mail messages. This leads into the situationin which the SOM can allocate more resourceson modeling useful e-mail. The same idea applies tomany application areas in which a particular point ofview is more relevant than some other.3.3 Quality assessment of medical websitesThe web has become an increasingly importantsource of medical information replacing much ofthe area of medical self-help literature. The consequencesof relying on medical information found onweb sites can be crucial in terms of making decisionsabout treatment. Hence there is need for assessing thequality of these web sites to help the layman decidewhich information he/she should rely on.Currently, quality labeling of medical web sites isdone by various organizations of medical professionals.The assessment process is usually done completelyby hand requiring a large amount of manualwork by trained physicians in searching web sites forthe required elements (including, for example, propercontact information). The EU funded project MedIEQ1 aims to develop tools to facilitate the processof web site quality labeling.The MedIEQ project applies mostly the current semanticweb technologies to describe the web site contents.However, already in the early stages of theproject, it has become apparent that some kinds ofcontents are difficult to analyze using the structured1 http://www.medieq.org/approach. For instance, the target audience of a medicalweb site is a property by which the site is beinglabeled in the assessment process. As it turns out, itis less than trivial to automatically detect whether asite is intended to be read by laymen or medical professionals.Further, the division of the target audiencetypes into crisp categories is often subjective by nature.4 ConclusionsWe have presented some motivation why the coretechnologies in Semantic Web should not solely relyon predicate logic and related formalisms. We haveargumented for a certain data-driven approach inwhich the original data is analyzed automaticallyrather than relying on hand-crafted ontologies andtheir use as a basis for choosing descriptors in themetadata. We have given examples of such an approachmainly using the Self-Organizing Map as thecore method.ReferencesM. J. Bates. Subject access in online catalog: a designmodel. Journal of the American Society ofInformation Science, 37(6):357–376, 1986.S.C. Deerwester, S.T. Dumais, T.K. Landauer, G.W.Furnas, and R.A. Harshman. Indexing by latentsemantic analysis. Journal of the American Societyof Information Science, 41:391–407, 1990.G. W. Furnas, T. K. Landauer, L. M. Gomez, andS. T. Dumais. The vocabulary problem in humansystemcommunication. Communications of theACM, 30(11):964–971, 1987.P. Gärdenfors. Conceptual spaces: The Geometry ofThought. MIT Press, 2000.C.L. Hardin. Color for Philosophers - Unweaving theRainbow. Hackett Publishing Company, 1988.M. Holi and E. Hyvönen. Soft Computing in Ontologiesand Semantic Web, chapter Modeling Uncertaintyin Semantic Web Taxonomies, pages 31–46.Springer, 2006.T. Honkela, V. Pulkki, and T. Kohonen. Contextualrelations of words in grimm tales analyzed by selforganizingmap. In F. Fogelman-Soulié and P. Gallinari,editors, Proceedings of ICANN’95, InternationalConference on Artificial Neural Networks,147

pages 3–7, Paris, France, October 1995. EC2 etCie, Paris.T. Honkela, S. Kaski, K. Lagus, and T. Kohonen.Newsgroup exploration with WEBSOM methodand browsing interface. Technical Report A32,Helsinki University of Technology, Laboratory ofComputer and Information Science, Espoo, Finland,1996.T. Honkela, S. Kaski, K. Lagus, and T. Kohonen.WEBSOM—self-organizing maps of documentcollections. In Proceedings of WSOM’97,Workshop on Self-Organizing Maps, pages 310–315. Espoo, Finland, 1997.T. Kohonen. Self-Organizing Maps. Springer, 2001.T. Kohonen, S. Kaski, and H. Lappalainen. Selforganizedformation of various invariant-featurefilters in the adaptive-subspace SOM. Neural Computation,9:1321–1344, 1997.M. Nikravesh. Soft Computing in Ontologies and SemanticWeb, chapter Beyond the Semantic Web:Fuzzy Logic-Based Web Intelligence, pages 149–209. Springer, 2006.H. Ritter and T. Kohonen. Self-organizing semanticmaps. Biological Cybernetics, 61(4):241–254,1989.P. Vogt. Cumulative cultural evolution: Can we everlearn more? In S. Nolfi et al., editor, Proceedingsof SAB 2006, From Animals to Animats 9, pages738–749, Berlin, Heidelberg, 2006. Springer.L. Vygotsky. Thought and language. MIT Press,1986, originally published in 1934.L. A. Zadeh. Fuzzy sets. Information and Control, 8:338–353, 1965.T. Kohonen, S. Kaski, K. Lagus, J. Salojärvi,J. Honkela, V. Paatero, and A. Saarela. Self organizationof a massive text document collection.In Kohonen Maps, pages 171–182. Elsevier, Amsterdam,1999.P. Koikkalainen and E. Oja. Self-organizing hierarchicalfeature maps. In Proc. IJCNN-90, InternationalJoint Conference on Neural Networks,Washington, DC, volume II, pages 279–285, Piscataway,NJ, 1990. IEEE Service Center.M. Koskela, J. Laaksonen, M. Sjöberg, and H. Muurinen.PicSOM experiments in TRECVID 2005.In Proceedings of the TRECVID 2005 Workshop,pages 267–270, 2005.J. Laaksonen, M. Koskela, and E. Oja. Picsom: Selforganizingmaps for content-based image retrieval.In Proceedings of IEEE International Joint Conferenceon Neural Networks (IJCNN’99), pages2470–2473, 1999.J. Laaksonen, M. Koskela, and E. Oja. Picsom -self-organizing image retrieval with mpeg-7 contentdescriptions. IEEE Transactions on NeuralNetworks, Special Issue on Intelligent MultimediaProcessing, 13(4):841–853, 2002.K. Lagus, S. Kaski, and T. Kohonen. Mining massivedocument collections by the WEBSOM method.Information Sciences, 163:135–156, 2004.Z. Ma. Soft Computing in Ontologies and SemanticWeb. Springer, 2006.New Developments in Artificial Intelligence and the Semantic WebProceedings of the 12th Finnish Artificial Intelligence Conference STeP 2006 148

Julkaisuluettelo – 2006 – List of PublicationsKonferenssijulkaisuja – Conference Proceedings €1. Hyvönen E., Seppänen J., Syrjänen M., eds. (1984):STeP­84 – Symposium Papers, Vol. 1. 62. Hyvönen E., Seppänen J., Syrjänen M., eds. (1984):STeP­84 – Tutorial and Industrial Papers, Vol. 2. 63. Karjalainen M., Seppänen J., Tamminen M., eds. (1986):STeP­86 – Symposium Papers: AI and Philosophy, Vol. 1 64. Karjalainen M., Seppänen J., Tamminen M., eds. (1986):STeP­86 – Symposium Papers: Methodology, Vol. 2. 65. Karjalainen M., Seppänen J., Tamminen M., eds. (1986):STeP­86 – Symposium Papers: Applications, Vol. 3. 66. Mäkelä M., Linnainmaa S., Ukkonen E., eds. (1988):STeP­88 – Contributed Papers: Applications, Vol. 1. 67. Mäkelä M., Linnainmaa S., Ukkonen E., eds. (1988):STeP­88 – Contributed Papers: Methodology, Vol. 2. 68. Syrjänen M., Oivo M., Salonen P., eds. (1990):STeP­90 – Tutorial and Symposium Papers 69. Hyvönen E., Seppänen J., Syrjänen M., eds. (1992):STeP­92 – New Directions in AI – Papers, Vol. 1 1010. Hyvönen E., Seppänen J., Syrjänen M., eds. (1992):STeP­92 – New Directions in AI – Symposia, Vol. 2 1011. Hyvönen E., Seppänen J., Syrjänen M., eds. (1992):STeP­92 – New Directions in AI – Workshops, Vol. 3 1012. Carlsson C., Järvi T., Reponen T., eds. (1994):STeP­94 – Multiple Paradigms for Artificial Intelligence 1013. Alander J., Honkela T., Jakobsson M., eds (1996):STeP­96 – Genes, Nets and Symbols 1214. Koikkalainen P., Puuronen S., eds (1998):STeP­98 – Human and Artificial Information Processing 1415. Hyötyniemi H., ed. (2000):Feedback to the Future Systems, Cybernetics andArtificial Intelligence 1416. Hyötyniemi H., ed. (2000):STeP 2000 – Millennium of Artificial Intelligence– 'AI of Today': Symposium on Applications 1417. Hyötyniemi H., ed. (2000):STeP 2000 – Millennium of Artificial Intelligence– 'AI of Tomorrow': Symposium on Theory 1418. Ala­Siuru P., Kaski S., eds. (2002):STeP 2002 – The Art of Natural and Artificial 1419. Hyötyniemi H., Ala­Siuru P., Seppänen J. eds. (2004):STeP 2004 – Life, Cybernetics and Systems Sciences 1420. Hyvönen E., Kauppinen T., Salminen M., Viljanen K.Ala­Siuru P, eds. (2004): STeP 2004 – Web Intelligence 1421. Alander J., Ala­Siuru P., Hyötyniemi H., eds. (2004):STeP­2004 – Origin of Life and Genetic Algorithms 1422. Honkela T., Raiko T., Kortela J., Valpola H., eds. (2006):Ninth Scandinavian Conference on Artificial Intelligence(SCAI 2006) 1823. Hyvönen E., Kauppinen T., Kortela J., Laukkanen M., Raiko T.,and Viljanen K., eds. (2006): STeP­2006 – New Developmentsin Artificial Intelligence and the Semantic Web 182. Karanta I., ed. (1989):Hyperteksti ja tekoäly 63. Hynynen J., Linnainmaa S., eds. (1990):The French­Finnish Colloquium on AI Applications 64. Karanta I., Seppänen J., eds. (1993):Kaaostutkimus ja tietotekniikka – katastrofeja, attraktoreja,ja fraktaaleja 125. Stark S., Tyystjärvi K., eds. (1990):Ajattelevatko koneet? – Tekoäly, tietotekniikka ja robotiikka 126. Hautamäki A., Nyman G., toim. (1992):Kognitiotiede ja koneäly 127. Lehtola., Jokiniemi J., eds. (1992):Embedded and Real­Time Systems 148. Kurenniemi E., ed. (1992):A Bibliography on SETI 109. Linnaluoto S., Seppänen J., eds. (1992):SETI – Search for Extraterrestrial Intelligence 1210. Bulsari A., Saxén B., eds. (1993):Neural Network Research in Finland 1411. Lehtola A., Jokiniemi J., eds. (1993):Conceptual Modelling and Object­Oriented Programming 1412. Hyvönen E., Seppänen J., toim. (1995):Keinoelämä – Artificial Life. Tekniikkaa, luonnontiedettä,filosofiaa ja taidetta 1413. Haaparanta L., Hyvönen E., Seppänen J., Silvonen J.,toim. (1995): Älyn ulottuvuudet ja oppihistoria –Matka logiikan, psykologian ja tekoälyn juurille 1414. Gefwert C., Orponen P., Seppänen J., toim. (1996):Logiikka, matematiikka ja tietokone – Logic, Mathematicsand the Computer 1415. Honkela T., toim. (1999): Pelit, tietokone ja ihminen– Games, Computers and People 14SCAI – Scandinavian AI Conference Proceedings €1. Jaakkola H., Linnainmaa S., eds. (1989):SCAI' 1989 – Proceedings of the Second ScandinavianConference on Artificial Intelligence, Vols. 1­2, Tampere 102. Grahn G., ed. (1997): SCAI' 1997 – Proceedings of the SecondScandinavian Conference on Artificial Intelligence, Vol. 1­2,Helsinki 103. Alander J., ed. (1997): Proceedings of the Third Nordic Workshopon Genetic Algorithms and their Applications (3NWGA), Vaasa 10(See also: SCAI 2006 proc. item 22 in Conference Proceedings above.)Kansainvälisiä kokouksia – International Conferences €1. Bulsari Abhay, Kallio Sirpa, eds. (1995): EngineeringApplications of Artificial Neural Networks. Proceedings of theInternational Conference EANN '95, Otaniemi,21­23 August 1995, Finland. 142. Pylkkänen Paavo, Pylkkö Pauli, eds. (1995): New Directions inCognitive Science. Proceedings of the International Symposium,Saariselkä, 4­9 August 1995, Lapland, Finland. 14Symposiojulkaisuja – Symposium Publications €1. Keskinen K., toim. (1989):Epävarmuus, päätöksenteko ja tietämystekniikka 6Tiedotuslehti – Arpakannus – Magazine €1986 alkaen 2­4 numeroa/v – 2­4 issues/year 1986­1996 61997 alkaen ajoittain – Since 1997 occasional issues 6To order, call +358 9 5617 830, send a fax to +358 9 5607 5365 or email to office@stes.fi.When ordering publications, please state the name(s) of the publication(s) and the amount ofcopies you want to buy. Please also give your post address (name, street address, zip/postal codeand country) for delivering the publications to you by post. Shipping costs 5€ will be added to thetotal sum of your order.Contact informationFinnish Artificial Intelligence Society, Susanna Koskinen, c/o Toimistopalvelu Hennax T:mi,Henrikintie 7 D, FIN­00370 HELSINKIWWW: http://www.stes.fiemail: office@stes.fiphone: +358 40 535 2617