Dissertation_Mahsa Chitsaz.pdf - DSpace@UM - University of Malaya

Medical Image Segmentation by Hybridizing Multi-Agent System and
Reinforcement Learning Agent
Mahsa Chitsaz
Supervisor: Dr. Woo Chaw Seng
Faculty of Computer Science and Information Technology,
University of Malaya
December 2009

UNIVERSITI MALAYA
ORIGINAL LITERARY WORK DECLARATION
Name of Candidate: (I.C/Passport No: )
Registration/Matric No:
Name of Degree:
Title of Project Paper/Research Report/Dissertation/Thesis (“this Work”):
Field of Study:
I do solemnly and sincerely declare that:
(1) I am the sole author/writer of this Work;
(2) This Work is original;
(3) Any use of any work in which copyright exists was done by way of fair dealing
and for permitted purposes and any excerpt or extract from, or reference to or
reproduction of any copyright work has been disclosed expressly and
sufficiently and the title of the Work and its authorship have been acknowledged
in this Work;
(4) I do not have any actual knowledge nor do I ought reasonably to know that the
making of this work constitutes an infringement of any copyright work;
(5) I hereby assign all and every rights in the copyright to this Work to the
University of Malaya (“UM”), who henceforth shall be owner of the copyright
in this Work and that any reproduction or use in any form or by any means
whatsoever is prohibited without the written consent of UM having been first
had and obtained;
(6) I am fully aware that if in the course of making this Work I have infringed any
copyright whether intentionally or otherwise, I may be subject to legal action or
any other action as may be determined by UM.
Candidate’s Signature Date
Subscribed and solemnly declared before,
Name:
Designation:
Witness’s Signature Date
ii

To My Beloved Mother and Father
iii

Abstract
Image segmentation is still a debatable problem although there have been many research
work done in the last few decades. First of all, every solution for image segmentation is
problem-based. Secondly, medical image segmentation methods generally have
restrictions because medical images have very similar gray level and texture among the
interested objects.
Therefore, this dissertation presents a framework to extract simultaneity several objects
of interest from head Computed Tomography images. The proposed method contains
two phases; training and testing. A Reinforcement-Learning method is proposed for the
training phase, and a new Multi-Agent system is proposed for the testing phase. In the
training phase, a few images are used as a trained image whereas the RL agent will find
the appropriate value of each object or region in the input image. The outcome of this
training phase is transferred to the next phase, testing phase. In this phase, the images
are segmented by some priori knowledge and the properties of local agent.
Proposed reinforcement learning model attains significant result in segmentation
accuracy; the accuracy is more than 95% for each region in the image and the mean
computation time of all datasets is less than 13 seconds. Moreover, the number of
training data set for PRLM can be one or a small number of images. Also, PRLM has
the ability to segment simultaneously an image into some distinct regions.
Proposed multi-agent model attains considerable result in segmentation accuracy; the
accuracy is more than 90% for each region in the image and the mean computation time
iv

of all datasets is less than 7 seconds. Furthermore, PMAM is capable to segment
simultaneously an image into some distinct regions.
v

Table of Contents
DECLARATION………………………………………………………………. ii
DEDICATION…………………………………………………………………. iii
ABSTRACT……………………………………………………………………. iv
TABLE OF CONTENTS………………………………………………………. vi
LIST OF FIGURES ……………………………………………………………. viii
LIST OF TABLES………………………………………………………...…… ix
LIST OF ABRIVATIONS AND SYMBOLS………………………………….. x
LIST OF PUBLICATIONS……………………………………………………. xii
ACKNOWLEDGEMENT……………………………………………………… xiii
1. Introduction………………………………………………………………… 1
1.1 Background………………………………………………………… 1
1.2 Motivation …………………………………………………………. 2
1.3 Problem Description………………………………………………... 4
1.4 Goal and Objectives………………………………………………... 4
1.5 Scope of the project………………………………………………… 5
1.6 Dissertation Organization…………………………………………... 5
2. Literature Review………………………………………………………….. 7
2.1 Overview……………………………………………………………… 7
2.2 Three-Dimensional Medical Imaging and Skull Anatomy…………… 8
2.2.1 Computed Tomography Images.…………………………………. 9
2.2.2 Magnetic Resonance Imaging……………………………………. 10
2.2.3 Skull Anatomy……………………………………………………. 12
2.2.3 Cranial Bones………….…………………………………. 12
2.2.3 Facial Bones…………...…………………………………. 13
2.2.3 Anatomical Structure that Used as the Case Study....……. 14
2.3 Segmentation Methods………………………………………………... 17
2.3.1 Several Classifications of Segmentation Methods……………….. 18
2.3.2 Brief Description of Main Segmentation Methods………………. 22
2.3.2.1 Thresholding…………………….……………………… 22
2.3.2.2 Region Growing………………...……………………… 23
2.3.2.3 Edge Detection………………….……………………… 24
2.3.2.4 Classifiers……………………….……………………… 24
2.3.2.5 Clustering……………………….……………………… 25
2.3.2.6 Deformable Models…………….……………………… 26
2.3.2.7 Neural Network…..…………….……………………… 26
2.3.3 Comparison of the Segmentation Methods………………………. 27
2.4 Agent and Multi-Agent System………………………………………. 28
2.5 Standard Reinforcement Learning Model…………………………….. 30
2.6 Image Segmentation Methods by Autonomous Agents in Multi-Agent
System………………………………………………………………….. 33
2.6.1 Kagawa et al method…………………………………………...… 33
2.6.2 Wang and Yuan method………………………………………..… 34
2.6.3 Gyohten method………………………………………………….. 35
2.6.4 Guillaud et al. method…………………………………………..... 37
2.6.5 Rodin et al. method……………………………………………..... 38
2.6.6 Melkemi et al. method……………...…………………………….. 39
2.6.7 Spinnu et al. method…………………………………...…………. 40
2.6.8 Boucher et al. method…………………………………..………… 42
vi

2.6.9 Liu and Tang method……………………………………...……... 43
2.6.10 Germond et al. method……………………………………..…… 44
2.6.11 Duchesnay et al. method………………………………………... 45
2.6.12 Khosla and Lai method……………………...………………….. 47
2.6.13 Richard et al. method…………………………………………… 49
2.6.14 Benamrane and Nassane method………………………………... 50
2.6.15 Discussion…………………………………………………......... 51
2.7 Image Segmentation Methods by Reinforcement Learning Model…... 55
2.7.1 Peng and Bhanu method………………………………………….. 55
2.7.2 Shokri method…………………………………………………..... 57
2.7.3 Sahba method…………………………………………………….. 57
2.8 Chapter Summary……………………………………………………... 59
3. Methodology………………………………………………………………... 61
3.1 Image Acquisition..…………………………………………………… 61
3.2 Image Segmentation…………………………………………………... 62
3.2.1 Training Phase……………………………………………………. 66
3.2.1.1 Definition of States …………….……………………… 69
3.2.1.2 Definition of Actions ….……….……………………… 72
3.2.1.3 Definition of Reward…………………………………… 73
3.2.1.4 Graphical User Interface for Training Phase …..……… 73
3.2.2 Testing Phase……………………………………………………... 74
3.2.2.1 Graphical User Interface for Testing Phase …..……….. 78
3.3 Chapter Summary……………………………………………………... 79
4. Experimental Results and Discussion …..………………………………... 81
4.1 Experiment Result of Training Procedure……………………….……. 81
4.1.1 Image Data Sets of PRLM…………….………..………………... 82
4.1.2 Qualitative Analysis of PRLM…………………………………… 83
4.1.3 Quantitative Analysis of PRLM………………………….………. 83
4.1.3.1 Accuracy of PRLM …..…………………… …..……… 85
4.1.3.2 Efficiency of PRLM ..…………………….. …..……… 88
4.2 Experiment Result of Testing Procedure……………………………... 88
4.2.1 Image Data Sets of PMAM….…………………………………... 91
4.2.2 Qualitative Analysis of PMAM …………………………….…... 91
4.2.3 Quantitative Analysis of PMAM ……………………………..… 93
4.2.3.1 Accuracy of PMAM ……………………….…..……… 93
4.2.3.2 Efficiency of PMAM .…………………….. …..……… 95
4.3 Chapter Summary……..…………………………………………..….. 96
5. Conclusions…………………………………………………………………. 97
5.1 The Proposed Reinforcement-Learning model ……………………..... 97
5.2 The Proposed Multi-Agent model………………………………….…. 99
5.3 Achievements………………………………………………………..... 101
5.4 Future work………………………………………………………….... 102
Bibliography…………………………………………………………………... 104
Appendix A: Experimental Results of the Training Phase…...……………... 110
Appendix B: TPVF and FPVF of the Experimental Results of the Training
Phase…………………………………………………………... 114
Appendix C: Experimental Results of the Testing Phase.…………………… 116
vii

Appendix D: TPVF and FPVF of the Experimental Results of the Testing
Phase…………………………………………………………... 119
viii

List of Figures
Figure 1.1: (a) A Slice of CT Image of a Human Head (b) Segmented Image of Figure
1.1(a)…………………………………………………………………………... 4
Figure 2.1: CT Scanner (Jeri 2008)………………………………………………………... 10
Figure 2.2: Human Head CT Slices at Axial View (Obaidellah 2006)…………………… 10
Figure 2.3: MRI Scanner (Garrobo 2006)…………………………………………………. 11
Figure 2.4: The Lateral (left) and the Anterior (right) View of Skull (Gray 1918)……….. 12
Figure 2.5: Cranial Bones of the Skull (Martini 2004)……………………….…………… 13
Figure 2.6: (a): The Maxillae Bone (b): the Mandible Bone (Gray 1918)………………… 14
Figure 2.7: The Top Side of Skull (Walter 2007)…………………………………………. 15
Figure 2.8: (a) CT Image of 1-3 Section (b) CT Image of 1-6 Section (Obaidellah
2006)………………………………………………………………….……….. 15
Figure 2.9: The Middle Part of Skull (Walter 2007)………………….………….………... 15
Figure 2.10: (a) CT Image of 1-10 Section (b) CT Image of 1-12 Section (c) CT Image of
1-15 Section (d) CT Image of 1-20 Section (Obaidellah
2006)…………………………………………………………………………... 16
Figure 2.11: The Inferior Part of the Skull (Walter 2007)………………………………….. 16
Figure 2.12: (a) CT Image of 1-22 Section (b) CT Image of 1-23 Section (Obaidellah
2006)…………………………………………………….…….......................... 17
Figure 2.13: Uniformity Predicate of the Segmentation (Awcock 1995)……..……………. 17
Figure 2.14: A Clustering Approach (Jain 1989)………………………………..……….…. 25
Figure 2.15: The Internal Structure of a Typical Agent (Chitsaz 2008)………………..…... 29
Figure 2.16: The Standard RL Model (Kaelbling 1996)…………..………………………... 31
Figure 2.17: Pseudocode for Q-Learning Algorithm (Watkins, 1989)……………..…… 32
Figure 2.18: The Proposed Method of Kagawa (Kagawa 1999) …………………………... 34
Figure 2.19: The Proposed Method of Gyothen (Gyohten 2000).......................…………… 36
Figure 2.20: The Procedure of Proposed Method by Guillaud et al. (Guillaud 2000) ……... 37
Figure 2.21: The MAS which Proposed by Rodin et al.(Rodin 2004) …………………...… 38
Figure 2.22: The Finite State Machine that is describing an Agent’s Behavior (Rodin
2004)…………………………………………………………………………... 39
Figure 2.23: The Architecture of the Proposed Method of Melkemi (Melkemi 2006) ……. 40
Figure 2.24: The Proposed Method using MAS by Spinu (Spinu 1996) …….…………….. 41
Figure 2.25: The Proposed MAS by Boucher et al.(Boucher 1998) …………….…………. 43
Figure 2.26: The Local Neighboring Region of an Agent at Location (i,j) (Liu 1999)…..… 44
Figure 2.27: A Global View of the Framework and Information Flow of the Proposed
Method by Germond (Germond 2000)……..……………………………….… 45
Figure 2.28: The Conceptual Framework of Duchesnay et al. (Duchesnay 2001) ………… 46
Figure 2.29: The Graphical Representations of the Seven Behaviors (Duchesnay 2003) …. 47
Figure 2.30: The Multi-Agent Optimization Model for Unstained Cell Images by Khosla
et al. (Khosla 2003) ……...……………………………………........................ 48
Figure 2.31: The Multi-Agent Soft Computing Model for Unstained Cell Images by Lai et
al. (Lai 2003) ……….……………………………………………………...…. 48
Figure 2.32: The Proposed Multi-Agent Framework by Richard et al.(Richard 2004) ……. 50
Figure 2.33: The Proposed Method by Benamrane et al. (Benamrane 2007) ……………… 51
Figure 2.34: The Conceptual Diagram of the Phoenix Segmentation Algorithm (Peng
1998 b)………………………………………………………………………… 56
Figure 2.35: The Segmentation Evaluation by RL (Bhanu 2000) ……................................. 56
Figure 2.36: The Standard Model of RL (Shokri, 2003) …………………………………... 57
Figure 2.37: The RL Model used in the Proposed Method by Sahba (Sahba 2006 b) …….. 58
Figure 2.38: The General Model used in the Proposed Method (Sahba 2008) ………..…… 58
Figure 3.1: The Global View of our Proposed Model...…………….…………………….. 63
Figure 3.2: An Example for Calculating the Number of the Agents within a Window Size
of 7�7 over an Image Size of 16�16…...……..…..………………………… 65
Figure 3.3: (a) The Original CT Image, (b) The Manually Segmented Image……………. 66
Figure 3.4: The Global View of our Proposed Method in Training Phase...……………… 67
Figure 3.5: The RL Agent’s Behavior.…………………………………………….…….... 68
Figure 3.6: The Example of the number of States for an Image with two Regions……….. 70
Figure 3.7: The Example of the number of States for an Image with three Regions; each
window shows a typical sub-image…………………………………………... 71
ix

Figure 3.8: An Example of Defining Action using the Maximum and Minimum
Thresholding Gray-scale Value of a Typical Sub-image………………...…… 72
Figure 3.9: GUI of the Training Phase..……………………..…………….………………. 74
Figure 3.10: The Global View of our Proposed Method in Testing Phase.......……………. 75
Figure 3.11: The Agents’ Behavior in Testing Phase….…..……………...………………... 77
Figure 3.12: GUI of the Testing Phase..……………………………………………………. 79
Figure 4.1: The Segmentation Example from Two Experiments and Four Different Slices
of 3D Images, (a) Input Image (b) Result from Proposed Method and (c)
Ground Truth Image…………………………………………………………... 84
Figure 4.2: ROC Curve for the First Data set…………………….……………………….. 87
Figure 4.3: ROC Curve for the Second Data set……….………………………………….. 87
Figure 4.4: GUI of the Training Phase to Suggest the User Thresholding Range of each
Region...……………………………………………………………………….. 90
Figure 4.5: GUI of the Testing Phase…………….……………………………………….. 90
Figure 4.6: The Segmentation Example from Two Experiments and Four Different Slices
of Data sets, (a) Result from our Method, (b) Input Image.…………………... 92
Figure 4.7: ROC Curve of the First Data set…………………………….………………… 94
Figure 4.8: ROC Curve of the Second Data set …………………………………………... 94
Figure 4.9: The Computation Time of all Data sets; X axis shows the image number
(identity) and Y axis shows the computation time …...…………………….… 96
x

List of Tables
Table 2.1 The Comparison among the Segmentation Methods (Chitsaz 2008) …………… 27
Table 2.2: The Comparison of the Segmentation Methods using Non-medical Images by
Agent Properties (Chitsaz 2008) ………………………………………………. 52
Table 2.3: The Comparison of the Segmentation Methods using Medical Images by Agent
Properties (Chitsaz 2008) ….……………………………………………………. 53
Table 2.4: The Comparison between Multi-Agent and Non-Agent Segmentation Methods
(Chitsaz 2008)…………………………………………………………………… 54
Table 4.1: Details of the Image Data set used in PRLM……………..…………………… 82
Table 4.2: TPVF and FPVF of PRLM…………………………..……………….……….. 86
Table 4.3: Efficiency of the PRLM……….………………………………….…..………. 88
Table 4.4: Details of the Image Data set used in PMAM…………...…………………… 91
Table 4.5: TPVF and FPVF for the Testing Phase of PMAM…………..……………….. 93
Table 4.6: Mean User-interaction Time and Computation Time of PMAM………………. 95
Table 5.1: Efficiency Comparison of Image Segmentation Methods………………………. 98
xi

A. List of Abbreviations
List of Abbreviations and Symbols
2D Two-Dimensional
3D Three-Dimensional
CT Computed Tomography
DICOM Digital Imaging and Communications in Medicine
EM Expectation-Maximization
FPS Frames per Second
FPVF False Positive Volume Fraction
GA Genetic Algorithm
GUI Graphical User Interface
HSV Hue, Saturation, and Value
ICA Intelligent Control Agent
IPA Image Processing Agents
KP Knowledge Processors
KS Knowledge Servers
MAS Multi-Agent System
MRI Magnetic Resonance Imaging
PMAM Proposed Multi-Agent Model
PRLM Proposed Reinforcement-Learning Model
RF Radiofrequency
ROC Receiver Operating Characteristic
RL Reinforcement Learning
SPRT Sequential Probability Ratio Test
SRLM Standard Reinforcement Learning Model
TPVF True Positive Volume Fraction
xii

B. List of Symbols
� Complete array of image pixels
f(x, y) Intensity of pixel at (x, y)
Q(s,a) Sum of future payoffs r obtained by taking action a from state s.
s State
a Action
r Reward
� Learning Rate
� Discount Factor
� Probability Factor of � -Greedy Algorithm
C M
d
Segmented Image
A� ( C,
f ) Image
C 2D rectangular array of pixels
f (c)
Intensity of any pixel c in C
U d
Binary image representation of a reference superset of pixels
xiii

List of Publications
Chitsaz, M., Woo, C.S. (2008). The Rise of Multi-Agent and R.L. Segmentation Methods
for Biomedical Images. The 4th Malaysian Software Engineering Conference
(MySEC’08), Kuala Terengganu, Malaysia.
Abstract-Image segmentation is an important operation in image analysis. We present a critical
assessment of conventional methods for image segmentation. Current segmentation approaches
have some common disadvantages. They are sensitive to noise and require manual fine-tuning. We
found that multi-agent and reinforcement learning (RL) methods are suitable for biomedical image
segmentation. They have shown very outstanding results and high adaptability in most of the
segmentation scenarios.
Chitsaz, M., Woo, C.S. (2009). Medical Image Segmentation using Reinforcement
Learning Agent. International Conference on Digital Image Processing (ICDIP'09),
Bangkok, Thailand, IEEE Computer Society Press.
Abstract—the principal goal of this work is to design a framework to extract one or more objects of
interest from Computed Tomography (CT) images. The learning phase is based on reinforcement
learning (RL). The input image is divided into several sub-images, and each RL agent works on it
to find the suitable value for each object in the image. For each state in the environment has been
defined some actions; also a reward function compute reward for each action of RL agent. Finally
the valuable information is stored in Q-Matrix, and the final result can be used to segment new
similar input images by applying it. The experimental results for cranial CT image show accuracy
of segmented image is more that 95%.
Chitsaz, M., Woo, C.S. (2009). A Multi-Agent System Approach for Medical Image
Segmentation. International Conference on Future Computer and Communication
(ICFCC'09), Kuala Lumpur, Malaysia, IEEE Computer Society Press.
Abstract— Image segmentation still requires improvements although there have been research
works since the last few decades. This is coming due to some issues. Firstly, most image
segmentation solutions are problem-based. Secondly, medical image segmentation methods
generally have restrictions because medical images have very similar gray level and texture among
the interested objects. The goal of this work is to design a framework to extract simultaneously
several objects of interest from Computed Tomography (CT) images by using some prioriknowledge.
Our method used properties of agent in a multi-agent environment. The input image is
divided into several sub-images, and each agent works on a sub-image and tries to mark each pixel
as a specific region by means of given priori-knowledge. During this time the local agent marks
each cell of sub-image individually. Moderator agent checks the outcome of all agents’ work to
produce final segmented image. The experimental results for cranial CT images demonstrated
segmentation accuracy around 90%.
xiv

Acknowledgement
I would like to thank Dr. Woo Chaw Seng, my supervisor, for patiently giving me good
advice over the course of this program, and for spending a lot of time talking to me
about ideas that led up to this works.
Financial support from Dr Woo along with an ample supply of awards and
Fellowship/research assistantship from, or through, Institute of Graduate Studies,
Institute of Research Management and Monitoring, Faculty of Computer Science and
Information Technology, and Department of Artificial Intelligence contributed greatly
toward the completion of this research.
I would like to thank all my professors and colleagues from whom I learned so much.
Special thanks go to my parents, my brothers and my friends, for their continual love,
support, and encouragement throughout my time in graduate school. A particular debt
of thanks is due to Hadi for his efforts in preparing many of the line drawings in my
dissertation.
Certain studies described in this thesis would not have been possible without images
from University of Malaya Hospital.
xv

Chapter 1
Introduction
1.1 Background
Medical images can represent in a two-dimensional array of picture element (pixel) or
in a 3D array (voxel). Medical images are normally stored accordingly to Digital
Imaging and Communications in Medicine (DICOM) standard for distributing and
viewing of any medical image regardless of its origin. Digital image processing is
concerned with the analysis and manipulation of images by computer. Enhancement,
segmentation, quantification, registration, visualization, and compression are the most
famous tasks in image processing and computer vision (Bankman, 2008). Image
segmentation is a pre-processing task for separating an image to several components. In
other words, segmentation is a process of identifying the objects in an image. For
example in a computed tomography (CT) image of the head, the components may
consist of bone, tooth, fat, etc. In another definition, image segmentation is categorizing
of the image to some disjoint partitions whereas the whole of partitions reconstruct the
primarily image again (Pham, 2000).

Image segmentation techniques have been an invaluable task in many domains such as
quantification of tissue volumes, diagnosis, localization of pathology, study of
anatomical structure, treatment planning, partial volume correction of functional
imaging data, and computer-integrated surgery (Pham, 2000). Segmentation algorithms
play a unique role in machine vision systems as it creates a bridge between the low-
level and high-level processing operations. Low-level operation is carried out on image
array of rows data thus adopts a bottom-up approach to image analysis. High-level
processing is important with the manipulation of high-level data, abstract data
presentation and thus favors a top-down approach. Segmentation can employ either or
both of these approaches (Awcock, 1995).
The image segmentation can perform manually but this is a time-consuming task.
Therefore, automatic segmentation is preferable. Although there have been many
researches carried out in last two decades, image segmentation is still a debatable
problem (Liu, 2006). Image segmentation suffers from two main problems, which these
generally caused the segmentation an unsolved problem. Firstly, every solution for
image segmentation is problem-based. Secondly, the noisy nature of medical images
makes it difficult to relate each pixel within an image to different texture classes
(Withey, 2006).
1.2 Motivation
There is a broad range of facial diseases that could cause face deformation. Facial
diseases involve facial area injuries and tumors, which inherit by syndromes such as
Crouzon-Syndrom, Apert-Syndrom and developmental diseases like distributed growth
of the jaw (Koch, 2002). Face is the first organ of body that has seen at the first sight.
Everybody solicits that own face appear gracefully in a first sight so these general
2

elieve his/her importance of facial diseases. Thus, the importance of facial surgery is
very tangible, and the risk of this kind of treatment operation is very high. The specialist
and the patient are very eligible to imagine the face after the surgery, however, the
specialist would be predicting the post-operation of face but the outcome is not very
clear for the patients. Consequently, simulating facial surgery and predicting the effect
of an operation is very pivotal. One of the primary preprocessing levels in every kind of
surgical simulation is image segmentation. Therefore, the accuracy of this level is
essential because the result will affect the overall outcome.
Our motivation caused that we propose a method that can segment the head CT image
of patient by the way that the result will be practical for the remainder of planning.
Because the accuracy of the image segmentation can be impressed in the reconstruction
of 3D image of patient face, the significant of the segmentation is very evident.
The other motivation for doing this research is the current methods in image
segmentation field have problems in some areas. The Multi-Agent System (MAS) has
used in several image segmentations that investigate in chapter 2. However, in practice
they still encounter the three main limitations:
a. Since the Reinforcement Learning (RL) system is an effective to find the
optimal result in the system. A few researchers have investigated this system
for image segmentation. Furthermore, no research has attempted in the field
of CT image segmentation.
b. Other researchers do still not do combination of the characteristics of pure
agent with the RL agent. We expect this can be resulted in a better
segmentation outcome.
3

c. Due to the lack of practical framework for segmentation, we endeavor to
present a structure for image segmentation, which it can use for some
different image modalities.
1.3 Problem Description
The problem in this dissertation is to segment CT image of the head, as shown in figure
1.1(a), which the outcome of this process will be used for reconstruction of 3D image of
the face, or other post-processing steps. The aim of our segmentation method is to
separate the skin tissue from the bone and the background. Figure 1.1(b) shows a
segmented image of Figure 1.1(a). The red color shows the background or air, green is
skin, and the blue color depicts the bone. The represented colors are optional. A new
framework for this kind of image segmentation proposes using the MAS and a special
type of agent, which calls RL agent.
Figure 1.1 (a): A Slice of CT Image of a Human Head, (b): Segmented Image of (a)
1.4 Goal and Objectives
(a) (b)
The goal of this dissertation is to propose a method to segment the CT image of a head.
For the image segmentation of head, we have the following objectives:
4

� Review of existing experimental studies and investigate the Agents Technology
for image segmentation;
� Propose a novel approach by RL agent in the multi-agent framework which will
be quicker, more accurate, and more robust;
� Evaluate the outcome of segmented image by the proposed method with an
appropriate estimation approach.
1.5 Project Scope
The scope in this thesis is the segmentation of head CT image for identification of
targeted tissues, structure and bone, this problem is carried out by RL agent in the MAS,
which they have used both the social ability of pure agent and the properties of RL
agent. Therefore, the scope of problem is just concerned with CT image of head and the
methodology which is proposed, is implemented by collaborating, cooperating and
negotiation of RL agent. The data sets, which used in this thesis, consist of two different
resources. The first data set includes 33 images from Hospital of University of Malaya.
The second data set contains 28 images that downloaded from
http://pubimage.hcuge.ch:8080/.
1.6 Dissertation Organization
The remainder of this thesis has the following components. The next chapter, literature
review, is a concise perusal of the past and recent methods in medical image
segmentation. Before summarizing and analysis those methods with the proposed
framework, the history of agent and MAS, and the face anatomy present. There are brief
descriptions of recent methods, and there are comparison between conventional
5

methods and agent-based methods. In addition, advantages and disadvantages of the
other methods mention and compare.
Chapter 3 brings about our proposed methodology which consists two phases. The first
phase bases on the RL approach. Another phase implements by multi-agent properties.
In discussion section, some contributions of our proposed framework discuss.
Chapter 4 discusses how we evaluate our proposed method, both qualitatively, through
image display, and quantitatively, through evaluation experiments. In addition, the
efficiency of proposed method compared to the other methods.
The last chapter provides discussion for each phase of the proposed method. It mentions
all contributions and weaknesses of proposed method. At the end, the achievement, and
ideas for future works present.
Finally, more results of our method provide in appendices.
6

Chapter 2
Literature Review
2.1 Overview
Based on the problems mentioned in chapter 1, this chapter presents review of recent
works in image segmentation. In addition, the backgrounds of research described here.
The Three-dimensional Medical Imaging section is about two commonly used
modalities for medical imaging; Computed Tomography (CT) and Magnetic Resonance
Imaging (MRI). The Skull Anatomy section described anatomy of the face, which uses
in our research to segment the CT image of head. Moreover, different CT images are
shown to clarify each image is related to different parts of head and its details.
The Segmentation Methods section represents general methods, which are widely used
in medical image segmentation. Moreover, those methods categorize based on the
characteristics of the proposed model. This classification takes from a number of
publications regarding medical image segmentation. Since the publications are quite
large the information intend to be representative rather than exhaustive.
7

The Agent and MAS section defines agent and multi-agent system. In the end of the
section, the advantages of using agent in a typical system are listed, on the other words,
the main reason which we used the agent for segmenting medical image is mentioned.
The Standard-Reinforcement Learning Model (SRLM) section explains a thorough
standard model of RL agent, and some basic characteristics of RL agent.
The Image Segmentation by Autonomous Agents in MAS section introduces the recent
segmentation methods, which employed agents to segment images. The Image
Segmentation by Reinforcement Learning Model discusses about the existing
segmentation literature, which employed RL model.
Finally, the summarizing of this chapter mentions in the last section with a discussion
about image segmentation methods using agent or RL agent and methods that do not
use agent.
2.2 Three-Dimensional Medical Imaging and Skull Anatomy
In this section, a brief explanation of two commonly used modalities for medical
imaging provides. Furthermore, the skull anatomies will explain to facilitate the
correlation with the anatomical images.
In general, CT is the modality of choice for bony details, and MRI is superior to CT for
soft tissue details. CT is superior to MRI for detecting classification, and it is the study
of choice for evaluation of foreign bodies. Moreover, MRI has not known biological
side effect. Each of the modalities images the body through using a form of energy to
8

map the internal structures. X-rays and radio waves are the energies used for CT, and
MRI, respectively (Valvassori, 1995).
2.2.1 Computed Tomography Images
CT is one of the most powerful diagnostic tools available in medicine nowadays.
Advantages of the CT system are in its ability to distinguish smaller differences in x-ray
attenuation between tissues, and to provide extremely high spatial resolution data. These
advantages have resulted in considerable improvement in image quality. One of the
major advances of the CT has been the ability to generate a comprehensive scan of a
region or even entire body (Walter, 2007).
The CT scanner, as shown in Figure 2.1, consists of a gantry that rotates around the
patient. The x-ray tube and detectors mount on the gantry, along with a host of
additional electronics and equipment. A table for the patient moves through a
cylindrical opening in the middle of the gantry. The gantry rotates at a high rate of
speed around the patient, who positions within the bore of the CT scanner. Data record
as the patient moves through the x-ray beam, creating what call ‘projections’. For each
individual image, multiple projections at various angles acquire. The data collected
from these multiple projections then transfer to a computer, which uses a mathematical
algorithm to reconstruct the CT image and store it in digital format (Walter, 2007).
The CT image is a digital file consisting of the pixels. The CT system calculates the
amount of the x-ray attenuation for each pixel. These attenuation values are
standardized and called ‘Hounsfield’ or ‘CT numbers’. Once the CT numbers calculate
they typically map to a shape of gray to create an image. The black areas represent
9

egions with lower CT attenuation (like air) while the white areas represent regions with
higher CT numbers (like bone), a sample of the CT images has shown in Figure 2.2.
Figure 2.1: CT scanner (Jeri, 2008)
Figure 2.2: Human Head CT Slices at Axial View (Obaidellah, 2006)
2.2.2 Magnetic Resonance Imaging
MRI provides an extremely high level of the detail and concerning the anatomy and
pathology in vivo with the radio waves and a strong magnetic field. The Magnetic
10

Resonance Image Scanner is a medical device used to generate images of the soft
tissues for the diagnosis of illnesses.
In MRI, the patient is placed within the bore of a large magnet. This magnet creates the
external magnetic field; a MRI scanner is shown in Figure 2.3. The signal used to create
an image is created by moving the group of spins out of alignment with external
magnetic field (the z-axis). An image is created by measuring the signal or echo of the
protons processing about external magnetic field after the application of a
radiofrequency (RF) pulse sequence. A specific RF pulse is broadcast into the body and
then moves the net magnetization vector so that it is processing in x-y plane. Motion
and flowing blood can detect on MR images. Thus, flowing blood can map as bright or
dark depending upon the pulse sequence used to obtain the images.
Figure 2.3: MRI Scanner (Garrobo, 2006)
The spatial information in MR images obtain by applying smaller external magnetic
gradients across the patient. These gradients determine what level or slice of tissue is to
be imaged. Through the appropriate application of these magnetic gradients, MR images
11

can be obtained in any plane throughout the body: coronal, sagittal, transaxial, or
oblique (Walter, 2007).
2.2.1 Skull Anatomy
The bones of the skull protect the brain and guard the entrances to the digestive and
respiratory systems. The skull contains the twenty-two bones; 8 from the cranium, or
braincase, and 14 are associated with the face (Marieb, 2000). Figure 2.4 shows the
anterior and lateral view of the skull.
Figure 2.4: The Lateral (left) and the Anterior (right) View of the Skull (Gray, 1918)
2.2.1.1 Cranial Bones
The cranium consists of the 8 cranial bones: the occipital bone, frontal bone, sphenoid,
ethmoid, and the paired parietal and temporal bones. Together, the cranial bones
enclose the cranial cavity, a chamber that supports the brain. The outer surface of these
bones provides an extensive area for the attachment of the muscles that move the eyes,
jaws, and head. Figure 2.5 represents the cranial bones.
12

2.2.1.2 Facial Bones
Figure 2.5: Cranial Bones of the Skull (Martini, 2004)
Facial bones protect and support the entrances to the digestive and respiratory tracts.
The superficial facial bones (the paired maxillary, lacrimal, nasal, and zygomatic bones
and the single mandible) provide areas for the attachment of the muscles that control
facial expressions and assist in manipulating food. The deeper facial bones (the palatine
bone, inferior nasal conchae, and vomer) help separate the oral and nasal cavities,
increase the surface area of the nasal cavities (Martini, 2004). The two maxillae fuse to
form the upper jaw. All facial bones except the mandible join the maxillae; thus the
maxillae are the main, or ‘keystone’, bone of the face (Marieb, 2000). The mandible,
forming the skeleton of the chin, is one of the largest bones of the skull and the only
moveable one (Hiatt, 1982). Figure 2.6 (a) and (b) show the maxillae and the mandible
bone of facial face respectively.
13

(a)
(b)
Figure 2.6: (a) The Maxillae Bone (b) The Mandible Bone (Gray, 1918)
2.2.1.3 Anatomical Structure that Used as the Case Study
In this section, we present some sketches of the human skull in order to show the case
studies of this research. Figure 2.7 shows the top side of the skull, through this sketch
the brain is visible, but in CT images the soft tissue is not very clear meanwhile the
Figure 2.8 (a) and (b) show one slice of this part of the skull, 1-3 and 1-6 section
respectively. Figure 2.9 depicts a wide spectrum of the skull, where the Figures 2.10(a-
d) shows the CT images of the specific section of Figure 2.9. In the Figure 2.11, the
14

inferior part of the skull is shown; also Figure 2.12(a-b) shows the CT image of the
particular section of Figure 2.11.
Figure 2.7: The Top Side of the Skull (Walter, 2007)
(a) (b)
Figure 2.8: (a) CT Image of 1-3 section (b) CT image of 1-6 section (Obaidellah, 2006)
Figure 2.9: The Middle Part of the Skull (Walter, 2007)
15

(a) (b)
(c) (d)
Figure 2.10: (a) CT Image of 1-10 Section (b) CT Image of 1-12 Section (c) CT Image
of 1-15 Section (d) CT Image of 1-20 Section (Obaidellah, 2006)
Figure 2.11: The Inferior Part of the Skull (Walter, 2007)
16

Figure 2.12: (a) CT Image of 1-22 Section (b) CT Image of 1-23 Section (Obaidellah,
2006)
2.3 Segmentation Methods
The objective of the segmentation method is to identify the disjoint objects into image,
which have certain uniformity. A formal definition of the segmentation in (Awcock,
1995) is presented in following.
Consider an image array of m columns by n rows, Figure 2.13(a). Let � denote this
complete array of the image pixels, the set of the pairs {i,j} where i=0,1,2,…,(m-1) and
j=0,1,2,…,(n-1).
(n-1)
(a) (b)
Rb
�
(m-1)
(a) (b)
Figure 2.13: Uniformity Predicate of the Segmentation (Awcock 1995)
R1
R2
Rg
R3
R4
Rh
Rt
17

Let Ra be a non-empty subset of R consisting of the sequential image pixels. A
uniformity predict, P(Ra), is a logical statement which assigns the value True or False to
Ra, depending on the properties related to the intensity matrix f(i,j) for the point of Ra.
A segmentation of the array of R, Figure 2.13(b), is a partition of R into disjoint non-
empty subsets R1, R2, R3… Rt and can be defined mathematically as:
I. � Rg � R for g=1,2,3,…,t.
II. R g
is a connected region; g=1,2,3,…,t.
III. Rg� Rh��
for all g and h; g � h.
IV. P ( Rg)
= True for g=1,2,3,…,t.
V. P( Rg�
Rh)
= False for g � h.
There are a number of the image features, which can be used in image segmentation
methods, such as gray-level values, color parameters, boundary and range information,
texture and motion. These features can be determined how much the segmentation
methods have achieved to the uniformity criteria.
2.3.1 Several Classifications of Segmentation Methods
In this section, it is presented some main approaches based on the opinion of
(Bovenkamp, 2004; Withey, 2006; Awcock, 1995; Umbaugh, 1998; Kagawa, 1999;
Pham, 2000) other researchers. The next section, a brief description of main
segmentation methods that are classified by the mentioned researchers brings out.
18

Awcock and Thomas (Awcock, 1995) have dichotomized the segmentation methods
into:
(a) Pixel-based or local or discontinuity methods;
(b) Region-based or global or similarity approaches.
These approaches are complementary; in practice the results are not the same in each
case. The first method (a) detects and enhances the edges within element and links these
to construct an object. They only used point-wise or nearest-neighbor local information
and no information is taken of the general properties of the whole region. The second
one seeks to create regions directly by collecting the common features of a group of the
pixels into areas or region of the uniformity.
Some methods mention for the approach (a) such as Edge Detection, and Boundary
Detection. The examples of the approach (b) are Region merging and splitting, and
Thresholding.
Umbaugh in his book (Umbaugh, 1998) mentioned the segmentation techniques into
three classification; region growing and shrinking, clustering methods, and boundary
detection. The one sample of region growing is, the segmentation starts with smallest
level and only merges, with no region splitting. In region shrinking, the entire image
consider as initial region, and then follows an algorithm, the image is only spitted. The
clustering techniques are image segmentation methods; based on some measure of the
similarity in image, the image is cluster to some groups. The boundary detection is
carried out by seeking the boundary between objects, this method is usually commenced
by marking pixel may be a part of an edge.
19

Pham et al., in their survey (Pham, 2000) divided the common segmentation methods
into eight categories:
(a) Thresholding approaches;
(b) Region growing approaches;
(c) Classifiers;
(d) Clustering approaches;
(e) Markov random field models;
(f) Artificial neural networks;
(g) Deformable models, and
(h) Atlas-guided approaches.
However, the thresholding, classifier, clustering, and Markov random field approaches
can consider as independent methods of the pixel classification.
Kagawa et al. in (Kagawa, 1999), considered the segmentation methods to the four main
approaches:
(a) Edge detection,
(b) Region growing,
(c) Method of clustering,
(d) Statistical methods.
Edge detection is a method, which detects the location of the feature pixel, is changed
precipitously. Region growing is an approach that it makes use of share common feature
that the regions have. Methods of clustering are the approach that uses the feature space
of the color information and so on. Statistical methods are the approach that it makes
use of the statistical and/or structural texture that the image has.
20

Bovenkamp et al., in (Bovenkamp, 2004) discussed the basic image interpretation
strategy. They distinguished them to three basic strategies:
(a) Bottom-up,
(b) Top-down,
(c) Hybrid.
The bottom-up strategy does not have any information of the object within image and
based on pixels in the image can achieve the segmentation goal. The top-down strategy
is to assume an object to be in the image and then to seek for it, for example using a
deformable model such as snakes and active shape model. For conveying limitation
which exists in the above two strategies, the hybrid strategy is posed. This strategy
combined both bottom-up and top-down process, a common implementation of the
hybrid strategy is to create a feedback loop from the symbols back to image, for
example locally re-segment the image given evidence from the image data and
reasoning process.
Withey in his thesis (Withey, 2006) classified the segmentation methods into three
generations; the first generation has carried out the most primary and lowest level of the
processing, the image models, optimization methods, and uncertainty models are used
in the second generation, and the third generation algorithms have capability of the
incorporating knowledge.
In the first generation, thresholding, region growing, region split/merge, edge detection,
and edge tracing methods are noticeable. The statistical methods, C-mean clustering,
Fuzzy connectedness, deformable models, watershed algorithm, neural networks, and
multi-resolution methods are citable for the second-generation algorithms. The method
combinations and knowledge-based segmentation is generated the third.
21

2.3.2 Brief Description of Main Segmentation Methods
2.3.2.1 Thresholding
The thresholding technique is the primitive technique in image segmentation. This
method produces regions of the uniformity within an image based on some threshold
criterion, T. the function T defines:
T = T{x, y, A(x, y), f(x, y)},
Where f(x, y) is the intensity of the pixel at (x, y), and A(x, y) denotes some local
property in the neighborhood of this pixel.
A threshold image g(x, y) defines:
g(x, y) =
�1
�
�0
if f ( x,
y)
� T �
� .
if f ( x,
y)
� T �
The thresholding technique can identify as:
(a) Global threshold: T = T{f(x, y)},
Where T is depended only on the intensity of the pixel at x, y.
(b) Local threshold: T = T{A(x, y), f(x, y)},
Where T is depended on a neighborhood property of the pixel as well as its
intensity.
(c) Dynamic threshold: T = T{x, y, A(x, y), f(x, y)},
Where T is depended on the pixel coordinates, and the other two criteria
Selection of the value of the threshold, T, is critical issue. It is common to study
histogram in order to find the appropriate threshold. One variation on the simple
threshold is interval threshold operation. A binary image is produced where all gray-
level values falling between two threshold values T1 and T2. However, a complex image
exhibits the more gray-level threshold in its histogram, for this kind, multiple
22

thresholding can be used to reduce the number of the gray-level values in the image
(Awcock, 1995).
2.3.2.2 Region Growing
Region growing algorithms based on the growth of a region whenever its interior is
homogeneous according to certain features as intensity, color or texture. The
implemented algorithm follows the strategy of a typical Region Growing: it is based on
the growth of a region by adding similar neighbors. Region Growing is one of the
simplest and most popular algorithms for region-based segmentation. The most
traditional implementation starts by choosing a starting point called seed pixel. Then,
the region grows by adding similar neighboring pixels according to a certain
homogeneity criterion, increasing gradually the size of the region. Therefore, the
homogeneity criterion has the function of deciding whether a pixel belongs to the
growing region or not. The decision of merging generally based only on the contrast
between the evaluated pixel and the region. However, it is not easy to decide when this
difference is small (or large) enough to take a decision.
The Split-and-Merge algorithm is related to the region growing; a typical split and
merge techniques consist of two basic steps. First, the whole image is considered as one
region. If this region does not satisfy a homogeneity criterion the region is split into four
quadrants (sub-regions) and each quadrant is tested in the same way; this process is
recursively repeated until every square region created in this way contains
homogeneous pixels. Next, in the second step, all adjacent regions with similar
attributes may be merged following other (or the same) criteria. The criterion of the
homogeneity is generally based on the analysis of the chromatic characteristics of the
region. A region with small standard deviation in the color of its members (pixels) is
23

considered homogeneous. The integration of the edge information allows adding to this
criterion another term to take into account. So, a region is considered homogeneous
when is very free of the contours (Chen, 1980).
2.3.2.3 Edge Detection
The edge detection methods attempt to sketch the object in image by boundary instead
of volume of it. The edge detection is not a pure segmentation methods, it is used by the
other methods for supplementing.
2.3.2.4 Classifiers
The classifier methods are pattern recognition techniques, which classify object into one
of several categories based on feature space. A feature space is the range space of any
function of the image, with the most common feature space is the image intensities.
The classifiers are known as supervised approach; those divide to distribution free or
statistical. Distribution free methods do not require knowledge of any priori probability
distribution functions and reasoning also heuristics are the basis of these. Statistical
techniques are based on probability distribution models.
Suppose there are K different objects or pattern classes S1, S2, …, Sk, …, SK. each class is
characterized by Mk prototype, which have N � 1 feature vectors
(k )
y m , m=1, …, Mk. A
simple classifier is k-nearest neighbor classifier, for classifying the image to Si, if
among a total of k nearest prototype neighbors, the maximum number of the neighbors
belong to class Si. In statistical classifiers techniques it is assumed the different object
classes and feature vector have a probability density. Let P(Sk) be a priori probability of
24

the occurrence of the class Sk and p(x) be the probability density function of the random
feature vector observed as x. The Bayes’ minimum-risk classifier is a kind of the
statistical classifiers that its objective is to minimize the average loss or risk in assigning
x to a wrong class.
There are also some sequential classification techniques such as sequential probability
ratio test (SPRT), where decision can be made initially using fewer than N features and
refined as more features are acquired sequentially (Jain, 1989; Pham, 2000).
2.3.2.5 Clustering
Clustering methods carry out the same function as classifier methods without the use of
the training data. In addition, they are termed unsupervised methods. A cluster is a set
of the feature space, which their local density is large in comparison with the density of
the feature point in the neighbors. In order to tackle the lack of the training data, as
shown in Figure 2.14, clustering methods iterate between segmenting the image and
characterizing the properties of the each class. In a sense, clustering methods train
themselves using the available data.
Input
data
Partition
Test and
Merge
Split
Conve
rgence
Figure 2.14: A Clustering Approach (Jain, 1989)
25

Three commonly used clustering algorithms are the K-means, the fuzzy c-means
algorithm, and the expectation-maximization (EM) algorithm. The K-means clustering
algorithm clusters data by iteratively computing a mean intensity for each class and
segmenting the image by classifying each pixel in the class with the closest mean. The
fuzzy c-means algorithm generalizes of K-means allowing for soft segmentations based
on fuzzy set theory. The EM algorithm applies the same clustering principles with the
underlying assumption that the data follows a Gaussian mixture model. It iterates
between computing the posterior probabilities and computing maximum likelihood
estimates of the means, co-variances, and mixing coefficients of the mixture model
(Jain, 1989; Pham, 2000).
2.3.2.5 Deformable Models
Deformable models are known techniques for boundary extraction and segmentation of
the medical images. One of the earlier active contours is snake. It formulated as a
parametric model, they consist of a curve, which can dynamically match to object
shapes in response to internal and external forces. To describe an object boundary in an
image, a closed curve or surface must first place near the desired boundary and then
allowed to undergo an iterative relaxation process. Internal forces are computed from
within the curve or surface to keep it smooth throughout the deformation. External
forces are usually derived from the image to drive the curve or surface towards the
desired feature of the interest (Giraldi, 2006; Pham, 2000).
2.3.2.6 Neural Network
An artificial neural network is a set of the parallel elements called neurons that emulate
a biological neural learning system. Each neuron can perform elementary computation
26

such that weights assign to the connections is achieved to learning (Pham, 2000). The
neural network acts as classifiers where a set of the features is determined for each
image pixel and presented as input to the neural networks (Withey, 2006).
2.3.3 Comparison of the Segmentation Methods
It is useful to compare all segmentation methods in the previous section. In Table 2.1,
the advantages and disadvantages of all methods list by reviewing of these literatures
(Pham, 2000; Freixenet, 2002; Kirbas, 2003; Withey, 2006).
Table 2.1: The Comparison among the Segmentation Methods (Chitsaz, 2008)
Methods Advantages Disadvantages
Thresholding Simple implementation.
Region Growing
Good for small and simple
structure, easy to detect the
global structure of the image.
Edge detection Useful for boundary detection
Classifiers
Clustering
Deformable models
Neural networks
Can apply to multiple-channel
image.
Do not need training data, fast
computation, robustness to
intensity inhomogeneities
Robustness to noise and spurious
edges,
Parallel, easily incorporate
spatial information to
classification procedures, ability
to learn
Sensitive to noisy image, can
not apply to multiple-channel
image.
Manual interaction is needed,
Over-segmentation, sensitive to
noisy image
It is difficult to detect edge in
complex image, sensitive to
noisy image.
Requirement to manual
interaction, sensitive to
intensity inhomogeneities
Dependency to the number of
the clusters and features,
dependency to initial
segmentation
Requirement to manual
interaction, dependency to
parameter value
require to train every time a
new feature is introduced the
network, difficult to debug the
performance of the network
27

2.4 Agent and Multi-Agent System
Although many people use the term of the agent and multi-agent who are working in
closely related areas. There are no widely accepted definitions of these terms, and the
definitions are still open challenge. However, in many literatures, some attributes of the
agent are similar.
The following properties are common for a hardware or software-based computer
system agent in a weak notation (Wooldridge, 1997; Kagawa, 1999):
� autonomy: agents accomplish without the direct interposition of the humans or
others, and having control over their actions and internal state;
� social ability: agents cooperate with other agents (and may be humans)
� reactivity: realizing their environment, and responding to changes that occur in it;
� Pro-activeness: having ability to exhibit goal-directed behavior by taking the
initiative;
� Robustness: should be prepared to learn and to recover from failure.
The other properties of the agent, which related to its context discussed. For example,
mobility is the ability of an agent to move around an electronic network, such as
moving agent from a computer to another. Veracity is the assumption that an agent will
not knowingly communicate false information. Benevolence is the assumption that
agents always attempt to do what is asked of it. Finally, rationality is the assumption
that an agent will strongly act in order to achieve its goals (Wooldridge, 1995).
28

The internal structure of an agent may consist of several units as shown in Figure 2.15.
In general, agents have the following units (Rares, 1999):
� Input units, for receiving incoming data;
� Output units, for delivering agent’s results;
� Planning units, for determining the processing strategy;
� Control units, which put into practice the plan elaborated by the planning units,
and coordinate the execution;
� Evaluation units, for checking the quality of the processing operations;
� Learning units, for knowledge acquisition and adaptive behavior.
Agent
Planning Unit
Figure 2.15: The Internal Structure of a Typical Agent (Chitsaz, 2008)
These units are varied by nature of each problem, and probability in some cases the
other units has been added or decreased.
Control Unit
Input Evaluation
Unit
Output
Learning Unit
MAS is overall a system with several entities which they have some mutual behavior
like cooperation, coordination and negotiation. In (Jennings, 1998) MAS is defined as
they are ideally suited to representing problems that have multiple problem solving
29

methods, multiple perspectives and/or multiple problem solving entities. Such systems
have the traditional advantages of the distributed and concurrent problem solving, but
have the additional advantage of the sophisticated patterns of interactions”. Therefore,
in MAS every agent is an ingredient in a massive system, and they just have knowledge
about their environs so by cooperating, coordinating and negotiating they can able to
achieve goal quickly.
The MAS is broadly used in variety fields, such as robotic, etc. we intend to use this
system as the skeleton of our framework, because of following reasons (Crevier, 1997):
� Ease of construction and maintenance. It is easier to set up and repair a
collection of independent modules than a single huge program
� Ability to benefit from parallel architectures.
� Focusing ability. Not all knowledge requires for all tasks. Modularizing
provides the ability to focus the system’s efforts in the most productive manner.
� Heterogeneous problem solving. The methods best appropriate to one part of a
problem may not be best for working on another part.
� Reliability. If one agent provides a wrong answer or clue, the consensus of other
agents may yet provide the true answer.
2.5 Standard Reinforcement Learning Model
Learning to act in ways that are rewarded is a sign of intelligence (Watkins, 1989). For
example, it is natural to train elephant in circus by rewarding it when the elephant acts
correctly in reaction of a command. That animal can learn to obtain more rewards than
punishment, and this aspect of animal intelligence has been studied extensively in
experimental psychology (Watkins, 1989).
30

In the standard RL model, an agent is interacted to its environment via perception and
action, as shown in Figure 2.16. On each step of interaction the agent receives as input,
i, the current state, s, of the environment; the agent then chooses an action, a, to
generate an output. The action changes the state of the environment and the value of this
state transition which are undertaken to the agent through a reinforcement signal
(reward/punishment), r. The agent's behavior, B, should choose actions that tend to
increase the overall sum of the rewards values. Agent can learn to do this over time by
systematic trial and error (Kaelbling, 1996). The RL agent does not have any knowledge
about environment; it is just trained by obtaining rewards or punishment based on its
action from environment. It is important that the agent gather useful experience about
the possible system states, actions, rewards and punishment actively to behave
optimally.
Figure 2.16: The Standard RL Model (Kaelbling, 1996)
As a whole, the model consists of:
� a discrete set of the environment states, S;
� a discrete set of the agent actions in turn of the states, A;
� a set of the scalar reward/punishment for each associated action or a sequence of
the actions ; typically [0,1], or the real numbers.
31

Q-learning (Watkins, 1989) is a recent form of the RL algorithm. Q-learning algorithm
works by estimating the values of the state-action pairs. The value Q(s,a) is defined to
be the expected sum of the future payoffs r obtained by taking action a from state s.
Once these values have learned, the optimal action from any state is the one with the
highest Q-value. After being initialized to arbitrary numbers, Q-values are estimated
based on the experience as shown in Figure 2.17:
1. From the current state s, select an action a. This will cause a receipt of an
immediate payoff r, and arrival at a next state s'.
2. Update Q(s,a) based upon this experience as follows:
Q(
s,
a)
� ( 1��
) Q(
s,
a)
��[
r ��
maxQ(
s�,
a�)]
e 2.17: Pseudocode for Q-Learning Algorithm (Watkins, 1989)
Figur
This algorithm is guaranteed to converge to the correct Q-values with the probability
one if the environment is stationary and depends on the current state and the action
taken in it; a lookup table (Q-Matrix) is used to store the Q-values, every state-action
pair continues to be visited, and the learning rate is decreased appropriately over time.
This exploration strategy does not specify which action to select at each step. In
practice, a method for choosing action is usually chosen that will ensure sufficient
exploration while still actions with higher value estimates.
a�
where � is the learning rate and 0 < � < 1 is the discount factor
3. Go to step 1.
32

2.6 Image Segmentation by Autonomous Agents in Multi-Agent System
There have been many researches to segment an image using agent approaches. In this
section, the previous works in image segmentation using the MAS will be considered.
The following will first look at previous attempt to image segmentation via MAS. The
literatures mentioned in this section are not limited to medical image segmentation. The
objective is to present all models, which are related to image segmentation, and multi-
agent models.
Different multi-agent approaches have presented lately for the segmentation of the
image or edge detection on image. The first category is suitable for non-medical images.
In addition, the second category is suitable for medical images.
2.6.1 Kagawa et al method
The method of Kagawa et al. (Kagawa, 1999) has been presented two basic phases;
region segmentation phase and region integrating phase, as shown Figure 2.18. The
agents are distributed in the image and calculate several features of every pixel in an
image. Subsequently, they move onto the pixel, which has the most similar features.
Following the inactive agents are modified in order to activate them again. When an
agent cannot find any pixel whose similarities are higher than a defined threshold, it has
been vanished. After the phase of the region segmentation, the segmented regions are
integrated into larger region, which are parts of the objects on the given image. The
result for this proposed approach is based on landscape images and the color feature on
the image.
33

Figure 2.18: The Proposed Method of Kagawa (Kagawa, 1999)
The proposed method by Kagawa et al. requires less amount of the calculation. In
addition, it could be applied to a wide variety of the natural images. Nevertheless, this
method can improve by utilizing not only the color plane, but also the frequency space
as the features agents.
2.6.2 Wang and Yuan method
Wang and Yuan (Wang Y., 2000; Wang Y., 2002a; Wang Y., 2002b) proposed face
detection model by evolutionary agents. Several agents are uniformly anchored in the
each pixel on 2D image environment to seek the skin-like pixel. The evolutionary agent
was defined as Agent= < p, d, a, f, fml, Diff, Rep, Die >. p denotes the position of an
agent in image. d represents its current diffusion direction. a stands for the age of an
agent. f symbolizes its fitness, which indicates the adaptability of an agent and can be
computed using the number of steps the agent takes to find a skin-like point. fml
represents the family index. The five states, which defined, are representing the internal
34

state of an agent while the Diff (diffusion), Rep (reproduction) and Die describe the
behavior of the agent. The agents have different behaviors; such as self-production,
diffusion and death. The researchers first investigate the color skin of 50 images and
they found the HSV (hue, saturation, and value) of the skin is in the following range:
0

The system assesses its state based on all agents’ intention. The intention is represented
at the knowledge that each agent experience. The information that should be included in
the agent knowledge is such as kind, type, plausibility, relationship, and search area.
Kind is a class of the objects to be extracted. The more small groups that object of the
same kind are classified into type. The measurement that evaluates how best the agent
satisfied the constraints is plausibility. The behavior of the agents based on their state is
different; such as producing son agents, decaying son agents, constructing parent agent,
changing agent knowledge and resolving overlap between agents. The proposed method
was applied to line drawing recognition and character segmentation.
The proposed method is a hierarchical multi-agent based method to extract object from
a given image. It can obtain the desired letter only with the knowledge on them. Then,
this method does not need direct control on agents. However, the computation time
should improve in future.
Figure 2.19: The Proposed Method of Gyothen (Gyohten, 2000)
36

2.6.4 Guillaud et al. method
Guillaud et al. (Guillaud, 2000) presented MAS for ring detection on fish otoliths. They
used two types of agent, the dark and light agent. Each agent should check pixels
around it in a circular neighborhood. If it finds that its neighborhood satisfies the
condition to be a region, the central pixel will be marked and new agents will generate
to grow the region. The agents can move in the gray scale image of the otolith, every
dark (light) agent try to find darker (lighter) pixel. They save the path. When the agent
has run over a loop, it can validate the path as a ring. In addition, the researchers have
added high-level information about the shape of the contour to improve the detection;
the procedure is shown in Figure 2.20.
The proposed method has an acceptable result to find the continuous ring of the otolith
image. Besides, detecting nucleus position is automated. However, the tuning of the
agents parameters is not easy.
Figure 2.20: The Procedure of the Proposed Method by Guillaud et al. (Guillaud ,
2000).
37

2.6.5 Rodin et al. method
Rodin et al. (Rodin, 2004) proposed MAS for biological image segmentation. There are
two types of the agents; lightening agent and darkening agent. In the system, each agent
can sense the environment. Then, based on its type, it marks the current located pixel.
After that, each agent records its path until it has rotated again. In following lifetime of
the agent, when an agent recognizes a path, the first discoverer agent kills the other
agents that located in the path; this procedure take place to avoid validation of the same
ring. At the end, the agent draws a polygon corresponding to the path it has just passed.
The finite state machine of describing the behavior of the agent is shown in the Figure
2.21. The color of this polygon is depended on type of the discoverer agent. In Figure
2.22, it can been seen the proposed model which the input data is brought from
environment and the behavior consist of decreasing or increasing brightness based on
type of the agent (darken or lighter), rotating movement, or go forward (Go to).
Figure 2.21: The MAS which Proposed by Rodin et al. (Rodin, 2004)
38

Figure 2.22: The Finite State Machine that is describing an Agent’s Behavior (Rodin,
2004)
As a conclusion, the proposed method was automated, and can be used to different type
of the images.
2.6.6 Melkemi et al. method
Melkemi et al. (Melkemi, 2004; Melkemi, 2005; Melkemi, 2006) proposed a model,
which is a hybridization of MAS and Markov Random Field and Genetic Algorithm, the
architecture is shown on Figure 2.23. The model has two types of agents, the
coordinator agent and the segmentation agent. The first step is that each segmentation
agent segments a part of the image by Iterated Conditional Modes procedure and the
initial sub-optimal configuration, however the initial configuration is created by agents
which they use K-means and a chaotic mapping. The second step is that the result of
every segmented agent transfers to the coordinator agent. It decides which result is
better and based on the initial configuration, Genetic Algorithm, the coordinator
produces some new initial configurations, and these configurations transmit to the
segmentation agents. This procedure iterates until the stable situation is achieved.
39

Consequently, experimental results of the proposed method are very encouraging which
show the feasibility, the convergence and the robustness of the method. In addition, the
method found to be much faster than traditional methods.
Figure 2.23: The Architecture of the Proposed Method of Melkemi (Melkemi, 2006)
From beginning of the section to this point, many literatures reviewed which uses agent
for segmenting non-medical images. Following the other approaches will summarize
regarding medical images.
2.6.7 Spinnu et al. method
Spinnu et al. (Spinu, 1996) proposed a multi-agent approach to edge detection in
medical images. They have defined two basic agent types; knowledge servers (KS), and
knowledge processors (KP). KS agents manage the problem elements that are
represented by objects and attributes. KP agents manage the processing and reasoning
methods. Any agent may get or set attribute values, create or delete object instances and
modify system configuration as well as dynamically creating new agents. The major
40

types of KP agent are KP noise, KP texture, KP config, KP operator, KP evaluation, and
KP split. KP noise and KP texture generate maps of estimated noise and texture
characteristics of the given image. KP config activates the created processing groups
that will start running concurrently and cooperatively. KP operator selects the
appropriate operator, for example, the Deriche operator may select for a zone affected
by additive noise. KP evaluation minimizes the estimated error and inconsistency error.
KP split evaluates the current result and proposes a partition of the region into sub-
region. The proposed MAS is shown in Figure 2.24.
Figure 2.24: The Proposed Method using MAS by Spinu (Spinu, 1996)
The proposed method achieves to its goals defined. However, there are some other
improvements to reach the optimal solution. For example, localization error can be
41

taken in formulation error. Besides, the contrast characteristic could be used in addition
of noise and texture characteristics.
2.6.8 Boucher et al. method
Boucher et al. (Boucher, 1996; Boucher, 1998) proposed MAS to segment image of the
living cells; the model is shown in Figure 2.25. The living cell image has four different
regions; nucleus, pseudopod, white halo and background. Therefore, these components
determine the type of the agents. Also, the internal manager agent is used which
manages the execution of the agents. The segmentation is based on region-growing
approach. Every agent assesses the four neighbor pixels. Then, each pixel value that has
highest evaluation pixel is labeled by a region. An evaluation function used for deciding
the highest evaluation pixel. This function uses six criteria like as variance similarity,
compact, gray level similarity, gradient direction similarity, cell and nucleus image
thresholding. If two types of the agent label a pixel then this pixel is added to event list
of the manager. The behavior of the agents is categorized to three kinds; Merging,
Negotiation and Reproducing. Merging occurs when two discovered regions are the
same so these regions merge and life of one of the agent is terminated. Negotiation
carries out when two agents are in conflict with the one region. For example, a region
has two different labels by two kind of the agent, in a situation agents negotiate about
the correct labeling. Reproduction means an agent can produce a new agent. This occurs
when an agent finds a new region in its environment for another kind agent so it can
produce an agent, which related to new region. In another example, when segmentation
sufficiency is mature, an agent can produce another agent so the new agent can access to
information of the previous frame, which the terminated agent have done.
42

Figure 2.25: The Proposed MAS by Boucher et al. (Boucher A., 1998)
Therefore, the proposed method is adaptable to the shape and size of the living cells to
distinguish them from image. In addition, this method provides richness information
from images. This richness comes from the outcome of each agent in duration of its
adaptability.
2.6.9 Liu and Tang method
Liu and Tang (Liu, 1999) proposed MAS to segment a MRI of brain. The brain has the
four basic elements; like as outline, branching region, enclosing region and tumor
region. For detecting each four regions, they assign some threshold range. The agent
behavior is one of these four types: breeding, pixel labeling, diffusion, and decay.
Breeding means when an agent is in a homogeneous segment it should be produce some
new agents in neighborhood pixel. The significant difference in this paper is that the
neighborhood region is determined by a sector of a circle with specified radius like
Figure 2.26. Diffusion is finding new homogeneous-segment pixels by moving to
43

neighborhood pixel. When an agent encounters with a new pixel from an existed
homogeneous segment, this agent labels this pixel and it will become inactivated. After
each agent passes its life span, it must be vanished or decayed.
The proposed method has less computation time in comparison with conventional
method. However, there is a problem to distribute agent over an image optimally.
Figure 2.26: The Local Neighboring Region of an Agent at Location (i,j) (Liu, 1999)
2.6.10 Germond et al. method
Germond et al. (Germond L., 2000) proposed a framework, which composed of MAS, a
deformable model, and an edge detector. The framework is shown in Figure 2.27; the
image is brain MRI. There are three different types of the agent; region agent, edge
agent, and scheduler. The region agents specialize for gray matter or for white matter
segmentation. Edge agents specialize for the brain boundary detection. The agents are
autonomous and concurrent. A shared memory is used for communicating of the agents.
The MAS carries out segmentation of MRI scans. The proposed method uses seeded-
region-growing method, a priori domain knowledge, and a statistical method whose
parameters are acquired at run time. The aim of the deformable model is to detect the
44

general boundary of the brain. The edge detector module is used for its ability to detect
a precise and robust localization of the boundaries for the all edges in a given image.
Figure 2.27: A Global View of the Framework and Information Flow of the Proposed
Method by Germond (Germond, 2000)
As a result, the proposed method has the mean quality percentage of 96%. However, the
method needs considerable user interaction.
2.6.11 Duchesnay et al. method
Duchesnay et al. (Duchesnay, 2001; Duchesnay, 2003) proposed MAS to organize and
structure the knowledge according to irregular pyramid as shown Figure 2.28, the used
image is mammography. The pyramid is a stack of the graphs recursively built from
base to the apex and it provides removing geometrical constraint due to the fixed
structure of the neighborhood. This method has two different types of the agent; region
agent, and an edge agent.
45

Figure 2.28: The Conceptual Framework by Duchesnay et al. (Duchesnay, 2001)
The agents can use seven behaviors; territory marking and feature extraction,
exploration, merging planning, cooperation and negotiation, which are consisted
decimation, reproduction and attachment. The procedure of this framework is as follow.
First, the image is separated into two partitions and several agents are stayed at different
parts of the image. After that, every agent seeks features around it and decides to merge
with other agents based on similarity in features. In some cases the agents cannot decide
due to the specified threshold is not fixed. Therefore, the agents cooperate and negotiate
with the other agent of the same type in order to decide how to merge. All behaviors of
the agents are presented in Figure 2.29.
Accordingly, the proposed method does not require substantial tuning effort. In
addition, it is completely autonomous. Furthermore, it does not require priori
46

information to segment images. Another interesting result is that this method can use to
segment some different images as well.
Figure 2.29: The Graphical Representations of the Agent’s Behaviors (Duchesnay,
2003)
2.6.12 Khosla and Lai method
Khosla and Lai (Khosla, 2003) would like to segment a Chinese Hamster Ovarian
image which recognizes the number of the cells on image. Manually the technicians
after inserting some chemical thing to the cells count the cells.
They proposed a framework with 3 components, as shown in Figure 2.30. In their own
framework, there are two types of the agent; Intelligent Control Agent (ICA) and Image
Processing Agents (IPA). IPA consists of the segmentation agent of water immersion
and the mathematical morphology segmentation agent. IPA segments the image with its
own algorithm. Then, the result will transfer to ICA. ICA is like human operator, it
collects the accepted segmentation and unaccepted one. Then, ICA decides which of
them is better so the accepted segmentation puts on the result. ICA can decide by means
of the neural network agent and moment invariant transformation agent.
47

Water Immersion
Segmentation IPA
(ICA)
Neural
Network
Agent
Intelligent Control Agent
Moment
Invariant
Transformation
Agent
Mathematical Morphology
Segmentation Agent
1……..N
Figure 2.30: The Multi-Agent Optimization Model for Unstained Cell Images by
Khosla et al. (Khosla, 2003)
Lai et al. in (Lai, 2003) carried out the same function as previous research of Khosla by
a little difference. They added a GA component to own model for initializing the IPA
and for deciding which result is accepted for segmentation, as shown in Figure 2.31.
The proposed method can achieve to segment cell images with accuracy of 100 percent.
Figure 2.31: The Multi-Agent Soft Computing Model for Unstained Cell Images by Lai
et al. (Lai, 2003)
48

2.6.13 Richard et al. method
Richard et al. (Richard, 2004) proposed MAS in which the aim is to segment the brain
MR images. The framework, as shown in Figure 2.32, based on parallel execution of the
agents. System manager launches agent executions in a sequential way. The agents are
autonomous and have ability of the cooperation. In their framework, three types of the
agents coexist such as global agent, local agent, and tissue-dedicated agents. The global
agent specifies particular task, which performs to the whole image then to create local
agents distributed over the image. The local agents create the tissue-dedicated agents, to
estimate model parameters and to encounter tissue models for final labeling decisions.
The tissue-dedicated agents execute tasks distributed by tissue type (gray matter, white
matter, and cerebro-spinal fluid). They acquire the tissue models from the neighborhood
and label the voxels using a region-growing process.
The proposed method shows the correct estimation of the tissue-intensity distribution in
different locations in the image, despite large intensity variations inside the same tissue.
In addition, in comparison with the other methods, the proposed method has the
significant performance in spite of the increasing non-uniformity of intensity.
49

Figure 2.32: The Proposed Multi-Agent Framework by Richard et al. (Richard, 2004)
2.6.14 Benamrane and Nassane method
Benamrane et al. (Benamrane, 2007) proposed a multi-agent approach permitting
segmenting brain MRI. They used two main types of the agent; global agent, and
region agent, as shown in Figure 2.33. Global agent has three basic behaviors; initial
segmentation, creating and launching the region agents, and coordinating of the region
agents. Region agent can behave one of these six types; discovering the neighborhood
agents, selection of the best fusion criterion from neighbor verifying, finding with
which agent merges, growing, and disappearing.
The proposed method is based on three steps. Firstly, the global agent segments image
by region growing approach. Secondly, iterative merging of the initial regions from the
previous step will merge the intermediate segmentation of the initial image. Finally,
segmentation of the intermediate segmentation by iterative merging of the intermediate
regions is obtained using a fusion criterion.
50

Figure 2.33: The Proposed Method by Benamrane et al. (Benamrane, 2007)
The proposed method has had acceptable results; each region presents clear-cut limits,
particularly the tumor regions, which correctly detected. However, the execution time is
exceedingly high.
2.6.15 Discussion
In Table 2.2 and Table 2.3, all researches related to segmentation using agent
approaches are compared based on the properties of the agent using non-medical and
medical images respectively. They are divided to class of the non-medical and medical
image segmentations. Furthermore, this table is based on our opinion. Subsequently, the
comparison results for each agent properties are concluded from the researchers work
and the definition of the agent properties.
Besides, Table 2.4 lists the advantages or disadvantages of some mentioned methods in
comparison with non-agent method. These advantages and disadvantages came from the
researchers beliefs.
51

Non Medical image segmentation
Table 2.2: The Comparison of the Segmentation Methods using Non-medical
Kagawa et
al.
Wang et
al.
Gyohten
Guillaud
et al.
Rodin et
al
Melkemi
et al
Images by Agent Properties (Chitsaz, 2008)
Landscape
photo
Face
photo
Document
image
fish
otoliths
fish
otoliths
Landscape
photo
Image
Modality
Yes
Yes
Yes
Yes
Yes
Yes
Autonomy
No
No
Yes
Yes
Yes
Yes
Social
ability
Agent
Properties
N/A
Yes
N/A
No
Yes
Yes
Reactivity
No
No
Yes
No
No
Yes
Pro-activity
N/A
N/A
5
2
2
2
Number of
Agent types
52

Table 2.3: The Comparison of the Segmentation Methods using Medical Images by
Medical image segmentation
Spinnu
et al.
Boucher
et al.
Liu and
Tang
Germond
et al.
Duchesnay et
al.
Khosla
and Lai
Richard
et al.
Benamrane
and
Nassane
Muscle
cell and
MRI
Living
cell
Brain
MR
images
Brain MR
images
Mammography
Chinese
Hamster
Ovarian
Cells
Brain
MR
images
Brain MR
images
Image
Modality
Agent Properties (Chitsaz, 2008)
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Autonomy
No
Yes
No
Yes
Yes
No
Yes
Yes
Social
ability
Agent
Properties
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Reactivity
No
Yes
No
No
Yes
No
No
Yes
Proactivity
8
5
4
3
2
3
3
2
Number
of Agent
types
53

Table 2.4: The Comparison between Multi-Agent and Non-Agent Segmentation
Methods
Researchers
Kagawa et al.
Wang et al.
Image size and
modality
Landscape image
with size of
160×240
30 fps for 160×120
video size
12 fps for 320×240
video size
Gyohten Document image
Guillaud et al.
Rodin et al.
Melkemi et al.
Spinnu et al.
Otolith image with
the size of 512×512
or 1024×1024
Otolith image with
the size of 150×150
Synthetic noisy
image
Muscle cell and
MRI
Boucher et al. Living cells
Liu an Tang
MRI of brain
612×792
Germond L. et al. MRI of brain
Duchesnay et al.
Khosla and Lai
192×192 images
including both
medical and non-
medical one
Chinese Hamster
Ovarian Cells
Richard et al. MRI of brain
Benamrane and
Nassane
MRI of brain that
contains tumors
Comparison with non-agent
methods
Less computation and can be applied
to different images variety.
No restriction to the face pose, face
moving direction and speed. Also, it is
much faster than template-based and
neural network-based methods.
Line recognition only with the
knowledge on image. More
computation time.
An acceptable result to find the
continuous ring of the otolith image.
Automated detecting of the nucleus
position. Difficult to tune the agents’
parameters.
Automated, and can be used to
different type of the images.
Experiment result shows the
feasibility, convergence and
robustness of this method. Faster than
traditional method.
Can find the optimal solution properly.
The method is adaptable and the result
has rich information.
Less computation time. Agent
distribution is not optimal.
The mean quality percentage is equal
to 96%. Considerable user interaction.
The approach does not require
substantial tuning effort and it is
completely autonomous. Not required
priori information.
The accuracy is 100%.
Adaptation to intensity non-uniformity
and noise.
Good success in image includes
heterogeneous, local and repartee
information.
54

2.7 Image Segmentation by Reinforcement Learning Model
The following describes the basic ideas of the researchers who contributed to the field
of the image segmentation regarding RL system. Some of the following methods also
used the MAS to implement a RL system in order to segment an image.
2.7.1 Peng and Bhanu method
Peng and Bhanu (Peng, 1998 a; Peng, 1998 b; Bhanu, 2000) proposed a framework for
object recognition using RL approach. Some pre-processing steps need to achieve
successful object recognition, like segmentation and feature extraction. The algorithm
was used for segmentation is Phoenix Segmentation Algorithm. This algorithm works
based on a recursive region splitting. It uses information from histogram of red, green
and blue image components to split the region based on a peak/valley analysis of each
histogram. The conceptual diagram is shown in Figure 2.34.
The evaluation framework for object recognition is implemented by RL. In every loop,
the best threshold for peak/valley selects and uses in segmentation algorithm; the
diagram is shown in Figure 2.35.
Consequently, the proposed method is capable of exploring a significant portion of the
search space, resulting in the discovery of the good solutions due to the stochastic
nature of RL. In general, this result cannot achieve by any deterministic or simple
supervised learning methods.
55

Figure 2.34: The Conceptual Diagram of the Phoenix Segmentation Algorithm (Peng,
1998b)
Figure 2.35: The Segmentation Evaluation by RL (Bhanu, 2000)
56

2.7.2 Shokri method
Shokri (Shokri, 2003) used concept of RL for finding best thresholding of an image.
Their approach was based on standard RL framework, as shown in Figure 2.36. The
model has states, actions and a matrix that saved the reward or punishment. The matrix
update on each iteration. In addition, for achieving global goal, states should examine
several times.
Figure 2.36: The Standard Model of RL (Shokri, 2003)
The model of the reward/punishment has two types; subjective and objective.
Subjective case means an experienced user will assign a reward/punishment to the
outcome image. Objective case is defined based on the black pixel ratio, the area of the
object, the tolerance for area deviation, and the number of the objects.
The proposed method achieves performance 87% for subjective method, and 60% for
objective method. Additionally, this method needs considerable user interaction to
achieve a better performance.
2.7.3 Sahba method
Sahba et al. (Sahba, 2008; Sahba, 2006 b) proposed a RL model to segment an ultra
sound image of the prostate. First, the image is divided to some sub-images. Then,
agents find the optimal threshold of all sub-images. After completing the segmentation
57

of all sub-images, the result has compared by a manual segmented image (gold image).
Subsequently, the reward or punishment is assigned to every agent. After training, the
agent finds the best threshold for the image and possible to segment another image of
the same type. In addition, researchers have used a simple deformable model for
extracting prostate from image; prostate has elliptical shape. The modified RL model is
shown in Figure 2.37. Figure 2.38 shows the proposed reinforcement model.
Figure 2.37: The RL Model used in the Proposed Method by Sahba (Sahba, 2006 b)
Figure 2.38: The General Model used in the Proposed Method (Sahba, 2008)
58

2.8 Chapter Summary
In this chapter, some background knowledge of our research mentions. Brief
explanations of two commonly used modalities for medical imaging are provided.
Furthermore, the skull anatomy is explained to facilitate the correlation with the
anatomical images. Subsequently, the basic image segmentation methods are elaborated
to compare the agent-based method with conventional methods.
In addition, researches, this related to our work, presents. The review covers most of the
image segmentation methods that uses MAS. These methods are employed to segment
medical images or non-medical images. These methods have acceptable results in
comparison to the conventional methods (Chitsaz, 2008). In addition, reviews of the
methods that use the RL system presents. Some of these works used solely the RL
method to segment an image. The results are satisfactory for these proposed methods
but are not very useful. This is because of the modified model of RL that was used in
image segmentation needs the manually segmented image. Therefore, if we want to
segment each image, it requires the manually segmented image too. Nevertheless, the
best threshold for an image can be found using the RL method and then we can segment
another image with the same characteristic using the result. Therefore, the RL method is
a pre-processing method employed by the other methods proposals.
As the result of reviewing the researches of other people, it is concluded that they
employed the MAS with the automatic agent, or used reinforcement agent without any
social ability of the agent. These disadvantages are our motivation to hybridize the
MAS with RL agents, which this framework uses both the properties of the automatic
agent and RL agent. Therefore, we want to propose a RL method to find the best
59

threshold value of the images, and then propose another method to use the agent
properties and the result of the RL method to segment another image.
60

Chapter 3
Methodology
This chapter contains studies on the proposed medical image segmentation algorithm
and represents a progression in its development. In conjunction with previous chapter,
which studied some of the current image segmentation methods by means of MAS and
RL model, this chapter discusses about developing and implementing our method based
on the properties of the local agent and the RL agent.
This chapter includes few sections. The next section describes the images, which used
in this project. After that, the methodology of this research globally describes. The
following section talks about the development of the system and how will initialize. The
last section summarized the overall description of our method; it also describes how the
research is concluded and why this conclusion can show the result is satisfactory.
3.1 Image Acquisition
The aim of this research is to segment the CT image of the head using RL agent
properties into MAS. These images have been collected from the University of Malaya
Hospital (Obaidellah, 2006), and collected from internet (DICOMsample).
61

The experiments are from two different data sets. In the first experimental data set, head
CT images from UM Hospital, are acquired on a CT scanner with an image size
512�512, and a pixel size of 0.5mm�0.5mm. Upper human body CT images for the
second experiment (DICOMsample) are acquired on the same machine. The imaging
protocol used is image size of 512�512, and a pixel size of 0.55mm � 0.55 mm.
3.2 Image Segmentation
Image segmentation has to employ important amount of the information, the importance
of this task will be more when segmentation has to be processed on a sequence of the
images. Complicated segmentation problems require sophisticated algorithm with more
priori-knowledge from image. Moreover, algorithm with learning ability needs more
training sets.
To solve these problems, a trainable and parallel processing approach has developed.
The proposed method consists of two disjoin phases; training phase, and testing phase
as shown in Figure 3.1. In the following paragraphs, each phase will elaborate in detail.
Figure 3.1 shows the relation between these two phases, training and testing. In the
training phase, a little image is used as a trained image whereas the RL agent will find
the appropriate value of each object or region in the input image. The outcome of this
training phase is transferred to the next phase, testing phase. In this phase, the images
are segmented by some priori knowledge and the properties of the local agent.
62

Figure 3.1: The Global View of our Proposed Model
In the training phase, new algorithm is proposed based on RL agent in order to segment
the CT images. The RL agent can learn to segment the images over time by systematic
trial and error. The RL agent is trained by obtaining rewards from its environment
whereas these signals are based on its actions on its environment. Because of the
dynamic nature of RL agent, it is appropriate to use this type of the agent to segment the
complex textured images. The goal of RL agent is to find out an optimal way to reach
the best answer with some signals, which obtained after each action. The best answer is
the most accurate segmented image.
For using the RL method in medical image segmentation, it is defined by some actions
to identify regions in an image. In addition, there are states based on the number of the
interest objects in image. Firstly, the agent takes the image and applies some values.
The input image is divided into several sub-images, and each RL agent works on it to
find the suitable value for each object in the image. Various actions are defined for each
state in the environment. Besides, a reward function computes reward for each action of
63

RL agent. Therefore, the agent tries to learn which actions can gain the highest reward.
Finally, the valuable gained information can be used to segment new similar images in
the next phase. The valuable information is the thresholding value of each object in the
image.
In the testing phase, new algorithm based on local agent in MAS is proposed to segment
the similar images of trained image. This method uses some priori knowledge, which
can obtain from training phase. Local agent has some important properties of the agent,
which described in Section 2.4, such as autonomy, reactivity, and social ability, less
robustness and pro-activity.
In the beginning of our proposed algorithm, the user should provide the number of the
extracted objects from an input image. Additionally, a near estimation of the extreme
thresholding of each object must be provided. The input image is divided into several
sub-images, and each local agent works on it to mark pixels for each object in the image
by means of the input data. If the thresholding range of the objects in the image has
overlapped, then the local agent cannot individually decide to mark the pixels.
Therefore, the local agent uses its social ability to make decision. In addition, this
algorithm works in parallel, so each local agent simultaneously works with other local
agents. Whenever agents have problem to mark a particular pixel, they would get some
help from their neighbors.
The number of the agent is directly depended on the window size and image size in both
phases. If we choose a window size of 7�7, then the number of the agent can be
acquired by dividing the image size to 49. Because the RL agent should stay on the
center of the window and the quotient is a real number, the number of the agents is less
64

than the quotient. In our framework, the image has 512�512=262144 pixels; the result
for division of 262144 by 49 is 5349.87. As a result, some agents, which locate in
rightmost and bottom side of the image, extend their areas to cover all pixels over
image, so the number of the agent is 5329.
For an example, Figure 3.2 shows an image with size of 16�16 pixels. We want to
calculate the number of the agents that should be appointed into image when the
window size is 7�7. As shown in the Figure 3.2, the agent 1, 2, 3, and 4 are placed in
center of yellow, dark blue, dark green and dark orange window respectively. However,
there are some uncovered pixels in rightmost and bottom of the image. Then, the
window size of agent 2, 3, and 4 should extend to cover all pixels in the image.
Therefore, the result of diving 16�16=256 to 49 is 5.22, but it only needs four agents to
cover all pixels of the exemplified image. Because of this, the number of the agent for
our framework, 5329, is less than the division result, 5349.
1 2
3 4
Figure 3.2: An Example for Calculating the Number of the Agents within a Window
Size of 7�7 over an Image Size of 16�16
65

3.2.1 Training Phase
The RL agent has already used in image processing task where it discussed in Chapter
2. In this section, it will show that the RL agent is suitable for segmenting medical
images in parallel. This method is specifically useful for medical images where there are
several images from a patient having most similar characteristics. In such a case, some
of the images can use as training images. Then, the appropriate parameter can find for
segmenting the other similar images.
In this phase, the image is divided into several sub-images. First, the user should
provide the number of the interest regions. Besides, the manually segmented image of
the input image has to provide. Figure 3.3 (a) and (b) show a cranial CT image and its
manually segmented version. They are applied as an input for the RL agent to obtain the
knowledge from image. The RL agent determines the local thresholding value for each
individual sub-image via dividing the maximum gray-scale of the input image by the
given number of objects within image. The maximum gray-scale of our experimental
result is 256. The Q-matrix will be constructed regards to states and actions.
(a) (b)
Figure 3.3: (a) The Original CT Image, (b) Manually Segmented Image
The RL agent needs three components to learn from its environment; states, actions and
reward. The RL agent starts its work using an input image and the manually segmented
66

input image. The RL agent tries to find the appropriate state of an image, after that it
chooses one of the defined actions. During this time, the RL agent changes the local
thresholding values for each sub-image individually. By taking each action, the agent
receives the corresponding reward for that state-action pair and updates the
corresponding value in Q-matrix. After this process, the agent has explored many
actions and has tried to exploit the most rewarding ones. The global view of this
procedure is shown in Figure 3.4. The state is perceived from images. Then, the RL
agent chooses one action to alter the image, after that it received a reward as evaluation
of its work.
Images
State
Action
Agents
Reward
Image
Processing
Tasks
Moderator
RL Agents
Figure 3.4: The Global View of our Proposed Method in Training Phase
Figure 3.5 shows the flowchart of RL agent behavior. At the beginning, the RL agent
discovers all pixels in the sub-image, and marks those pixels based on some fixed
thresholding range. This fixed thresholding range could be acquired by dividing the
maximum gray-scale of the image by the number of the acquired regions in image.
67

Begin
Discover every pixel in a 7x7 window and mark each pixel
State selecting, the number of different type in
window determines state
Action selecting, changing the thresholding of each region
No
Receiving Reward
Evaluation
Is accuracy in
window
satisfactory?
Yes
Death
End
Figure 3.5: The RL Agent’s Behavior
68

As an example, if the image has three object for segmenting, the first thresholding range
will be [0, 85] for the first region, [85,170] for the second region, and [170,256] for
third region. After marking all pixels in the window, RL agent would find its state.
There are some actions for each state; RL agent should select one of them. � -Greedy is
a method, which helps the RL agent to choose better possible action. Where � is a
probability to choose action with most Q-value. A lookup table of Q-Matrix is used to
store the Q-values. If � is less than a predefined parameter, the RL agent selects action
with most Q-value, otherwise it will select action randomly.
After choosing the action, the RL agent alters the primary thresholding value of each
region by means of the maximum and the minimum thresholding in the current sub-
image. A reward function calculates the number of the true-segmented pixels of the
image. Then, information of this state-action is saved in the Q-matrix by following
formula:
Q(
s,
a)
�(1�α)
Q(
s,a)
�α[
r�γ
maxQ(
s�,
a�)]
a�
(3.1)
After evaluating the RL agent work, if the result is satisfactory, the RL agent lifetime is
finished. The satisfactory result is obtained when the accuracy of the segmented sub-
image is more than 95%.
In the following paragraphs, the definition of RL components presents. State, action and
reward are described in depth.
3.2.1.1 Definition of States
The number of regions in the sub-image identifies the number of states. There are
pixels, which related to a particular region in each sub-image. For example, there are
three different regions such as bone, skin, and air in Figure 3.3(a).
69

Refer to Figure 3.6; if the number of acquired regions in an image identified as two,
then the states will be three as follows:
� the sub-image that consists of the pixels of the first region type, as shown in
Figure 3.6(a);
� the sub-image that consists of the pixels of the second region type, as shown in
Figure 3.6(b);
� The sub-image consists of the pixels of both region types, as shown in
Figure 3.6(c).
Sub-image 1: Sub-image 2:
(a) (b) (c)
Figure 3.6: The Example of the number of States for an Image with two Regions
Refer to Figure 3.7; if the number of the acquired region in image identified as three,
then the states is seven as follows:
Region I:
Region II:
� the sub-image that includes the pixels of the first region type, as shown in
Figure 3.7(a);
� the sub-image that includes the pixels of the second region type, as shown in
Figure 3.7(b);
� the sub-image that includes the pixels of the third region type, as shown in
Figure 3.7(c);
� the sub-image that includes the pixels of both first and second region type, as
shown in Figure 3.7(d);
70

� the sub-image that includes the pixels of both first and third region type, as
shown in Figure 3.7(e);
� the sub-image that includes the pixels of both second and third region type, as
shown in Figure 3.7(f);
� the sub-image that includes the pixels of all region types, as shown in
Figure 3.7(g);
Figure 3.7: The Example of the number of States for an Image with three Regions; each
window shows a typical sub-image
Therefore, the number of states can find by the following formula:
n �
�n
N � � � �
�
� i
i �1
�
�
�
�
�
�
(3.2)
Where N is the total number of states, and n is the number of acquired objects in the
image.
(a) (b) (c)
(d)
(e) (f)
(g)
Region I:
Region II:
Region III:
71

3.2.1.2 Definition of Actions
Actions change the local thresholding value of each sub-image. There are some actions
for each state. An action employs the maximum and minimum gray-scale value in the
image. This distance can be divided into several intervals with a predefined parameter.
Each action would be defined as one of the intervals of this distance to threshold the
sub-image. The number of actions depends on the predefined parameter, and the state.
For example, if the predefined parameter is 2. The minimum and maximum the gray-
scale value of the sub-image are 25 and 27 respectively, as shown in Figure 3.8. The
actions for the state that shown in Figure 3.6(c) is choosing one of these sets {[25,25],
[25,27]}, {[25,26], [26,27]}, or {[25,27], [27,27]}. One of these sets would choose as an
action to segment the sub-image. Hence, the total number of actions is three. These sets
can find via dividing the distance between maximum and minimum gray-scale value of
image by the predefined parameter. In this example, (27-25)/2 is one, so there are six
different intervals; [25,25] , [25,26] , [25,27] , [26,26] , [26,27] , and [27,27]. Because
the numbers of interest regions are two, it should select two intervals from those.
Therefore, there are three sets.
25 26 27
Figure 3.8: An Example of Defining Action using the Maximum and Minimum
Thresholding Gay-sale Value of a Typical Sub-image
If the minimum and maximum values are similar for an action interval of n, it means
pixels in the sub-image do not included the region number of n. For example, the first
interval in the first set of the previous example is [25,25]. The minimum and maximum
values are similar. It means there are no pixels, which marked by region type I in the
72

sub-image, therefore all the pixels are from region type II. For another example, in this
set {[25,27], [27,27]}, there is an interval with the same value at the first and the last
interval, so it means there are no pixels with the type of region II in the sub-image.
3.2.1.3 Definition of Reward
The reward shows how well the image was segmented by RL agent. As a result an
appropriate segmented of truth is needed for evaluation in place of true delineation. The
reward function is defined as the number of pixels, which are segmented correctly. This
function is like True-Positive, which discuss in Chapter 4.
3.2.1.4 Graphical User Interface for Training Phase
A GUI has developed for the training phase. There are some defined images used in the
application. Therefore, all defined images are shown in a drop-down list to select one of
them. Then, after pushing the ‘segment’ button, the true positive volume fraction
(TPVF) and false positive volume fraction (FPVF) of the segmented image are written
in the box. Additionally, the estimation of the thresholding value for each region has
shown in another textbox. Figure 3.9 shows GUI for training phase, whereas the input
image and the segmented one are shown in different box. In addition, the TPVF and
FPVF are written in different box.
73

3.2.2 Testing Phase
Figure 3.9: GUI of the Training Phase
Autonomous agents have already been used in image processing tasks were discussed in
Chapter 2. In this section, different MAS are proposed for segmenting medical images
in parallel way using priori-knowledge from the training phase.
Figure 3.10 shows a global view of this framework where there are three basic
components; input materials, agent and its environment, and image processing task. The
input materials are similar to the input images of the training phase. Additionally, the
user gives an estimation of the thresholding for each region within an input image. This
estimation came from the training phase. There are two main agent types, Moderator
74

agent and Local Agent. These agents have the responsibility to segment the input image
using the priori-knowledge and image processing tasks. Image processing tasks are
procedures that related to segmentation of the images. We use these terms
interchangeably in this thesis: local agent, autonomous agent, or agent.
Input Image,
Thresholding
for each region
of input image
Local Agents
Figure 3.10: The Global View of our Proposed Method in Testing Phase
In this phase, the image was divided into several sub-images like the training phase. As
shown in Figure 3.10, there are two different agents here. The moderator agent has a
managing role in the framework, and the local agents are like labor.
There is a moderator agent to create and initialize the local agents, after that each local
agent commences its life. The moderator agent decides to create local agents in different
parts of an image. Moreover, the moderator agent terminates the lifetime of a particular
local agent if there is no progress for that local agent. Besides, after terminating the
lifetime of all agents, moderator agent forces that local agents marked all pixels in the
image. If there are some undiscovered pixels, the moderator agent will create a second
generation of the local agents in the unmarked area.
There are many local agents to segment the image. First, an estimation of the
thresholding for each region within the image should enter. This priori knowledge can
be derived from the training phase. At the final state of the training phase, RL agent
Moderator
Image Processing
Tasks
75

finds the thresholding for each region in sub-images. Therefore, there is little user-
interaction to obtain the overall thresholding for each region within the image. For this
attempt, our designed interface would help the user to find better thresholding for each
region, by changing the thresholding and viewing the result.
After the local agent had obtained the input materials, the local agent tried to mark each
pixel in sub-images by means of the thresholding input. In the duration of the marking
procedure, each agent should make a decision about label of each pixel in the sub-
image. A local agent can do it by the given priori knowledge, but there are overlapped
or gapped areas between given thresholding ranges of each region. In this situation, an
agent uses its properties to negotiate with the other agents. It means that if an agent
cannot find the type of a pixel, it negotiates with its neighbor agent to find an
appropriate region type. Nevertheless, if no agent exists in the neighborhood of the
current agent, or a neighbor agent has not known proper information yet, the agent will
leave that pixel as unmarked for further processing of the other agents. Figure 3.11
shows all the behavior of agents in the testing phase.
76

Yes
Death of
Local agent
Start
Moderator agent creates many local agents in different part of image.
No
Are all
pixels
marked?
Local agent marks the pixel in its sub-image.
Can mark
pixel?
Local agent uses one of the
negotiation approaches to mark pixel.
Can mark
pixel?
Mark the pixel to
unable-to-mark
Moderator agent checks all the pixels in the image. If there
are undiscovered or unable-to-mark pixels, then it generates
another generation of local agents.
No
Yes
Yes
Are all
pixels
discovered?
End
No
No
Yes
Figure 3.11: The Agents’ Behavior in Testing Phase
77

A local agent tries to find the appropriate label for the current pixel. If it cannot find the
meaningful label, it goes to negotiate. The negotiation term is to calculate the mean
value of the 3�3 window of the negotiable pixel. If this mean value is in the range of
discovered thresholding, then the negotiable pixel can mark by this mean value. The
discovered thresholding means the thresholding range is not in the overlapped or
gapped distance. The other approach for negotiation is to count the number of
discovered neighboring pixels. The major type in the counting approach specifies the
label of the negotiable pixel. For example, there are 8 pixels around the negotiable
pixel. If there are 3 pixels labeled as Region I, 4 pixels labeled as Region II, and 1 pixel
labeled as Region III. Then, the outcome from this approach is to mark the negotiable
pixel as Region II because majority of its neighbors were marked as Region II.
3.2.2.1Graphical User Interface for Testing Phase
A GUI has developed for the testing phase too. There are some pre-defined images for
use in the application. Therefore, all defined images are shown in a drop-down list to
allow a user select one of them. Then, the user should give the estimation of the
thresholding value. This priori knowledge can find from the training phase. By pushing
the ‘segment’ button, TPVF and FPVF of the segmented image are written in a box.
Additionally, the input image and the segmented one of it will be shown in another
window. Figure 3.12 shows the GUI for the testing phase, the input image and the
segmented one are shown in different windows. In addition, the TPVF and FPVF are
written in the box.
78

3.3 Chapter Summary
Figure 3.12: GUI of the Testing Phase
The main purpose of this work is to segment medical images simultaneously with some
different interest regions. Bearing in mind the obstacles of the medical image
segmentation, the training methods need a huge training sample. Agent can learn to
perform segmentation over time by systematic trial and error. The RL agent is trained
by obtaining rewards or punishment based on its action in an environment. The goal of
the RL agent is to find out an optimal way to reach the best answer given some signals
obtained after each action. The state and action should be defined for using RL method
in medical image segmentation. Firstly, the agent takes an image and applies some
values. The input image is divided into several sub-images, and each RL agent works on
it to find the suitable value for each object in the image. Each state in the environment is
associated with some actions. A reward function computes the reward for each action of
79

RL agent. Therefore, the agent tries to learn which actions can gain the highest reward.
Finally, the gained valuable information will use to segment new similar images. Due to
the dynamic nature of RL agent, it is suitable for segmenting images with high
complexity.
Another algorithm based on MAS is proposed to use the valuable information from the
previous method. Local agent can use its properties to perform segmentation over time.
The goal of the agent is to find out appropriate label for each pixel in an image. Firstly,
a moderator agent creates and initializes the local agents within the image. The local
agent takes a sub-image and applies some values. The input image is divided into
several sub-images, and each agent works on it and tries to mark each pixel in sub-
images by means of the given priori knowledge. During this time, the local agent marks
each cell of a sub-image individually. Finally, the moderator agent checks the outcome
of all agents’ work to produce a final segmented image.
Finally, it is required to show how well our algorithms are. Therefore, the outcome
evaluation of these two proposed methods will discuss in the next chapter in order to
show how well the algorithms are proposed.
80

Chapter 4
Experimental Results and Discussion
Based on the methods described in Chapter 3, we carried out experiments and analyzed
the results. This chapter has two major parts, i.e., training phase results and testing
phase results. Two separate groups of images are used for training and testing; in each
group there are two data sets, one is from UM and another is from public domain on the
internet. The experimental results will discuss, analyze qualitatively, and quantitatively.
4.1 Experimental Results of Training Procedure
In this section, we consider the experiments and results for the training phase, both
qualitatively, and quantitatively. Image display uses for qualitative evaluation, and
statistical analysis uses for quantitative evaluation. We name the proposed training
method Proposed Reinforcement Learning Model (PRLM). For evaluating this phase,
we used 33 different images of the same modality, size, and tissues in image.
RL agent has a remarkable role in PRLM. As mentioned in the methodology chapter,
every RL agent should work on a 7�7 window over the image. After marking every
81

pixels of the window, it should receive a reward based on the training data set, and
manually segmented image of the corresponded image.
The heart of RL in the system uses the eq. 4.1 to learn:
Q(
s,
a)
� ( 1 � � ) Q(
s,
a)
� �[
r � � max Q(
s�,
a�)]
(4.1)
a�
Where s is a state, a is an action, r is a reward, � is a learning rate that was initialized to
0.9, and � is the discount factor that was set to 0.1.
We have used the � -Greedy method for choosing an action, � set to 0.7. The number
of iterations, to examine every action of a specified state, is another fixed parameter
employs. We found in our experiment by trial and error, the reasonable number is 200.
This number of the iterations should not be bigger because the processing time would
increase. In addition, if the number of iterations becomes smaller, the RL agent cannot
examine more actions to reach steady state. In other words, the Q-matrix would not be
full for all state-action pairs
4.1.1 Image Data Sets of PRLM
The images used in the experiments list in Table 4.1. In the first experiment, head CT
images are acquired on a CT scanner with an image size 512�512, and a pixel size of
0.5mm�0.5mm. Upper human body CT images for the second experiment
(DICOMsample) are acquired on the same machine. The imaging protocol used is
image size of 512�512, and a pixel size of 0.55mm � 0.55mm.
Table 4.1: Details of the Image Data set used in the Experiments
Data Modality Object Image size Pixel size No. of
set
images
First CT Head 512�512 0.55mm�0.55mm 18
Second CT Upper human
body
512�512 0.55mm�0.55mm 15
82

In overall, 33 slices has selected from full 3D CT images; these slices are at the same
location in the body and the same orientation with respect to the body. Each image set
has used for the training phase and testing phase. The numbers of training data are 33
images, and all slices are used as a test set.
4.1.2 Qualitative Analysis of PRLM
A subjective inspection discovered that in all experiments and in all data, the results are
close to their manually segmented images. Some examples are displayed in Figure 4.1.
In addition, initialization was done in identical manner for all experiments images to
evaluate the results of the input images.
Figure 4.1 shows the segmented image by our proposed algorithm, the input image, and
the ground truth image. Each row shows different slice of the 3D images from data set.
The first two rows are from the first experiment data set, and the two remaining rows
are from the second data set. More results are shown in Appendix A.
4.1.3 Quantitative Analysis of PRLM
The objective of the quantitative segmentation evaluation is to assess segmentation
methods, or compare some segmentation methods with each other. The inter-technique
and intra-technique are two classes of the quantitative analysis. The inter-technique
reveals the performance of different techniques in segmenting the same type of images.
The intra-technique recognizes the behavior of the considered technique in segmenting
various kinds of the images (Zhang, 2001). In this research, we assess the proposed
methods through various kinds of the images.
83

(a) (b) (c)
(a)
(a)
(a)
(b)
(b)
Figure 4.1: The Segmentation Example from Two Experiments and Four Different
Slices of 3D CT Images, (a) Input Image (b) Result from Proposed method and (c)
Ground Truth Image
(c)
(c)
(b) (c)
84

Another classification of the evaluation methods consist of analytical methods,
goodness methods, and discrepancy methods. For assessing the complexity of
algorithm, the analytical methods employ the algorithm for segmenting image by
considering the requirement, variants, etc. The goodness methods evaluate the
segmented image by measuring the intra-region uniformity, inter-region contrast, and
region shape. The discrepancy methods compute the difference between segmented
image and ground truth of the considered image. In other words, these methods try to
determine the number of missed-segmented pixels, position of missed-segmented
pixels, feature value of segmented objects, and miscellaneous quantities (Zhang, 2001).
Choosing an appropriate evaluation method is an important task. However, it is obvious
now that two types of metrics must measure for evaluation: accuracy, and efficiency
(Fenster, 2005). Accuracy of a segmentation technique is to show how far actually
segmented image differs from the manually segmented one. Efficiency of a segmented
method is the segmentation time that all aspects of the user interaction should consider.
4.1.3.1 Accuracy of PRLM
The difference between the actually segmented image and the manually segmented
image determine as accuracy of a segmentation technique. As a result, an appropriate
segmented ground truth requires for evaluation in place of the true delineation. In all
experiments, all data sets have manually segmented in the domain. For any
image A� ( C,
f ) , where C is a 2D (or higher-dimensional) rectangular pixels array, and
f (c)
denotes the intensity of any pixel c in C, letC M
be the segmentation result of
d
method M which obtained from C, and Ctd is the true delineation. U is a binary
d
image representation of a reference superset of pixels that is used to express the two
85

measures as a fraction. We have used true positive volume fraction (TPVF) and false
positive volume fraction (FPVF) from (Udupa, 2006). Eq. 4.2 and 4.3 are sufficient to
describe the accuracy of the method:
TPVF
M
d
�
C
M
d
C
U
�
C
M
td
C
C
C
td
�100
(4.2)
�
M d td
FPVF � �100
, (4.3)
d
�
d
td
The sample cases were 33 slices of the CT images; therefore, Table 4.2 listed the mean
and standard deviation of TPVF and FPVF achieved in our experiments by the proposed
method. The TPVF of all data sets are above 96%, and their FPVF do not exceed 0.9%.
The TPVF and FPVF obtain for each slice; the results of first and second data set are
shown in Appendix B.
The Receiver Operating Characteristic curve (ROC) is a plot of the true positive fraction
against the false positive fraction. The closer the curve follows the left-hand border and
then the top border of the ROC space, the more accurate the test. The closer the curve
comes to the 45-degree diagonal of the ROC space, the less accurate the test. In
addition, the area under the curve is a measure of the accuracy. Figure 4.2 and Figure
4.3 demonstrate the ROC curve for head CT and upper human body data sets
respectively. As the result of this curve, the result is accurate; also, the skin result is
more accurate than the segmented result from bone.
Data Set
Head CT
Images
Upper human
body
Table 4.2: TPVF and FPVF of PRLM
TPVF (%) FPVF (%)
BG Skin Bone BG Skin Bone
97.63
99.94 �
� 0.9
0.07
6
96.66 � 1.26 0.34 � 0.16 0.38 � 0.20 0.60 � 0.25
99.96 � 0.03 99.55 � 1.5 96.10 � 1.41 0.17 � 0.05 0.17 � 0.07 0.07 � 0.03
86

Figure 4.2: ROC Curve for the First Data set
Figure 4.3: ROC Curve for the Second Data set
87

4.1.3.2 Efficiency of PRLM
In determining the efficiency of a segmentation method time, all aspects of the user
interaction should consider (Fenster, 2005). PRLM implements on 2.00 GHz Intel Core
2 Duo and 2.00 GB RAM. The efficiency of our segmentation method provides
information on the sensible use of the algorithm. In the proposed algorithm, a user does
not interact with the program in the training phase. Therefore, the time for user
interaction has ignored. The computation time directly relates to the image size and the
number of iteration for filling the Q-matrix. However, in the training phase, the reward
function needs the manually segmented image of current image, therefore there is in
average 10 minutes to segment an image by means of the imaging software like
Photoshop.
Table 4.3 depicts the mean computation time for all data sets. For every set, the
program ran 15 times for each slide and the mean computation time is measured.
Table 4.3: Efficiency of the PRLM
Data set Computation time
Head CT Image 13 Seconds
Upper human body 7 Seconds
4.2 Experimental Results of Testing Procedure
In this section, we demonstrate both qualitatively, through image display, and
quantitatively, through evaluation experiments for the next phase of our algorithm. In
training phase, the RL agent achieved to discover the thresholding range for each
region, so the result can use in testing phase to segment the similar images from the
88

input images of the training phase. We name our proposed testing method Proposed
Multi-Agent Model (PMAM).
As mentioned in Chapter 3, local agent starts to segment the image using RL agent
result. In this phase, each local agent through marked-pixel table can negotiate with
other local agent. Therefore, the social ability, reactivity, and autonomy of the agent
properties have been satisfied. However, it is required to adjust the result of the training
phase a few times, so some user interactions are necessary in this phase. In the training
phase, the exact thresholding range obtains for each window in an image. Meanwhile
some user interaction is required to find the exact thresholding range for each region
globally.
Predetermined parameter does not use through this phase. We have a global marked-
pixel table, which can refer in experiment of other local agent. Figure 4.4 shows the
GUI, which brings out the thresholding value for each window and the mean value of
each row. After consideration of those numbers, a user should suggest the thresholding
for each region. Figure 4.5 depicts the GUI of the testing phase, which after each
suggestion the TPVF, FPVF, and segmented image have shown, then the user can
suggest to get better results.
89

Figure 4.4: GUI of the Training Phase to Suggest the User Thresholding Range of each
Region
Figure 4.5: GUI of the Testing Phase
90

4.2.1 Image Data Sets of PMAM
The image data sets used in the experiments are exactly the same properties that briefly
described in section 4.1.1. In this phase, 28 images as test sets are used. Table 4.4 shows
the details of data sets. For testing each image of data set, only one image from training
phase is used.
Table 4.4: Details of the Image Data set used in PMAM
Data Modality Object Image size Pixel size No. of
set
images
First set CT Head 512�512 0.55mm�0.55mm 15
Second CT Upper 512�512 0.55mm�0.55mm 13
set
human body
4.2.2 Qualitative Analysis of PMAM
A subjective inspection discovered that in all experiments and in all data, the results are
satisfactory. Figure 4.6 shows the segmented image using our proposed algorithm in
the second phase, the input image, and the ground truth. Each row shows different slice
of the 3D images from the CT image of the head. The first two rows are from the first
experiment data set, and the two remaining rows are from the second data set. For the
first two rows, the result of RL agent from the first slice, Figure 4.1 (the first row), have
used. Meanwhile for the second two rows of Figure 4.6, the result of RL agent from the
253 rd slice, Figure 4.1 (the third row), was employed. More results of first and second
data set from testing phase are shown in Appendix C.
91

(a) (b)
(a) (b)
(a) (b)
(a) (b)
Figure 4.6: The Segmentation Example from Two Experiments and Four Different
Slice of the Data set, (a) Result from our Method, (b) Input Image
92

4.2.3 Quantitative Analysis of PMAM
For evaluating PMAM, we used those two types of metrics: accuracy, and efficiency
(Fenster, 2005). The following are comprehensive explanations for each of these
metrics.
4.2.3.1 Accuracy of PMAM
TPVF and FPVF were defined in Eq. (4.2) and (4.3) at Section 4.1.3. Table 4.5 lists the
mean and standard deviation values of TPVF and FPVF achieved in the two
experiments. All data sets have TPVF above 90% except bone tissue in the second data
set and FPVF less than 1.5%. The TPVF and FPVF slice obtains for each slice. The
results of first and second data set are shown in Appendix D.
The ROC curve is a plot of the true positive fraction against the false positive fraction.
Figure 4.7 demonstrates the ROC curve for the head CT data set, and Figure 4.8 depicts
the ROC curve for upper human body CT data set.
Table 4.5: TPVF and FPVF for the Testing Phase of PMAM
D a t a S e t
TPVF (%) FPVF (%)
BG Skin Bone BG Skin Bone
Head CT
Images
95.55
99.13 �
� 2.5
0.53
8
91.63 � 2.33 0.88 � 0.82 1.41 � 0.70 1.28 � 0.76
Upper
human body
98.82
99.61 �
� 0.6
0.54
6
85.53 � 3.53 0.26 � 0.21 1.02 � 0.52 0.21 � 0.17
93

Figure 4.7: ROC Curve of the First Data set
Figure 4.8: ROC Curve of the Second Data set
94

4.2.3.2 Efficiency of PMAM
PMAM implements on a 2.00 GHz Intel Core 2 Duo CPU, and 2.00 GB RAM personal
computer. The efficiency of our segmentation method provides information on the
sensible use of the algorithm. In the testing phase, a user only needs minimal interaction
with the program. The time for user interaction has been denoted TU. In addition, the
computation time, TC, which directly related to the image size and the number of agents,
has considered.
Table 4.6 depicts the mean user interaction time and the mean computation time for
each image. For every slice in each data set, 15 times the program ran, so the mean
computation time is measured. In addition, Figure 4.9 shows the bar chart of the
computation time for all data sets.
Table 4.6: Mean User-interaction Time and Computation Time of PMAM
Data set TU TC
Head CT Images 60 Seconds 7 Seconds
Upper human body 60 Seconds 7 Seconds
95

Figure 4.9: The Computation Time of all Data sets; X axis shows the slice number
(identity) and Y axis shows the computation time (second).
4.3 Chapter Summary
In this chapter, evaluation of our proposed method is discussed both qualitatively
through image display, and quantitatively through evaluation experiments.
In the training phase, PRLM is based on RL model has demonstrated impressive results.
The accuracy is more than 95% for each region and the computation time is less than 13
seconds
In the testing phase, PMAM is implemented. The result is satisfactory; the accuracy
found is more than 85% for each region and the computation time recorded in less than
6 seconds. However, the achieved accuracy is less than PRLM, but the efficiency is
much better. Therefore, the result is satisfactory because there is a balance between
accuracy and efficiency.
96

Chapter 5
Conclusions
5.1 The Proposed Reinforcement-Learning Model
Section 3.2 contains a description of the proposed RL model (PRLM), it is also identify
a progression of the development. Section 4.1 presents the evaluation of PRLM.
PRLM utilizes a standard RL model to segment images. State, action, and reward
function also defined. The RL agents have used them to learn from the image. Every
state has one or more actions. RL agent in each state decides to choose one action.
Therefore, RL agent marked the image. It means each pixel in the image labels as a
specific tissue such as bone, skin, etc. Finally, a reward function evaluates the accuracy
of the segmented image, and gives a reward to the RL agent.
Quantitative comparison of PRLM results in the training phase does not anticipate;
however, having the manually segmented image for side-by-side comparison gives an
opportunity to consider the advantages of our proposed method. Furthermore, the
97

qualitative comparison shows encouraging results. The accuracy of PRLM is more than
95% for each region in the image. PRLM is almost automatic. It only requires the
manually segmented image for the reward function. The most significant advantage of
our proposed method is segmentation of an image into more than two regions in a
parallel way. It means the regions of the interest can be more than one and with
complex characteristics. For example, the CT image of the head consists of three
different regions, such as the air (background), bone, and skin. PRLM segments the
image into these three different objects simultaneously. In addition, the number of
training data set decreases in comparison with the neural networks approaches, as well
as, the other learning methods. The efficiency illustrates this method is quick in
comparison to the other method. The achieve computation time of PRLM is less than 13
seconds. In Table 5.1, the efficiency of some segmentation methods lists. Some of them
have a better efficiency; it is because of the smaller image sizes that used in their
framework.
Table 5.1: Efficiency Comparison of the Image Segmentation Methods
Researcher Method Data Set PC Specification Efficiency
Liang and
Rodriguez
(Liang, 1996)
Pan and Lu
(Pan, 2007)
Lu and Bao
(Lu, 2006)
Chitsaz and
woo (Chitsaz,
2009 a)
Chitsaz and
woo (Chitsaz,
2009 b)
Fuzzy C-Mean
Region-Growing
Extended Image
Force Model of
Snakes
Reinforcement
Learning
Multi-Agent
System
MR images of a
patient’s head
(256�256)
Skull CT image
(256�256) and
liver (384�384)
Heart (160�169)
Lung (225�211)
2 different data
sets of the CT
image of head
(512�512)
2 different data
sets of the CT
image of head
(512�512)
Sun SPARCstation
10/50
Windows 2000 and
VC++ 6.0 platforms
Not available
2.00 GHz Intel Core
2 Duo and Java
platform
2.00 GHz Intel Core
2 Duo and Java
platform
Pixel-based:
9.7 (min)
Region-based:
0.73 (min)
Skull: more than
10 second
Liver: less than
5 second
Heart: 8.3 s
Lung: 30.2 s
First Data set: 13s
Second Dataset: 7s
First Data set: 7s
Second Dataset: 7s
Although the qualitative result shows accuracy of PRLM, the method does not work for
all images in the data set because of some reasons. First of all, at the beginning of the
98

algorithm, some predetermined conditions, such as the number of iterations,� ,� ,
and� , need to be set. These conditions cannot change in the middle of program
execution. Therefore, these predetermined conditions may not suit some specific images
in the duration of running the program. For example, the number of iteration has been
set to 200 cycles, for images with a much narrower histogram, this number is not
sufficient to fill all the cells in Q-Matrix.
Moreover, states define based on gray-scale value. This should improve to cover more
image features like texture, or shape in the future. In addition, the numbers of states
depend on the number of objects to recognize in the image.
Finally, the number of actions for each state is rigid. For each image window, which
covers a bigger range of gray-scale, the actions are not satisfactory. Since the maximum
and minimum thresholding of the gray-scale in the window is too large, finding the
appropriate gray-scale would not be achievable for each region in window because that
involve huge computation cycles. Meanwhile, the choosing method of the action is � -
Greedy in our framework. It can change to a comprehensive method to obtain better
performance in the accuracy.
5.2 The Proposed Multi-Agent model
In section 3.3, we have proposed a multi-agent model (PMAM) to segment an image, by
input of the maximum and the minimum gray-scale value of each region in image.
Section 4.2 evaluated PMAM both qualitatively and quantitatively.
Quantitative assessment of PMAM shows the results are aggravated in comparison with
PRLM result. The achieved accuracy from PMAM is more than 85% in each region of
99

the image. Furthermore, the efficiency time is less than 7 seconds for all data sets.
Nevertheless, it is conceivable to improve result by some morphological operations.
Furthermore, the qualitative comparison shows interesting result, and better
computation time. The proposed method is almost automatic. It requires a little
adjustment on the result from PRLM outcome. The most significant advantage is
segmenting image to more than two regions in a parallel way. PRLM has this property
too. It means the interest regions can be more than one and with different
characteristics. For example, the CT image of the head consists of three different
regions, such as air, bone, and skin. PMAM segments the image into three different
objects simultaneously. Moreover, the efficiency illustrates PMAM is very fast in
comparison to PRLM. It is possible to compare the result from testing phase with the
other methods’ result in Table 5.1.
However, the qualitative result shows high accuracy of our proposed method; but the
method has a few defects because of some reasons. First, PMAM is such a simple
approach that used for images with less noise. When there is some noise or foreign
object such tooth filling material in the CT image, the noise would label as bone tissue
instead of background or air, as shown in Figure 4.6. Of course, it is possible to improve
the method by putting more constraint in the model. For example, we can improve the
social ability of each agent. In PMAM, the local agent uses the gray-scale value of
neighborhood pixels and the mean value of neighborhood pixels for making decision to
mark each pixel. This straightforward manner can modify to a comprehensive one in the
future. Finally, the adjustment of gray-scale value from training phase is a tedious work;
the expert should consider the mean value of each low or high extreme for every region
to conclude an appropriate gray-scale range. However, the GUI can help the user to see
100

each segmented image after each adjustment of the thresholding, as shown in Figure 4.4
and 4.5.
5.3 Achievements
In summary, we have shown that the proposed methods can be used to segment
different anatomic structure in medical images. Thus, the methods fulfilled our
objectives mentioned in the first chapter. For the image segmentation of the head, we
have the some objectives. The first objective is to review the existing experimental
studies were investigating the Agents Technology for image segmentation. We have
done this objective in Chapter 2 that we have presented some important methods that
related to RL system and MAS. Another objective is to propose a novel approach by RL
agent in the multi-agent framework that will be quicker, more accurate, and more
robust. In Chapter 3, the two proposed method explain in detail. The final objective is to
evaluate the outcome of segmented image by the proposed method with an appropriate
estimation approach. In Chapter 4, there are a comprehensive evaluation of our method
to show the accuracy and efficiency of it.
In the following are the main results of PRLM:
� PRLM attains significant result in segmentation accuracy; the accuracy is
more than 95% for each region in the image.
� PRLM achieves satisfactory result in computation time; the mean
computation time of all datasets is less than 13 seconds.
� The number of training data set for PRLM can be one or a small number of
images.
� PRLM has the ability to segment simultaneously an image into some
distinct regions.
101

Furthermore, we have shown that PMAM can be used to segment different anatomic
structure in medical images, which the input data obtained from PRLM. The main
results of this method summarizes below:
� PMAM attains significant result in segmentation accuracy; the accuracy is
more than 85% for each region in the image.
� PMAM achieves satisfactory result in computation time; the mean
computation time of all datasets is less than 7 seconds.
� PMAM is capable to segment simultaneously an image into some distinct
regions.
Simulating facial surgery and predicting the effect of an operation is very pivotal. One
of the primary preprocessing levels in every kind of surgical simulation is image
segmentation. Therefore, the accuracy of this level is essential because the result will
affect the overall outcome. This work shows the achieved accuracy is such impressive
that it can use in simulating facial surgery as a primary preprocessing level.
5.4 Future work
To tackle the remaining problems in PRLM that mentioned in Section 5.1, three ideas
stand out for future works. The first would be an investigation to use comprehensive
state model, which can cover more features of an image such as tissue, shape, etc. This
would increase the scope of images that can be used to segment. For reducing the
problem of action choosing, it is conceivable to use some existing method, which is not
simple. This would convey the problem of choosing the optimal action of each state.
Finally, the action model can modify to involve more neighboring pixels in the
processing window, not only making decision by the maximum and minimum gray-
scale value of each region in the window but also getting help from neighboring pixels.
102

Another future work is an improvement of PMAM; consequently, two ideas were
inspired. First, the Multi-Agent model should modify to a comprehensive one; the agent
properties mentioned in chapter 2.5 should employ to improve the model. In addition,
the finding of optimal input data is a time-consuming work; therefore, the model should
be less dependent to the input data.
103

Bibliography
Alan, L., T. Frank, et al. (2003). A Survey of Surgical Simulation: Applications,
Technology, and Education. Presence: Teleoperators and Virtual Environments,
12(6), 599-614.
Awcock, G. J., Thomas, R. (1995). Applied Image Processing: McGraw-Hill.
Bankman, I. (2008). Handbook of Medical Image Processing and Analysis: Academic
Press.
Bathe, K. J. (1996). Finite Element Procedures: Prentice-Hall.
Benamrane, N., Nassane, S. (2007). Medical Image Segmentation by a Multi-Agent
System Approach. Proceedings of the 5th German Conference on Multiagent
System Technologies, Leipzig, Germany, Springer-Verlag.
Bettega, G., Y. Payan, et al. (2000). A Simulator for Maxillofacial Surgery Integrating
3D Cephalometry and Orthodontia. Journal of Image Guided Surgery, 5(3), 156-
165.
Bhanu, B., Peng, J. (2000). Adaptive Integrated Image Segmentation and Object
Recognition. IEEE Transaction on Systems, Man, and Cybernetics, 30(4), 427-
441.
Boucher, A., Garbay, C. (1996). A Multi-Agent System to Segment Living Cells.
Proceedings of the 13th International Conference on Pattern Recognition, 3,
558-562.
Boucher, A., A. Doisy, et al. (1998). A Society of Goal-Oriented Agents for the
Analysis of Living Cells. Artificial Intelligence in Medicine, 14(1-2), 183-196.
Bovenkamp, E. G. P., J. Dijkstra, et al. (2004). Multi-Agent Segmentation of IVUS
images. Pattern Recognition Society, 37, 647-653.
Chen, P., Pavlidis, T. (1980). Image Segmentation as an Estimation Problem. Computer
Graphics and Image Processing, 12(2), 153.
Chitsaz, M., Woo, C.S. (2008). The Rise of Multi-Agent and R.L. Segmentation Methods
for Biomedical Images. The 4th Malaysian Software Engineering Conference
(MySEC’08), Kuala Terengganu, Malaysia.
Chitsaz, M., Woo, C.S. (2009). Medical Image Segmentation by using Reinforcement
Learning Agent. International Conference on Digital Image Processing
(ICDIP'09), Bangkok, Thailand, IEEE Computer Society Press.
Chitsaz, M., Woo, C.S. (2009). A Multi-Agent System Approach for Medical Image
Segmentation. International Conference on Future Computer and
104

Communication (ICFCC'09), Kuala Lumpur, Malaysia, IEEE Computer Society
Press.
Coleman, G. B., Andrews, H.C. (1979). Image Segmentation by Clustering.
Proceedings of the IEEE, 67(5), 773-795.
Crevier, D., Lepage, R. (1997). Knowledge-Based Image Understanding Systems: A
Survey. Computer Vision and Image Understanding, 67(2), 161-185.
DICOMsample. "DICOM Files." Retrieved July 2008, from
http://pubimage.hcuge.ch:8080/.
Duchesnay, E., J. J. Montois, et al. (2001). An Agent-based Implementation of Irregular
Pyramid for distributed Image Segmentation. Proceedings of Eighth
International Conference on Emerging Technologies and Factory, 509-504.
Duchesnay, E., J. J. Montois, et al. (2003). Cooperative Agents Society Organized as an
Irregular Pyramid: a Mammography Segmentation Application. Pattern
Recognition Letters, 24(14), 2435-2445.
Felippa, C. (2007). "Introduction to Finite Element Methods " Retrieved March 2008,
from http://www.colorado.edu/engineering/CAS/courses.d/IFEM.d/.
Fenster, A., Chiu, B. (2005). Evaluation of Segmentation algorithms for Medical
Imaging. 27th Annual International Conference of the Engineering in Medicine
and Biology Society.
Freixenet, J., X. Mu˜noz, et al. (2002). Yet Another Survey on Image Segmentation:
Region and Boundary Information Integration. Proceedings of the 7th European
Conference on Computer Vision-Part III, Springer-Verlag.
Garrobo, A. (2006). "MRI Scanner." Retrieved January, 2009, from
http://media.collegepublisher.com/media/paper871/stills/45359417825c3-89-
2.jpg.
Germond, L., M. Dojat, et al. (2000). A Cooperative Framework for Segmentation of
MRI Brain Scans. Artificial Intelligence in Medicine, 20(1), 77-93.
Giraldi, G. A., P. S. Rodrigues, et al. (2006). Implicit Dual Snakes for Medical Imaging.
the 28th Annual International Conference of the Engineering in Medicine and
Biology Society.
Gray, H. (1918). "Gray's Anatomy of Human Body." Retrieved January, 2009, from
http://www.bartleby.com.
Guillaud , A., H. Troadec, et al. (2000). Continuity Perception using a Multi-Agent
System an Application to Growth Ring Detection on Fish Otoliths. The 15th
International Conference on Pattern Recognition Proceedings. , 2, 519-521.
Gyohten, K. (2000). Optimization-Based Image Analysis Dealing with Symbolic
Constraints using Hierarchical Multi-Agent System. IEEE International
Conference on Systems, Man, and Cybernetics, 4, 2794-2799.
105

Hiatt, J. L., Gartner, L.P. (1982). Textbook of Head and Neck Anatomy. New York:
Appleton-Century-Crofts.
Jain, A. K. (1989). Fundamentals of Digital Image Processing Printice Hall.
Jennings, N. R., K. Sycara, et al. (1998). A Roadmap of Agent Research and
Development. International Journal of Autonomous Agents and Multi-Agent
Systems, 1(1), 7-38.
Jeri. (2008). "CT Scanner." Retrieved January, 2009, from
www.smugpuppies.com/category/health/.
Kaelbling, L. P., M. L. Littman, et al. (1996). Reinforcement Learning: A Survey.
Artificial Intelligence Research, 4, 237-285.
Kagawa, H., M. Kinouchi, et al. (1999). Image Segmentation by Artificial Life
Approach using Autonomous Agents. International Joint Conference on Neural
Networks.
Khosla, R., C. Lai, et al. (2003). Optimising the Performance of Soft Computing Agents
for Classification of Unstained Mammalian Cell Images. International
Symposium on Computational Intelligence for Measurement Systems and
Applications, 163-168.
Kincaid , D., Cheney W. (2002). Numerical Analysis: Mathematics of Scientific
Computing.
Kirbas, C., Quek, F. K. H. (2003). Vessel Extraction Techniques and Algorithms : A
Survey. The Third IEEE Symposium on BioInformatics and BioEngineering.
Koch, R. M., S. H. M. Roth, et al. (2002). A Framework for Facial Surgery Simulation.
Proceedings of the 18th Spring Conference on Computer graphics, Budmerice,
Slovakia, ACM.
Lai, C., R. Khosla, et al. (2003). GA Based Optimisation of a Multi-Agent Soft
Computing Model for Segmentation and Classification of Unstained
Mammalian Cell Images. The Congress on Evolutionary Computation, 2(7),
1192.
Liang, T., Rodriguez, J.J. (1996). MR Cranial Image Segmentation-A Morphological
and Clustering approach. Proceeding of the IEEE Southwest Symposium on
Image Analysis and Interpretation.
Liu, J., Tang, Y.Y. (1999). Adaptive Image Segmentation with Distributed Behaviour-
Based Agents. IEEE Transaction on Pattern Analysis and Machine Intelligence,
21(6), 8.
Liu, J. (2006). Synergistic hybrid Image Segmentation: Combining Model and Image-
Based Startegies, University of Pennsylvania. PhD. Dissertation.
106

Lu, H., Bao, S. (2006). An Extended Image Force Model of Snakes for Medical Image
Segmentation and Smoothing. 8th International Conference on Signal
Processing.
Marieb, E. N. (2000). Essentials of Anatomy and Physiology: Addison Welsely
Longman.
Martini, F. H., C. W. Garrison, et al. (2004). Fundamental of Anatomy and Physiology:
Pearson.
Melkemi, K. E., M. Batouche, et al. (2004). MRF and Multi-Agent System bsed
Approach for Image Segmentation. IEEE International Conference on Industrial
Technology.
Melkemi, K. E., M. Batouche, et al. (2005). Chaotic MultiAgent System Approach for
MRF-based Image Segmentation. Proceedings of the 4th International
Symposium on Image and Signal Processing and Analysis, 268-273.
Melkemi, K. E., M. Batouche, et al. (2006). A Multi-Agent System Approach for Image
Segmentation using Genetic Algorithms and Extremal Optimization Heuristics.
Pattern Recognition Letters, 27, 1230-1238.
Nikishov, G. P. (2007). Introduction to The Finite Element Method: McGraw-Hill
Science/Engineering/Math.
Obaidellah, U. H. B. (2006). A Finite Element Approach for the Planning and
Simulation of 3D Mandibular Osteotomy for Orthoganathic Surgery. University
of Malaya. Master Dissertation.
Pan, Z., Lu, J. (2007). A Bayes-Based Region-Growing Algorithm for Medical Image
Segmentation. Computing in Science & Engineering, 9(4), 32-38.
Peng, J., Bhanu, B. (1998 a). Delayed Reinforcement Learning for Adaptive Image
Segmentation and Feature Extraction. IEEE Transactions on Systems, Man, and
Cybernetics, Part C: Applications and Reviews, 28(3), 482-488.
Peng, J. and B. Bhanu (1998 b). Closed-Loop Object Recognition Using Reinforcement
Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence,
20(2), 139-154.
Pham, D. L., C. Xu, et al. (2000). A Survey of Current Methods in Medical Image
Segmentation. Annual Review of Biomedical Engineering, 2, 315-337.
Rares, A., M. J. T. Reinders, et al. (1999). Mapping Image Analysis Problems on Multi-
Agent Systems, Information and Communication Theory Group TU Delft.
Richard, N., M. Dojat, et al. (2004). Automated Segmentation of Human Brain MR
Images using a Multi-Agent Approach. Artificial Intelligence in Medicine,
30(2), 24.
107

Rodin, V., A. Benzinou, et al. (2004). An Immune Oriented Multi-Agent System for
Biological Image Processing. Pattern Recognition Society, 37.
Sahba, F., H. R. Tizhoosh, et al. (2006 a). Increasing Object Recognition Rate using
Reinforced Segmentation. IEEE International Conference on Image Processing.
Sahba, F., H. R. Tizhoosh, et al. (2006 b). A Reinforcement Learning Framework for
Medical Image Segmentation. International Joint Conference on Neural
Networks.
Sahba, F., H. R. Tizhoosh, et al. (2007). Application of Opposition-Based
Reinforcement Learning in Image Segmentation. IEEE Symposium on
Computational Intelligence in Image and Signal Processing.
Sahba, F., H. R. Tizhoosh, et al. (2008). A Reinforcement Agent for Object
Segmentation in Ultrasound Images. Expert Systems with Applications, 35(3),
772-780.
Shariatpanahi, H. F., N. Batmanghelich, et al. (2006). Distributed behavior-based
Multi-agent System for Automatic Segmentation of brain MR Images.
International joint Conference on Neural network.
Shokri, M. and H. R. Tizhoosh (2003). Using Reinforcement Learning for Image
Thresholding. IEEE Canadian Conference on Electrical and Computer
Engineering.
Spinu, C., C. Garbay, et al. (1996). A Multi-Agent Approach to Edge Detection as a
Distributed Optimization Problem. Proceedings of the 13th International
Conference on Pattern Recognition.
Tanasoca, R. C. M. (2006). Finite Element Modeling of the Anthropoid Mandible:
Mandible Model, Experimental Validation, and Anthropologic Application,
University of Florida. PhD. Dissertation.
Udupa, J. K., V. R. LeBlanc, et al. (2006). A Framework for Evaluating Image
Segmentation Algorithms. Computerized Medical Imaging and Graphics, 30(2),
75-87.
Umbaugh, S. E. (1998). Computer Vision and Image Processing: A Practical Approach
using Cviptools: Prentice Hall.
Valvassori, G. E., M. F. Mafee, et al. (1995). Imaging of the Head and Neck. Stuttgart:
Georg Thieme Verlag.
Walter, J. B., J. J. Carr, et al. (2007). Basic Atlas of Sectional Anatomy with Correlated
Imaging: Saunders.
Wang, Y., Yuan, B. (2000). Face Detection Using Evolutionary agents. 5th
International Conference on Signal Processing Proceedings.
108

Wang, Y., Yuan, B. (2002 a). Fast Method for Face Location and Tracking by
Distributed Behaviour-Based Agents. Vision, Image and Signal Processing,
149(3), 173- 178.
Wang, Y., Yuan, B. (2002 b). Face Location and Tracking by Distributed Behaviour-
Based Agents. 6th International Conference on Signal Processing Proceedings.
Watkins, C. J. C. H. (1989). Learning from Delayed Rewards. Cambridge, UK,
University of Cambridge. PhD Dissertation.
Withey, D. (2006). Dynamic Edge Tracing: Recurcive Methods for Medical Image
Segmentation, University of Alberta. PhD Dissertation.
Wooldridge, M. (1997). Agent-Based Software Engineering. IEE Proceedings on
Software Engineering.
Wooldridge, M. J., Jennings, N. R. (1995). Intelligent Agents: Theory and Practice The
Knowledge Engineering Review, 10(2), 115-152.
Zhang, Y. J. (2001). A Review of Recent Evaluation Methods for Image Segmentation.
Sixth International Symposium on Signal Processing and its Applications.
109

Appendix A
Experimental Results of the Training Phase - the first data set
Slice No. 0 Slice No. 3
Slice No. 10 Slice No. 13
Slice No. 15 Slice No. 20
Slice No. 23 Slice No. 28
Slice No. 30 Slice No. 33
110

Slice No. 35 Slice No. 38
Slice No. 40 Slice No. 45
Slice No. 47 Slice No. 50
Slice No. 53 Slice No. 55
Slice No. 57 Slice No. 60
111

Experimental Results of the Training Phase - the Second data set
Slice No. 253 Slice No. 260
Slice No. 263 Slice No. 269
Slice No. 271 Slice No. 278
Slice No. 284 Slice No. 287
Slice No. 290 Slice No. 301
112

Slice No. 310 Slice No. 319
Slice No. 330 Slice No. 350
Slice No. 360
113

Appendix B
TPVF and FPVF of the Experimental Results of the Training Phase - the first data
set
Image No.
TPVF (%) FPVF (%)
BG Skin Bone BG Skin Bone
0 99.99 98.29 96.43 0.34 0.22 0.37
3 99.99 97.44 98.34 0.36 0.1 0.56
13 99.99 95.76 94.74 0.45 0.37 0.92
15 99.99 98.16 98.85 0.72 0.08 0.22
20 99.99 95.97 94.65 0.37 0.37 0.89
23 99.99 97.55 97.66 0.55 0.18 0.47
28 99.97 97.99 96.44 0.2 0.34 0.53
30 99.99 97.32 94.08 0.19 0.57 0.78
33 99.99 98.61 97.52 0.1 0.26 0.42
35 99.99 98.83 97.87 0.13 0.22 0.34
40 99.95 98.3 97.41 0.3 0.29 0.46
45 99.94 98.63 96.4 0.26 0.32 0.41
47 99.92 98.7 95.92 0.33 0.33 0.39
50 99.86 96.85 96.43 0.41 0.63 0.91
53 99.73 95.95 97.12 0.43 0.66 1.14
55 99.92 97.8 96.53 0.12 0.84 0.55
57 99.87 97.45 96.48 0.54 0.54 0.75
60 99.86 97.79 97.11 0.37 0.46 0.75
114

TPVF and FPVF of the Experimental Results of the Training Phase - the second
data set
Image No.
TPVF (%) FPVF (%)
BG Skin Bone BG Skin Bone
253 99.9 99.27 97.23 0.25 0.21 0.12
260 99.99 99.28 96 0.1 0.21 0.14
263 99.96 99.69 97.29 0.14 0.14 0.05
269 99.88 99.74 94.55 0.17 0.19 0.06
271 99.96 99.65 94.95 0.15 0.12 0.07
278 99.99 99.63 96.5 0.11 0.11 0.05
284 99.99 99.74 96.52 0.11 0.09 0.03
287 99.98 99.52 92.41 0.16 0.22 0.06
290 99.99 99.61 95.39 0.17 0.12 0.04
301 99.99 99.69 96.8 0.18 0.09 0.04
310 99.98 99.56 96.1 0.28 0.19 0.08
319 99.91 99.65 96.69 0.2 0.19 0.11
330 99.99 99.6 97.19 0.15 0.21 0.07
350 99.95 99.38 95.6 0.14 0.37 0.15
360 99.97 99.46 98.29 0.27 0.12 0.08
115

Appendix C
Experimental Results of the Testing Phase - the first data set
Slice No. 0 Slice No. 3
Slice No. 13 Slice No. 23
Slice No. 30 Slice No. 35
Slice No. 40 Slice No. 47
Slice No. 50 Slice No. 55
116

Slice No. 57 Slice No. 60
Experimental Results of the Testing Phase - the second data set
Slice No. 253 Slice No. 260
Slice No. 271 Slice No. 278
Slice No. 284 Slice No. 290
Slice No. 301 Slice No. 310
117

Slice No. 330 Slice No. 360
118

Appendix D
TPVF and FPVF of the Experimental Results of the Testing Phase - the first data set
Image No.
TPVF (%) FPVF (%)
BG Skin Bone BG Skin Bone
0 99.97 98.95 94.87 0.18 0.34 0.23
3 99.34 97.97 93.54 1.23 0.73 0.34
13 98.55 93.43 92.13 3.08 0.95 1.29
15 98.68 92.07 89.94 2.86 0.89 1.75
23 98.54 97.57 91.94 0.67 1.35 0.87
28 98.62 91.94 97.76 1.64 0.81 97.76
30 98.60 91.26 91.70 1.63 1.25 2.68
35 99.05 98.37 90.09 0.44 1.70 0.46
40 99.66 98.48 90.78 0.52 1.133 0.38
47 99.71 94.05 88.62 0.63 1.04 1.96
50 99.40 95.91 91.86 0.00 1.95 1.32
53 98.99 93.73 89.53 0.00 3.12 1.83
55 99.68 94.60 91.02 067 1.74 1.53
57 99.73 97.51 91.11 0.72 1.46 0.68
60 98.70 96.10 89.60 0.04 2.41 1.69
119

TPVF and FPVF of the Experimental Results of the Testing Phase - the second data
set
Image No.
TPVF (%) FPVF (%)
BG Skin Bone BG Skin Bone
253 98.41 0 98.17 1.9 91.61 0.43
260 99.35 0.06 98.12 1.39 85.66 0.4
263 99.28 0.01 98.41 1.2 88.14 0.33
269 98.96 0.15 97.86 1.33 84.64 0.43
271 98.8 0.12 97.93 1.43 86.03 0.43
278 99.99 0.12 99.73 0.33 88.25 0.03
284 99.99 0.19 99.54 0.27 88.06 0.05
290 100 0.49 99.07 0.41 82.82 0.08
301 99.96 0.51 98.61 0.51 83.48 0.22
310 99.98 0.57 98.95 0.84 84.94 0.13
319 99.91 0.65 99.27 1.3 79.3 0.02
330 99.99 0.18 99.51 1.5 80.19 0.09
360 99.92 0.24 99.68 1.27 84.05 0.02
120