To the Graduate Council: I am submitting herewith a thesis written by ...

Chapter 4: Algorithm Overview 564.2.4 Information MeasureInformation theory is a relatively new branch of mathematics that began only in the1940’s. The term “information theory” still does not possess a unique definition butbroadly deals with the study of problems concerning systems that involve informationprocessing, information storage, information retrieval and decision making.The first studies in this direction were undertaken by Nyquist in 1924 and by Hartleyin 1928 (Equation 4.17) that recognized the logarithmic nature of the measure ofinformation. In 1948, Shannon published his seminal paper on the properties ofinformation sources and of the communication channels used to transmit the outputsof these sources and the important definition of entropy as the measure of information(Equation 4.18).HHartley( p1 , p2,...pn) = log |{i; pi> 0,1≤i ≤ n }|(4.19)HShannon= −ni=pilog pi1 (4.20)In the past fifty years, literature on information theory has grown quite voluminousand apart from communication theory it has found deep applications in many social,physical and biological sciences, economics, statistics, accounting, language,psychology, ecology, pattern recognition, computer sciences and fuzzy sets.A key feature of Shannon’s information theory is the term “information” that can oftenbe given a mathematical meaning as a numerically measurable quantity, on the basisof a probabilistic model. This important measure has a very concrete operationalinterpretation for the communication engineers. We would like to summarize thevarious definitions of entropy in the literature as Table 4.2.The list that we have presented in Table 4.2 is not an exhaustive one though we havespanned a few important definitions involving parameters and weights. Pap in [Pap,2002] briefs the history of information theory and discusses various measures ofinformation while Reza [Reza, 1994] approaches information theory from the codingaspect of communication theory. We would like to emphasize that the difference inusing a discrete random variable and a continuous random variable. The analogousdefinition of Shannon’s entropy in the continuous case is called the differentialentropy (Equation 4.21).Hdifferenti al∞−∞= p( x )log( p( x )) dx(4.21)