Diversity as a potential for surprise

Diversity as a potential for surprise
An information theoretic measure
of effective product diversity
Stefan Baumgärtner 1
Interdisciplinary Institute for Environmental Economics,
University of Heidelberg, Germany
7 May 2004 (Version 4.2)
Abstract: In the face of uncertainty, diversity causes surprise. Taking the view of
a consumer with incomplete knowledge of the choice set, I propose to take the expected
surprise when sequentially observing an allocation as a measure of effective
diversity of that allocation. Expected surprise can be quantified as the expected
information content of an observation (entropy). In this approach, effective product
diversity is determined by (i) the pure number of different products and (ii)
the evenness of distribution of their market shares. The concept is illustrated by
an empirical analysis of the allocation of passenger cars in Germany.
JEL-classification: D11, L11, L62
Key words: auto industry, entropy, diversity, information, market concentration,
product variety, surprise, uncertainty
Correspondence:
Dr. Stefan Baumgärtner, Interdisciplinary Institute for Environmental Economics,
University of Heidelberg, Bergheimer Str. 20, D-69115 Heidelberg, Germany,
phone: +49.6221.54-8012, fax: +49.6221.54-8020, email: baumgaertner@uni-hd.de,
http://www.stefan-baumgaertner.de
1 I am grateful to Switgard Feuerstein, Ralph Winkler and seminar participants in Heidelberg
and Stockholm (EEA2003) for critical discussion and helpful comments on earlier drafts; to Eva
Kiesele for research assistance with the data evaluation used in Section 4; to the Energy and
Resources Group at the University of California/Berkeley, where the first draft of this manuscript
was written, for their hospitality in 2001/2002; to the Deutsche Forschungsgemeinschaft (DFG)
for financial support; and – last, not least – to Clemens Puppe for most stimulating discussions
on diversity.

1 Introduction
In the face of uncertainty, diversity causes surprise. As an illustration by contrast,
imagine an economy where there exists only one make of car, now and forever. (As
a matter of history, the formerly socialist economies of central and eastern Europe
were pretty close to this extreme.) Living in such an economy, you would exactly
know which car you would be driving for the rest of your life. You would exactly
know which car your kids would get one day. If you were to win the main prize in
the lottery – a brand new car – you would exactly know what car that would be.
And if you were to die in a traffic accident, guess what car would have caused it.
Without diversity, you really cannot expect much surprise.
While the view of diversity as a potential for surprise may seem elusive, a
number of more direct economic reasons have been proposed why diversity matters:
• Diversity of a portfolio of options provides insurance for risk averse agents,
e.g. when investing in financial assets (Eichberger and Harper 1997) or when
devising corporate strategies (Berry 1975, Jacquemin and Berry 1979).
• Diversity of current technology and knowledge is a potential resource base
for future invention and innovation (Mowery and Rosenberg 1998, Schiller
2001).
• Diversity allows a competitive economy to efficiently adapt to unforeseen contingencies,
such as evolutionary changes in preferences, technology, resource
scarcity and institutions (Dekel et al. 1998, Matutinović 2001).
• Diversity is tantamount for the freedom of choice of autonomous persons
(Puppe 1996, Sen 1988, Weikard 1999).
All of these arguments share the view that the economic relevance of diversity is
intimately related to uncertainty or even fundamental ignorance about the future.
And surprise is just one expression of such uncertainty (Shackle 1955, Faber et al.
1992).
The simplest measure of diversity – and for that sake let’s return to product diversity
– is the number of different products available in an economy. This measure
is used in the strand of literature on product differentiation under monopolistic
competition and increasing returns to scale, which goes back to the seminal contributions
of Spence (1976) and Dixit and Stiglitz (1977). It underlies the influential
works on economic growth through product or factor differentiation (Ethier 1982,
Romer 1987, 1990) and on gains from international trade in differentiated products
(Dixit and Norman 1980, Ethier 1982, Helpman and Krugman 1985).
The pure number of different products, however, does not tell anything about
how different they are. In a more fundamental sense, diversity stems from the
dissimilarity between different products in terms of some characteristic properties
or attributes (Lancaster 1966). The diversity of a set of products may be defined as
an aggregate measure of their dissimilarity. Recently, Weitzman (1992, 1998), Gans
2

and Hill (1997), Bernhofen (2001) and Nehring and Puppe (2002) have suggested
various approaches for measuring (product) diversity in such a way.
All of these approaches are static and furthermore assume complete knowledge
about the number and properties of the different products available. Yet, the
relevance of diversity stems exactly from not knowing all the available products
and all their attributes. Such incomplete knowledge, I shall argue, should be
constituent for the definition and measurement of diversity. I want to suggest
that product diversity can be defined and measured as the expected surprise that
an uninformed person encounters when sequentially observing which products are
present in the actual allocation of an economy. For simplification, I adopt the
assumption that all the different products are pairwise equally dissimilar. Thus,
diversity becomes a matter merely of the number of different products.
A distinction can then be made between pure product diversity and effective
product diversity. Pure product diversity denotes the number of different products
existent in an economy and is a technological, or objective, measure of diversity.
Effective product diversity, on the other hand, adopts the consumers’ point of view
and captures an uninformed economic agents’ perception of a diverse allocation.
It, thus, is a subjective measure of product diversity.
Consider, for example, an allocation with five different car models. Assume
they have market shares of 50%, 40%, 6%, 3% and 1%. What effective diversity
would you attribute to that allocation? In other words, if you were out there
in the street and would watch the cars come by, one by one, how much surprise
can you expect? The pure product diversity which, in principle, exists in the
allocation is five different car models. Effectively, however, you should not expect
the same degree of surprise, or effective diversity, as if there were five different and
equally abundant car models. The two with the highest market share dominate the
allocation, and they also dominate your impression of diversity in that allocation.
Effectively, your perceived diversity will be little more than two car models, as
you may probably not even see one of the car models with negligible market share
when observing the actual aggregate allocation.
In order to quantify the expected surprise of uninformed consumers when observing
an allocation, I will employ concepts from information theory, in particular
the concepts of an ‘information function’ and ‘entropy’ (Shannon 1948, Rényi
1961), to derive a general measure of effective product diversity (Section 2). It
will turn out that this measure of effective product diversity is intimately linked
to common measures of market concentration (Section 3). It is identical to the
class of measures of market concentration suggested by Hannah and Kay (1977),
and is a subclass of the general and unifying class of concentration indices that
has been described in an axiomatic way by Foster and Shneyerov (1999). Thus,
the interesting relation between effective product diversity and market concentration
may also shed new light on the interpretation of different measures of market
concentration, which have been criticized in the past for their apparent ‘lack of
content’ (Shepherd 1987b: 639). The information theoretic approach to measuring
effective product diversity put forward here may provide such an economic ratio-
3

nale for the use of certain concentration indices. Finally, I will apply the measure
of effective product diversity in an empirical analysis of the allocation of passenger
cars in Germany 2003 (Section 4).
2 Expected surprise from sequentially observing
a diverse allocation
2.1 Setting and notation
Consider a market for the different varieties of a differentiated product, e.g. different
models of passenger cars. Let n ∈ IN be the total number of different product
varieties which are in principle available on the market. Let x ∈ IR n be the actual
aggregate allocation realized in the market and ¯x = ∑ n
i=1 x i the total amount of
output sold, where output is measured in physical units, i.e. number of individual
cars sold of a particular make and model. Then s i = x i /¯x is the market share of
product variety i (i = 1, . . . , n). Without loss of generality assume that the different
product varieties are numbered in the sequence of decreasing market share,
such that s i ≤ s i+1 for all i = 1, . . . , n − 1. For given n ∈ IN the set of possible
market share distributions is D n , with a typical element s = (s 1 , . . . , s n ) with
0 ≤ s i ≤ 1 for all i = 1, . . . , n and ∑ n
i=1 s i = 1. The set of all possible market
share distributions is D = ⋃ n∈IN Dn . For any function f : D → IR the restriction
of f on D n is denoted by f n . The n coordinated vector of ones is denoted by
1 n = (1, . . . , 1) ∈ IR n .
Now imagine a person who has the statistical information about the aggregate
allocation, i.e. she knows the number of different car models, n ∈ IN, and their
market shares, s ∈ D n . What can we say about the impression of diversity that she
will experience when observing that allocation? What is her degree of uncertainty,
or expected surprise, about the actual allocation? Imagine that this person sets out
to actually observe the allocation. For example, she might take a chair, sit down
next to a busy street and just watch the cars come by. That way, her observations
of the allocation are sequential, i.e. she observes the individual cars in the allocation
one by one. Any single car coming by is a new observation. It reveals which event
has taken place, i.e. which car model has actually occurred. If place and time are
suitably chosen all events will be random draws from the population of all cars in
the allocation. The probability for the next event to be car model i is then given
by that model’s market share s i (i = 1, . . . , n). Actually, the vector of market
shares s ∈ D n has all the properties of a probability distribution over a complete
set of n mutually exclusive events.
2.2 Expected information gain, expected surprise, and effective
diversity
In order to construct a measure of effective product diversity of the allocation
described above (Section 2.1), this section introduces and applies a number of
4

concepts well-known from information theory (e.g. Aczél and Daróczy 1975, Theil
1967). The actual measure of effective product diversity is then constructed in the
following Section 2.3
If it is known that the occurrence of car model i has probability s i and if the
observation reveals that it is indeed model i that has occurred, how large will then
be the observer’s surprise about this finding? Well, when s i = 0.99 she should not
be surprised at all to observe model i since it was practically sure that the next
car to come by would be of model i. In that case the observation has very little
information content. If, on the other hand, s i = 0.01 she will be greatly surprised
to observe model i since it was practically sure that this would not happen. In
that case, the observation has a very large information content.
The information content of an observation thus depends on the probability that
some event would take place before the actual observation was made. It can be
measured by an information function which specifies the information gain from observing
an event with prior probability p with 0 ≤ p ≤ 1. The information content
of an observation also quantifies the observer’s uncertainty prior to actually making
the observation. In this sense, expected information and uncertainty are ‘dual
concepts’ (Theil 1967: 25). While uncertainty prevails prior to the observation,
information is supplied by the observation, thus resolving the uncertainty. Assuming
that the information gain from observing a particular event depends only on
the prior probability of that event, an information function is usually defined in
the following axiomatic way (e.g. Theil 1967: 6).
Definition 1. A function h : ]0, 1] → IR with h = h(p) is an information function
if and only if it satisfies the following properties:
(h-Continuity) h(p) is a continuous function of p for all 0 < p ≤ 1.
(h-Monotonicity) h(p 1 ) > h(p 2 ) if 0 < p 1 < p 2 ≤ 1.
(h-Additivity) h(p 1 p 2 ) = h(p 1 ) + h(p 2 ) if 0 < p 1 , p 2 ≤ 1.
(h-Range) lim p→0 h(p) = +∞ and h(1) = 0.
Continuity states that the gain in information from observing an event does not
change drastically when the prior probability of that event only changes slightly.
Monotonicity states that the higher the prior probability of an event, the lower is
your surprise when actually observing it. Additivity states that the information
gain from observing two stochastically independent events happening jointly is
simply the sum of information gains from observing these two events separately.
While Continuity, Monotonicity and Additivity are the core of any axiomatic foundation
of an information function, there is some arbitrariness in choosing the range
of this function. The Range property serves to put absolute numbers on the quantitative
measure of surprise. It states that your surprise should be infinite when
observing an event that was to occur with zero probability. Conversely, you should
not be surprised at all when observing an event that was to occur for sure. Together
with Monotonicity the Range property implies that h(p) yields nonnegative
5

values for all 0 < p ≤ 1. This can be interpreted as saying that the information
gain from an observation cannot be negative.
The four properties uniquely (up to a positive multiplicative constant) specify
a functional form of the information function.
Proposition 1. A function h : ]0, 1] → IR is an information function if and only
if it is given by
h(p) = − log p, (1)
where any positive real number may be chosen as the base of the logarithm.
Proof : It is easy to verify that h(p) = − log p satisfies all the properties of an
information function. For the proof that it is the only functional form that does
so, see Khinchin (1957: 9-13).
✷
The logarithm in the functional representation of h is due to the requirement of
Additivity. In information theory it has become customary to use the base 2 for the
logarithm since this is consistent with an interpretation of information content in
terms of bits, i.e. the number of answers to yes/no-questions. But as the definition
is only unique up to a positive multiplicative constant one may choose any positive
real number as a base. I will follow the general tendency in economics when dealing
with logarithms and use the natural logarithm, i.e. the base e. If we express, as
we will do in the end, effective diversity not as the expected surprise in terms of
expected information gain, but convert this value into a numbers-equivalent, the
choice of the base does not matter anyway.
Having now an idea of the ‘information content’ of an observation and the
surprise of the observer, we can return to the person sitting next to the street and
watching the cars come by. Knowing the probability distribution s, she will assign
an information content of h(s i ) = − log s i to the event that a car of model i comes
by before making the actual observation. As there are n different possible events
with a probability distribution s = (s 1 , . . . , s n ), the expected information content
of the observation is:
n∑
n∑
H n (s) = s i h(s i ) = − s i log s i , (2)
i=1
i=1
which is the well-known Shannon (1948) entropy. 2 Measuring the expected information
content of an observation, it can also be taken as a measure of expected
surprise or uncertainty. Thus, it is ‘a measure of observational variety or actual
(as opposed to logically possible) diversity’ (Krippendorff 1986: 15), taking into
account that there are n different car models in the allocation and that they may
be unevenly distributed.
2 The choice of the name ‘entropy’ is due to its formal resemblance to the entropy expression
from Statistical Thermodynamics. It should be noted, however, that the name entropy was
chosen by Shannon based on a purely formal analogy and that formula (2) does not have any
substantial relation with the physical entropy concept as known from thermodynamics.
6

For s i = 0 the product s i log s i is not defined. It is therefore customary to
define
0 log 0 := lim
si →0 s i log s i = 0.
Obviously, H n (s) is non-negative for all n ∈ IN and for all s ∈ D n . Its minimum
value is zero and is attained, for given n, when s i = 1 for some i ∈ {1, . . . , n} and
s j = 0 for all j ∈ {1, . . . , n}\{i}. In words, when event i is known to occur with
certainty there is no surprise to be expected when observing which event actually
has occurred. H n (s) assumes its maximal value, for given n, when s = 1 n 1n , i.e.
s i = 1/n for all i = 1, . . . , n. 3 In words, when all events are equally likely the
uncertainty prior to the observation is maximal, and so is the expected surprise by
the observation. Evaluating H n (s) as given by Equation (2) for s = 1 n 1n reveals
that this maximum value is log n. For given n the expected surprise is therefore
bounded by
0 ≤ H n (s) ≤ log n. (3)
The maximum thus increases with n, the number of different possible events. This
is in accordance with the intuition that the uncertainty prior to the observation,
and the expected surprise experienced in the observation, increases with the number
of possibilities. For given n the value of H n (s) increases with the uniformity
or evenness of the distribution. The more evenly the probabilities of different
events are distributed, the higher is the uncertainty prior to the observation. For
example, if there are two different car models and their probabilities are (0.50,
0.50), your uncertainty about which model to observe next is higher than if the
probabilities were (0.95, 0.05), in which case you will be little surprised to observe
the first model.
The expected information content of the observation, Equation (2), has been
calculated as the expectation value of the information functions (1) of the individual
events,
h(s i ) = − log s i for 0 < s i ≤ 1.
In analogy to this expression, Equation (2) can be written as
H n (s) = − log G n (s) for s ∈ D n , (4)
where G n may be interpreted as a kind of average probability. From Equations (2)
and (4) it follows that G n is given by the geometric mean of the probabilities s i
(i = 1, . . . , n) weighted with the same probabilities as weight:
G n (s) =
n∏
i=1
s s i
i for s ∈ D n . (5)
It is possible to define more general weighted means G n and thus to obtain
from equation (4) more general entropies, of which Shannon-entropy is then only
3 This is easily confirmed by maximizing expression (2) over all s ∈ D n for given n ∈ N subject
to the constraint that ∑ n
i=1 s i = 1.
7

a special case (Aczél and Daróczy 1975: Chap. 5). For example, with
( n
)
G n ∑ 1/(α−1)
α(s) = s α i for all α > 0, n ∈ IN, s ∈ D n , (6)
i=1
as a generalized weighted mean of order α where, again, the weights are given
by the probabilities s i , one obtains the following class of one-parameter functions
H α : D → IR, which is due to Rényi (1961):
Definition 2. The following one-parameter functions H α : D → IR are called
entropies of order α, or Rényi-entropies, of the probability distribution s, for all
α > 0:
⎧
⎨
Hα(s) n =
⎩
1
log 1−α (∑ n
i=1 s α i ) ; α > 0, α ≠ 1
lim α→1
1
1−α log (∑ n
i=1 s α i ) ; α = 1
for all n ∈ IN (7)
For α = 1 one obtains Expression (2) for the Shannon-entropy as a special
case. 4 One can show that all the Hα(s) n have basically the same properties as
Shannon-entropy H1 n (s) (Rényi 1961, Aczél and Daróczy 1975: Chap. 5). They
may therefore legitimately be regarded as general measures of expected information.
Proposition 2. The Rényi-entropies, Equation (7), have the following properties
for all α > 0, n ∈ IN and s ∈ D n :
(H-Continuity) Hα(s) n is a continuous function of s.
(H-Symmetry) Hα(s) n = Hα(P n s) for all permutation matrices P .
(H-Maximum) Hα( n 1 n 1n ) > H n (s) for all s ∈ D n \{ 1 n 1n }.
(H-Additivity) Hα mn (r ∗ s) = H m (r) + H n (s) for all m ∈ IN, r ∈ D m ,
s ∈ S n where r ∗ s is the probability distribution consisting
of probabilities r i s j for all i = 1, . . . , m, j = 1, . . . , n.
Proof : Easy to verify.
✷
The Symmetry property states that H n α is symmetric in all its arguments, i.e.
it does not matter for the measurement of expected information gain in what sequence
the different possible events are numbered. The Maximum property states
4 Using the theorem of Bernoulli and l’Hospital, by which lim α→1 φ(α)/ψ(α) =
lim α→1 φ ′ (α)/ψ ′ (α) if φ(1) = ψ(1) = 0 and the derivatives of φ and ψ exist at α = 1 and
ψ ′ (1) ≠ 0, and noting that ∑ n
i=1 s i = 1, one has
lim
α→1
log ( ∑ n
i=1 sα i ) = lim
1 − α
∑ n
i=1 (sα i log s i)
n∑
α→1 (−1) ∑ n = − s i log s i .
i=1 sα i
i=1
8

that H n α reaches its (unique) maximum for a completely uniform distribution. Its
minimum value is zero when one event has probability one, and its maximum
value, attained for an absolutely uniform probability distribution, is log n. The
parameter α determines how much weight is given to n and how much to the evenness
of the probability distribution s in calculating an overall expected information
value based on both, number of different events and evenness of their probability
distribution. I will discuss the properties of expression (7) in more detail in the
next section.
Rényi’s generalized entropy (Equation 7) is a general measure of expected information
content of an observation. By that token it is also a measure of the
expected surprise when sequentially observing an allocation that is only known in
statistical terms. As I have suggested to conceptualize effective diversity as expected
surprise, it seems reasonable to take Expression (7) as a measure of effective
diversity.
2.3 Numbers-equivalent measure of effective diversity
One can establish direct and meaningful comparability of H n α(s) with the pure
diversity n by constructing a numbers-equivalent measure of effective diversity in
the following way. H n α(s) is a logarithmic measure; it yields log n as its maximal
and 0 as its minimal value. In order to compare it directly and in a meaningful
manner to the pure product variety, n, which is defined on a linear scale, we should
essentially take the exponential of H n α(s) to be the numbers-equivalent measure of
effective product diversity. Formally, this is achieved in the following way.
Definition 3. For all n ∈ IN the numbers equivalent measure of effective diversity
of an actual aggregate allocation x ∈ IR n with pure diversity n is the equivalent
number ν of different (hypothetical) products which would yield the same expected
surprise as the actual allocation x when the total output ¯x were evenly distributed
over these ν products.
Formally, taking Rényi’s generalized entropy (Equation 7) as a measure for the
expected surprise, ν is implicitly defined by
H ν α( 1 ν 1ν ) = H n α(s) for all n ∈ IN, s ∈ D n . (8)
The solution of this defining condition is explicitly stated in the following proposition.
Proposition 3. With Rényi’s generalized entropy (Equation 7) as a measure of
the expected surprise by an observation, the numbers equivalent measure of effective
diversity is a function ν : D → IR with:
⎧
⎪⎨ ( ∑ n
να(s) n i=1 s α i ) 1/(1−α) ; α > 0, α ≠ 1
=
⎪⎩ lim a→1 ( ∑ for all n ∈ IN. (9)
n
i=1 s α i ) 1/(1−α) ; α = 1.
9

Proof : As the entropy of a distribution 1 ν 1ν of ν different products with equal
probability 1/ν is given from Equation (7) by Hα( ν 1 ν 1ν ) = log ν it follows that
ν = exp Hα(s) n is the number we are looking for. With expression (7) for Hα(s),
n
expression (9) follows immediately.
✷
We now have the measure we are looking for. While the generalized entropy
H n α (Equation 7) measures the effective product diversity of an allocation in terms
of the expected information gain or expected surprise when sequentially observing
the allocation, the numbers-equivalent ν n α (Equation 9) expresses the effective diversity
by a number which can be directly compared to the pure diversity of that
allocation, n. Both measures are fully equivalent, of course, since ν n α = exp H n α.
3 Properties of effective product diversity
While H n (Equation 7) is for all α > 0, n ∈ IN, s ∈ D n bounded by 0 ≤ Hα(s) n ≤
log n, the corresponding numbers-equivalent να n (Equation 9) is bounded by 1 ≤
να
n ≤ n. Both increase with n and with the evenness of the of the probability
distribution from which they are calculated.
3.1 Effective diversity and market concentration
As ν n α is bounded by 1 ≤ ν n α ≤ n the following identity holds for all α > 0, n ∈ IN,
s ∈ D n :
ν n α(s) ≡ η n α(s) · n, (10)
where the fraction
η n α(s) = νn α(s)
n
= νn α(s)
ν n α(s) | max (11)
is an index of evenness of the distribution s with 1/n ≤ ηα(s) n ≤ 1. At the
same time, the ratio η indicates to what fraction of its pure value n diversity gets
effectively reduced due to unevenly distributed relative shares. While both H and
ν are extensive quantities, i.e. their value increases with n, the ratio η is normalized
to n and thus is an intensive quantity, i.e. its value only depends on the evenness of
s and not on the dimension n of s. If the distribution is absolutely even, s = 1 n 1n ,
the index (11) assumes its maximal value, ηα(s) n = 1, and the effective diversity
of the allocation is given by its pure diversity, να(s) n = n. On the other hand, if
the allocation is maximally uneven, s i = 1 for some i (i = 1, . . . , n) and s j = 0
for all j ≠ i, then the evenness index (11) assumes its minimal value for given n,
s = 1/n, and the effective diversity of the allocation is one, να(s) n = 1.
In a world where every firm produces exactly one product variety the evenness
of the distribution s is equivalent to the market concentration. Identity (10) thus
expresses one key result of the approach developed here: market concentration
effectively reduces the diversity of an allocation. At the same time, identity (10)
provides a simple tool to calculate the effective product diversity from empirical
information about the pure product diversity and the market concentration.
10

Hence it becomes apparent that the measure of effective product diversity proposed
here (Equation 9) is intimately linked to measures of market concentration.
In fact, it is identical to the class of (numbers-equivalent) measures of market concentration
suggested by Hannah and Kay (1977) and derived from the normative
framework of Blackorby et al. (1982) by Chakravarty (1988). It is a subclass of
the general and unifying class of (numbers-equivalent) concentration indices that
has been described in an axiomatic way by Foster and Shneyerov (1999). This will
become obvious from looking at various special cases in the following.
3.2 The role of α
One can show that H n α(s) is continuous, differentiable and decreasing in the parameter
α for all α > 0 (Aczél and Daróczy 1975).
One of the properties of ν n α(s) (Equation 9) is that for given n and s the value of
ν n α(s) decreases with α. As the most widely used diversity or concentration indices
(see below) can all be expressed as special cases of Equation (9) for different values
of α, it becomes evident that the results for the effective species number yielded
by these indices are related in the following way:
ν 0 ≥ ν 1 ≥ ν 2 ≥ ν +∞ , (12)
where equality only holds in the case of equal market shares.
3.3 Special cases
For particular values of α one can recover from Equations (7) and (9) a number
of different special cases which correspond to well-known market concentration
indices. For α = 1 and α = 2 one obtains from expressions (7) and (9) indices
of effective product diversity which correspond to the two indices most widely
used to measure industry concentration, the Shannon (1948)-entropy index and
the Hirschman (1945, 1961)-Herfindahl (1951) index. In general, for 0 < a < +∞
Equations (7) and (9) yield an index of effective diversity which takes into account
both the pure number of different product varieties, n, and the (un)evenness of the
distribution of their shares s. The different H n α(s) and ν n α(s) differ in the extent to
which they include or exclude the relatively less abundant product varieties. The
smaller α, the more are less abundant product varieties included in the measure
of effective diversity, with α = 0 being the extreme case in which all varieties
are equally included. The larger α, the more emphasis is given to more common
product varieties in the estimate of the effective diversity, with α = +∞ being the
extreme case in which only the most common one is taken into account.
3.3.1 The total number of firms (α = 0)
Obviously, for α = 0 Equation (9) reduces to ν n 0 (s) = n. That is, the zeroth order
effective product diversity is just the pure diversity, n, the number of different
products in the allocation. Correspondingly, H n 0 (s) = log n. This means, to zeroth
11

order the indices (7) and (9) take into account all products equally, irrespective of
their market shares.
3.3.2 The Shannon index (α = 1)
Substituting α = 1 into Equations (7) and (9) yields
H1 n (s) =
n∑
− s i log s i , (13)
i=1
ν n 1 (s) = exp H n 1 (s) = Π n i=1
( 1
s i
) si
(14)
where H1
n is the Shannon-expression for entropy (Equation 2) introduced above.
For a given distribution of market shares the Shannon-index (Equation 14)
increases with n, the total number of different product varieties in the allocation.
It yields its maximal value when all n different varieties have equal market share,
s i = 1/n (i = 1, . . . , n). In that case, ν1
n = n, which means that the effective
number of different product varieties equals their pure number. With unequal
market shares, the index yields values smaller than n. The index assumes its
minimal value when an allocation is dominated by one single product variety, with
all others having negligible relative share. In that case, s i ≈ 0 for all i = 1, . . . , n
except i = i ∗ , where i ∗ denotes the dominant product variety, s i ∗ ≈ 1. In that
case, ν1 n ≈ 1, which means that the effective number of different product varieties
is only negligibly larger than one. In general, for given value of n the value of
ν1 n (s) can vary from one to n depending on the variation in market shares, s.
Table 1 illustrates the working of the Shannon-index for different hypothetical
allocations. In a hypothetical allocation A 1 with four different product varieties
(Table 1, column 2), where all four have equal market share 0.25, the Shannonindex
assumes its maximal value, ν1
4 = 4, and thus equals the total number of
different product varieties in that market, n = 4. Similarly, if the number of
equally abundant product varieties increases to n = 5, then ν1 5 = n = 5 (Table 1,
column 3). Allocations A 3 and A 4 (Table 1, columns 4 and 5) illustrate that with
n − 1 equally abundant product varieties and one much less abundant variety,
i = 5 in the example, the Shannon index of effective product diversity will be only
slightly greater than n − 1. The smaller s 5 , the closer ν1
5 approaches n − 1. A
comparison of allocations A 2 and A 6 (Table 1, columns 3 and 7) shows that between
two allocations the effective diversity, ν1 n , can decrease although pure diversity, n,
actually increases This is due to the increase in concentration outweighing the
increase in pure diversity.
3.3.3 The Hirschman-Herfindahl index (α = 2)
With α = 2 one obtains from Equation (9) the following index:
n∑
ν2 n (s) = 1/ s 2 i . (15)
i=1
12

product variety i
share s i in allocation
A 1 A 2 A 3 A 4 A 5 A 6
i = 1 0.25 0.20 0.24 0.249 0.50 0.50
i = 2 0.25 0.20 0.24 0.249 0.30 0.30
i = 3 0.25 0.20 0.24 0.249 0.10 0.10
i = 4 0.25 0.20 0.24 0.249 0.07 0.07
i = 5 - 0.20 0.04 0.004 0.03 0.01
i = 6 - - - - - 0.01
i = 7 - - - - - 0.01
α = 0: ν0 n = n 4 5 5 5 5 7
( )
α = 1: ν1 n = Π n si 1
i=1 s i
4.00 5.00 4.48 4.08 3.42 3.53
α = 2: ν2 n = 1/ ( ∑ n
i=1 s 2 i ) 4.00 5.00 4.31 4.03 2.81 2.82
α = +∞: ν+∞ n = 1/s 1 4.00 5.00 4.17 4.02 2.00 2.00
Table 1: Effective product diversity να
n (Equation 9) for different values of the
parameter α and for different hypothetical allocations A j (j = 1, . . . , 6) which
are characterized by different number n and different relative shares s i of product
varieties.
This is the well known (inverse) market concentration index due to Hirschman
(1945, 1961) and Herfindahl (1951). Its properties are qualitatively similar to
those of the Shannon index, ν1 n (Equation 14). As in the case of the Shannon index,
higher values of ν2
n represent a greater effective product diversity in the sense of
a combination of a higher pure product diversity, n, and a more homogeneous
distribution of market shares, s. Also like the Shannon index, the Hirschman-
Herfindahl index gives less weight to less abundant product varieties than to more
abundant ones in calculating the effective diversity. Table 1 presents values of
ν2
n for different hypothetical allocations, which may be compared directly to the
Shannon index, ν1 n .
The Hirschman-Herfindahl index is strongly weighted towards the most abundant
product variety in the market while being less sensitive to differences in
small market shares and in the pure number of product varieties, as can be seen
from comparing allocations A 5 and A 6 in Table 1 (columns 6 and 7). Being a
logarithmic measure of effective diversity, the Shannon index is more sensitive to
differences in small market shares than the Hirschman-Herfindahl index. On the
other hand, it is less sensitive to small differences in large market shares, whereas
the Hirschman-Herfindahl index responds more substantially to these differences.
3.3.4 The market leader’s concentration ratio (α → +∞)
As α approaches infinity, ν n α(s) goes to 1/s 1 , the inverse market share of the product
variety with the highest market share:
ν n +∞(s) = 1/s 1 . (16)
13

This index, ν n +∞(s), corresponds to the (inverse) concentration ratio CR1 (Shepherd
1987a). It can be interpreted as an effective number of product varieties in
the sense that 1/s 1 gives the equivalent number of equally abundant (hypothetical)
product varieties with the same market share as the leader. If, for example, in an
allocation with n = 5 different product varieties the leading one has a share of
s 1 = 0.5, with the other four having smaller shares, then the effective number of
product varieties in that allocation would be ν 5 +∞ = 1/0.5 = 2 (Table 1, column 6).
ν n +∞ obviously only pays attention to the relative dominance of the most abundant
product variety, neglecting all others.
4 Illustration: Passenger cars in Germany
To illustrate the concepts introduced above let us now turn to one particular
example of a concentrated allocation of a differentiated product, namely passenger
cars in Germany. If you were observing the actual aggregate allocation one by
one, you should not be much surprised to observe a Volkswagen Golf/Bora, since
this is the most abundant car model with a relative share of 10.4% of all cars.
Your surprise should be much greater if you spotted a Chrysler Le Baron or a
Ferrari F355, since these have relative shares below 0.003%. But how large is your
expected surprise overall, and thus the effective diversity of that allocation?
The total stock of passenger cars in Germany on January 01, 2003 is reported
by the Federal Office of Motor Vehicles (Kraftfahrt-Bundesamt 2004) for each
make and model. 5 In total, 44.657.303 passenger cars were registered effective
this date. Of these, ¯x = 38.587.685 can be attributed to a particular model
of car. In the following, all calculations are based on this smaller number of
explicitly attributable registrations. They comprise a pure product diversity of
n = 395 different models. The respective numbers x i and relative shares s i (i =
1, . . . , 395) for each individual model are reported in the appendix. They range
from 4,025,254 registrations for the Volkswagen Golf (market share 10.4%) down
to 1,083 registrations (market share 0.0028%) for the Ford Mustang Convertible.
The mean and median number of registrations per model are 97,690 and 17,012,
with a standard deviation of 305,002.
Figure 1: Total stock of passenger cars in Germany on January 01, 2003 by model.
x i denotes the total number for model i (i = 1, . . . , 395).
Figure 1 shows the total stock of passenger cars by model, where x i denotes the
total number for model i (i = 1, . . . , 395). The different models have been numbered
according to their rank from most to least abundant. The most abundant
one is the Volkswagen Golf/Bora (x 1 = 4, 025, 254, s 1 = 0.1043), second is Opel
Astra (x 2 = 2, 097, 364, s 2 = 0.0544), and third is Opel Corsa (x 3 = 1, 521, 435,
5 A detailed description of this data set is given in the appendix.
14

s 3 = 0.0394). As the figure shows, the market leader has almost twice the market
share of the second placed, the third placed has a relative share only about
one third of the leader’s one, and the number of individual cars of each model
go down very fast with every less abundant model. The leading models together
strongly dominate the whole allocation, with the top 15 models 6 (3.8% of all models)
accounting already for more than half of the entire allocation (20,119,044
registrations, or 52.1%). The other half of the allocation (18,468,641 registrations,
or 47.9%) is shared by 381 models (96.2% of all models) with correspondingly
much lower relative shares.
Figure 2: Relative market share (s i ) for each model of passenger car (i =
1, . . . , 395) in Germany on January 01, 2003.
The strong dominance of the more abundant models over the less abundant
ones becomes apparent again in Figure 2 which shows the relative share (s i ) for
each model (i = 1, . . . , 395) on a logarithmic scale, where the different models
have again been numbered according to their rank from most to least abundant.
The negative slope of the curve indicates how the shares of less abundant models
quickly drop by orders of magnitude. Overall, the curve displays a shape consistent
with a log-normal distribution of shares (or absolute number of registrations) over
rank.
Table 2 reports for different values of the parameter α the expected surprise
H (Equation 7) when observing this allocation. It also reports the corresponding
indices ν of effective diversity (Equation 9) and evenness η (Equation 11). Taking
pure number Shannon Herfindahl CR1
α = 0 α = 1 α = 2 α = +∞
expected surprise Hα 395 (s) 5.98 4.44 3.61 2.26
effective diversity να 395 (s) 395 84.94 36.84 9.59
evenness index ηα 395 (s) 1.00 0.22 0.09 0.02
Table 2: Pure and effective diversity of the allocation s of passenger cars in Germany
on January 01, 2001 for different values of the parameter α. H n α is the generalized
entropy (Equation 7), ν n α is the corresponding numbers-equivalent measure
of effective diversity (Equation 9), η n α is the corresponding evenness index (Equation
11).
α = 0 reproduces the pure diversity (n = 395) of the allocation. With α = 1
6 The top 15 models are (in this sequence): Volkswagen Golf/Bora, Opel Astra, Opel Corsa,
Volkswagen Passat, BMW 3-series, Volkswagen Polo, Audi A4/S4, Opel Vectra, Mercedes C-
class, Ford Fiesta/Fusion, Mercedes E-class, Ford Escort, Ford Mondeo, BMW 5-series, Audi
A6/S6.
15

one obtains the Shannon entropy, H1 395 (s) = 4.44, or the corresponding numbersequivalent,
ν1 395 (s) = 84.94. By this number the effective diversity of the allocation
is less than one quarter (η1 395 (s) = 0.22) of its pure diversity, due to the unevenness
of the distribution of market shares. It is this number that describes the subjective
impression of diversity (in the sense of expected surprise) that an observer
experiences when sequentially observing the allocation. Taking α = 2 yields the
Hirschman-Herfindahl index values H2 395 (s) = 3.61 and ν2 395 (s) = 36.84, which indicates
that the effective diversity of the allocation is less than 10% (η2 395 (s) = 0.09)
of its pure diversity, due to the unevenness of the distribution of market shares.
Obviously, a higher value of α places more weight on how uneven distribution
diminishes diversity. As a consequence, the Hirschman-Herfindahl values indicate
a lower effective diversity than the Shannon values. In number terms, going from
α = 1 to α = 2 means a drop in effective diversity index by more than one half.
Figure 3: Effective product diversity να
395 (s) (Equation 9) of the allocation s of
passenger cars in Germany on January 01, 2001 as a function of the parameter α.
Figure 3 shows how να
395 (s) depends on the parameter value α. As α approaches
infinity, να
395 (s) approaches the inverse market share of the leading model (Volkswagen
Golf/Bora, 10.4%), ν+∞(s) 342 = 11.7. This index of effective diversity only
takes into account the dominance of the leading product variety over all others.
It can be interpreted as the number of equally abundant (hypothetical) product
varieties that would cause the same expected surprise than the actual heterogenous
allocation with 395 different product varieties.
5 Conclusion
Diversity causes surprise. I have proposed to take this everyday experience as a
guideline for an economically meaningful definition and measurement of effective
diversity. Expected surprise may be quantified by information theoretic tools as the
expected information gain from an observation of an actual economic allocation.
I have suggested to use Rényi’s (1961) generalized entropy (Equation 7), or the
corresponding numbers-equivalent (Equation 9), as such a measure of effective
diversity. The effective diversity of an actual aggregate allocation increases with
the pure number of different product varieties and the evenness of the distribution
of their relative shares in the allocation. In general, when the different product
varieties are unevenly distributed the effective diversity of that allocation will be
smaller than its pure diversity. Thus, market concentration effectively reduces
diversity below its pure value. I have illustrated this concept in an empirical
analysis of the diversity of passenger cars in Germany.
The conclusion from this analysis is that the effective diversity as experienced
by an observer may be considerably smaller than the underlying pure diversity.
The exact quantitative amount by which pure and effective diversity differ because
16

of unevenly distributed market shares depends on how much weight is given to
pure diversity and how much to evenness when calculating the index of effective
diversity.
When deriving the entropy measure of effective diversity I have made the simplifying
assumption that all product varieties are pairwise equally dissimilar. Thus
the pure diversity of an allocation is simply the number of different product varieties.
Effective diversity measures by how much that number is diminished due to
the uneven distribution of relative shares of product varieties. In future research
one could apply the idea of diversity as a potential for surprise to measures of pure
diversity which are based on pairwise dissimilarity between product varieties, such
as those of Weitzman (1992, 1998), Gans and Hill (1997), Bernhofen (2001) and
Nehring and Puppe (2002).
Another line of research could address the question of what are the benefits and
costs of effective (as opposed to pure) product diversity for economic agents who
reach consumption decisions over time and under uncertainty. On the one hand,
higher diversity generally allows consumers to reach a higher utility level as the
range of choice is expanded or consumers may have preferences for diversity per
se. On the other hand, with higher diversity also the costs of making consumption
decisions are higher. For example, these costs may be search costs due to the time
spent when identifying the different available product options or finding dealers
that offer the preferred variety. The information theoretic measure of effective
diversity presented here can be used to quantify the value of information gained
from observing the market allocation and thus the decision costs associated with
a given level of pure diversity.
Economists have been analyzing product diversity for quite some time now.
Most of the conceptions of diversity used so far are static and assume complete
knowledge of all the options at hand. As the economic relevance of diversity
ultimately stems from uncertainty over the future it seems logical to now move
on, and address the role of economic diversity in a dynamic context and under
uncertainty. The information theoretic approach presented here may be a fruitful
starting point.
References
[1] Aczél, J. and Z. Daróczy (1975). On Measures of Information and their Characterizations.
Academic Press, New York.
[2] Bernhofen, D.M. (2001). ‘Measuring diversity in product space: the missing
axiom’, Economics Letters 70: 69-72.
[3] Berry, C.H. (1975). Corporate Growth and Diversification. Princeton University
Press, Princeton.
17

[4] Blackorby, C., D. Donaldson and J.A. Weymark (1982). ‘A normative approach
to industrial-performance evaluation and concentration indices’, European
Economic Review 19: 89-121.
[5] Chakravarty, S.R. (1988). ‘On the separable industry performance evaluation
function’, Economics Letters 27: 179-182.
[6] Chakravarty, S.R. and J.A. Weymark (1988). ‘Axiomatizations of the entropy
numbers equivalent index of industrial concentration’, in: W. Eichhorn (ed.),
Measurement in Economics. Theory and Applications of Economic Indices,
Physica-Verlag, Heidelberg, pp. 383-397.
[7] Dekel, E., B.L. Lipman and A. Rustichini (1998). ‘Recent developments in
modeling of unforeseen contingencies’, European Economic Revue 42: 523-
542.
[8] Dixit, A.K. and V. Norman (1980). Theory of International Trade. Cambridge
University Press, Cambridge.
[9] Dixit, A.K. and J.E. Stiglitz (1977). Monopolistic competition and optimum
product diversity. American Economic Review 67(3): 297-308.
[10] Eichberger, J. and I.R. Harper (1997). Financial Economics. Oxford University
Press, Oxford.
[11] Ethier, W. (1982). ‘National and international returns to scale in the modern
theory of international trade’, American Economic Review 72: 389-405
[12] Faber, M., R. Manstetten and J.L.R. Proops (1992). ‘Humankind and the
environment: an anatomy of surprise and ignorance’, Environmental Values
1: 217-241.
[13] Foster, J.E. and A.A. Shneyerov (1999). ‘A general class of additively decomposable
inequality measures’, Economic Theory 14: 89-111.
[14] Gans, J. and R. Hill (1997). ‘Measuring product diversity’, Economics Letters
55: 145-150.
[15] Gehrig, W. (1988). ‘On the Shannon-Theil concentration measure and its
characterizations’, in: W. Eichhorn (ed.), Measurement in Economics. Theory
and Applications of Economic Indices, Physica-Verlag, Heidelberg, pp. 413-
427.
[16] Hannah, L. and J.A. Kay (1977). Concentration in Modern Industry. Theory,
Measurement and the U.K. Experience. Macmillan London.
[17] Helpman, E. and P.R. Krugman (1985). Market Structure and Foreign Trade:
Increasing Returns, Imperfect Competition, and the International Economy.
Cambridge University Press, Cambridge.
18

[18] Herfindahl, O. (1950). Concentration in the Steel Industry. Unpublished Ph.D.
Dissertation, Columbia University.
[19] Hirschman, A.O. (1945). National Power and the Structure of Foreign Trade.
University of California Press, Berkeley and Los Angeles.
[20] Hirschman, A.O. (1964). ‘The paternity of an index’, American Economic
Revue 54: 761.
[21] Jacquemin, A.P. and C.H. Berry (1979). ‘Entropy measure of diversification
and corporate growth’, Journal of Industrial Economics 27: 359-369.
[22] Khinchin, A.I. (1957). Mathematical Foundations of Information Theory.
Dover, New York.
[23] Kraftfahrt-Bundesamt (2004). ‘Bestand an Personenkraftwagen am 1. Januar
2003 nach Segmenten und Typgruppen’, Press Release, January 2004,
Kraftfahrt-Bundesamt (German Federal Office of Motor Vehicles), Flensburg.
[24] Krippendorff, K. (1986). Information Theory. Structural Models for Qualitative
Data. Sage, Beverly Hills.
[25] Lancaster, K.J. (1966). ‘A new approach to consumer theory’, Journal of
Political Economy 74: 132-157.
[26] Matutinović, I. (2001). ‘The aspects and the role of diversity in socio-economic
systeme: an evolutionary perspective’, Ecological Economics 39: 239-256.
[27] Mowery, D.C. and N. Rosenberg (1998). Paths of Innovation. Technological
Change in 20th-Century America. Cambridge University Press, Cambridge.
[28] Nehring, K. and C. Puppe (2002). ‘A theory of diversity’, Econometrica 70:
1155-1198.
[29] Puppe, C. (1996). ‘An axiomatic approach to ”preference for freedom of
choice”’, Journal of Economic Theory 68: 174-199.
[30] Rényi, A. (1961). ’On measures of entropy and information’, in: J. Neyman
(ed.), Proceedings of the Fourth Berkeley Symposium on Mathematical
Statistics and Probability, Vol. I, University of California Press, Berkeley, pp.
547-561.
[31] Romer, P.M. (1987). ‘Growth based on increasing returns due to specialization’,
American Economic Review 77(2): 56-62.
[32] Romer, P.M. (1990). ‘Endogenous Technological Change’, Journal of Political
Economy 98(5, part II): S71-S102.
19

[33] Schiller, J. (2001). Umweltprobleme und Zeit. Bestände als konzeptionelle
Grundlage ökologischer Ökonomik. Ph.D. Dissertation, University of Heidelberg.
[34] Sen, A. (1988). ‘Freedom of choice: concept and content’, European Economic
Review 32: 269-294.
[35] Shackle, G.L.S. (1955). Uncertainty in Economics and Other Reflections.
Cambridge University Press, Cambridge.
[36] Shannon, C.E. (1948). ‘A mathematical theory of communication’, Bell System
Technical Journal 27: 379-423 and 623-656.
[37] Shepherd, W.G. (1987a). ‘Concentration ratios’, in: J.M. Eatwell, M. Milgate
and R. Newman (eds), The New Palgrave. A Dictionary of Economics, Vol.
1, Macmillan, London, pp. 563-564.
[38] Shepherd, W.G. (1987b). ‘Herfindahl index’, in: J.M. Eatwell, M. Milgate
and R. Newman (eds), The New Palgrave. A Dictionary of Economics, Vol.
2, Macmillan, London, p. 639.
[39] Spence, A.M. (1976). ‘Product selection, fixed costs, and monopolistic competition’,
Review of Economic Studies 43: 217-236.
[40] Theil, H. (1967). Economics and Information Theory. North-Holland, Amsterdam.
[41] Weikard, H.-P. (1999). Wahlfreiheit für zukünftige Generationen. Neue
Grundlagen für eine Ressourcenökonomik. Metropolis, Marburg.
[42] Weitzman, M. (1992). ‘On diversity’, Quarterly Journal of Economics 107:
363-405.
[43] Weitzman, M. (1998). ‘The Noah’s Ark problem’, Econometrica 66: 1279-
1298.
Appendix: Data used in Section 4
The following table reports the data on the stock of passenger cars in Germany
on January 01, 2003 that have been used in Section 4. The data set has been
assembled from an official report published by the German Federal Office of Motor
Vehicles (Kraftfahrt-Bundesamt 2004). The data reported there have been
calculated by an inventory-update-method from the monthly number of new registrations,
transcriptions, and deregistrations since January 01, 1990. The report
explicitly lists only models with a minimum number of 1,000 cars registered at the
end of period date (January 01, 2003). Models with fewer registrations, and registrations
for which the vehicle identification number (“FIN”) could not properly be
20

processed, are grouped together as “others”. Because of the space constraint only
a part of the entire data set is reproduced here. The full data set can be obtained
from the author upon request.
rank make model registrations market share
i x i s i [%]
1 Volkswagen Golf/Bora 4,025,254 10.4315
2 Opel Astra 2,097,364 5.4353
3 Opel Corsa 1,521,435 3.9428
4 Volkswagen Passat 1,521,385 3.9427
5 BMW 3-series 1,339,196 3.4705
6 Volkswagen Polo 1,325,186 3.3368
7 Audi A4/S4 1,287,576 3.0477
8 Opel Vectra 1,176,037 3.0477
9 Mercedes C-class 1,023,111 2.6514
10 Ford Fiesta/Fusion 993,508 2.5747
11 Mercedes E-class 937,928 2.4306
12 Ford Escort 831,033 2.1536
13 Ford Mondeo 789,338 2.0456
14 BMW 5-series 682,132 1.7678
15 Audi A6/S6 568,561 1.4734
. .
.
.
.
66 Skoda Fabia 100,533 0.2605
67 Skoda Felicia 98,418 0.2551
68 Ford Scorpio 97,010 0.2514
69 Citroen AX 96,270 0.2495
. .
.
.
.
196 Isuzu Trooper/Monterey 17,402 0.0451
197 Volvo 80 17,138 0.0444
198 Alfa Romeo Spider 17,012 0.0441
199 Suzuki Samurai 16,876 0.0437
200 Toyota Picnic 16,829 0.0436
. .
.
.
.
389 Chrysler Le Baron 1,123 0.0029
390 Lada 110, 111, 112 1,121 0.0029
391 Honda Integra 1,114 0.0029
392 Toyota Paseo 1,110 0.0029
393 Citroen Visa 1,108 0.0029
394 Ferrari F355 1,089 0.0028
395 Ford Mustang (convertible) 1,083 0.0028
Subtotal 38,587,685 100.0000
Others Others 6,069,618
Total 44,657,303
21