Markov Chains

Markov Chains
1 Discrete or Continuous Markov Chain
What is the difference between a continuous time Markov Chain and a discrete Markov
Chain?
See Ex Moshe Zukerman or Google for more detail.
2 Two state Markov Chain
Consider a communications system which transmits digits 0 and 1. Each digit transmitted
must pass through several stages, at each of which there is a probability p that
the digit entered will be unchanged when it leaves. Letting X n denote the digit entering
the nth stage, then {X n , n = 0, 1, .....} is a two state Markov Chain.
a) Draw the two state Markov Chain and set up the transition probability Matrix.
p
1-p
p
o 1
1-p
Figure 1: State diagrams
[
P = p ij =
p 1 − p
1 − p p
]
.
1

) Solve the state equations to find the limiting probabilities.
π 0 + π 1 = 1
Which implies that
π 0 = π 0 p + π 1 (1 − p)
π 1 = π 2 = 0.5
3 One server with blocking
Rejected Customers
Figure 2: State diagrams
Fig. 2 shows a system that consists of one server that can only handle one customer
at a time. This means that if a customer arrives and the server is busy, the customer
will be rejected. Let I be the number in the system I = 0, 1. Assume that at every
time unit a new customer will arrive with probability α and the probability that a
customer in service will complete service is β. This means that if there is both a
service completion and an arriving customer in a time unit, the system will remain
unchanged.
a) The probability r is the probability that when the system is in state 1, the system
will remain in this state. Determine r.
r = αβ + α(1 − β) + (1 − β)(1 − α)
b) Draw the state transition diagram with all the transition probabilities.
c) Set up the state transition matrix and use this to find the stationary probabilities
P (I = 0) and P (I = 1).
[ ]
0.74 0.25
P = p ij =
.
0.375 0.625
π 0 = 0.6andπ 1 = 0.4
2

1-α
α
r
o 1
β(1-α)
Figure 3: State diagrams
d) Show that you will get the same answer by using cut-equations.
e) What is the expected number of customers in service (the Carried traffic)?
A ′ = 1 ∗ π 1
4 Negative exponential distribution
Let X 1 , ..., X n be independent exponential random variables with mean µ i
E(X i ) = 1 , i = 1, ..., n.
µ
Define
and
Y n = min(X 1 , ..., X n )
Z n = max(X 1 , ..., X n )
.
a) Determine the distribution of Y n and Z n
3

See Moshe Zukerman page 19 for some hints. The variables are independent so :
Use that
n∏
P (Y n > x) = P (X i > x)
i=1
and
n∏
P (Z n ≤ x) = P (X i ≤ x)
Then just plug in P (X i > x = e µ−it and P (X i ≤ x = 1 − e muit
i=1
b) Show that the probability that X i is the smallest one among X 1 , ..., X n is equal
µ
to i
µ 1+...+µ n
, i = 1, ..., n
Using the continuous version of total probability:
P (X i = min(X 1 , ...., X n )) =
∫ ∞
0
∫ ∞
0
∏
P (X i > x)f Xi (x)dx
j≠i
e −(µ1+µ2+..µn)x µ i e −µix dx =
µ 1 + µ 2 + ... + µ n
µ i
5 Continuous Time Markov Chain
Do problem 2.21 in Moshe Zukerman.
Birth rate: b i (negative Exponential distribution)
Death rate: d i (negative Exponential distribution)
P(increase by 1): this means that a birth happens before a death. Since the birth
rate and death rate is negative exponential distributions:
Similarly:
d i
P (increase by 1) =
b i + d i
b i
P (decrease by 1) =
b i + d i
See above problem for the derivation of the probability of one negative exponential
bigger than another and also Zukerman p. 19.
4