2G1318 Queuing theory and teletraffic systems - KTH

EP2200 Queuing theory and teletraffic systems
3rd lecture
Markov chains
Birth-death process
- Poisson process
Discrete time Markov chains
Viktoria Fodor
KTH EES
1

Outline for today
• Markov processes
– Continuous-time Markov-chains
– Graph and matrix representation
• Transient and steady state solutions
• Balance equations – local and global
• Pure Birth process – Poisson process as special case
• Birth-death process as special case
• Outlook: Discrete time Markov-chains
EP2200 Queuing theory and teletraffic
systems
2

• Stochastic process
– p i(t)=P(X(t)=i)
Markov processes
• The process is a Markov process if the future of the process depends on the
current state only - Markov property
– P(X(t n+1)=j | X(t n)=i, X(t n-1)=l, …, X(t 0)=m) = P(X(t n+1)=j | X(t n)=i)
– Homogeneous Markov process: the probability of state change is unchanged
by time shift, depends only on the time interval
P(X(t n+1)=j | X(t n)=i) = p ij(t n+1-t n)
• Markov chain: if the state space is discrete
– A homogeneous Markov chain can be represented by a graph:
• States: nodes
• State changes: edges
EP2200 Queuing theory and teletraffic
systems
0 1
M
3

Continuous-time Markov chains
(homogeneous case)
• Continuous time, discrete space stochastic process, with Markov
property, that is:
P(
X ( t
P(
X ( t
n�1
n�1
) �
) �
j | X ( t ) � i,
X ( t
n
j | X ( t ) � i),
n
t
n�1
0
) � l,
�X
( t
• State transition can happen in any point of time
• Example:
– number of packets waiting at the output buffer of a router
– number of customers waiting in a bank
� t
1
��
� t
• The time spent in a state has to be exponential to ensure Markov
property:
– the probability of moving from state i to state j sometime between
t n and t n+1 does not depend on the time the process already spent
in state i before t n.
n�1
EP2200 Queuing theory and teletraffic
systems
n
0
) � m)
�
� t
states
t 0 t 1 t 2 t 3 t 4 t 5
4

Continuous-time Markov chains
(homogeneous case)
• State change probability depends on the time interval:
P(X(t n+1)=j | X(t n)=i) = p ij(t n+1-t n)
• Characterize the Markov chain with the state transition rates instead:
q
ij
q
ii
P(X(t � Δt) � j|X(t) � i)
� lim
, i � j - rate (intensity) of state change
Δt � 0 Δt
� � � q - defined to easy calculation later on
ij
j � i
Transition rate matrix Q:
� q
�
�
�
Q �
�
�
��
qM
00
0
q
01
�
�
q
�
M ( M �1)
q
q
0M
( M �1)
M
q
MM
�
�
�
�
�
��
0
q 01=4
q 10=6
EP2200 Queuing theory and teletraffic
systems
1
��
4
Q
� �
� 6
4 �
� 6
�
�
5

Outline for today
• Markov processes
– Continuous-time Markov-chains
– Graph and matrix representation
• Transient and steady state solutions
• Balance equations – local and global
• Pure Birth process – Poisson process as special case
• Birth-death process as special case
• Outlook: Discrete time Markov-chains
EP2200 Queuing theory and teletraffic
systems
6

q
ij
i
i
�
�t�0
p ( t � �t)
�
p ( t)
� p ( t)
pi
( t � �t)
� pi
( t)
�
�t
i
i
i
i
j
p
j�i
p ( t � �t)
� p ( t)
� p ( t)
q �t
�
dp(
t)
�
dt
P(
X ( t � �t)
� j | X ( t)
� i)
lim
�t
�
�
p(
t)
Q, p(
t)
� p(
0)
�e
j
ii
q
ij
( t)
q
�t
�
ji
�
j�i
Qt
Transient solution
• The transient - time dependent – state probability distribution
• p(t)={p 0(t), p 1(t), p 2(t),...} – probability of being in state i at time t,
given p(0).
�
j�i
p
j
p
�(
�t)
�
�t
j
�
( t)
q
( t)
q
ji
�
P(
X ( t � �t)
�
ji
�t
��
( �t),
leaves the state arrives to the state
�t
��
( �t)
�
dpi
( t)
�
dt
�
j
�
j
�t�0
p
p
j | X ( t)
� i)
� q
�(
�t)
lim � 0
�t
j
j
( t)
q
( t)
q
Transient solution
EP2200 Queuing theory and teletraffic systems
ji
ji
ij
�t
��
( �t)
�t
��
( �t)
( �
�
j�i
q
ij
�
q
ii
)
7

0.8
0.6
0.4
0.2
0
1
q 01=4
q 10=6
1
p 0(t)
p 1(t)
Example – transient solution
��
4
� �
� 6
4 �
� 6
�
�
0.2 0.4 0.6 0.8 1
dp(
t)
p(
t)
Q
�
dt
p(
t)
� p(
0)
�e
Q Qt
0.8
0.6
0.4
0.2
EP2200 Queuing theory and teletraffic systems
1
Transient
state
p 0(t)
p 1(t)
Stationary / steady
state
0.2 0.4 0.6 0.8 1
A: p 0(0)=0, p 1(0)=1 B: p 0(0)=1, p 1(0)=0
8

• Matrix exponential
• Matrix exponentials are defined as:

• Def: stationary state probability distribution (stationary solution)
– p � lim p(
t)
exists
t�� Stationary solution (steady state)
– p is independent from p(0)
• The stationary solution p has to satisfy:
dp(
t)
( t)
Q � � 0,
� p ( t)
�1
dt
p i
Note: the rank of Q MM is M-1!
0
q 01=4
q 10=6
1
�p , p � � �0, 0�
p
0
0
�
1
0.
6,
��
4
�
� 6
� q
�
�
�
Q
�
�
�
��
qM
p
4 �
� 6
�
�
0.
4
EP2200 Queuing theory and teletraffic systems
1
�
00
0
q
01
�
�
q
,
�
M ( M �1)
p
0
�
q
q
( M �1)
M
q
p
0M
MM
1
�
�
�
�
�
��
�1
10

Important theorems – without the proof
• Stationary solution exists, if
– The Markov chain is irreducible (there is a path between any two states) and
pQ
� 0, p�1
�1
Stationary solution (steady state)
– has positive solution
• Equivalently, stationary solution exists, if
– The Markov chain is irreducible
– For all states: the mean time to return to the state is finite
• Finite state, irreducible Markov chains always have stationary solution.
• Markov chains with stationary solution are also ergodic:
– p i gives the portion of time a single realization spends in state i, and
– the probability that one out of many realizations are in state i at arbitrary
point of time
EP2200 Queuing theory and teletraffic systems
11

Outline for today
• Markov processes
– Continuous-time Markov-chains
– Graph and matrix representation
• Transient and steady state solutions
• Balance equations – local and global
• Pure Birth process – Poisson process as special case
• Birth-death process as special case
• Outlook: Discrete time Markov-chains
EP2200 Queuing theory and teletraffic systems
12

Balance equations
• How can we find the stationary solution? pQ=0
0 � pQ
�
State
0 � �(
q
q
21
12
2
State
0 � q
q
p
p
1
1:
12
12
� ( q
2 :
p
1
� q
� q
32
12
� q
p
3
14
21
) p
� q
p
14
2
� q
1
) p
21
� q
1
� q
p
32
2
21
flow in flow out
p
p
3
2
q 14
q12 P1 P2 P3 q 21
• Global balance conditions
– In equilibrium (for the stationary solution)
– the transition rate out of a state – or a group of states - must equal
the transition rate into the state (or states)
• flow in = flow out
– defines a global balance equation
P 4
q 43
q 32
EP2200 Queuing theory and teletraffic systems
��
( q12
� q
�
� �
q21
Q
� 0
�
� 0
14
)
q
12
� q
q
32
0
21
0
0
� q
q
43
32
q
14
0
0
� q
43
�
�
�
�
�
�
13

Group work
• Global balance equation for state 1 and 2:
0 � pQ
State
0 � �(
q
q
21
12
2
State
0 � q
q
p
p
1
1:
12
12
� ( q
2 :
p
1
� q
32
�
� q
12
� q
p
3
14
21
) p
� q
p
14
2
� q
1
) p
21
� q
1
� q
p
32
2
21
p
p
3
2
q 12
q 14
EP2200 Queuing theory and teletraffic systems
P 4
q 43
P 1 p 2 P 3
q 21
• Is there a global balance equation for the circle around
states 1 and 2?
q 32
14

Balance equations
• Local balance conditions in equilibrium
– the local balance means that the total flow from one part of the chain
must be equal to the flow back from the other part
– for all possible cuts
– defines a local balance equation
– The local balance equation is the same as a global balance equation
around a set of states!
q 14
P 4
q12 P1 p2 P3 q 21
q 43
q 32
EP2200 Queuing theory and teletraffic systems
15

Balance equations
• Set of linear equations instead of a matrix equation
0
0
q
�
�
12
p
pQ
q
1
12
p
1
� q
32
�
� q
p
• Global balance :
– flow in = flow out around a state
– or around many states
• Local balance equation:
– flow in = flow out across a cut
q43p4 �
q32p3
• M states
– M-1 independent equations
– �pi=1 3
21
�
p
2
q
� q
21
p
32
2
p
flow in flow out
3
EP2200 Queuing theory and teletraffic
systems
q 12
q 14
P 4
q 43
P 1 p 2 P 3
q 21
q 12
q 14
P 4
q 43
q 32
P 1 p 2 P 3
q 21
q 32
16

Outline for today
• Markov processes
– Continuous-time Markov-chains
– Graph and matrix representation
• Transient and steady state solutions
• Balance equations – local and global
• Pure Birth process – Poisson process as special case
• Birth-death process as special case
• Outlook: Discrete time Markov-chains
EP2200 Queuing theory and teletraffic
systems
17

Pure birth process
• Continuous time Markov-chain, infinite state space
• Transitions occur only between neighboring states
– State independent birth intensity: � � �,
�i
k-1 k k+1
• No stationary solution
• Transient solution:
� k-1=� � k=�
− p k(t)=P(system in state k at time t)
− number of events (births) in an interval t
EP2200 Queuing theory and teletraffic
systems
i
Q
�
��
�
�
�
0
� 0
�
� 0
�
� �
0
0
0
�
� �
0
��
0
�
�
� �
�
��
18

Pure birth process
• Transient solution – number of events (births) in an interval (0,t]
p'(
t)
� p(
t)
Q,
p'
p'
�
p'
0
1
k
( t)
� ��p
( t)
� �p
0
( t)
� �p
0
k �1
�k-1=� �k=� k-1 k k+1
( t)
p
0
( 0)
( t)
� �p
( t)
1
( t)
� �p
k
�1,
( t)
�
p
�
k
p
( 0)
0
p
( t)
� e
k
� 0
��t
� p'
( t)
� �e
1
��t
( �t)
( t)
�
k!
� �p
( t)
• Pure birth process gives Poisson process! – time between state
transitions is Exp(λ)
for
k
��t
EP2200 Queuing theory and teletraffic
systems
e
k
1
� 0
� p ( t)
� �te
1
��t
Q
�
��
�
�
�
0
� 0
�
� 0
�
� �
0
0
0
�
� �
0
��
0
�
�
� �
�
��
19

Equivalent definitions of Poisson process
1. Pure birth process with intensity �
2. The number of events in period (0,t] has Poisson distribution with
parameter �
3. The time between events is exponentially distributed with parameter �
P(
X � t)
�1�
e
��t
previous slide
number of events
Poisson distribution
pure birth process
time between events
exponential
previous lecture
EP2200 Queuing theory and teletraffic
systems
check in the binder
20

Pure death process
• Continuous time Markov-chain, infinite state space
• Transitions occur only between neighboring states
– State independent death intensity: � � �,
�i
� 0
• No stationary solution
k-1 k k+1
�
�
• Pure death process gives Poisson process until reaching state 0
• Time between state transitions is Exp(µ)
EP2200 Queuing theory and teletraffic
systems
i
21

Outline for today
• Markov processes
– Continuous-time Markov-chains
– Graph and matrix representation
• Transient and steady state solutions
• Balance equations – local and global
• Pure Birth process – Poisson process as special case
• Birth-death process as special case
• Outlook: Discrete time Markov-chains
EP2200 Queuing theory and teletraffic
systems
22

Birth-death process
• Continuous time Markov-chain
• Transitions occur only between neighboring states
i�i+1 birth with intensity λ i
i�i-1 death with intensity μ i (for i>0)
�
��
q
�
� q10
Q �
� q20
�
� �
0 j
�
q
�
q
01
21
1 j
�
�
02
12
2 j
��
��
�0
� �
��
� �
�1
��
� 0
� �
��
�
modells population
�k-1 �k k-1 k k+1
q
q
q
q
� k
� ( � � � )
EP2200 Queuing theory and teletraffic
systems
�
1
�
0
2
1
� ( � � � )
• State holding time – length of time spent in a state k
– Until transition to states k-1 or k+1
– Minimum of the times to the first birth or first deaths � minimum of two
Exponentially distributed random variables: Exp(� k+� k)
� k+1
2
0
�
1
2
��
0
�
�
� � 2
�
��
23

B-D process - stationary solution
• Local balance equations, like for general Markov-chains
• Stability: positive solution for p (since the MC is irreducible)
Cut 1:
�k
�1
pk
�1
� �k
pk
�
�k
�1
pk
� pk
�1
�
Cut 2 : � p
�
�
�
p
0
p
p
k
k
�
� 1
�
1�
k
�
1
�
k
�
1
k �1
�
� �
�0
��
� ��
k �1
k
�i
�
k �1
i�0
k �1 i�1
p
,
p
0
k �1
�
k �1
�
i�0
�
�i
�
i�1
p
k �1
p
0
,
�
k
�k
�
k �1
k-1
k �1
� k-1
EP2200 Queuing theory and teletraffic
systems
p
k
�
�k�k
�1
� �
k
p
� k
k �1
k
Cut 1 Cut 2
� k
� k+1
k+1
Group work: stationary solution for
state independent transition rates:
�i � �,
�i
� �.
24

Markov-chains and queuing systems
• Why do we like Poisson and B-D processes?
How are they related to queuing systems?
– If arrivals in a queuing system can be modeled as Poisson
process � also as a pure birth process
– If services in a queuing systems can be modeled with
exponential service times � also as a (pure) death process
– Then the queuing system can be modeled as a birth-death
process
EP2200 Queuing theory and teletraffic
systems
λ
λ
μ μ
25

Summary – Continuous time Markov-chains
• Markovian property: next state depends on the present state
only
• State lifetime: exponential
• State transition intensity matrix Q
• Stationary solution: pQ=0, or balance equations
• Poisson process
– pure birth process (�)
– number of events has Poisson distribution, E[X]=�t
– interarrival times are exponential E(�)=1/�
• Birth-death process: transition between neighboring states
• B-D process may model queuing systems!
EP2200 Queuing theory and teletraffic
systems
26

Discrete-time Markov-chains
(detour)
• Discrete-time Markov-chain: the time is discrete as well
– X(0), X(1), … X(n), …
– Single step state transition probability for homogeneous MC:
P(X(n+1)=j | X(n)=i) = p ij, �n
• Example
– Packet size from packet to packet
– Number of correctly received bits in a packet
– Queue length at packet departure instants …
(get back to it at non-Markovian queues)
p 01
p1m pm1 0 1
M
p 10
p 0m
p m0
2G1318 Queuing theory and teletraffic systems
27

• Transition probability matrix:
– The transitions probabilities can be represented in a matrix
– Row i contains the probabilities to go from i to state j=0, 1, …M
p 01
Discrete-Time Markov-chains
• P ii is the probability of staying in the same state
0 1
M
p 10
p 0m
p m0
p1m pm1 � p00
p01
� p0M
�
�
p p
�
� 10 11 �
P � �,
� pij
�1
� � � � � j
�
�
� pM
0 � pMM
�
2G1318 Queuing theory and teletraffic systems
�i
28

• The probability of finding the process in state j at time n is denoted by:
– p j (n) = P(X(n) = j)
– for all states and time points, we have:
� � ) (
) ( ( n)
n
p p p
p �
�
( n)
n
0 1
M
• The time-dependent (transient) solution is given by:
p
p
( n�1)
i
( n�1)
�
p
Discrete-Time Markov-chains
� pi
pii
��
p
j�i
( n)
j
p
( n)
( n�1)
P � p PP � �
ji
�
p
( 0)
P
n�1
2G1318 Queuing theory and teletraffic systems
p 01
p1m pm1 0 1
M
p 10
p 0m
p m0
29

• Steady (or stationary) state exists if
– The limiting probability vector exists
– And is independent from the initial probability vector
lim �
n��
• Stationary state probability distribution is give by:
• Note also:
�p p p �
( n)
p � p 0 1 �
M
p � p P,
�
j�
0
p
j
�1
M
– The probability to remain in a state j for m time units has geometric
distribution
�1 p 1�
m
jj
Discrete-Time Markov-chains
� p �
jj
� ( n�1)
( n)
p � p P�
– The geometric distribution is a memoryless discrete probability
distribution (the only one)
2G1318 Queuing theory and teletraffic systems
30

Summary
• Continuous-time Markov chains
• Balance equations (global, local)
• Pure birth process and Poisson process
• Birth-death process
• Discrete time Markov chains
EP2200 Queuing theory and teletraffic
systems
31