Lecture Notes - Department of Mathematics and Statistics - Queen's ...

10 CHAPTER 2. CONTROLLED MARKOV CHAINS
2.1.2 Classes of Control Policies
Admissible Control Policies
Let H 0 := X, H t = H t−1 × K for t = 1, 2, . . .. We let h t denote an element of H t , where h t = {x [0,t] , u [0,t−1] }.
A deterministic admissible control policy Π is a sequence of functions {γ t } from H t → U; in this case
u t = γ t (h t ). A randomized control policy is a sequence Π = {Π t , t ≥ 0} such that Π : H t → P(U) (with P(U)
being the space of probability measures on U) such that
Π t (u t ∈ U(x t )|h t ) = 1, ∀h t ∈ H t .
Markov Control Policies
A policy is randomized Markov if
P Π x 0
(u t ∈ C|h t ) = Π t (u t ∈ C|x t ),
C ∈ B(U).
Hence, the control action only depends on the state and the time, and not the past history. If the control strategy
is deterministic, that is if
Π t (u t = f t (x t )|x t ) = 1.
for some function f t , the control policy is said to be deterministic Markov.
Stationary Control Policies
A policy is randomized stationary if
P Π x 0
(u t ∈ C|h t ) = Π(u t ∈ C|x t ),
C ∈ B(U).
Hence, the control action only depends on the state, and not the past history or on time. If the control strategy is
deterministic, that is if Π(u t = f(x t )|x t ) = 1 for some function f, the control policy is said to be deterministic
stationary.
2.2 Markov Chain Induced by a Markov Policy
The following is an important result:
Theorem 2.2.1 Let the control policy be randomized Markov. Then, the controlled Markov chain becomes a
Markov chain in X, that is, the state process itself becomes a Markov chain:
P Π x 0
(x t+1 ∈ B|x t , x t−1 , . . .,x 0 ) = Q Π t (x t+1 ∈ B|x t ),
B ∈ B(X), t ≥ 1, P.a.s.
Proof: Let us consider the case where U is countable. Let B ∈ B(X). It follows that,
Px Π 0
(x t+1 ∈ B|x t , x t−1 , . . . , x 0 )
= ∑ Px Π 0
(x t+1 ∈ B, u t |x t , x t−1 , . . . , x 0 )
u t
= ∑ Px Π 0
(x t+1 ∈ B|u t , x t , x t−1 , . . . , x 0 )Px Π 0
(u t |x t , x t−1 , . . .,x 0 )
u t
= ∑ u t
Q Π x 0
(x t+1 ∈ B|u t , x t )π t (u t |x t )
= ∑ u t
Q Π x 0
(x t+1 ∈ B, u t |x t )
= Q Π t (x t+1 ∈ B|x t ) (2.3)