Lecture Notes - Department of Mathematics and Statistics - Queen's ...

12 CHAPTER 2. CONTROLLED MARKOV CHAINS
is independent of the initial distribution (or initial condition) on x 0 .
The last two results are computationally very important, as there are powerful computational algorithms that
allow one to develop such stationary policies.
In the following set of notes, we will first consider further properties of Markov chains, since under a Markov
control policy, the controlled state becomes a Markov chain. We will then get back to the Controlled Markov
Chains and the development of optimal control policies.
The classification of Markov Chains in the next topic will implicitly characterize the set of problems for which
Stationary policies contain optimal admissible policies.
2.3.1 Partially Observed Model
Consider the following model.
x t+1 = f(x t , u t , w t ), y t = g(x t , v t )
Here, as before, x t is the state, u t ∈ U is the control, (w t , v t ) ∈ W × V are second order, zero-mean, i.i.d noise
processes and w t is independent of v t . In addition to the previous fully observed model, y t denotes an observation
variable taking values in Y, a subset of R n in the context of this review. The controller only has causal access to
the second component {y t } of the process. An admissible policy {Π} is measurable with respect to σ({y s , s ≤ t}).
We denote the observed history space as: H 0 := P, H t = H t−1 × Y × U. Hence, the set of (wide-sense) causal
control policies are such that P(u(h t ) ∈ U|h t ) = 1 ∀h t ∈ H t .
One could transform a partially observable Markov Decision Problem to a Fully Observed Markov Decision
Problem via an enlargement of the state space. In particular, we obtain via the properties of total probability
the following dynamical recursion
π t (A) : = P(x t ∈ A|y [0,t] , u [0,t−1] )
∫
X
=
π t−1(dx t−1 )r(y t |x t )P(dx t |x t−1 , u t−1 )
∫ ∫X π t−1(dx t−1 )r(y t |x t )P(dx t |x t−1 , u t−1 ) ,
X
where we assume that ∫ B r(y|x)dy = P(y t ∈ B|x t = x) for any B ∈ B(Y) and r denotes the density process.
The conditional measure process becomes a controlled Markov chain in P(X), which we endow with the weak
convergence topology.
Theorem 2.3.1 The process {π t , u t } is a controlled Markov chain. That is, under any admissible control policy,
given the action at time t ≥ 0 and π t , π t+1 is conditionally independent from {π s , u s , s ≤ t − 1}.
Let the cost function to be minimized be
T∑
−1
t=0
E Π x 0
[c(x t , u t )],
where E Π x 0
[] denotes the expectation over all sample paths with initial state given by x 0 under policy Π.
We ∫ transform the system into a fully observed Markov model as follows. Define the new cost as ˜c(π, u) =
X
c(x, u)π(dx), π ∈ P(X). The stochastic transition kernel q is given by:
∫
q(dx, dy|π, u) = P(dx, dy|x ′ , u)π(dx ′ ), π ∈ P(X)
And, this kernel can be decomposed as q(dx, dy|π, u) = P(dy|π, u)P(dx|π, u, y).
X
The second term here is the filtering equation, mapping (π, u, y) ∈ (P(X) × U × Y) to P(X). It follows that
(P(X), U, K, ˜c) defines a completely observable controlled Markov process. Here, we have
∫
K(B|π, u) = 1 (P(.|π,u,y)∈B) P(dy|π, u), ∀B ∈ B(P(X)),
with 1 (.) denoting the indicator function.
Y