Foundations of Data Science

andom walk
graph
vertex
strongly connected
aperiodic
strongly connected
and aperiodic
undirected graph
Markov chain
stochastic process
state
persistent
aperiodic
ergotic
time reversible
Table 5.1: Correspondence between terminology of random walks and Markov chains
Aspects of the theory of random walks were developed in computer science with an
important application in defining the pagerank of pages on the World Wide Web by their
stationary probability. An equivalent concept called a Markov chain had previously been
developed in the statistical literature. A Markov chain has a finite set of states. For each
pair of states x and y, there is a transition probability p xy of going from state x to state y
where for each x, ∑ y p xy = 1. A random walk in the Markov chain starts at some state. At
a given time step, if it is in state x, the next state y is selected randomly with probability
p xy . A Markov chain can be represented by a directed graph with a vertex representing
each state and an edge with weight p xy from vertex x to vertex y. We say that the Markov
chain is connected if the underlying directed graph is strongly connected. That is, if there
is a directed path from every vertex to every other vertex. The matrix P consisting of the
p xy is called the transition probability matrix of the chain. The terms “random walk” and
“Markov chain” are used interchangeably. The correspondence between the terminologies
of random walks and Markov chains is given in Table 5.1.
A state of a Markov chain is persistent if it has the property that should the state ever
be reached, the random process will return to it with probability one. This is equivalent
to the property that the state is in a strongly connected component with no out edges.
For most of the chapter, we assume that the underlying directed graph is strongly connected.
We discuss here briefly what might happen if we do not have strong connectivity.
Consider the directed graph in Figure 5.1b with three strongly connected components,
A, B, and C. Starting from any vertex in A, there is a nonzero probability of eventually
reaching any vertex in A. However, the probability of returning to a vertex in A is less
than one and thus vertices in A, and similarly vertices in B, are not persistent. From
any vertex in C, the walk eventually will return with probability one to the vertex, since
there is no way of leaving component C. Thus, vertices in C are persistent.
A connected Markov Chain is said to be aperiodic if the greatest common divisor of
the lengths of directed cycles is one. It is known that for connected aperiodic chains, the
probability distribution of the random walk converges to a unique stationary distribu-
140