Foundations of Data Science

This is a linear program.
As we remarked earlier, the equation A = BC will not hold exactly. A more practical
model views A as a matrix of probabilities rather than exact frequencies. In this model,
each document is generated by picking its terms in independent trials. Each trial for
document j picks term 1 with probability a 1j ; term 2 with probability a 2j , etc. We are
not given entire documents; instead we are given s independent trials for each document.
Our job is to find B and C. We do not discuss the details of either the model or the
algorithms. In this new situation, algorithms are known to find B and C when there exist
anchor terms, even with a small number s of trials.
At the heart of such an algorithm is the following problem:
Approximate NMF Given a n×m matrix A and the promise that there is a n×r matrix
B and a r × m matrix C, both with nonnegative entries, such that ||A − BC|| F ≤ ∆, find
B ′ and C ′ of the same dimensions, with nonnegative entries such that ||A − B ′ C ′ || F ≤ ∆ ′ .
Here, ∆ ′ is related to ∆ and if the promise does not hold, the algorithm is allowed to
return any answer.
Now for the case when anchor words exist, this reduces to the problem of finding which
rows of A have the property that no point close to the row is positively dependent on
other rows. It is easy to write the statement that there is a vector y close to a i which is
positively dependent on the other rows as a convex program:
∣ ∑ ∣∣∣∣
∃x 1 , x 2 , . . . , x i−1 , x i+1 , . . . , x n such that
x j a j − a i ≤ ε.
∣
| ∑ j≠i x ja j − a i | is convex function of x j and hence this problem can be solved efficiently.
9.2 Hidden Markov Model
A hidden Markov model, HMM, consists of a finite set of states with a transition
between each pair of states. There is an initial probability distribution α on the states
and a transition probability a ij associated with the transition from state i to state j. Each
state has a probability distribution p(O, i) giving the probability of outputting the symbol
O in state i. A transition consists of two components. A state transition to a new state
followed by the output of a symbol. The HMM starts by selecting a start state according
to the distribution α and outputting a symbol.
Example: An example of a HMM is the graph with two states q and p illustrated below.
j≠i
303