CLC-Conference-Proceeding-2018

As is usual in Mathematics, when dealing
with a non-linear problem, the possibility of
linearizing is analyzed. A very natural idea
emerges that gives rise to the Alternating Least
Squares xxiii . The idea is to initialize one of the
matrices, and this would be related to the third
question of 1.1, and optimize with respect to the
other.
The problem is solved considering the
other matrix as the variable and in the next step
the calculated solution is taken as initialization
for the matrix and optimized with respect to the
first matrix. This is an alternating procedure and
the advantage is that each step is solving a linear
Least Squares problem with inequality
constraints (non negativity) with convex
objective function for which there are methods
such as, for example, Lawson and Hanson xxiv .
The general idea of the ALS is the following xxv :
Initialize the matrix A, A (0) randomly or using
any other strategy.
For k = 1,2, ..., until satisfying a stopping
criterion
Solve
(1)
Solve min‖Y − AX ( k) ‖; A ( k+1)
(2) A≥0
As it is observed, the solution of a non-linear
optimization problem has been replaced by the
solution in each step of two linear problems with
constraints.
Another way to describe this algorithm appears
in xxvi and is as follows:
Initialize A randomly or with any other strategy.
Estimate X of the matrix equation A T AX = A T Y
and solve the problem
A.
with fixed matrix
Assign all the negative coefficients of X the
value 0 or a chosen small positive value ε. 4.
Estimate A of the matrix equation XX T A T =
XY T solving the problem
matrix X.
with fixed
5. Assign to all the negative coefficients of A the
value 0 or a chosen small positive value ε.
This procedure can be rewritten as follows: X ←
max{ε, (A T A) −1 AY} = [A † Y]
+
A ← max{ε, YX T (XX T ) −1 AY} = [YX † ]+
where A † is the Moore-Penrose pseudoinverse
xxvii of A, ε is a small constant (often 10 −6 ) to
force the coefficients to be positive.
In the Alternating Least Squares,
however, the convergence to a global minimum
is not guaranteed, not even to a stationary point,
only to a point at which the objective function
stops decreasing. This procedure can be
improved as can be seen in Chapter 4 of xxviii It is
interesting to notice that the NMF can be
considered as an extension of the Non-negative
Least Squares (NLS), which is a particular case
of the Least Squares with inequality constraints
(LSI) as described below:
Given a matrix A ∈ R IxJ and a data set y ∈ R I ,
find a non-negative vector x ∈ R J that minimizes
the cost function , that is, ,
, subject to x ≥ 0.
In xxix , the problem of Non-negative Least
Squares is formulated as a particular case of
Least Squares with inequality constraints (LSI).
If the matrix A has full rank then we are in the
presence of a strictly convex problem and the