Mathematics in Independent Component Analysis

Chapter 14. Proc. ICASSP 2006 193
S n−1
p
∇�.� p p
y
t
∇�.−x� 2
2
Fig. 1. Constrained gradient t on the p-sphere, given by the projection
of the unconstrained gradient∇�.−x� 2
2 onto the tangent space
that is orthogonal to∇�.� p p, see equation (6).
2.2. Projection onto a p-sphere
Let S n−1
p := {x ∈ Rn |�x�p = 1} denote the (n−1)-dimensional
sphere with respect to the p-norm (p>0). A scaled version of this
unit sphere is given by cS n−1
p :={x∈R n |�x�p= c}. The spheres
are smoothC 1-submanifolds ofR n for p≥2. For p0.
p
Now, in the case p=2, the projection is simply given by
πS n−1 (x)=x/�x�2. (2)
2
In the case p=1, the sphere consists of a union of hyperplanes
being orthogonal to (±1,...,±1). Considering only the first quadrant
(i.e. xi> 0), this means thatπ S n−1 (x) is given by the projection onto
1
the hyperplane H :={x∈R n |〈x, e〉=n −1/2 } and setting resulting
negative coordinates to 0; here e := n−1/2 (1,...,1). So with x+ := x
if x≥0and 0 otherwise, we get
πS n−1 (x)=
1
� x+(n −1/2 −〈x, e〉)e �
+ . (3)
In the case of arbitrary p>0, the projection is given by the unique
solution of
�x−y� 2
2 . (4)
π S n−1
p
(x)=argmin y∈S n−1
p
Unfortunately, no closed-form solution exists in the general case,
so we have to numerically determine the solution. We have experimented
with a) explicit Lagrange multiplier calculation and minimization,
b) constrained gradient descent and c) constrained fixedpoint
algorithm (best). Ignoring the singular points at the coordinate
hyperplanes, let us first assume that all xi > 0. Then at
a regular solution y of equation (4), the gradient of the function
to be minimized is parallel to the gradient of the constraint, i.e.
y−x=λ∇�.� p �
�
p�
for some Lagrange-multiplierλ∈R, which can be
y
calculated from the additional constraint equation�y� p p= 1. Using
the notation y⊙p := (y p
1 ,...,yp n) ⊤ for the componentwise exponentiation,
we therefore get
y−x=λp y ⊙(p−1)
�
and y p
i = 1 (5)
i
Algorithm 1: Projection onto S n−1
p by constrained gradient descent.
Commonly, the iteration is stopped after the update stepsize
lies below some given threshold.
Input: vector x∈R n , learning rateη(i)
Output: Euclidean projection y=π S n−1 (x)
p
Initialize y∈S n−1
1
p randomly.
for i←1, 2,... do
df← y−x, dg← p sgn(y)|y| ⊙(p−1)
2
t←df− df ⊤ dgdg/(dg ⊤ 3
dg)
4 y←y−η(i)t
5 y←y/�y�p
end
For p�{1, 2}, these equations cannot be solved in closed form,
hence we propose an alternative approach to solving the constrained
minimization (4). The goal is to minimize f (y) :=�y−x� 2
2 under the
constraint g(y) :=�y� p p= 1. This can for example be achieved by
gradient-descent methods, taking into account that the gradient has
to be calculated on the submanifold given by the S n−1
p -constraint, see
figure 1 for an illustration. The projection of the gradient∇ f onto
the tangent space of S n−1
p at y can be easily calculated as
t=∇ f−〈∇ f,∇g〉∇g/�∇g� 2
2 . (6)
Here, the explicit gradients are given by∇ f (y)=y−x and∇g(y)=
p sgn(y)|y| ⊙(p−1) , where sgn(y) denotes the vector of the componentwise
signs of y, and|y| := sgn(y)y the componentwise absolute
value. The projection algorithm is summarized in algorithm 1. Iteratively,
after calculating the constrained gradient (lines 2 and 3), it
performs a gradient-descent update step (line 4) followed by a pro-
jection onto S n−1
p (line 5) to guarantee that the algorithm stays on the
submanifold.
The method performs well, however as most gradient-descentbased
algorithms, without further optimization it takes quite a few iterations
to achieve acceptable convergence, and the choice of an optimal
learning rateη(i) is non-trivial. We therefore propose a second
projection method employing a fixed-point optimization strategy. Its
idea is based on the fact that at local minima y of f (y) on S n−1
p , the
gradient∇ f (y) is orthogonal to S n−1
p , so∇ f (y)∝∇g(y). Ignoring
signs for illustrative purposes, this means that y−x∝ p y⊙(p−1) , so
y can be calculated from the fixed-point iteration y←λp y⊙(p−1) + x
with additional normalization. Indeed, this can be equivalently derived
from the previous Lagrange equations (5), also yielding equations
for the proportionality factorλ: we can simply determine it
from one component of equation (5), or to increase numerical robustness,
as mean from the total set. Taking into account the signs
of the gradient (which we ignored in equation (5)), this yields an
estimate ˆλ := 1�
n
i=1(yi−xi)/(p sgn(yi)|yi| p−1 ). Altogether, we get
n
the fixed-point algorithm 2, which in experiments turns out to have
a considerably higher convergence rate than algorithm 1.
In table 1, we compare the algorithms 1 and 2, namely with respect
to the number of iterations they need to achieve convergence
below some given threshold. As expected, the fixed-point algorithm
outperforms gradient-descent always except for the case of higher
dimensions and p>2 (non-sparse case). In the following we will
therefore use the fixed-point algorithm for projection onto S n−1
1 .
2.3. Projection onto convex sets
If M is a convex set, then the Euclidean projectionπM(x) for any
x∈R n is already unique, soX(M)=∅and the operatorπM is called