Nonnegativity Constraints in Numerical Analysis - CiteSeer

More documents

Recommendations

Info

$Math 733: Vector Fields, Differential Forms, and Cohomology$

However, when applied in a straightforward manner to large scale NNLS problems, thisalgorithm’s performance is found to be unacceptably slow owing to the need to perform theequivalent of a full pseudo-inverse calculation for each observation vector. More recently,Bro and de Jong [15] have made a substantial speed improvement to Lawson and Hanson’salgorithm for the case of a large number of observation vectors, by developing a modifiedNNLS algorithm.Fast NNLS fnnls :In the paper [15], Bro and de Jong give a modification of the standard algorithm forNNLS by Lawson and Hanson. Their algorithm, called Fast Nonnegative Least Squares,fnnls, is specifically designed for use in multiway decomposition methods for tensor arrayssuch as PARAFAC and N-mode PCA. (See the material on tensors given later in this paper.)They realized that large parts of the pseudoinverse could be computed once but usedrepeatedly. Specifically, their algorithm precomputes the cross-product matrices that appearin the normal equation formulation of the least squares solution. They also observed that,during alternating least squares (ALS) procedures (to be discussed later), solutions tend tochange only slightly from iteration to iteration. In an extension to their NNLS algorithmthat they characterized as being for “advanced users”, they retained information about theprevious iterations solution and were able to extract further performance improvements inALS applications that employ NNLS. These innovations led to a substantial performanceimprovement when analyzing large multivariate, multiway data sets.Algorithm fnnls :Input: A ∈ R m×n , b ∈ R mOutput: x ∗ ≥ 0 such that x ∗ = arg min ‖Ax − b‖ 2 .Initialization: P = ∅, R = {1, 2, · · · , n}, x = 0, w = A T b − (A T A)xrepeat1. Proceed if R ≠ ∅ ∧ [max i∈R (w i ) > tolerance]2. j = arg max i∈R (w i )3. Include the index j in P and remove it from R4. s P = [(A T A) P ] −1 (A T b) P4.1. Proceed if min(s P ) ≤ 04.2. α = − min i∈P [x i /(x i − s i )]4.3. x := x + α(s − x)4.4. Update R and P4.5. s P = [(A T A) P ] −1 (A T b) P4.6. s R = 08
5. x = s6. w = A T (b − Ax)While Bro and de Jongs algorithm precomputes parts of the pseudoinverse, the algorithmstill requires work to complete the pseudoinverse calculation once for each vectorobservation. A recent variant of fnnls, called fast combinatorial NNLS [4], appropriatelyrearranges calculations to achieve further speedups in the presence of multiple observationvectors b i , i = 1, 2 · · · l. This new method rigorously solves the constrained least squaresproblem while exacting essentially no performance penalty as compared with Bro and Jong’salgorithm. The new algorithm employs combinatorial reasoning to identify and group togetherall observations b i that share a common pseudoinverse at each stage in the NNLS iteration.The complete pseudoinverse is then computed just once per group and, subsequently,is applied individually to each observation in the group. As a result, the computationalburden is significantly reduced and the time required to perform ALS operations is likewisereduced. Essentially, if there is only one observation, this new algorithm is no different fromBro and Jong’s algorithm.In the paper [24], Dax concentrates on two problems that arise in the implementationof an active set method. One problem is the choice of a good starting point. The secondproblem is how to move away from a “dead point”. The results of his experiments indicatethat the use of Gauss-Seidel iterations to obtain a starting point is likely to provide largegains in efficiency. And also, dropping one constraint at a time is advantageous to droppingseveral constraints at a time.However, all these active set methods still depend on the normal equations, renderingthem infeasible for ill-conditioned. In contrast to an active set method, iterative methods,for instance gradient projection, enables one to incorporate multiple active constraints ateach iteration.3.2.2 Algorithms Based On Iterative MethodsThe main advantage of this class of algorithms is that by using information from a projectedgradient step along with a good guess of the active set, one can handle multiple activeconstraints per iteration. In contrast, the active-set method typically deals with only oneactive constraint at each iteration. Some of the iterative methods are based on the solutionof the corresponding LCP (8). In contrast to an active set approach, iterative methods likegradient projection enables the incorporation of multiple active constraints at each iteration.Projective Quasi-Newton NNLS (PQN-NNLS)In the paper [46], Kim, et al. proposed a projection method with non-diagonal gradientscaling to solve the NNLS problem (6). In contrast to an active set approach, gradientprojection avoids the pre-computation of A T A and A T b, which is required for the use of9
Page 1: Nonnegativity Constraints in Numeri
Page 5: NNLS Problem:Given a matrix A ∈ R
Page 10 and 11: the active set method fnnls. It als
Page 12 and 13: Block principal pivoting algorithm:
Page 14 and 15: 4.1 Nonnegative Matrix Factorizatio
Page 16 and 17: algorithms can be very fast. The im
Page 18 and 19: Furthermore, the product T (m×np)
Page 20 and 21: 5.2 Image Processing and Computer V
Page 22 and 23: 5.4 Environmetrics and Chemometrics
Page 24 and 25: Figure 3: Artist rendition of a JSA
Page 26 and 27: References[1] S. Bellavia, M. Macco
Page 28 and 29: [29] V. Franc, V. Hlavč, and M. Na
Page 30 and 31: [60] J. G. Nagy, Z. Strakoš, Enfor
Page 32: [89] R. S. Varga, Matrix Iterative

Nonnegativity Constraints in Numerical Analysis - CiteSeer

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?