Nonnegativity Constraints in Numerical Analysis - CiteSeer

More documents

Recommendations

Info

$Math 733: Vector Fields, Differential Forms, and Cohomology$

4.1 Nonnegative Matrix FactorizationIn Nonnegative Matrix Factorization (NMF), an m × n (nonnegative) mixed data matrix Xis approximately factored into a product of two nonnegative rank-k matrices, with k smallcompared to m and n, X ≈ WH. This factorization has the advantage that W and H canprovide a physically realizable representation of the mixed data. NMF is widely used in avariety of applications, including air emission control, image and spectral data processing,text mining, chemometric analysis, neural learning processes, sound recognition, remotesensing, and object characterization, see, e.g. [9].NMF problem: Given a nonnegative matrix X ∈ R m×n and a positive integer k ≤min{m, n}, find nonnegative matrices W ∈ R m×k and H ∈ R k×n to minimize the functionf(W, H) = 1‖X − 2 WH‖2 F , i.e.k∑min f(H) = ‖X − W (i) ◦ H (i) ‖ subject to W, H ≥ 0 (15)Hi=1where ′ ◦ ′ denotes outer product, W (i) is ith column of W, H (i) is ith column of H TFigure 1: An illustration of nonnegative matrix factorization.See Figure 1 which provides an illustration of matrix approximation by a sum of rankone matrices determined by W and H. The sum is truncated after k terms.Quite a few numerical algorithms have been developed for solving the NMF. The methodologiesadapted are following more or less the principles of alternating direction iterations,the projected Newton, the reduced quadratic approximation, and the descent search. Specificimplementations generally can be categorized into alternating least squares algorithms[64], multiplicative update algorithms [40, 51, 52], gradient descent algorithms, and hybridalgorithms [67, 69]. Some general assessments of these methods can be found in [20, 56]. Itappears that there is much room for improvement of numerical methods. Although schemesand approaches are different, any numerical method is essentially centered around satisfyingthe first order optimality conditions derived from the Kuhn-Tucker theory. Note that thecomputed factors W and H may only be local minimizers of (15).14
Theorem 1. Necessary conditions for (W, H) ∈ R m×p+ × R p×n+ to solve the nonnegativematrix factorization problem (15) areW. ∗ ((X − W H)H T ) = 0 ∈ R m×p ,H. ∗ (W T (X − W H)) = 0 ∈ R p×n ,(X − W H)H T ≤ 0,W T (X − W H) ≤ 0,(16)where ′ .∗ ′ denotes the Hadamard product.Alternating Least Squares (ALS) algorithms for NMFSince the Frobenius norm of a matrix is just the sum of Euclidean norms over columns(or rows), minimization or descent over either W or H boils down to solving a sequenceof nonnegative least squares (NNLS) problems. In the class of ALS algorithms for NMF, aleast squares step is followed by another least squares step in an alternating fashion, thusgiving rise to the ALS name. ALS algorithms were first used by Paatero [64], exploitingthe fact that, while the optimization problem of (15) is not convex in both W and H, it isconvex in either W or H. Thus, given one matrix, the other matrix can be found with NNLScomputations. An elementary ALS algorithm in matrix notation follows.ALS Algorithm for NMF:Initialization: Let W be a random matrix W = rand(m, k) or use another initializationfrom [50]repeat: for i = 1 : maxiterend1. (NNLS) Solve for H in the matrix equation W T WH = W T X by solvingwith W fixed,minH f(H) = 1 2 ‖X − WH‖2 F subject to H ≥ 0, (17)2. (NNLS) Solve for W in the matrix equation HH T W T = HX T by solvingwith H fixed.minW f(W) = 1 2 ‖XT − H T W T ‖ 2 F subject to W ≥ 0 (18)Compared to other methods for NMF, the ALS algorithms are more flexible, allowingthe iterative process to escape from a poor path. Depending on the implementation, ALS15
Page 1: Nonnegativity Constraints in Numeri
Page 5: NNLS Problem:Given a matrix A ∈ R
Page 8 and 9: However, when applied in a straight
Page 10 and 11: the active set method fnnls. It als
Page 12 and 13: Block principal pivoting algorithm:
Page 16 and 17: algorithms can be very fast. The im
Page 18 and 19: Furthermore, the product T (m×np)
Page 20 and 21: 5.2 Image Processing and Computer V
Page 22 and 23: 5.4 Environmetrics and Chemometrics
Page 24 and 25: Figure 3: Artist rendition of a JSA
Page 26 and 27: References[1] S. Bellavia, M. Macco
Page 28 and 29: [29] V. Franc, V. Hlavč, and M. Na
Page 30 and 31: [60] J. G. Nagy, Z. Strakoš, Enfor
Page 32: [89] R. S. Varga, Matrix Iterative

Nonnegativity Constraints in Numerical Analysis - CiteSeer

Create successful ePaper yourself

Delete template?

Save as template?