The Parallel Solver Packages MUMPS, PaStiX & SuperLU

It’s Not A Disease:The Parallel Solver PackagesMUMPS, PaStiX & SuperLUA. WindischPhD Seminar: High Performance Computing IIG. HaaseMarch 29 th , 2012, Graz

Outline1 MUMPS2PaStiX3SuperLU4Summary and OutlookAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

MUMPSMUltifrontal Massively Parallel sparse directSolverAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Some historical facts...1996 – 1999(and others)> 1999Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Getting MUMPS is easy (it’s PD)Linkhttp://mumps.enseeiht.frDebian, Ubuntu$ sudo apt-get installlibmumps-4.9.2Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Using MUMPSInterfacesMUMPS (FORTRAN90):1C2MATLAB3Octave4ScilabAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

MUMPS: Relevant literatureP. R. Amestoy, I. S. Duff, J. Koster and J. Y. L’Excellent,SIAM Journal on Matrix Analysis and Applications 23(2001) 15-41P. R. Amestoy, A. Guermouche, J. Y. L’Excellent andS. Pralet, Parallel Computing 32 (2006) 136-156I. S. Duff and J. K. Reid, ACM Transactions onMathematical Software 9 (1983) 302-305I. S. Duff, A. M. Erisman and J. K. Reid, Oxford UniversityPress, London (1986)J. W. H.Liu, SIAM Review 34 (1992) 82-109Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

So, what is MUMPS, and how does itwork?SolvesAx = bDirect Solver based onMultifrontal ApproachA square sparse matrix1unsymmetric2symmetric positivedefinite3general symmetricFactorizationA = LUSymmetric AA = LDL TAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

So, what is MUMPS, and how does itwork?SolvesAx = bDirect Solver based onMultifrontal ApproachA square sparse matrix1unsymmetric2symmetric positivedefinite3general symmetricFactorizationA = LUSymmetric AA = LDL TAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

A = ∑ l A[l]a ij = a ij + a [l]ijAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

A = ∑ l A[l]a ij = a ij + a [l]ijAssemblyAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

A = ∑ l A[l]a ij = a ij + a [l]ijAssemblyFully summedAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

a (k+1)ij= a (k)ij− a (k)ika(k) kk−1a(k)kjGEAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

a (k+1)ij= a (k)ij− a (k)ika(k) kk−1a(k)kjAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Assemble A, B4 8 1 5 • • • • • •4 × × × ×8 × × ×1 × × ×5 × × × ×••••••Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Assemble A, BEliminate 44 8 1 5 • • • • • •4 u u u u8 l × ×1 l × ×5 l × × ×••••••Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Assemble A, BEliminate 4Assemble C4 8 1 5 2 • • • • •4 u u u u8 l × × ×1 l × × × ×5 l × × × ×2 × × ×•••••Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Assemble A, BEliminate 4Assemble CPermute 1 ↔ 84 1 8 5 2 • • • • •4 u u u u1 l × × × ×8 l × × ×5 l × × × ×2 × × ×•••••Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Assemble A, BEliminate 4Assemble CPermute 1 ↔ 8Eliminate 14 1 8 5 2 • • • • •4 u u u u1 l u u u u8 l l × × 5 l l × × ×2 l × ×•••••Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

4 1 8 5 2 • • • • •4 u u u u1 l u u u u8 l l × × 5 l l × × ×2 l × ×•••••Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Frontal((· · · ((A [1] + A [2] ) + A [3] ) +A [4] ) + · · · )Multifrontal((A [1] + A [2] ) + (A [3] + A [4] ) +(A [5] + A [6] ) + (A [7] + A [8] ))Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

How MUMPS solves a problemAnalysis1 PreprocessingFactorization2 Ordering3 Symbolic factorizationSolutionAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

How MUMPS solves a problemAnalysisFactorizationSolution1 Elimination tree→ nodes2 Numerical factorization: frontalmatrices3 Factor matrices distributedAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

How MUMPS solves a problemAnalysis1 LUx = b, LDL T x = bFactorization2 Forward: Ly = b or LDy = b3 Backward: Ux = y or L T x = ySolutionAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

MUMPSFurthermore...➣ Interfaces to PORD, SCOTCH,METIS➣ Parallel version requires MPI,BLAS, BLACS and ScaLAPACK➣ Error analysis➣ Detection of null-pivots➣ Schur complementAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

PaStiXParallel Sparse matriX packageAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Getting PaStiXLinkhttp://pastix.gforge.inria.frAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Depending onsymmetry:➣ A = LL T➣ A = LDL TPaStiX Steps1 Reordering to reducefill-in2 Symbolic factorization3 Distribute matrix blocksto processors4 Decomposition of A5 Solve system6 Refine solution (staticpivoting)Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

1. Ordering➣ SCOTCH (or METIS)➣ Halo ApproximateMinimum Degree➣ Tree representsdependencies2. Symbolic factorization➣ Structure of factorizedmatrix from A➣ Cheap step➣ # of off-diag blocks3. Distribution➣ Partitioning: large blocksdistributed to severalprocessorsProcessor candidates:local communication➣ Distribution: blocks to nodesUse elimination treeScheduling comm.&comp.➣ Levels of ParallelismCoarse: independendparts of treeMedium: block decomp.Fine: BLAS3Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

4. Factorization➣ Calculate LL T or LDL T➣ multi-frontal vs.super-nodal➣ PaStiX: super-nodal(left-looking)5. Solve➣ distribution kept➣ cheap6. Refinement (opt)➣ GMRES by Y. Saad➣ iterative refinement➣ conjugate gradientAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

SuperLUAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Getting SuperLULinkhttp://crd-legacy.lbl.gov/ xiaoye/SuperLU/Debian, Ubuntu$ sudo apt-get install libsuperlu3Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Three packages1 Sequential SuperLUSequential processorsOne or more layers of memory2 Multithreaded SuperLU (SuperLU_MT)Shared memory multiprocessor (SMPs)Can use 16-32 parallel processors3 Distributed SuperLU (SuperLU_DIST)Distributed memory parallel processorsMPICan use hundreds of processorsAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

SummaryAndreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

MUMPS➣ Symbolic Factorization➣ Distribution➣ Factorization throughmultifrontal methodPaStiX➣ Symbolic Factorization➣ Distribution➣ Factorization throughsupernodal methodSuperLU➣ To be investigated...Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU

Thank You ForYour Attention!Andreas Windisch, University of GrazMUMPS,PaStiX & SuperLU