NLPy - gerad

NLPy: Launching Pad forOptimization in PythonD. OrbanGERAD and École Polytechnique, Montréaldominique.orban@gerad.ca

OutlineWriting optimization codes is difficultThe Python programming languageOptimization and PythonNLPy building blocksExamplesOutlook

Writing Optimization CodesNeed a “good” programming languageOften need a modeling languageDifficult because programming details distract fromalgorithmic logicTypically rely on existing building blocksAt best, code is as good as building blocksOnce in place, building blocks difficult to swap

The Way We WereFortran 77 dominated the landscapeMinority of codes written in C

The Way We Just WereCodes in Matlab (which interface with C)More codes in CFortran 77 in decline, Fortran 90/95 risingThe dark shadow of C++Java. Really? No, not really.

The Way We AreFortran 77 codes still widely used but no longerdevelopedFortran 90/95 very activeLots of CLots of C++Lots of MatlabAnd...

The Way We Will BeMixed compiled-interpreted languagesMex is already widely usedPython is overwhelmingly widely used(But surely, Fortran will stick around in scientificprogramming)

What is Python?An open-source interpreted languageObject-oriented if you want it toA real, complete, programming languageEasy to read, learn, program and portable!Large amounts of resources and documentationBatteries included, e.g., Enthought EditionLots of scientific extensions, e.g., Numpy, Scipy, SAGE, PyAMG,SfePy, etc.

Optimization and Pythonminimize f(x) subject to h(x) =0,c(x) ≥ 0x∈R nScipy: Interfaces to several Netlib and other codesPyomo: Modeling language for (N)LP + standard solversOpenOpt: Interfaces to existing codesCVX[OPT/MOD]: Build and solve convex problemsPyIpopt (now using NLPy)

NLPy Design GoalsNeed: Devise numerical methods for NLP, analyzethem, then implement them in the right environmentFor a given class of problems, we normally know whichbuilding blocks are efficientNLPy offers access to the appropriate building blocksfor optimization from Python and lets you concentrateon the logic

NLPy Design GoalsResearchers in optimization vs. Optimization usersSeamlessly access and build upon basic blocksUse from command line or from more elaborate scripts

Building BlocksLine search (csrch, mcsrch)Sparse matrices (pysparse)Krylov and related methods (native)Trust-region subproblem (GLTR, native truncated CG)Cholesky factorization (HSL MA57, ICFS)Perhaps LU but more likely QR (UMFPACK, SuperLU)Symmetric indefinite factorization (HSL MA57)

More Building BlocksSPD Preconditioner (LBFGS, ICFS)Performance profiles (native)Modeling language (AMPL, native but no AD)MINRES (native)LSQR (native)… and more

Problems AdressedLinear ProgramsQuadratic ProgramsLinear and Nonlinear Least-SquaresUnconstrained Nonlinear ProgramsConstrained Nonlinear ProgramsConvex or Non-convex Nonlinear ProgramsDegenerate Problems

NLPy Data FlowSolver written in PythonMore elaborate Python classesPython classesPythonPythonCCCC++ICFSFortranAMPLCFortranand CNumpyPysparseC++ Lib

AmplPyInterface the AMPL Solver LibraryRead data from a given NL file (need AMPL)Represent problem as abstract AmplModel classMembers obj, grad, jac, hess, cons, hprod, ...

from nlpy.model import amplpynlp = amplpy.AmplModel(‘someNlFile’) # Created by AMPLx0 = nlp.x0 # Initial pointpi0 = nlp.pi0 # Initial multipliersprint ‘%d variables, %d constraints’ % (nlp.n, nlp.m)print 'Lower bounds on x: ', nlp.Lvar # Numpy arrayprint 'Upper bounds on x: ', nlp.Uvar # Numpy arrayprint 'f(x0) = ', nlp.obj(x0)print 'grad f(x0) = ', nlp.grad(x0) # Numpy arrayif nlp.m > 0:print 'Lower constraint bounds: ', nlp.Lconprint 'Upper constraint bounds: ', nlp.Uconc = nlp.cons(x0)J = nlp.jac(x0) # Pysparse sparse matrixH = nlp.hess(x0, pi0)# Pysparse sparse matrixprint ' nnzJ = ', J.nnzprint ' nnzH = ', H.nnz

Example: L-BFGSfrom nlpy.model amplpy import AmplModelfrom nlpy.optimize.solvers.lbfgs import LBFGSFrameworkfor ProblemName in [‘a.nl’, ‘b.nl’, ‘c.nl’]:nlp = AmplModel(ProblemName)for m in [1,2,3,4,5,10,15,20]:lbfgs = LBFGSFramework(nlp, npairs=m, scaling=True)lbfgs.solve()nlp.close()Note: nlp need not come from an AMPL model

Augmented SystemsSymmetric indefinite systemsSymmetric quasi-definite systemsAbstract solver: SILSDerived classes: pyMa27Context, pyMa57ContextUse keyword sqd=True for SQD system

# import order not importantfrom nlpy.linalg.pyma27 import PyMa27Contextfrom pysparse.sparse import spmatrixfrom numpy import ones, empty# Create SQD matrix KA = spmatrix.ll_mat_from_mtx('Data/bcsstk18.mtx')C = spmatrix.ll_mat_from_mtx('Data/bcsstk11.mtx')nA = A.shape[0]nC = C.shape[0]K = spmatrix.ll_mat_sym(nA+nC, A.nnz+C.nnz+min(nA,nC))K[:nA,:nA] = AK[nA:,nA:] = -Cfor i in range(min(nA,nC)): K[nA+i,i] = 1.0# Right-hand side rhs=K*ee = ones(nA+nC)rhs = empty(nA+nC)K.matvec(e,rhs) # Also interface to write rhs=K*ones()LBL = PyMa27Context(K, sqd=True)LBL.solve(rhs, get_resid=False) # Solution is in LBL.x

Projected Krylov MethodsSolve augmented systems iterativelyFactorize projection matrixAbstract class: ProjectedKrylovDerived classes: ProjectedCG, ProjectedBCGSTAB,ProjectedTFQMR [Orban, 2008]ProjectedCG used for optimization, Stokes problemsOther two can be used in PDE solvers [Dufour, Orban, 2008]

Projected CGminimizex∈R n c T x + 1 2 xT Hx subject to Ax = bfrom nlpy.krylov.ppcg import ProjectedCGPCG = ProjectedCG(c, A=Jac, rhs=b, H=Hess)# or PCG = ProjectedCG(c, A=Jac, rhs=b, matvec=lambda p: H*p)PCG.Solve() # Solution is in PCG.xminimizex∈R n c T x + 1 2 xT Hx subject to Ax =0, ‖x‖ 2 ≤ ∆from nlpy.krylov.ppcg import ProjectedCGPCG = ProjectedCG(c, A=Jac, H=Hess, radius=10)PCG.Solve() # Solution is in PCG.x

Projected BiCGSTAB[H ATA 0][ up]=[ qr]Here, (1,1) block need not be symmetric.Usage is identical as Projected CG.Only matrix-vector products with (1,1) block

Complete Code: TrunkUnconstrained minimization, matrix freeQuadratic trust-region subproblemsNocedal-Yuan linesearchEssentially 13 lines of code

Complete Code: PsuperbSolve degenerate problems with mixed interiorexteriorpenalty and elastic variables[Gould, Orban, Toint, 2003]Well suited to MPECs, MPVCs or unstructureddegenerate problems[Coulibaly, Orban, 2008], [Curatolo, Orban, 2008]Uses ProjectedCG, PyMa57, AmplPy, among others

Performance ProfilesCommand line or OO interface

How Can YOU Use NLPy?Use it in your researchHave your students use itWith a programming effort comparable to that ofMatlab (and getting better), get an efficient sparse codeFor free!Use it in the classroomTell your friends, students, colleagues about it

What’s Missing?Lots of documentationTrue open-source linear algebra kernelsInterfaces to more linear algebra tools (SuiteSparse?)Automatic differentiationBound-constrained linear least-squares

Thank you!http://nlpy.sf.netdominique.orban@gerad.ca