Methods for Constrained Optimization - MIT Certificate Error

x kFigure 14. Typical user input vectorsBecause each vector is of dimension 16, the problem becomes very complicated. Myimplementation of the quadratic programming method uses the method of Theil and Vande Panne, which iterates through all the different combinations of inequality constraintsto find the active set. This method grows factorially with the number of inequalityconstraints. In this problem, the inequality constraints are 0 ≤ αk≤ Const,k = 1,2, K,pto allow for misclassifications (see [Cortes 1993] for more detail on soft marginclassifiers), and thus there are p × 2 = 32 inequality constraints. The 32 inequalityconstraints can form over a billion different combinations to form the active set ofconstraints. Thus it is infeasible for my implementation of the QP method to performoptimization on this problem.Since the penalty function method is iterative, it does not scale with the number ofconstraints, and thus still performs the optimization in a reasonably short amount of time.The MATLAB implementation of quadratic programming with inequality constraintsactually applies an active set method that uses projections, which is a much leaner way offinding the active constraints. Implementing this more advanced version of the quadraticprogramming method is beyond the scope of this paper, but for future reference, thismethod should be considered for further investigation.At any rate, I performed classification on the 20 handwritten numerals training data usingboth the penalty function optimizer and MATLAB’s QP, and tested the classifier on over700 test data. The performance results are shown in Figure 15. The penalty functionmethod misclassified about 7% of the test data, and MATLAB’s QP misclassified about4.5% of the test data. Again, reducing the penalty function’s convergence criteria mayhelp reduce the misclassification error.