Constraint Reduction for Linear Programs with Many Constraints

Constraint Reduction for Linear Programs with
Many Constraints
Work supported by the Department of Energy,
under Grant DEFG0204ER25655.
P.I.s: A.L. Tits and D.P. O’Leary (Univ. of Maryland)
Collaborators:
P.A. Absil (Cambridge University, UK)
W. Woessner (Univ. of Maryland)
S. Nicholls (Univ. of Maryland)
Presented at Sandia, Livermore, 29 Sep 2005.
1

Consider the following linear program in dual standard form
max b T y subject to A T y ≤ c, (1)
where A is m × n. Suppose n ≫ m.
Observations:
• Interior-Point (IP) methods use search directions whose computation involves
all constraints at every iteration.
• Normally, only a few constraints (no more than m under nondegeneracy
assumptions) are active at the solution. The others are “redundant”.
b
Objective:
• Compute IP search direction based on a reduced Newton-KKT system,
by adaptively selecting a small subset of critical columns of A.
Hope:
– Significantly reduced cost per iteration.
– No drastic increase in the number of iterations.
– Preserve theoretical convergence properties.
2

Outline
1. Background
• Some related work
• Notation
• Primal-dual framework
• Operation count – Reduced Newton-KKT system
2. Reduced, Dual-Feasible PD Affine Scaling: µ = 0
• Algorithm statement
• Observations
• Numerical experiments: Matlab
• Convergence properties
3. Reduced Mehrotra-Predictor-Corrector
• Algorithm statement
• Numerical experiments: Matlab
4. Invariance under Transformations?
5. Concluding Remarks
3

Background
Some related work
• “Indicators” (to identify early the zero components of primal solution
x ∗ ): El-Bakry et al.[1994], Facchinei et al.[2000].
• “Column generation”, “build-up”, “build-down”: Ye [1992], den Hertog
et al.[1992, 1994, 1995], Goffin et al.[1994], Ye [1997], Luo et al.[1999].
– Focus is on complexity analysis;
– Good numerical results on discretized semi-infinite programming
problems; but typically, many more than m columns of A are retained.
Notation
n := {1, . . .,n},
A = [a 1 , . . .,a n ], e = [1, ..., 1] T ,
Given Q ⊆ n,
A Q := col[a i : i ∈ Q],
X Q := diag(x i : i ∈ Q),
x Q := [x i : i ∈ Q] T
S Q := diag(s i : i ∈ Q). s Q := [s i : i ∈ Q] T
4

Background (cont’d)
Primal-dual framework
Primal-dual LP pair in standard form:
min c T x subject to Ax = b, x ≥ 0
max b T y subject to A T y + s = c, s ≥ 0. (2)
Perturbed (µ ≥ 0) KKT conditions of optimality:
A T y + s − c = 0 (3)
Ax − b = 0 (4)
XSe = µe (5)
x, s ≥ 0, (6)
Given µ ≥ 0, µ-perturbed Newton-KKT system:
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
0 A T I ∆x −r c
⎢
⎣A 0 0 ⎥ ⎢
⎦ ⎣∆y⎥
⎦ = ⎢
⎣ −r b
⎥
⎦
S 0 X ∆s −Xs + µe,
with r b := Ax − b (primal residue) and r c := A T y + s − c (dual residue).
5

Background (cont’d)
Primal-dual framework (cont’d)
Equivalently, (∆x, ∆y, ∆s) satisfy the normal equations:
AS −1 XA T ∆y = −r b + A(−S −1 Xr c + x − µS −1 e)
∆s = −A T ∆y − r c
∆x = −x + µS −1 e − S −1 X∆s
Simple interior-point iteration: Given x > 0, s > 0, y,
• Select a value for µ (≥ 0);
• Solve Newton-KKT system for ∆x, ∆y, ∆s;
• Set x + := x + α P ∆x > 0, s + := s + α D ∆s > 0, y + := y + α D ∆y with
appropriate α P , α D (possibly forced to be equal).
Note: If (y, s) is dual feasible, then (y + , s + ) also is.
6

Background (cont’d)
Operation count – Reduced Newton-KKT system
• Operation count (for a dense problem):
- Forming H := AS −1 XA T : m 2 n;
- Forming v := r b + A(x − S −1 (Xr c + µe)); 2mn;
- Solving H∆y = v: m 3 /3 (Cholesky);
- Computing ∆s = −A T ∆y − r c : 2mn;
- Computing ∆x = −x + S −1 (−X∆s + µe): 2n;
• Benefit in replacing A with A Q : n replaced with |Q|
• Assume n ≫ m and m ≫ 1. Then main gain can be achieved in line 1,
i.e., by merely replacing H with H Q := A Q S −1
Q X QA T Q , and leaving the
rest unchanged. This is done in the sequel.
∗ Key question: How to select Q for
– Significantly reducing the work per iteration (|Q| small);
– Avoiding a dramatic increase in number of iterations;
– Preserving theoretical convergence properties.
7

Reduced, Dual-Feasible PD Affine Scaling: µ = 0
Algorithm statement
Iteration rPDAS.
Parameters. β ∈ (0, 1), x max > 0, x > 0, M ≥ m.
Data. y, with A T y < c; s := c − A T y (> 0) (i.e., r c = 0); x > 0;
Q ⊆ n, including the indices of M smallest entries of s.
Step 1. Compute search direction.
Step 2. Updates.
Solve
and compute
A Q S −1
Q X QA T Q ∆y = b
∆s ← −A T ∆y
˜x ← −S −1 X∆s
(i) Primal update (appropriately clip the components of ˜x). Set
x + i ← min{max{min{‖∆y‖ 2 + ‖˜x − ‖ 2 , x}, ˜x i }, x max }, ∀i ∈ n, (8)
where (˜x − ) i := min{˜x i , 0}.
(ii) Dual update (step along ∆y). Set
⎧
⎨
∞ if ∆s i ≥ 0 ∀i ∈ n,
t D ←
⎩ min{(−s i /∆s i ) : ∆s i < 0, i ∈ n} otherwise.
(9)
Set ˆt D ← min{max{βt D , t D −‖∆y‖}, 1}.
Set y + ← y + ˆt D ∆y, s + ← s + ˆt D ∆s. (So r c remains at 0.)
8

Reduced, Dual-Feasible PD Affine Scaling: µ = 0
Observations
• (∆x Q , ∆y, ∆s Q ) constructed by iteration rPDAS, also satisfy
∆s Q = −A T Q ∆y
∆x Q = −x Q − S −1
Q X Q∆s Q ,
(10a)
(10b)
i.e., they satisfy the full set of normal equations associated with the
constraint-reduced system. Equivalently, they satisfy the Newton system
(with µ = 0 and r c = 0)
⎡ ⎤⎡
⎤ ⎡ ⎤
0 A T Q I ∆x Q 0
⎢
⎣A Q 0 0 ⎥⎢
⎦⎣
∆y ⎥
⎦ = ⎢
⎣b − A Q x Q
⎥
⎦ .
S Q 0 X Q ∆s Q −X Q s Q
(This is a key ingredient to the local convergence analysis.)
• Update rule (8), in particular the lower bound ‖∆y‖ 2 + ‖˜x − ‖ 2 , is instrumental
to our convergence analysis. (As step along ∆x would not always
allow such lower bound.)
9

Reduced, Dual-Feasible PD Affine Scaling: µ = 0
Numerical experiments: Matlab
Heuristic used for Q: For given M ≥ m,
Q = indexes of M smallest components of s.
Parameter value: β = .99, x = 10 −5 , x max = 10 15 .
Selection of x 0 : Based on Mehrotra’s [SIOPT, 1992] scheme.
Test problems (with dual-feasible initial point)
• Polytopic approximation of unit sphere (Semi-infinite problem)
– entries of b ∼ N(0, 1);
– columns of A uniformly distributed on the unit sphere;
– c = e, y 0 = 0;
– m = 50, n = 20, 000.
• “Fully random” problem
– entries of A and b ∼ N(0, 1);
– components of y 0 and s 0 uniformly distributed on (0,1);
– c = A T y 0 + s 0 to ensure dual feasibility;
– m = 50, n = 20, 000.
• SCSD1 (m = 77, n = 760) and SCSD6 (m = 147, n = 1350) from
Netlib.
10

Reduced, Dual-Feasible PD Affine Scaling: µ = 0
Numerical experiments: Matlab (cont’d)
The points on the plots correspond to different runs of Algorithm rPDAS
on the same problem. The runs only differ by the number M of constraints
that are retained in Q; this information is indicated on the horizontal axis
in relative value. The rightmost point thus corresponds to the experiment
without constraint reduction, while the points on the far left correspond to
the most drastic constraint reduction.
Observations:
• In most cases, surprisingly, the number of iterations does NOT increase
as M is reduced. Thus any gain in CPU per iteration directly translates
into the same relative gain in overall CPU.
• Displayed CPU values are purely indicative. Indeed, they will strongly depend
on the implementation (in particular, how the product A Q S −1
Q X QA T Q
is computed), and on the possible sparsity of the data.
• The algorithm sometimes fails for small |Q|. This is due to A Q losing
rank, and accordingly A Q SQ −1X
QA T Q
becoming singular. (Note that this
will almost surely not happen when A is generated randomly.) Schemes
to bypass this difficulty are being investigated.
11

80
Number of iterations
60
40
20
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
8
6
4
2
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rPDAS on Polytopic approximation of unit sphere; m = 50, n = 20, 000.
100
Number of iterations
80
60
40
20
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
15
10
5
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rPDAS on “Fully random”; m = 50, n = 20, 000.
12

40
Number of iterations
30
20
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
0.2
0.15
0.1
0.05
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rPDAS on SCSD1: m = 77, n = 760.
80
Number of iterations
60
40
20
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
0.5
0.4
0.3
0.2
0.1
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rPDAS on SCSD6: m = 147, n = 1350.
13

Reduced, Dual-Feasible PD Affine Scaling: µ = 0
Convergence properties
Let F := {y : A T y ≤ c}. For y ∈ F, let I(y) := {i : a T i y = c i}.
Assumption 1. All m × M submatrices of A have full (row) rank.
Assumption 2. The dual (y) solution set is nonempty and bounded.
Assumption 3. For all y ∈ F, the set {a i : i ∈ I(y)} is linear independent.
Theorem. {y k } converges to the dual solution set.
Assumption 4. The dual solution set is a singleton, say, {y ∗ }, and the
associated KKT multiplier x ∗ satisfies x ∗ i < x max for all i.
Theorem. {(x k , y k )} converges to (x ∗ , y ∗ ) Q-quadratically.
• The global convergence analysis focuses on the monotone increase of the
dual objective function b T y.
• The lower bound ‖∆y‖ 2 + ‖˜x − ‖ 2 in the primal update formula (8) is essential
as it keeps the Newton-KKT matrix away from singularity as long
as KKT points are not approached. (A step along the primal direction
∆x would not allow for this.)
14

Reduced Mehrotra-Predictor-Corrector
Algorithm statement
Iteration rMPC.
Parameters. β ∈ (0, 1), M ≥ m.
Data. y ∈ R m , s > 0, x > 0; µ := x T s/n;
Q ⊆ n, including the indices of M leftmost components of
c − A T y.
Step 1. Compute affine scaling step.
Solve
A Q S −1
Q X QA T Q ∆y = −r b + A(−S −1 Xr c + x)
and compute
∆s ← −A T ∆y − r c
∆x ← −x − S −1 X∆s
t aff
P ← arg max{t ∈ [0, 1] | x + t∆x ≥ 0}
t aff
D ← arg max{t ∈ [0, 1] | x + t∆s ≥ 0}
Step 2. Compute centering parameter.
µ aff ← (x + t aff
P ∆x)T (s + t aff
D ∆s)/n
σ ← (µ aff /µ) 3
Step 3. Compute centering/corrector direction.
Solve A Q S −1
Q X QA T Q∆y cc = −AS −1 (σµe − ∆X∆s)
and compute ∆s cc ← −A T ∆y cc
∆x cc ← S −1 (σµe − ∆X∆s) − S −1 X∆s cc
15

Step 4. Compute MPC step.
∆x mpc ← ∆x + ∆x cc
∆y mpc ← ∆y + ∆y cc
∆s mpc ← ∆s + ∆s cc
t max
P ← arg max{t ∈ [0, 1] | x + t∆x mpc ≥ 0}
t max
D ← arg max{t ∈ [0, 1] | s + t∆s mpc ≥ 0}
t P
← min{βt max
P , 1}
t D ← min{βt max
D , 1}
Step 5. Updates.
x + ← x + t P ∆x mpc
y + ← y + t D ∆y mpc
s + ← s + t D ∆s mpc
Numerical experiments with dual-feasible initial point
Algorithm rMPC was run on the same problems as rPDAS, with the same
(dual-feasible) initial points. The results are reported in the next few slides.
16

50
Number of iterations
40
30
20
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
10
8
6
4
2
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on Polytopic approx. of unit sphere, w/ dual-feasible init. pt; m = 50, n = 20, 000.
40
Number of iterations
30
20
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
8
6
4
2
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on “Fully random”, with dual-feasible initial point; m = 50, n = 20, 000.
17

Number of iterations
20
15
10
5
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
0.2
0.15
0.1
0.05
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on SCSD1, with dual-feasible initial point: m = 77, n = 760.
Number of iterations
35
30
25
20
15
10
5
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
0.4
0.3
0.2
0.1
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on SCSD6, with dual-feasible initial point: m = 147, n = 1350.
18

Reduced Mehrotra-Predictor-Corrector (cont’d)
Numerical experiments with infeasible initial point
The next few slides report results obtained on the same problem, but with
the (usually infeasible) initial point as recommended by Mehrotra [SIOPT,
1992].
19

Number of iterations
70
60
50
40
30
20
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
8
6
4
2
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on Polytopic approx. of unit sphere, w/ infeasible init. pt; m = 50, n = 20, 000.
80
Number of iterations
60
40
20
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
15
10
5
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on “Fully random”, with infeasible initial point: m = 50, n = 20, 000.
20

40
Number of iterations
30
20
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
0.25
0.2
0.15
0.1
0.05
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on SCSD1, with infeasible initial point: m = 77, n = 760.
50
Number of iterations
40
30
20
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
CPU time (sec.)
0.8
0.6
0.4
0.2
total
form H Q
solves
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
|Q|/n
rMPC on SCSD6, with infeasible initial point: m = 147, n = 1350.
21

Invariance under Transformations?
Let P ∈ R m×m , nonsingular; v ∈ R m ; R ∈ R n×n > 0, diagonal. 1
Consider the transformation
x ← R −1 x; s ← Rs; y ← P −T y + v.
(Also, A ← PAR; b ← Pb; c ← Rc + RA T P T v.)
• MPC: (P, v, R)-invariant.
• rMPC: (P, v)-invariant.
– R-invariance is affected by the rule for selection of Q.
– It can be recovered by substituting (c i −a T i y)/(c i−a T i y0 ) for (c i −a T i y).
• PDAS: (P, v)-invariant with orthogonal P, i.e., invariant under Euclidean
transformations of the dual space. (Not general P: due to use of
‖∆y‖. Not R: due to use of ‖˜x − ‖, x, and x max .)
– Nonsingular diagonal P can be allowed by replacing ‖∆y‖ with
‖(∆Y 0 ) −1 ∆y‖;.
– Scalar×orthogonal P can be allowed by replacing ‖∆y‖ with ‖∆y‖/‖∆y 0 ‖.
– R-invariance recovered by replacing ‖˜x − ‖ with ‖(∆X 0 ) −1˜x − ‖, and
similarly for the fixed bounds.
• rPDAS: R-invariance can be recovered as in rMPC.
1 R must be thus restricted in order for x ≥ 0 to be preserved.
22

Concluding Remarks
“Reduced” version of an primal-dual affine scaling algorithm (rPDAS) and
of Mehrotra’s predictor-corrector algorithm (rMPC) were proposed.
• When n ≫ m and m ≫ 1, for both rPDAS and rMPC, major reduction
in CPU per iteration can be achieved.
• Under nondegeneracy assumptions, rPDAS is proved to converge quadratically
in the primal-dual space; a convergence proof for rMPC is lacking
at this time.
• Numerical experiments show that
– The number of iterations to convergence remains essentially constant
as |Q| decreases, down to a small multiple of m.
– On some problems (e.g., SCSD1, SCSD6), when |Q| is reduced below
a certain value, the algorithm fails due to A Q losing rank. Schemes
to bypass this difficulty are being investigated.
This presentation can be downloaded from
http://www.isr.umd.edu/~andre/sandia050929.pdf
and the full paper and Matlab script from
http://www.csit.fsu.edu/~absil/Publi/reducedLP.htm
23