Relax and Randomize: From Value to Algorithms

Lemma 13. Consider the case when X is the l N ∞ ball and F is the l N 1 unit ball. Let f ∗ =
argmin ⟨f, R⟩, then for any random vector R,
f∈F
E
R
[sup
x∈X
Proof. Let f ∗ = argmin
f∈F
Hence note that
f ′ ∈F x∈X
{⟨f ∗ , x⟩ + ∥R + x∥ ∞
}] ≤ E [inf
R f∈F
sup
x∈X
sup {⟨f, x⟩ + ∥R + x∥ ∞
}] + 4 P (∥R∥ ∞
≤ 4)
x
⟨f, R⟩. We start by noting that for any f ′ ∈ F,
{⟨f ′ , x⟩ + ∥R + x∥ ∞
} = sup
x∈X
f ′ ∈F f∈F
= sup
f∈F
= sup
f∈F
{⟨f ′ , x⟩ + sup ⟨f, R + x⟩}
f∈F
sup {⟨f ′ , x⟩ + ⟨f, R + x⟩}
x∈X
{sup ⟨f ′ + f, x⟩ + ⟨f, R⟩}
x∈X
= sup {∥f ′ + f∥ 1
+ ⟨f, R⟩}
f∈F
inf sup {⟨f ′ , x⟩ + ∥R + x∥ ∞
} = inf sup {∥f ′ + f∥ 1
+ ⟨f, R⟩} (36)
≥ inf {∥f ′ − f ∗ ∥ 1
− ⟨f ∗ , R⟩} ≥ inf {∥f ′ − f ∗ ∥ 1
+ ∥R∥ ∞
} = ∥R∥ ∞
f ′ ∈F
f ′ ∈F
(37)
On the other hand note that, f ∗ is the vertex of the l 1 ball (any one which given by argmin ∣R[i]∣
i∈[d]
with sign opposite as sign of R[i] on that vertex). Since the l 1 ball is the convex hull of the 2d
vertices, any vector f ∈ F can be written as f = αh − βf ∗ some h ∈ F such that ∥h∥ 1
= 1 and
⟨h, R⟩ = 0 (which means that h is 0 on the maximal co-ordinate of R specified by f ∗ ) and for some
β ∈ [−1, 1], α ∈ [0, 1] s.t. ∥αh − βf ∗ ∥ 1
≤ 1. Further note that the constraint on α, β imposed by
requiring that ∥αh − βf ∗ ∥ 1
≤ 1 can be written as α + ∣β∣ ≤ 1. Hence,
sup
x∈X
{⟨f ∗ , x⟩ + ∥R + x∥ ∞
} = sup {∥f ∗ + f∥ 1
+ ⟨f, R⟩}
f∈F
= sup
sup
sup
α∈[0,1] h⊥f ∗ ,∥h∥ 1 =1 β∈[−1,1],∥αh−βf ∗ ∥ 1 ≤1
= sup
sup
sup
α∈[0,1] h⊥f ∗ ,∥h∥ 1 =1 β∈[−1,1],∥αh−βf ∗ ∥ 1 ≤1
= sup
≤
≤
sup
α∈[0,1] β∈[−1,1]∶∣β∣+α≤1
sup
β∈[−1,1]
sup
β∈[−1,1]
{∣1 − β∣ + α + β ∥R∥ ∞
}
{∣1 − β∣ + 1 − ∣β∣ + β ∥R∥ ∞
}
{2∣1 − β∣ + β ∥R∥ ∞
}
= sup {2∣1 − β∣ + β ∥R∥ ∞
}
β∈{−1,1}
= max {∥R∥ ∞
, 4 − ∥R∥ ∞
}
≤ ∥R∥ ∞
+ 4 1 {∥R∥ ∞
≤ 4}
Hence combining with equation 36 we can conclude that
E
R
[sup
x
{⟨f ∗ , x⟩ + ∥R + x∥ ∞
}] ≤ E [inf
R f∈F
= E [inf
R f∈F
sup
x
{∥(1 − β)f ∗ + αh∥ 1
+ β ⟨f ∗ , R⟩ + α ⟨h, R⟩}
{∣1 − β∣ ∥f ∗ ∥ 1
+ α ∥h∥ 1
+ β ∥R∥ ∞
}
{⟨f, x⟩ + ∥R + x∥ ∞
}] + 4 E [1 {∥R∥ ∞
≤ 4}]
R
sup {⟨f, x⟩ + ∥R + x∥ ∞
}] + 4 P (∥R∥ ∞
≤ 4)
x
18