Relax and Randomize: From Value to Algorithms

which can be re-written as
ŷ t (ɛ) = (sup
f∈F
{
T
∑
i=t+1
ɛ i f[i] − 1
2L L t−1(f) + 1 2
f[t]} − sup
f∈F
{
T
∑
i=t+1
By this choice of ŷ t (ɛ), plugging back in Equation (44) we see that
sup { E
y t
⎡
≤ E
ɛ ⎢
⎣
[l(ŷ t , y t )] + E
ŷ t∼q t ɛ
[sup
f∈F
{2L
T
∑
i=t+1
ɛ i f[i] − L t (f)}]}
ɛ i f[i] − 1
2L L t−1(f) − 1 2 f[t]})
T
⎤
sup {r t ⋅ ŷ t (ɛ) + sup {2L ∑ ɛ i f[i] − L t−1 (f) − r t ⋅ f[t]}}
r t∈{±L}
f∈F
⎥
i=t+1
⎦
⎡
T
⎤
= E
ɛ ⎢
inf
sup {r t ⋅ ŷ t + sup {2L ∑ ɛ i f[i] − L t−1 (f) − r t ⋅ f[t]}}
⎣ ŷ t r t∈{±L}
f∈F
⎥
i=t+1
⎦
⎡
T
⎤
= E
ɛ ⎢
inf
sup E rt∼p t
{r t ⋅ ŷ t + sup {2L ∑ ɛ i f[i] − L t−1 (f) − r t ⋅ f[t]}}
⎣ ŷ t p t∈∆({±L})
f∈F
⎥
i=t+1
⎦
The expression inside the supremum is linear in p t , as it is an expectation. Also note that the term
is convex in ŷ t , and the domain ŷ t ∈ [− sup f∈F ∣f[t]∣, sup f∈F ∣f[t]∣] is a bounded interval (hence,
compact). We conclude that we can use the minimax theorem, yielding
⎡
T
⎤
E
sup inf E [r
ɛ ⎢
t ⋅ ŷ t + sup {2L ∑ ɛ i f[i] − L t−1 (f) − r t ⋅ f[t]}]
⎣p t∈∆({±L}) ŷ t r t∼p t f∈F
⎥
i=t+1
⎦
⎡
T
⎤
= E
sup {inf E [r
ɛ ⎢
t ⋅ ŷ t ] + E [sup {2L ∑ ɛ i f[i] − L t−1 (f) − r t ⋅ f[t]}]}
⎣p t∈∆({±L}) ŷ t r t∼p t r t∼p t f∈F
⎥
i=t+1
⎦
⎡
T
⎤
= E
sup { E [sup {inf E [r
ɛ ⎢
t ⋅ ŷ t ] + 2L ∑ ɛ i f[i] − L t−1 (f) − r t ⋅ f[t]}]}
⎣p t∈∆({±L}) r t∼p t f∈F ŷ t r t∼p t
⎥
i=t+1
⎦
⎡
T
⎤
≤ E
sup { E [sup { E [r
ɛ ⎢
t ⋅ f[t]] + 2L ∑ ɛ i f[i] − L t−1 (f) − r t ⋅ f[t]}]}
⎣p t∈∆({±L}) r t∼p t f∈F r t∼p t
⎥
i=t+1
⎦
In the last step, we replaced the infimum over ŷ t with f[t], only increasing the quantity. Introducing
an i.i.d. copy r t ′ of r t ,
⎡
T
⎤
= E
sup { E [sup {2L
ɛ ⎢
∑ ɛ i f[i] − L t−1 (f) + ( E [r t ] − r t ) ⋅ f[t]}]}
⎣p t∈∆({±L}) r t∼p t f∈F i=t+1
r t∼p t
⎥
⎦
⎡
T
⎤
≤ E
⎧⎪
⎫⎪
sup ⎨ E [sup {2L ∑ ɛ i f[i] − L t−1 (f) + (r ′ t − r t ) ⋅ f[t]}] ⎬
ɛ ⎢p ⎣ t∈∆({±L}) r ⎪⎩ t,r t ′ ∼pt f∈F i=t+1
⎪⎭
⎥
⎦
Introducing the random sign ɛ t and passing to the supremum over r t , r t, ′ yields the upper bound
⎡
T
⎤
E
sup {E
ɛ ⎢
rt,r ′ E [sup {2L ∑ ɛ
t
⎣
∼pt i f[i] − L t−1 (f) + (r ′ t − r t ) ⋅ f[t]}]}
p t∈∆({±L})
ɛ t f∈F
⎥
i=t+1
⎦
⎡
T
⎤
≤ E
sup {E [sup {2L ∑ ɛ i f[i] − L t−1 (f) + ɛ t (r ′ ɛ ⎢r ⎣ t,r t ′ ∈{±L} t − r t ) ⋅ f[t]}]}
ɛ t f∈F i=t+1
⎥
⎦
⎡
T
≤ E
sup {E [sup {L ∑ ɛ i f[i] − 1 ⎤
ɛ ⎢r ⎣ t,r t ′ ∈{±L} ɛ t f∈F i=t+1 2 L t−1(f) + ɛ t r t ′ ⋅ f[t]}]}
⎥
⎦
⎡
T
+ E
sup {E [sup {L ∑ ɛ i f[i] − 1 ⎤
ɛ ⎢r ⎣ t,r t ′ ∈{±L} ɛ t f∈F i=t+1 2 L t−1(f) − ɛ t r t ⋅ f[t]}]}
⎥
⎦
In the above we split the term in the supremum as the sum of two terms one involving r t and other
r t ′ (other terms are equally split by dividing by 2), yielding
⎡
T
⎤
E
sup {E [sup {2L
ɛ ⎢
∑ ɛ i f[i] − L t−1 (f) + 2 ɛ t r t ⋅ f[t]}]}
⎣r t∈{±L} ɛ t f∈F
⎥
i=t+1
⎦
24