Relax and Randomize: From Value to Algorithms

A
PROOFS
Proof of Proposition 1. By definition,
T
∑
t=1
T
E ft∼q t
l(f t , x t ) − inf ∑ l(f, x t ) ≤ ∑ E ft∼q t
l(f t , x t ) + Rel T (F∣x 1 , . . . , x T ) .
f∈F t=1
Peeling off the T -th expected loss, we have
T
∑
t=1
T
t=1
T −1
E ft∼q t
l(f t , x t ) + Rel T (F∣x 1 , . . . , x T ) ≤
∑
t=1
T −1
≤ ∑
t=1
E ft∼q t
l(f t , x t ) + {E ft∼q t
l(f t , x t ) + Rel T (F∣x 1 , . . . , x T )}
E ft∼q t
l(f t , x t ) + Rel T (F∣x 1 , . . . , x T −1 )
where we used the fact that q T is an admissible algorithm for this relaxation, and thus the last
inequality holds for any choice x T of the opponent. Repeating the process, we obtain
T
∑
t=1
T
E ft∼q t
l(f t , x t ) − inf ∑ l(f, x t ) ≤ Rel T (F) .
f∈F t=1
We remark that the left-hand side of this inequality is random, while the right-hand side is not. Since
the inequality holds for any realization of the process, it also holds in expectation. The inequality
V T (F) ≤ Rel T (F)
holds by unwinding the value recursively and using admissibility of the relaxation. The highprobability
bound is an immediate consequences of (6) and the Hoeffding-Azuma inequality for
bounded martingales. The last statement is immediate.
Proof of Proposition 2. Denote L t (f) = ∑ t s=1 l(f, x s ). The first step of the proof is an application
of the minimax theorem (we assume the necessary conditions hold):
inf
q t∈∆(F) x t∈X
= sup
sup { E [l(f t , x t )] + sup
f t∼q t x
p t∈∆(X ) f t∈F
E ɛt+1∶T sup
f∈F
inf { E [l(f t , x t )] + E sup
x t∼p t x t∼p t x
[2
T
∑
s=t+1
E ɛt+1∶T sup
f∈F
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − L t (f)]}
[2
T
∑
s=t+1
For any p t ∈ ∆(X ), the infimum over f t of the above expression is equal to
E sup E ɛt+1∶T sup [2
x t∼p t x
f∈F
≤ E sup E ɛt+1∶T sup [2
x t∼p t x
f∈F
T
∑
s=t+1
T
∑
s=t+1
≤ E sup E ɛt+1∶T sup [2
x t,x ′ t ∼pt x
f∈F
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − L t−1 (f) + inf
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − L t (f)]}
f t∈F
E [l(f t , x t )] − l(f, x t )]
x t∼p t
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − L t−1 (f) + E [l(f, x t )] − l(f, x t )]
x t∼p t
T
∑
s=t+1
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − L t−1 (f) + l(f, x ′ t) − l(f, x t )]
We now argue that the independent x t and x ′ t have the same distribution p t , and thus we can introduce
a random sign ɛ t . The above expression then equals to
E E
x t,x ′ t ∼pt
ɛ t
sup
x
E ɛt+1∶T sup
f∈F
[2
≤ sup E sup E ɛt+1∶T sup [2
x t,x ′ t ∈X ɛ t x
f∈F
T
∑
s=t+1
T
∑
s=t+1
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − L t−1 (f) + ɛ t (l(f, x ′ t) − l(f, x t ))]
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − L t−1 (f) + ɛ t (l(f, x ′ t) − l(f, x t ))]
where we upper bounded the expectation by the supremum. Splitting the resulting expression into
two parts, we arrive at the upper bound of
2 sup E
x t∈X
sup
ɛ t x
E ɛt+1∶T sup
f∈F
[
T
∑
s=t+1
ɛ s l(f, x s−t (ɛ t+1∶s−1 )) − 1 2 L t−1(f) + ɛ t l(f, x t )] = R T (F∣x 1 , . . . , x t−1 ) .
10