TheoryofDeepLearning.2022

30 theory of deep learning
The proof of the Claim follows in straightforward fashion from
implementing the message passing protocol as an acyclic circuit.
Next we show how to compute ∇ 2 f (z) · v where v is a given fixed
vector. Let g(z) = 〈∇ f (z), v〉 be a function from R d → R. Then by
the Claim above, g(z) can be computed by a network of size O(V + E).
Now apply the Claim again on g(z), we obtain that ∇g(z) can also be
computed by a network of size O(V + E).
Note that by construction,
∇g(z) = ∇ 2 f (z) · v.
Hence we have computed the Hessian vector product in network size
time.