TheoryofDeepLearning.2022

Contents
1 Basic Setup and some math notions 11
1.1 List of useful math facts 12
1.1.1 Probability tools 12
1.1.2 Singular Value Decomposition 13
2 Basics of Optimization 15
2.1 Gradient descent 15
2.1.1 Formalizing the Taylor Expansion 16
2.1.2 Descent lemma for gradient descent 16
2.2 Stochastic gradient descent 17
2.3 Accelerated Gradient Descent 17
2.4 Local Runtime Analysis of GD 18
2.4.1 Pre-conditioners 19
3 Backpropagation and its Variants 21
3.1 Problem Setup 21
3.1.1 Multivariate Chain Rule 23
3.1.2 Naive feedforward algorithm (not efficient!) 24
3.2 Backpropagation (Linear Time) 24
3.3 Auto-differentiation 25
3.4 Notable Extensions 26
3.4.1 Hessian-vector product in linear time: Pearlmutter’s trick 27