TheoryofDeepLearning.2022

Recommendations

Info

612 Representation Learning 11113 Examples of Theorems, Proofs, Algorithms, Tables, Figures 11313.1 Example of Theorems and Lemmas 11313.2 Example of Long Equation Proofs 11313.3 Example of Algorithms 11413.4 Example of Figures 11513.5 Example of Tables 11613.6 Exercise 116
List of Figures3.1 Why it suffices to compute derivatives with respect to nodes. 223.2 Multivariate chain rule: derivative with respect to node z can be computedusing weighted sum of derivatives with respect to all nodesthat z feeds into. 233.3 Vector version of above 266.1 Steepest descent w.r.t ‖.‖4/3: the global minimum to which steepest descentconverges to depends on η. Here w 0 = [0, 0, 0], w‖.‖ ∗ = arg min ψ∈G ‖w‖4 /3denotes the minimum norm global minimum, and wη→0 ∞ denotes the solutionof infinitesimal SD with η → 0. Note that even as η → 0, the expectedcharacterization does not hold, i.e., wη→0 ∞ ̸= w∗ ‖.‖ . 447.1 Obstacles for nonconvex optimization. From left to right: local minimum,saddle point and flat region. 578.1 Convergence rate vs. projections onto eigenvectors of the kernel matrix.738.2 Generalization error vs. complexity measure. 749.1 Optimization landscape (top) and contour plot (bottom) for a singlehidden-layer linear autoencoder network with one dimensional inputand output and a hidden layer of width r = 2 with dropout,for different values of the regularization parameter λ. Left: for λ =0 the problem reduces to squared loss minimization, which is rotationinvariant as suggested by the level sets. Middle: for λ > 0 theglobal optima shrink toward the origin. All local minima are global,and are equalized, i.e. the weights are parallel to the vector (±1, ±1).Right: as λ increases, global optima shrink further. 9310.1 Visualization of Pearson’s Crab Data as mixture of two Gaussians.(Credit: MIX homepage at McMaster University.) 102
Page 1: C O N T R I B U T O R S : R A M A N
Page 4 and 5: 44 Basics of generalization theory
Page 8 and 9: 810.2 Autoencoder defined using a d
Page 11: IntroductionThis monograph discusse
Page 14 and 15: 14 theory of deep learning• Train
Page 17 and 18: 2Basics of OptimizationThis chapter
Page 19 and 20: basics of optimization 19where the
Page 21 and 22: basics of optimization 21Therefore,
Page 23 and 24: 3Backpropagation and its VariantsTh
Page 25 and 26: backpropagation and its variants 25
Page 31 and 32: 4Basics of generalization theoryGen
Page 33 and 34: basics of generalization theory 33p
Page 35 and 36: basics of generalization theory 35w
Page 37: basics of generalization theory 37N
Page 41 and 42: 6Algorithmic RegularizationLarge sc
Page 43 and 44: algorithmic regularization 43minimi
Page 45 and 46: algorithmic regularization 45update
Page 47 and 48: algorithmic regularization 476.2 Ma
Page 49 and 50: algorithmic regularization 496.3.2
Page 51 and 52: algorithmic regularization 51Given
Page 53 and 54: algorithmic regularization 53to the
Page 55: algorithmic regularization 55Since
Page 58 and 59:
58 theory of deep learning7.1 Preli
Page 60 and 61:
60 theory of deep learningcreasing
Page 62 and 63:
62 theory of deep learning“gradie
Page 64 and 65:
64 theory of deep learning7.4 Case
Page 66 and 67:
66 theory of deep learningWhen x =
Page 68 and 69:
68 theory of deep learningproof is
Page 70 and 71:
70 theory of deep learningthe input
Page 72 and 73:
72 theory of deep learningas the av
Page 74 and 75:
74 theory of deep learningSeveral r
Page 76 and 77:
76 theory of deep learningFigure 8.
Page 78 and 79:
78 theory of deep learningNote the
Page 81 and 82:
9Inductive Biases due to Algorithmi
Page 83 and 84:
inductive biases due to algorithmic
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
Page 93 and 94:
Page 95 and 96:
Page 97 and 98:
Page 99 and 100:
Page 101 and 102:
Page 103 and 104:
10Unsupervised learning: OverviewMu
Page 105 and 106:
unsupervised learning: overview 105
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
11Generative Adversarial NetsChapte
Page 113:
12Representation Learning
Page 116 and 117:
116 theory of deep learning13.3 Exa
Page 118:
118 theory of deep learning13.5 Exa
show all

TheoryofDeepLearning.2022

Create successful ePaper yourself

Delete template?

Save as template?