TheoryofDeepLearning.2022

5
8 Ultra-wide Neural Networks and Neural Tangent Kernel 67
8.1 Evolving Equation on Predictions 67
8.2 Coupling Ultra-wide Neural Networks and NTK 69
8.3 Explaining Optimization and Generalization of Ultra-wide Neural Networks via NTK
72
8.4 NTK formula for Multilayer Fully-connected Neural Network 74
8.5 NTK in Practice 77
8.6 Exercises 77
9 Inductive Biases due to Algorithmic Regularization 79
9.1 Matrix Sensing 80
9.1.1 Gaussian Sensing Matrices 82
9.1.2 Matrix Completion 85
9.2 Deep neural networks 87
9.3 Landscape of the Optimization Problem 90
9.3.1 Implicit bias in local optima 92
9.3.2 Landscape properties 94
9.4 Role of Parametrization 100
10 Unsupervised learning: Overview 101
10.0.1 Possible goals of unsupervised learning 101
10.1 Training Objective for Density estimation: Log Likelihood 103
10.2 Variational methods 104
10.3 Autoencoders 105
10.3.1 Sparse autoencoder 105
10.3.2 Topic models 106
10.4 Variational Autoencoder (VAE) 106
10.4.1 Training VAEs 107
10.5 Main open question 108
11 Generative Adversarial Nets 109
11.1 Basic definitions 109