Foundations of Data Science

Note ∇ wi J(W ) is a vector. Since w i is a vector, each component of ∇ wi J(W ) is the
derivative with respect to one component of the vector w i .
Over fitting is a major concern in deep learning since large networks can have hundreds
of millions of weights. In image recognition, the number of training images can
be significantly increased by random jittering of the images. Another technique called
dropout randomly deletes a fraction of the weights at each training iteration. Regularization
is used to assign a cost to the size of weights and many other ideas are being explored.
Deep learning is an active research area. Some of the ideas being explored are what
do individual gates or sets of gates learn. If one trains a network twice from starting with
random sets of weights, do gates learn the same features? In image recognition, the early
convolution layers seem to learn features of images rather than features of the specific set
of images they are being trained with. Once a network is trained on say a set of images
one of which is a cat one can freeze the weights and then find images that will map to the
activation vector generated by the cat image. One can take an artwork image and separate
the style from the content and then create an image using the content but a different style
[]. This is done by taking the activation of the original image and moving it to the manifold
of activation vectors of images of a given style. One can do many things of this type.
For example one can change the age of a child in an image or change some other feature. []
For more information about deep learning, see [?]. 27
6.14 Further Current directions
We now briefly discuss a few additional current directions in machine learning, focusing
on semi-supervised learning, active learning, and multi-task learning.
6.14.1 Semi-supervised learning
Semi-supervised learning refers to the idea of trying to use a large unlabeled data set U to
augment a given labeled data set L in order to produce more accurate rules than would
have been achieved using just L alone. The motivation is that in many settings (e.g.,
document classification, image classification, speech recognition), unlabeled data is much
more plentiful than labeled data, so one would like to make use of it if possible. Of course,
unlabeled data is missing the labels! Nonetheless it often contains information that an
algorithm can take advantage of.
As an example, suppose one believes the target function is a linear separator that
separates most of the data by a large margin. By observing enough unlabeled data to estimate
the probability mass near to any given linear separator, one could in principle then
27 See also the tutorials: http://deeplearning.net/tutorial/deeplearning.pdf and
http://deeplearning.stanford.edu/tutorial/.
228