Machine Learning - DISCo

simple functions such as XOR could not be represented or learned with singlelayer
perceptron networks, and work on ANNs receded during the 1970s.
During the mid-1980s work on ANNs experienced a resurgence, caused in
large part by the invention of BACKPROPAGATION and related algorithms for training
multilayer networks (Rumelhart and McClelland 1986; Parker 1985). These
ideas can be traced to related earlier work (e.g., Werbos 1975). Since the 1980s,
BACKPROPAGATION has become a widely used learning method, and many other
ANN approaches have been actively explored. The advent of inexpensive computers
during this same period has allowed experimenting with computationally
intensive algorithms that could not be thoroughly explored during the 1960s.
A number of textbooks are devoted to the topic of neural network learning.
An early but still useful book on parameter learning methods for pattern recognition
is Duda and Hart (1973). The text by Widrow and Stearns (1985) covers
perceptrons and related single-layer networks and their applications. Rumelhart
and McClelland (1986) produced an edited collection of papers that helped generate
the increased interest in these methods beginning in the mid-1980s. Recent
books on neural network learning include Bishop (1996); Chauvin and Rumelhart
(1995); Freeman and Skapina (1991); Fu (1994); Hecht-Nielsen (1990); and Hertz
et al. (1991).
EXERCISES
4.1. What are the values of weights wo, wl, and w2 for the perceptron whose decision
surface is illustrated in Figure 4.3? Assume the surface crosses the xl axis at -1,
and the x2 axis at 2.
4.2. Design a two-input perceptron that implements the boolean function A A -. B. Design
a two-layer network of perceptrons that implements A XO R B.
4.3. Consider two perceptrons defined by the threshold expression wo + wlxl+ ~2x2 > 0.
Perceptron A has weight values
and perceptron B has the weight values
True or false? Perceptron A is more-general~han perceptron B. (more-general~han
is defined in Chapter 2.)
4.4. Implement the delta training rule for a two-input linear unit. Train it to fit the target
concept -2 + XI+ 2x2 > 0. Plot the error E as a function of the number of training
iterations. Plot the decision surface after 5, 10, 50, 100, . . . , iterations.
(a) Try this using various constant values for 17 and using a decaying learning rate
of qo/i for the ith iteration. Which works better?
(b) Try incremental and batch learning. Which converges more quickly? Consider
both number of weight updates and total execution time.
4.5. Derive a gradient descent training rule for a single unit with output o, where